feat: Make GAPIC Bazel rules production ready by vam-google · Pull Request #402 · googleapis/gapic-generator-python

vam-google · 2020-04-25T06:30:25Z

Shoud fix #400 and #390, plus a bunch of other not-yet-opened issues.

This includes:

Fix long initial load time (5+ min). This was caused by python_rules buildling grpcio dependency from sources in one core (which was super slow). Switched to using bazel-native "@com_github_grpc_grpc//src/python/grpcio/grpc:grpcio" target instead, which is not only much faster, but is also already used in googleapis, so there is no additional cost for reusing it in microgenerator rules.
Properly handle pandoc dependency (platform-sepcific version of pandoc is properly pulled by bazel itself using toolchains).
Add simplistic version of the py_gapic_assembly_pkg rule, to make output of microgenerator compatible with GAPICBazel class in synthtool.
Add plugin_args argument for python_gapic_library rule to pass custom argumetns to the plugin (similar to PHP rules).
Add compatibility with python3.6 runtime (otherwise python3.7 is minimum because of dependency on dataclasses module). Python 3.6 compatibility can be enabled by adding --define=gapic_gen_python=3.6 command line argument to bazel build command.
Add support for Python runtimes installed with pyenv. To tell bazel using Python3 installed via pyenv add --extra_toolchains=@gapic_generator_python//:pyenv3_toolchain argument to bazel build command.

This includes: 1) Fix long time initial load time (5+ min). This was caused by python_rules buildling `grpcio` dependency from sources in one core (which was super slow). Switched to using bazel-native `"@com_github_grpc_grpc//src/python/grpcio/grpc:grpcio"` target instead, which is not only much faster, but is also already used in googleapis, so there is no additional cost for reusing it in microgenerator rules. 2) Properly handle `pandoc` dependency (platform-sepcific version of pandoc is properly pulled by bazel itself using toolchains). 3) Add simplistic version of the `py_gapic_assembly_pkg` rule, to make output of microgenerator compatible with `GAPICBazel` class in synthtool. 4) Add `plugin_args` argument for python_gapic_library rule to pass custom argumetns to the plugin (similar to PHP rules).

vam-google · 2020-04-25T06:31:27Z

@software-dov @aohren PTAL

software-dov

I'm assuming the bazel bits are all correct. Just a few nits to pick

software-dov · 2020-04-27T20:59:06Z

WORKSPACE

+# Import grpc as a native bazel dependency. This avoids duplication and also
+# spedds up loading phase a lot (otherwise python_rules will be building grpcio
+# from sources in a single-core speed, which takes around 5 minutes on a regular
+# workstation)


Wat? Why is it building single-cored?

I dont know how exactly it does it, but based on time it takes and the CPU load on the build machine I assumed it must be single core

software-dov · 2020-04-27T21:03:44Z

rules_python_gapic/py_gapic_pkg.bzl

+    srcjar_srcs = []
+    for dep in ctx.attr.deps:
+        for f in dep.files.to_list():
+            if f.extension in ("srcjar", "jar", "zip"):
+                srcjar_srcs.append(f)


Does skylark allow list comprehensions? Personal preference, feel free to ignore, but I think

srcjar_srcs = [ dep_file for dep in ctx.attr.deps for dep_file in dep.files.to_list() if dep_file.extension in ("srcjar", "jar", "zip") ]

is more readable because of the indentation.

It is a reduced verison of more complicated for loops copied from the other rules. Starlak support list comprehension. For sake of style consistency with the other rules lets keep it like that.

software-dov · 2020-04-27T21:05:41Z

WORKSPACE

+
+#
+# Import grpc as a native bazel dependency. This avoids duplication and also
+# spedds up loading phase a lot (otherwise python_rules will be building grpcio


Nit: speeds

software-dov · 2020-04-29T21:54:16Z

Any updates or increased bandwidth?

vam-google · 2020-04-30T20:05:28Z

@software-dov Still trying to make CI work with gapic-generator-python (python3.7 requirement issue). Before that issue is solved this PR can't be used (can't integrate it in googleapis).

This includes: 1) Fix long time initial load time (5+ min). This was caused by python_rules buildling `grpcio` dependency from sources in one core (which was super slow). Switched to using bazel-native `"@com_github_grpc_grpc//src/python/grpcio/grpc:grpcio"` target instead, which is not only much faster, but is also already used in googleapis, so there is no additional cost for reusing it in microgenerator rules. 2) Properly handle `pandoc` dependency (platform-sepcific version of pandoc is properly pulled by bazel itself using toolchains). 3) Add simplistic version of the `py_gapic_assembly_pkg` rule, to make output of microgenerator compatible with `GAPICBazel` class in synthtool. 4) Add `plugin_args` argument for python_gapic_library rule to pass custom argumetns to the plugin (similar to PHP rules). 5) Add python3.6 support via `--define=gapic_gen_python.3.6` command line argument for for `bazel build command`. Otherwise gapic-generator-python can be executed only on systems with python3.7+.

vam-google · 2020-05-02T07:44:49Z

@software-dov PTAL. I finally made it wokr from kokoro under pyenv 3.6.1 python (the available on a kokoro instance)

software-dov

LG, just a few nits/questions.

software-dov · 2020-05-04T17:10:03Z

BUILD.bazel

+        ":gapic_gen_python_3_6": [requirement("dataclasses")],
+        "//conditions:default": [],
+    }),
+)


Nit: no newline

BUILD.bazel

pyenv3wrapper.sh

This PR uses using documentai as an example. Depends on googleapis/gapic-generator-python#402 PiperOrigin-RevId: 309824146

This PR uses using documentai as an example. Depends on googleapis/gapic-generator-python#402 PiperOrigin-RevId: 309824146 Source-Author: Google APIs <noreply@google.com> Source-Date: Mon May 4 15:06:44 2020 -0700 Source-Repo: googleapis/googleapis Source-Sha: e0f9d9e1f9de890db765be46f45ca8490723e3eb Source-Link: googleapis/googleapis@e0f9d9e

…ary generator. (#255) * Integrate Python GAPIC Microgenerator in googleapis. This PR uses using documentai as an example. Depends on googleapis/gapic-generator-python#402 PiperOrigin-RevId: 309824146 Source-Author: Google APIs <noreply@google.com> Source-Date: Mon May 4 15:06:44 2020 -0700 Source-Repo: googleapis/googleapis Source-Sha: e0f9d9e1f9de890db765be46f45ca8490723e3eb Source-Link: googleapis/googleapis@e0f9d9e * fix: add resource reference to proto files chore: update copyright and comments PiperOrigin-RevId: 309990843 Source-Author: Google APIs <noreply@google.com> Source-Date: Tue May 5 11:51:12 2020 -0700 Source-Repo: googleapis/googleapis Source-Sha: 4bef0001ac7040431ea24b6187424fdec9c08b1b Source-Link: googleapis/googleapis@4bef000

This PR uses using documentai as an example. Depends on googleapis/gapic-generator-python#402 PiperOrigin-RevId: 309824146 Source-Author: Google APIs <noreply@google.com> Source-Date: Mon May 4 15:06:44 2020 -0700 Source-Repo: googleapis/googleapis Source-Sha: e0f9d9e1f9de890db765be46f45ca8490723e3eb Source-Link: googleapis/googleapis@e0f9d9e

Update comments on protos to clarify size limits on values. * Integrate Python GAPIC Microgenerator in googleapis. This PR uses using documentai as an example. Depends on googleapis/gapic-generator-python#402 PiperOrigin-RevId: 309824146 Source-Author: Google APIs <noreply@google.com> Source-Date: Mon May 4 15:06:44 2020 -0700 Source-Repo: googleapis/googleapis Source-Sha: e0f9d9e1f9de890db765be46f45ca8490723e3eb Source-Link: googleapis/googleapis@e0f9d9e * fix(python): use python installed by pyenv for bazel (#520) Source-Author: Jeff Ching <chingor@google.com> Source-Date: Mon May 4 15:52:28 2020 -0700 Source-Repo: googleapis/synthtool Source-Sha: a2c1c4f89a5e220e7b39420ebea33623c7c72804 Source-Link: googleapis/synthtool@a2c1c4f * Fix documentation for string_value. The value does not have to be at least 1MB, but rather, at most 1MB. PiperOrigin-RevId: 309845930 Source-Author: Google APIs <noreply@google.com> Source-Date: Mon May 4 16:55:46 2020 -0700 Source-Repo: googleapis/googleapis Source-Sha: 1b5a8d2bbb69c5a04db26bd546d2888e609c6bab Source-Link: googleapis/googleapis@1b5a8d2 * fix: make args optional (#521) Source-Author: Summer Ji <summerji@google.com> Source-Date: Tue May 5 09:52:17 2020 -0700 Source-Repo: googleapis/synthtool Source-Sha: ea3ae392fced25d7d6f5c7ef9c8a1f24768a0936 Source-Link: googleapis/synthtool@ea3ae39 * fix: make .kokoro-autosynth executable (#522) to fix googleapis/synthtool#519 Source-Author: Jeffrey Rennie <rennie@google.com> Source-Date: Tue May 5 10:12:11 2020 -0700 Source-Repo: googleapis/synthtool Source-Sha: ab883569eb0257bbf16a6d825fd018b3adde3912 Source-Link: googleapis/synthtool@ab88356 * fix: add resource reference to proto files chore: update copyright and comments PiperOrigin-RevId: 309990843 Source-Author: Google APIs <noreply@google.com> Source-Date: Tue May 5 11:51:12 2020 -0700 Source-Repo: googleapis/googleapis Source-Sha: 4bef0001ac7040431ea24b6187424fdec9c08b1b Source-Link: googleapis/googleapis@4bef000

This PR uses using documentai as an example. Depends on googleapis/gapic-generator-python#402 PiperOrigin-RevId: 309824146 Source-Author: Google APIs <noreply@google.com> Source-Date: Mon May 4 15:06:44 2020 -0700 Source-Repo: googleapis/googleapis Source-Sha: e0f9d9e1f9de890db765be46f45ca8490723e3eb Source-Link: googleapis/googleapis@e0f9d9e

* Integrate Python GAPIC Microgenerator in googleapis. This PR uses using documentai as an example. Depends on googleapis/gapic-generator-python#402 PiperOrigin-RevId: 309824146 Source-Author: Google APIs <noreply@google.com> Source-Date: Mon May 4 15:06:44 2020 -0700 Source-Repo: googleapis/googleapis Source-Sha: e0f9d9e1f9de890db765be46f45ca8490723e3eb Source-Link: googleapis/googleapis@e0f9d9e * ci(java): switch to GitHub Actions (#533) * ci: switch to GitHub Actions * update based on comment Source-Author: Stephanie Wang <stephaniewang526@users.noreply.github.com> Source-Date: Mon May 11 12:54:00 2020 -0400 Source-Repo: googleapis/synthtool Source-Sha: 55cdc844877d97139f25004229842624a6a86a02 Source-Link: googleapis/synthtool@55cdc84 * chore: move ci.yaml to workflows dir (#535) Source-Author: Stephanie Wang <stephaniewang526@users.noreply.github.com> Source-Date: Mon May 11 16:24:04 2020 -0400 Source-Repo: googleapis/synthtool Source-Sha: 98c50772ec23295c64cf0d2ddf199ea52961fd4c Source-Link: googleapis/synthtool@98c5077

This PR uses using documentai as an example. Depends on googleapis/gapic-generator-python#402 PiperOrigin-RevId: 309824146 Source-Author: Google APIs <noreply@google.com> Source-Date: Mon May 4 15:06:44 2020 -0700 Source-Repo: googleapis/googleapis Source-Sha: e0f9d9e1f9de890db765be46f45ca8490723e3eb Source-Link: googleapis/googleapis@e0f9d9e

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Apr 25, 2020

software-dov reviewed Apr 27, 2020

View reviewed changes

software-dov self-requested a review April 28, 2020 18:55

vam-google added 6 commits April 30, 2020 15:07

set LC_ALL to fix kokoro build

1172bba

Merge remote-tracking branch 'upstream/master'

a72432d

Fix typo

bb5e605

add pyenv3_toolchain

8066385

add pyenv3_toolchain

6fae027

software-dov approved these changes May 4, 2020

View reviewed changes

add newline character

b69b8fc

vam-google merged commit d18ed41 into googleapis:master May 4, 2020

lukesneeringer pushed a commit to googleapis/googleapis that referenced this pull request May 4, 2020

Integrate Python GAPIC Microgenerator in googleapis.

e0f9d9e

This PR uses using documentai as an example. Depends on googleapis/gapic-generator-python#402 PiperOrigin-RevId: 309824146

software-dov mentioned this pull request May 13, 2020

No way to specify python_gapic_opt flag with py_gapic_library rule #390

Closed

This was referenced Jan 18, 2022

chore(master): release reverted #1133

Merged

chore(master): release 0.39.1 #1146

Closed

This was referenced Jan 26, 2022

chore(main): release 0.39.1 #1160

Closed

chore(main): release 0.39.1 #1162

Closed

chore(main): release 0.61.0 #1163

Merged

Conversation

vam-google commented Apr 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vam-google commented Apr 25, 2020

Uh oh!

software-dov left a comment

Choose a reason for hiding this comment

Uh oh!

software-dov Apr 27, 2020

Choose a reason for hiding this comment

Uh oh!

vam-google May 2, 2020

Choose a reason for hiding this comment

Uh oh!

software-dov Apr 27, 2020

Choose a reason for hiding this comment

Uh oh!

vam-google May 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

software-dov Apr 27, 2020

Choose a reason for hiding this comment

Uh oh!

software-dov commented Apr 29, 2020

Uh oh!

vam-google commented Apr 30, 2020

Uh oh!

vam-google commented May 2, 2020

Uh oh!

software-dov left a comment

Choose a reason for hiding this comment

Uh oh!

software-dov May 4, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vam-google commented Apr 25, 2020 •

edited

Loading

vam-google May 2, 2020 •

edited

Loading