API Client Generator for Python

A generator for protocol buffer described APIs for and in Python 3.

This tool is a client library generator that implements the client library generators specification.

It accepts an API specified in protocol buffers and generates a client library, which can be used to interact with that API. It is implemented as a plugin to protoc, the protocol buffer compiler.

Getting Started

This code generator is implemented as a plugin to protoc, the compiler for protocol buffers, and will run in any environment that Python 3.6+ and protocol buffers do.

Because dependency management and such can be a significant undertaking, we offer a Docker image and interface which requires you only to have Docker installed and provide the protos for your API. Alternatively, the generator is also invocable via bazel rules.

It is also possible to install the tool locally and run it through protoc, and this approach is fully supported.

Note

The Docker approach is recommended for users new to this ecosystem, or those which do not have a robust Python environment available. If you want to experiment with generating client libraries but do not want to make changes to the generator itself, try the Docker image first.

The bazel approach is recommended for established pipelines. It is more lightweight than the Docker image but may take some more effort to set up.

Docker Image

If you are just getting started with code generation for protobuf-based APIs, or if you do not have a robust Python environment already available, we recommend using our Docker image to build client libraries.

However, this tool offers first-class support for local execution using protoc: Local Installation. It is still reasonably easy, but initial setup will take a bit longer.

Note

If you are interested in contributing, using a local installation is recommended.

Installing

Docker

In order to use a Docker image, you must have Docker installed. Docker is a container management service, and is available on Linux, Mac, and Windows (although most of these instructions will be biased toward Linux and Mac).

Install Docker according to their installation instructions.

Note

This image requires Docker 17.05 or later.

Pull the Docker Image

Once Docker is installed, simply pull the Docker image for this tool:

$ docker pull gcr.io/gapic-images/gapic-generator-python:latest

Usage

To use this plugin, you will need an API which is specified using protocol buffers. Additionally, this plugin makes some assumptions at the margins according to Google API design conventions as described in AIPs, so following those conventions is recommended.

Example

If you want to experiment with an already-existing API, one example is available. (Reminder that this is still considered experimental, so apologies for this part being a bit strange.)

You need to clone the googleapis repository from GitHub:

$ git clone https://github.com/googleapis/googleapis.git

It is possible to generate libraries for most (possibly all) APIs described here. The API we use as an example is the Google Cloud Vision API, available in the google/cloud/vision/v1/ subdirectory. This will be used for the remainder of the examples on this page.

Compiling an API

Note

If you are running code generation repeatedly, executing the long docker run command may be cumbersome. While you should ensure you understand this section, a shortcut script is available to make iterative work easier.

Compile the API into a client library by invoking the Docker image.

It is worth noting that the image must interact with the host machine (your local machine) for two things: reading in the protos you wish to compile, and writing the output. This means that when you run the image, two mount points are required in order for anything useful to happen.

In particular, the input protos are expected to be mounted into /in/, and the desired output location is expected to be mounted into /out/. The output directory must also be writable.

Note

The /in/ and /out/ directories inside the image are hard-coded; they can not be altered where they appear in the command below.

Docker requires the output directory to pre-exist; create a directory where you want the generated code to go:

$ mkdir dest/

Perform the actual code generation step with docker run:

# This is assumed to be run from the `googleapis` project root.
$ docker run \
  --mount type=bind,source=$(pwd)/google/cloud/vision/v1/,destination=/in/google/cloud/vision/v1/,readonly \
  --mount type=bind,source=$(pwd)/dest/,destination=/out/ \
  --rm \
  --user $UID \
  gcr.io/gapic-images/gapic-generator-python

Warning

protoc is very picky about paths, and the exact construction here matters a lot. The source is google/cloud/vision/v1/, and then the destination is that full directory path after the /in/ root; therefore: /in/google/cloud/vision/v1/.

This matters because of how proto imports are resolved. The import statement imports a file, relative to a base directory or set of base directories, called the proto_path. This is assumed (and hard-coded) to /in/ in the Docker image, and so any directory structure present in the imports of the proto files must be preserved beneath this for compilation to succeed.

Generating Samples

In addition to generating client libraries, the generator can also create standalone executable code samples.

The user can specify individual sample config files or can pass paths to directories that contain sample configs. Directories are searched recursively, and any file that is not a sample config is ignored.

A full description of the sample config, generated manifest, and generated samples is outside the scope of this documentation. We will provide links to such documentation when it is ready.

Samples and manifests are always generated in a ‘samples’ subdir of the destination directory.

# Multiple sample paths or directories can be passed simultaneously by duplicating
# the 'samples' option.
# If no 'samples' option is passed, the generator does not generate a manifest.
$ docker run \
  --mount type=bind,source=$(pwd)/path/to/proto/dir,destination=/in/path/to/proto,readonly \
  --mount type=bind,source=$(pwd)/dest/,destination=/out/ \
  --rm \
  --user $UID \
  gcr.io/gapic-images/gapic-generator-python \
  --samples path/to/sample/config.yaml \
  --samples path/to/sample/dir/

Verifying the Library

Once you have compiled a client library, whether using a Docker image, local installation or bazel, it is time for the fun part: actually running it!

Create a virtual environment for the library:

$ virtualenv ~/.local/client-lib --python=`which python3.7`
$ source ~/.local/client-lib/bin/activate

Next, install the library:

$ cd dest/
$ pip install --editable .

Now it is time to play with it! Here is a test script:

# This is the client library generated by this plugin.
from google.cloud import vision

# Instantiate the client.
#
# If you need to manually specify credentials, do so here.
# More info: https://cloud.google.com/docs/authentication/getting-started
#
# If you wish, you can send `transport='grpc'` or `transport='http'`
# to change which underlying transport layer is being used.
ia = vision.ImageAnnotatorClient()

# Send the request to the server and get the response.
response = ia.batch_annotate_images({
    'requests': [{
        'features': [{
            'type': vision.Feature.Type.LABEL_DETECTION,
        }],
        'image': {'source': {
            'image_uri': 'https://images.pexels.com/photos/67636'
                         '/rose-blue-flower-rose-blooms-67636.jpeg',
        }},
    }],
})
print(response)

Local Installation

If you are just getting started with code generation for protobuf-based APIs, or if you do not have a robust Python environment already available, it is probably easier to get started using Docker: Docker Image

However, this tool offers first-class support for local execution using protoc. It is still reasonably easy, but initial setup will take a bit longer.

Note

If you are interested in contributing, setup according to these steps is recommended.

Installing

protoc

This tool is implemented as a plugin to the protocol buffers compiler, so in order to use it, you will need to have the protoc command available.

The release page on GitHub contains the download you need.

Note

You may notice both packages that designate languages (e.g. protobuf-python-X.Y.Z.tar.gz) as well as packages that designate architectures (e.g. protoc-X.Y.Z-linux-x86_64.zip). You want the one that designates an architecture; your goal here is to have a CLI command.

It is likely preferable to install protoc somewhere on your shell’s path, but this is not a strict requirement (as you will be invoking it directly). protoc is also quirky about how it handles well-known protos; you probably also want to copy them into /usr/local/include

To ensure it is installed propertly:

$ protoc --version
libprotoc 3.6.0

pandoc

This generator relies on pandoc to convert from Markdown (the lingua franca for documentation in protocol buffers) into ReStructured Text (the lingua franca for documentation in Python).

Install this using an appropriate mechanism for your operating system. Multiple installation paths are documented on the pandoc installation page.

API Generator for Python

This package is provided as a standard Python library, and can be installed the usual ways. It fundamentally provides a CLI command, protoc-gen-python_gapic, (yes, the mismatch of kebob-case and snake_case is weird, sorry), so you will want to install using a mechanism that is conducive to making CLI commands available.

Additionally, this program currently only runs against Python 3.6 or Python 3.7, so you will need that installed. (Most Linux distributions ship with earlier versions.) Use pyenv to get Python 3.7 installed in a friendly way.

# Due to its experimental state, this tool is not published to a
# package manager; you should clone it.
# (You can pip install it from GitHub, not not if you want to tinker.)
git clone https://github.com/googleapis/gapic-generator-python.git
cd gapic-generator-python/

    # Install a version of python that is supported by the microgenerator.
    # We use 3.8.6 as an example.
    # You may need to install additional packages in order to
    # build python from source.
    # Setting a 'global' python is convenient for development but may interfere
    # with other system activities. Adjust as your environment requires.
    pyenv install 3.8.6 && pyenv global 3.8.6

    # Install the tool. This will handle the virtualenv for you, and
# make an appropriately-aliased executable.
# The `--editable` flag is only necessary if you want to work on the
# tool (as opposed to just use it).
python -m pip install --editable .

To ensure the tool is installed properly:

$ which protoc-gen-python_gapic
/path/to/protoc-gen-python_gapic

Usage

To use this plugin, you will need an API which is specified using protocol buffers. Additionally, this plugin makes some assumptions at the margins according to Google API design conventions as described in AIPs, so following those conventions is recommended.

Example

If you want to experiment with an already-existing API, one example is available. (Reminder that this is still considered experimental, so apologies for this part being a bit strange.)

You need to clone the googleapis repository from GitHub:

$ git clone https://github.com/googleapis/googleapis.git

It is possible to generate libraries for most (possibly all) APIs described here. The API we use as an example is the Google Cloud Vision API, available in the google/cloud/vision/v1/ subdirectory. This will be used for the remainder of the examples on this page.

You will also need the common protos, which define certain client-specific annotations. These are in the api-common-protos repository. Clone this from GitHub also:

$ git clone https://github.com/googleapis/api-common-protos.git

Compiling an API

Compile the API into a client library by invoking protoc directly. This plugin is invoked under the hood via. the --python_gapic_out switch.

# This is assumed to be in the `googleapis` project root, and we also
# assume that api-common-protos is next to it.
$ protoc google/cloud/vision/v1/*.proto \
    --proto_path=../api-common-protos/ --proto_path=. \
    --python_gapic_out=/dest/

Note

A reminder about paths.

Remember that protoc is particular about paths. It requires all paths where it expects to find protos, and order matters. In this case, the common protos must come first, and then the path to the API being built.

Generating Samples

In addition to generating client libraries, the generator can also create standalone executable code samples.

The user can specify individual sample config files or can pass paths to directories that contain sample configs. Directories are searched recursively, and any file that is not a sample config is ignored.

A full description of the sample config, generated manifest, and generated samples is outside the scope of this documentation. We will provide links to such documentation when it is ready.

Samples and manifests are always generated in a ‘samples’ subdir of the destination directory.

# Multiple sample paths or directories can be passed simultaneously by duplicating
# the 'samples' option. Options are comma delimited.
# If no 'samples' option is passed, the generator does not generate a manifest.
$ protoc path/to/api/protos/*.proto \
              --proto_path=../api-common-protos/ \
              --proto_path=. \
              --python_gapic_opt="samples=sample_config.yaml,samples=sample_dir/" \
              --python_gapic_out=/dest/

Verifying the Library

Once you have compiled a client library, whether using a Docker image, local installation or bazel, it is time for the fun part: actually running it!

Create a virtual environment for the library:

$ virtualenv ~/.local/client-lib --python=`which python3.7`
$ source ~/.local/client-lib/bin/activate

Next, install the library:

$ cd dest/
$ pip install --editable .

Now it is time to play with it! Here is a test script:

# This is the client library generated by this plugin.
from google.cloud import vision

# Instantiate the client.
#
# If you need to manually specify credentials, do so here.
# More info: https://cloud.google.com/docs/authentication/getting-started
#
# If you wish, you can send `transport='grpc'` or `transport='http'`
# to change which underlying transport layer is being used.
ia = vision.ImageAnnotatorClient()

# Send the request to the server and get the response.
response = ia.batch_annotate_images({
    'requests': [{
        'features': [{
            'type': vision.Feature.Type.LABEL_DETECTION,
        }],
        'image': {'source': {
            'image_uri': 'https://images.pexels.com/photos/67636'
                         '/rose-blue-flower-rose-blooms-67636.jpeg',
        }},
    }],
})
print(response)

Bazel Build

This generator can be called from Bazel, which is a recommended way of using it inside a continuous integration build or any other automated pipeline.

Installing

Bazel

You will need Bazel version 3.0+. Please check the Bazel website for the available installation options.

Bazel is distributed in a form of a single binary, so one of the easiest ways to install it is simply downloading the binary and making it executable:

curl -L https://github.com/bazelbuild/bazel/releases/download/3.2.0/bazel-3.2.0-linux-x86_64 -o bazel
chmod +x bazel

Python and Dependencies

Bazel build is mostly hermetic, with a few exceptions for Python generator. Specifically it expects Python 3.7+ with the python dev packages to be installed.

On Linux, to install those, simply run:

sudo apt-get install \
    python-dev \
    python3-dev

Usage

To use this plugin, you will need an API which is specified using protocol buffers. Additionally, this plugin makes some assumptions at the margins according to Google API design conventions as described in AIPs, so following those conventions is recommended.

Example

To generate a client library with Bazel you will need a Bazel workspace. An example of such workspace would be googleapis. It is already integrated with this this generator in its WORKSPACE file.

You need to clone the googleapis repository from GitHub:

$ git clone https://github.com/googleapis/googleapis.git

The API we use as an example is the Document AI API, available in the google/cloud/documentai/v1beta2/ subdirectory.

Creating the Targets

To build something with bazel you need to create the corresponding targets in your BUILD.bazel file. You can use the Python section of the Document AI BUILD.bazel file as an example:

load(
    "@gapic_generator_python//rules_python_gapic:py_gapic.bzl",
    "py_gapic_library"
)

load(
    "@gapic_generator_python//rules_python_gapic:py_gapic_pkg.bzl",
    "py_gapic_assembly_pkg"
)

py_gapic_library(
    name = "documentai_py_gapic",
    srcs = [":documentai_proto"],
)

py_gapic_assembly_pkg(
    name = "documentai-v1beta2-py",
    deps = [
        ":documentai_py_gapic",
    ],
)

Compiling an API

To generate the client library simply run the bazel command from the repository root, specifying the py_gapic_assembly_pkg target name as the argument:

bazel build //google/cloud/documentai/v1beta2:documentai-v1beta2-py

This will generate a tar.gz archive with the generated library packaged in it. To unpack it in dest location simply run the following command from the Bazel workspace root:

tar -xzpf bazel-bin/google/cloud/documentai/v1beta2/documentai-v1beta2-py.tar.gz -C dest

Verifying the Library

Once you have compiled a client library, whether using a Docker image, local installation or bazel, it is time for the fun part: actually running it!

Create a virtual environment for the library:

$ virtualenv ~/.local/client-lib --python=`which python3.7`
$ source ~/.local/client-lib/bin/activate

Next, install the library:

$ cd dest/
$ pip install --editable .

Now it is time to play with it! Here is a test script:

# This is the client library generated by this plugin.
from google.cloud import vision

# Instantiate the client.
#
# If you need to manually specify credentials, do so here.
# More info: https://cloud.google.com/docs/authentication/getting-started
#
# If you wish, you can send `transport='grpc'` or `transport='http'`
# to change which underlying transport layer is being used.
ia = vision.ImageAnnotatorClient()

# Send the request to the server and get the response.
response = ia.batch_annotate_images({
    'requests': [{
        'features': [{
            'type': vision.Feature.Type.LABEL_DETECTION,
        }],
        'image': {'source': {
            'image_uri': 'https://images.pexels.com/photos/67636'
                         '/rose-blue-flower-rose-blooms-67636.jpeg',
        }},
    }],
})
print(response)

How Code Generation Works

This page gives a brief decription of how this code generator works. It is not intended to be the final treatise on how to write any code generator. It is meant to be a reference for those who wish to contribute to this effort, or to use it as a reference implementation.

There are two steps: a parse step which essentially involves reorganizing data to make it more friendly to templates, and a translation step which sends information about the API to templates, which ultimately write the library.

The protoc contract

This code generator is written as a protoc plugin, which operates on a defined contract. The contract is straightforward: a plugin must accept a CodeGeneratorRequest (essentially a sequence of FileDescriptor objects) and output a CodeGeneratorResponse.

If you are unfamiliar with protoc plugins, welcome! That last paragraph likely sounded not as straightforward as claimed. It may be useful to read plugin.proto and descriptor.proto before continuing on. The former describes the contract with plugins (such as this one) and is relatively easy to digest, the latter describes protocol buffer files themselves and is rather dense. The key point to grasp is that each .proto file compiles into one of these proto messages (called descriptors), and this plugin’s job is to parse those descriptors.

That said, you should not need to know the ins and outs of the protoc contract model to be able to follow what this library is doing.

Entry Point

The entry point to this tool is gapic/cli/generate.py. The function in this module is responsible for accepting CLI input, building the internal API schema, and then rendering templates and using them to build a response object.

Parse

As mentioned, this plugin is divided into two steps. The first step is parsing. The guts of this is handled by the API object, which is this plugin’s internal representation of the full API client.

In particular, this class has a build() method which accepts a sequence of FileDescriptor objects (remember, this is protoc’s internal representation of each proto file). That method iterates over each file and creates a Proto object for each one.

Note

An API object will not only be given the descriptors for the files you specify, but also all of their dependencies. protoc is smart enough to de-duplicate and send everything in the correct order.

The API object’s primary purpose is to make sure all the information from the proto files is in one place, and reasonably accessible by Jinja templates (which by design are not allowed to call arbitrary Python code). Mostly, it tries to avoid creating an entirely duplicate structure, and simply wraps the descriptor representations. However, some data needs to be moved around to get it into a structure useful for templates (in particular, descriptors have an unfriendly approach to sorting protobuf comments, and this parsing step places these back alongside their referent objects).

The internal data model does use wrapper classes around most of the descriptors, such as Service and MessageType. These consistently contain their original descriptor (which is always spelled with a _pb suffix, e.g. the Service wrapper class has a service_pb instance variable). These exist to handle bringing along additional relevant data (such as the protobuf comments as mentioned above) and handling resolution of references (for example, allowing a Method to reference its input and output types, rather than just the strings).

These wrapper classes follow a consistent structure:

• They define a __getattr__ method that defaults to the wrapped desctiptor unless the wrapper itself provides something, making the wrappers themselves transparent to templates.
• They provide a meta attribute with metadata (package information and documentation). That means templates can consistently access the name for the module where an object can be found, or an object’s documentation, in predictable and consistent places (thing.meta.doc, for example, prints the comments for thing).

Translation

The translation step follows a straightfoward process to write the contents of client library files.

This works by reading in and rendering Jinja templates into a string. The file path of the Jinja template is used to determine the filename in the resulting client library.

More details on authoring templates is discussed on the Templates page.

Exit Point

Once the individual strings corresponding to each file to be generated is collected into memory, these are pieced together into a CodeGeneratorResponse object, which is serialized and written to stdout.

Templates

This page provides a description of templates: how to write them, what variables they receive, and so on and so forth.

In many cases, it should be possible to provide alternative Python libraries based on protocol buffers by only editing templates (or authoring new ones), with no requirement to alter the primary codebase itself.

Jinja

All templates are implemented in Jinja, Armin Ronacher’s excellent templating library for Python. This document assumes that you are already familiar with the basics of writing Jinja templates, and does not seek to cover that here.

Locating Templates

Templates are included in output simply on the basis that they exist. There is no master list of templates; it is assumed that every template should be rendered (unless its name begins with a single underscore).

Note

Files beginning with an underscore (_) are not rendered by default. This is to allow them to be used with extends and include. However, __init__.py.j2 is rendered.

The name of the output file is based on the name of the template, with the following string replacements applied:

• The .j2 suffix is removed.
• $namespace is replaced with the namespace specified in the client, converted to appropriate Python module case. If there is no namespace, this segment is dropped. If the namespace has more than one element, this is expanded out in the directory structure. (For example, a namespace of ['Acme', 'Manufacturing'] will translate into acme/manufacturing/ directories.)
• $name is replaced with the client name. This is expected to be present.
• $version is replaced with the client version (the version of the API). If there is no specified version, this is dropped.
• $service is replaced with the service name, converted to appropriate Python module case. There may be more than one service in an API; read on for more about this.

Note

$name_$version is a special case: It is replaced with the client name, followed by the version. However, if there is no version, both it and the underscore are dropped.

Context (Variables)

Every template receives one variable, spelled api. It is the API object that was pieced together in the parsing step.

Most APIs also receive one additional variable depending on what piece of the API structure is being iterated over:

• Services. APIs can (and often do) have more than one service. Therefore, templates with $service in their name are rendered once per service, with the $service string changed to the name of the service itself (in snake case, because this is Python). These templates receive a service variable (an instance of Service) corresponding to the service currently being iterated over.
• Protos. Similarly, APIs can (and often do) have more than one proto file containing messages. Therefore, templates with $proto in their name are rendered once per proto, with the $proto``string changed to the name of the proto file. These templates receive a ``proto variable (an instance of Proto) corresponding to the proto currently being iterated over.

Filters

Additionally, templates receive a limited number of filters useful for writing properly formatted templates.

These are:

• rst (rst()): Converts a string to ReStructured Text. If the string appears not to be formatted (contains no obvious Markdown syntax characters), then this method forwards to wrap.
• sort_lines (sort_lines()): Sorts lines of text, optionally de-duplicating if there are duplicates. This works best with the Jinja {% filter sort_lines %} style syntax.
• snake_case (to_snake_case()): Converts a string in any sane case system to snake case.
• wrap (wrap()): Wraps arbitrary text. Keyword arguments on this method such as offset and indent should make it relatively easy to take an arbitrary string and make it wrap to 79 characters appropriately.

Custom templates

It is possible to provide your own templates.

To do so, you need a folder with Jinja templates. Each template must have a .j2 extension (which will be stripped by this software when writing the final file; see above). Additionally, when you provide your own templates, the filename substitutions described above still occur.

Building Locally

To specify templates, you need to provide a --python_gapic_opt argument to protoc, with a key-value pair that looks like:

–python_gapic_opt=”python-gapic-templates=/path/to/templates”

It is also possible to specify more than one directory for templates (in which case they are searched in order); to do this, provide the argument multiple times:

–python_gapic_opt=”python-gapic-templates=/path/to/templates” –python_gapic_opt=”python-gapic-templates=/other/path”

If you provide your own templates, the default templates are no longer consulted. If you want to add your own templates on top of the default ones provided by this library, use the special DEFAULT string:

–python_gapic_opt=”python-gapic-templates=/path/to/templates” –python_gapic_opt=”python-gapic-templates=DEFAULT”

Building with Docker

When building with Docker, you instead provide the --python-gapic-templates argument after the docker run command:

$ docker run \
  --mount type=bind,source=google/cloud/vision/v1/,destination=/in/google/cloud/vision/v1/,readonly \
  --mount type=bind,source=dest/,destination=/out/ \
  --mount type=bind,source=/path/to/templates,destination=/templates/,readonly \
  --rm \
  --user $UID \
  gcr.io/gapic-images/gapic-generator-python \
  --python-gapic-templates /templates/ \
  --python-gapic-templates DEFAULT

As before, to provide more than one location for templates, specify the argument more than once.

Warning

If you are using custom templates with Docker, be sure to also mount the directory with the templates into the Docker image; otherwise the generator will not be able to read that directory. When specifying the --python-gapic-templates argument, it is the path inside the Docker image that matters!

Features and Limitations

Nice things this client does:

• Implemented in pure Python, with language-idiomatic templating tools.
• It supports multiple transports: both gRPC and protobuf over HTTP/1.1. A JSON-based transport would be easy to add.
• It uses a lighter-weight configuration, specified in the protocol buffer itself.

As this is experimental work, please note the following limitations:

• The output only works on Python 3.5 and above.
• The configuration annotations are experimental and provided in an awkward location.
• gRPC must be installed even if you are not using it (this is due to some minor issues in api-core).
• No support for samples yet.

Reference

Below is a reference for the major classes and functions within this module.

It is split into three main sections:

• The schema module contains data classes that make up the internal representation for an API. The API contains thin wrappers around protocol buffer descriptors; the goal of the wrappers is to mostly expose the underlying descriptors, but make some of the more complicated access and references easier in templates.
• The generator module contains most of the logic. Its Generator class is the thing that takes a request from protoc and gives it back a response.
• The utils module contains utility functions needed elsewhere, including some functions that are sent to all templates as Jinja filters.

Note

Templates are housed in the templates directory, which is a sibling to the modules listed above.

generator

The generator module contains the code generation logic.

The core of this work is around the Generator class, which divides up the processing of individual templates.

class gapic.generator.generator.Generator(opts: gapic.utils.options.Options)

A protoc code generator for client libraries.

This class provides an interface for getting a CodeGeneratorResponse for an API schema object (which it does through rendering templates).

Parameters:

• opts (Options) – An options instance.
• templates (str) – Optional. Path to the templates to be rendered. If this is not provided, the templates included with this application are used.

get_response(api_schema: gapic.schema.api.API, opts: gapic.utils.options.Options) → google.protobuf.compiler.plugin_pb2.CodeGeneratorResponse

Return a CodeGeneratorResponse for this library.

This is a complete response to be written to (usually) stdout, and thus read by protoc.

Parameters:

• api_schema (API) – An API schema object.
• opts (Options) – An options instance.

Returns:

A response describing appropriate files and contents. See plugin.proto.

Return type:

CodeGeneratorResponse

schema

The schema module provides a normalized API representation.

In general, this module can be considered in three parts: wrappers, metadata, and a roll-up view of an API as a whole.

These three parts are divided into the three component modules.

api

This module contains the “roll-up” class, API. Everything else in the schema module is usually accessed through an API object.

class gapic.schema.api.API(naming: gapic.schema.naming.Naming, all_protos: Mapping[str, gapic.schema.api.Proto], service_yaml_config: google.api.service_pb2.Service, subpackage_view: Tuple[str, ...] = <factory>)

A representation of a full API.

This represents a top-down view of a complete API, as loaded from a set of protocol buffer files. Once the descriptors are loaded (see load()), this object contains every message, method, service, and everything else needed to write a client library.

An instance of this object is made available to every template (as api).

classmethod build(file_descriptors: Sequence[google.protobuf.descriptor_pb2.FileDescriptorProto], package: str = '', opts: gapic.utils.options.Options = Options(name='', namespace=(), warehouse_package_name='', retry=None, sample_configs=(), autogen_snippets=False, templates=('DEFAULT',), lazy_import=False, old_naming=False, add_iam_methods=False, metadata=False, transport=[], service_yaml_config={}, PYTHON_GAPIC_PREFIX='python-gapic-', OPT_FLAGS=frozenset({'lazy-import', 'transport', 'service-yaml', 'retry-config', 'metadata', 'autogen-snippets', 'warehouse-package-name', 'samples', 'add-iam-methods', 'old-naming'})), prior_protos: Mapping[str, Proto] = None) → gapic.schema.api.API

Build the internal API schema based on the request.

Parameters:

• file_descriptors (Sequence[FileDescriptorProto]) – A list of FileDescriptorProto objects describing the API.
• package (str) – A protocol buffer package, as a string, for which code should be explicitly generated (including subpackages). Protos with packages outside this list are considered imports rather than explicit targets.
• opts (Options) – CLI options passed to the generator.
• prior_protos (Proto) – Previous, already processed protos. These are needed to look up messages in imported protos. Primarily used for testing.

enums

Return a map of all enums available in the API.

http_options

Return a map of API-wide http rules.

messages

Return a map of all messages available in the API.

protos

Return a map of all protos specific to this API.

This property excludes imported protos that are dependencies of this API but not being directly generated.

services

Return a map of all services available in the API.

subpackages

Return a map of all subpackages, if any.

Each value in the mapping is another API object, but the protos property only shows protos belonging to the subpackage.

top_level_enums

Return a map of all messages that are NOT nested.

top_level_messages

Return a map of all messages that are NOT nested.

class gapic.schema.api.Proto(file_pb2: google.protobuf.descriptor_pb2.FileDescriptorProto, services: Mapping[str, gapic.schema.wrappers.Service], all_messages: Mapping[str, gapic.schema.wrappers.MessageType], all_enums: Mapping[str, gapic.schema.wrappers.EnumType], file_to_generate: bool, meta: gapic.schema.metadata.Metadata = <factory>)

A representation of a particular proto file within an API.

classmethod build(file_descriptor: google.protobuf.descriptor_pb2.FileDescriptorProto, file_to_generate: bool, naming: gapic.schema.naming.Naming, opts: gapic.utils.options.Options = Options(name='', namespace=(), warehouse_package_name='', retry=None, sample_configs=(), autogen_snippets=False, templates=('DEFAULT', ), lazy_import=False, old_naming=False, add_iam_methods=False, metadata=False, transport=[], service_yaml_config={}, PYTHON_GAPIC_PREFIX='python-gapic-', OPT_FLAGS=frozenset({'lazy-import', 'transport', 'service-yaml', 'retry-config', 'metadata', 'autogen-snippets', 'warehouse-package-name', 'samples', 'add-iam-methods', 'old-naming'})), prior_protos: Mapping[str, Proto] = None, load_services: bool = True, all_resources: Optional[Mapping[str, gapic.schema.wrappers.MessageType]] = None) → gapic.schema.api.Proto

Build and return a Proto instance.

Parameters:

• file_descriptor (FileDescriptorProto) – The protocol buffer object describing the proto file.
• file_to_generate (bool) – Whether this is a file which is to be directly generated, or a dependency.
• naming (Naming) – The Naming instance associated with the API.
• prior_protos (Proto) – Previous, already processed protos. These are needed to look up messages in imported protos.
• load_services (bool) – Toggle whether the proto file should load its services. Not doing so enables a two-pass fix for LRO response and metadata types in certain situations.

disambiguate(string: str) → str

Return a disambiguated string for the context of this proto.

This is used for avoiding naming collisions. Generally, this method returns the same string, but it returns a modified version if it will cause a naming collision with messages or fields in this proto.

enums

Return top-level enums on the proto.

messages

Return top-level messages on the proto.

module_name

Return the appropriate module name for this service.

Returns:

The module name for this service (which is the service

name in snake case).

Return type:str

names

Return a set of names used by this proto.

This is used for detecting naming collisions in the module names used for imports.

python_modules

Return a sequence of Python modules, for import.

The results of this method are in alphabetical order (by package, then module), and do not contain duplicates.

Returns:The package and module pair, intended for use in a from package import module type of statement. Return type:Sequence[Tuple[str, str]]

resource_messages

Return the file level resources of the proto.

metadata

The metadata module defines schema for where data was parsed from. This library places every protocol buffer descriptor in a wrapper class (see wrappers) before loading it into the API object.

As we iterate over descriptors during the loading process, it is important to know where they came from, because sometimes protocol buffer types are referenced by fully-qualified string (e.g. method.input_type), and we want to resolve those references.

Additionally, protocol buffers stores data from the comments in the .proto in a separate structure, and this object model re-connects the comments with the things they describe for easy access in templates.

class gapic.schema.metadata.Address(name:str='', module:str='', module_path:Tuple[int, ...]=<factory>, package:Tuple[str, ...]=<factory>, parent:Tuple[str, ...]=<factory>, api_naming:gapic.schema.naming.Naming=<factory>, collisions:FrozenSet[str]=<factory>)

child(child_name: str, path: Tuple[int, ...]) → gapic.schema.metadata.Address

Return a new child of the current Address.

Parameters:child_name (str) – The name of the child node. This address’ name is appended to parent. Returns:The new address object. Return type:Address

module_alias

Return an appropriate module alias if necessary.

If the module name is not a collision, return empty string.

This method provides a mechanism for resolving naming conflicts, while still providing names that are fundamentally readable to users (albeit looking auto-generated).

proto

Return the proto selector for this type.

proto_package

Return the proto package for this type.

python_import

Return the Python import for this type.

rel(address: gapic.schema.metadata.Address) → str

Return an identifier for this type, relative to the given address.

Similar to __str__(), but accepts an address (expected to be the module being written) and truncates the beginning module if the address matches the identifier’s address. Templates can use this in situations where otherwise they would refer to themselves.

Parameters:address (Address) – The address to compare against. Returns:The appropriate identifier. Return type:str

resolve(selector: str) → str

Resolve a potentially-relative protobuf selector.

This takes a protobuf selector which may be fully-qualified (e.g. foo.bar.v1.Baz) or may be relative (Baz) and returns the fully-qualified version.

This method is naive and does not check to see if the message actually exists.

Parameters:selector (str) – A protobuf selector, either fully-qualified or relative. Returns:An absolute selector. Return type:str

sphinx

Return the Sphinx identifier for this type.

subpackage

Return the subpackage below the versioned module name, if any.

with_context(*, collisions: FrozenSet[str]) → gapic.schema.metadata.Address

Return a derivative of this address with the provided context.

This method is used to address naming collisions. The returned Address object aliases module names to avoid naming collisions in the file being written.

class gapic.schema.metadata.BaseAddress(name:str='', module:str='', module_path:Tuple[int, ...]=<factory>, package:Tuple[str, ...]=<factory>, parent:Tuple[str, ...]=<factory>)

class gapic.schema.metadata.FieldIdentifier(ident:gapic.schema.metadata.Address, repeated:bool)

class gapic.schema.metadata.Metadata(address:gapic.schema.metadata.Address=<factory>, documentation:google.protobuf.descriptor_pb2.Location=<factory>)

doc

Return the best comment.

This property prefers the leading comment if one is available, and falls back to a trailing comment or a detached comment otherwise.

If there are no comments, return empty string. (This means a template is always guaranteed to get a string.)

with_context(*, collisions: FrozenSet[str]) → gapic.schema.metadata.Metadata

Return a derivative of this metadata with the provided context.

This method is used to address naming collisions. The returned Address object aliases module names to avoid naming collisions in the file being written.

naming

class gapic.schema.naming.Naming(name: str = '', namespace: Tuple[str, ...] = <factory>, version: str = '', product_name: str = '', proto_package: str = '', _warehouse_package_name: str = '')

Naming data for an API.

This class contains the naming nomenclature used for this API within templates.

An concrete child of this object is made available to every template (as api.naming).

static build(*file_descriptors, opts: gapic.utils.options.Options = Options(name='', namespace=(), warehouse_package_name='', retry=None, sample_configs=(), autogen_snippets=False, templates=('DEFAULT', ), lazy_import=False, old_naming=False, add_iam_methods=False, metadata=False, transport=[], service_yaml_config={}, PYTHON_GAPIC_PREFIX='python-gapic-', OPT_FLAGS=frozenset({'lazy-import', 'transport', 'service-yaml', 'retry-config', 'metadata', 'autogen-snippets', 'warehouse-package-name', 'samples', 'add-iam-methods', 'old-naming'}))) → gapic.schema.naming.Naming

Return a full Naming instance based on these file descriptors.

This is pieced together from the proto package names as well as the google.api.metadata file annotation. This information may be present in one or many files; this method is tolerant as long as the data does not conflict.

Parameters:file_descriptors (Iterable[FileDescriptorProto]) – A list of file descriptor protos. This list should only include the files actually targeted for output (not their imports). Returns:

A Naming instance which is provided to

templates as part of the API.

Return type:Naming Raises:ValueError – If the provided file descriptors contain contradictory information.

long_name

Return an appropriate title-cased long name.

module_name

Return the appropriate Python module name.

module_namespace

Return the appropriate Python module namespace as a tuple.

namespace_packages

Return the appropriate Python namespace packages.

versioned_module_name

Return the versiond module name (e.g. apiname_v1).

If there is no version, this is the same as module_name.

warehouse_package_name

Return the appropriate Python package name for Warehouse.

class gapic.schema.naming.NewNaming(name: str = '', namespace: Tuple[str, ...] = <factory>, version: str = '', product_name: str = '', proto_package: str = '', _warehouse_package_name: str = '')

versioned_module_name

Return the versiond module name (e.g. apiname_v1).

If there is no version, this is the same as module_name.

class gapic.schema.naming.OldNaming(name: str = '', namespace: Tuple[str, ...] = <factory>, version: str = '', product_name: str = '', proto_package: str = '', _warehouse_package_name: str = '')

versioned_module_name

Return the versiond module name (e.g. apiname_v1).

If there is no version, this is the same as module_name.

wrappers

Module containing wrapper classes around meta-descriptors.

This module contains dataclasses which wrap the descriptor protos defined in google/protobuf/descriptor.proto (which are descriptors that describe descriptors).

These wrappers exist in order to provide useful helper methods and generally ease access to things in templates (in particular, documentation, certain aggregate views of things, etc.)

Reading of underlying descriptor properties in templates is okay, a __getattr__ method which consistently routes in this way is provided. Documentation is consistently at {thing}.meta.doc.

class gapic.schema.wrappers.CommonResource(type_name:str, pattern:str)

class gapic.schema.wrappers.EnumType(enum_pb: google.protobuf.descriptor_pb2.EnumDescriptorProto, values: List[gapic.schema.wrappers.EnumValueType], meta: gapic.schema.metadata.Metadata = <factory>)

Description of an enum (defined with the enum keyword.)

ident

Return the identifier data to be used in templates.

options_dict

Return the EnumOptions (if present) as a dict.

This is a hack to support a pythonic structure representation for the generator templates.

with_context(*, collisions: FrozenSet[str]) → gapic.schema.wrappers.EnumType

Return a derivative of this enum with the provided context.

This method is used to address naming collisions. The returned EnumType object aliases module names to avoid naming collisions in the file being written.

class gapic.schema.wrappers.EnumValueType(enum_value_pb: google.protobuf.descriptor_pb2.EnumValueDescriptorProto, meta: gapic.schema.metadata.Metadata = <factory>)

Description of an enum value.

class gapic.schema.wrappers.Field(field_pb: google.protobuf.descriptor_pb2.FieldDescriptorProto, message: Optional[MessageType] = None, enum: Optional[EnumType] = None, meta: gapic.schema.metadata.Metadata = <factory>, oneof: Optional[str] = None)

Description of a field.

ident

Return the identifier to be used in templates.

inner_mock(stack, visited_fields) → str

Return a repr of a valid, usually truthy mock value.

is_primitive

Return True if the field is a primitive, False otherwise.

map

Return True if this field is a map, False otherwise.

name

Used to prevent collisions with python keywords

primitive_mock(suffix: int = 0) → Union[str, bytes, int, float, List[Any], None]

Generate a valid mock for a primitive type. This function returns the original (Python) type.

If a suffix is provided, generate a slightly different mock using the provided integer.

primitive_mock_as_str() → str

Like primitive mock, but return the mock as a string.

proto_type

Return the proto type constant to be used in templates.

repeated

Return True if this is a repeated field, False otherwise.

Returns:Whether this field is repeated. Return type:bool

required

Return True if this is a required field, False otherwise.

Returns:Whether this field is required. Return type:bool

resource_reference

Return a resource reference type if it exists.

This is only applicable for string fields. Example: “translate.googleapis.com/Glossary”

type

Return the type of this field.

with_context(*, collisions: FrozenSet[str], visited_messages: FrozenSet[MessageType]) → gapic.schema.wrappers.Field

Return a derivative of this field with the provided context.

This method is used to address naming collisions. The returned Field object aliases module names to avoid naming collisions in the file being written.

class gapic.schema.wrappers.HttpRule(method: str, uri: str, body: Optional[str])

Representation of the method’s http bindings.

path_fields(method: ~.Method) → List[Tuple[gapic.schema.wrappers.Field, str, str]]

return list of (name, template) tuples extracted from uri.

sample_request(method: ~.Method) → str

return json dict for sample request matching the uri template.

class gapic.schema.wrappers.MessageType(message_pb: google.protobuf.descriptor_pb2.DescriptorProto, fields: Mapping[str, gapic.schema.wrappers.Field], nested_enums: Mapping[str, EnumType], nested_messages: Mapping[str, MessageType], meta: gapic.schema.metadata.Metadata = <factory>, oneofs: Optional[Mapping[str, Oneof]] = None)

Description of a message (defined with the message keyword).

get_field(*field_path, collisions: FrozenSet[str] = frozenset()) → gapic.schema.wrappers.Field

Return a field arbitrarily deep in this message’s structure.

This method recursively traverses the message tree to return the requested inner-field.

Traversing through repeated fields is not supported; a repeated field may be specified if and only if it is the last field in the path.

Parameters:field_path (Sequence[str]) – The field path. Returns:A field object. Return type:Field Raises:KeyError – If a repeated field is used in the non-terminal position in the path.

ident

Return the identifier data to be used in templates.

map

Return True if the given message is a map, False otherwise.

recursive_field_types

Return all composite fields used in this proto’s messages.

resource_path

If this message describes a resource, return the path to the resource. If there are multiple paths, returns the first one.

with_context(*, collisions: FrozenSet[str], skip_fields: bool = False, visited_messages: FrozenSet[MessageType] = frozenset()) → gapic.schema.wrappers.MessageType

Return a derivative of this message with the provided context.

This method is used to address naming collisions. The returned MessageType object aliases module names to avoid naming collisions in the file being written.

The skip_fields argument will omit applying the context to the underlying fields. This provides for an “exit” in the case of circular references.

class gapic.schema.wrappers.Method(method_pb: google.protobuf.descriptor_pb2.MethodDescriptorProto, input: gapic.schema.wrappers.MessageType, output: gapic.schema.wrappers.MessageType, lro: Optional[gapic.schema.wrappers.OperationInfo] = None, retry: Optional[gapic.schema.wrappers.RetryInfo] = None, timeout: Optional[float] = None, meta: gapic.schema.metadata.Metadata = <factory>)

Description of a method (defined with the rpc keyword).

field_headers

Return the field headers defined for this method.

grpc_stub_type

Return the type of gRPC stub to use.

http_opt

Return the (main) http option for this method.

e.g. {‘verb’: ‘post’

‘url’: ‘/some/path’ ‘body’: ‘*’}

http_options

Return a list of the http bindings for this method.

idempotent

Return True if we know this method is idempotent, False otherwise.

Note: We are intentionally conservative here. It is far less bad to falsely believe an idempotent method is non-idempotent than the converse.

ident

Return the identifier data to be used in templates.

is_deprecated

Returns true if the method is deprecated, false otherwise.

legacy_flattened_fields

top level fields only, required fields first

Type:Return the legacy flattening interface

paged_result_field

Return the response pagination field if the method is paginated.

path_params

Return the path parameters found in the http annotation path template

query_params

Return query parameters for API call as determined by http annotation and grpc transcoding

void

Return True if this method has no return value, False otherwise.

with_context(*, collisions: FrozenSet[str]) → gapic.schema.wrappers.Method

Return a derivative of this method with the provided context.

This method is used to address naming collisions. The returned Method object aliases module names to avoid naming collisions in the file being written.

class gapic.schema.wrappers.Oneof(oneof_pb: google.protobuf.descriptor_pb2.OneofDescriptorProto)

Description of a field.

class gapic.schema.wrappers.OperationInfo(response_type: gapic.schema.wrappers.MessageType, metadata_type: gapic.schema.wrappers.MessageType)

Representation of long-running operation info.

with_context(*, collisions: FrozenSet[str]) → gapic.schema.wrappers.OperationInfo

Return a derivative of this OperationInfo with the provided context.

This method is used to address naming collisions. The returned OperationInfo object aliases module names to avoid naming collisions in the file being written.

class gapic.schema.wrappers.PrimitiveType(meta: gapic.schema.metadata.Metadata, python_type: Optional[type])

A representation of a Python primitive type.

classmethod build(primitive_type: Optional[type])

Return a PrimitiveType object for the given Python primitive type.

Parameters:primitive_type (cls) – A Python primitive type, such as int or str. Despite not being a type, None is also accepted here. Returns:The instantiated PrimitiveType object. Return type:PrimitiveType

class gapic.schema.wrappers.PythonType(meta: gapic.schema.metadata.Metadata)

Wrapper class for Python types.

This exists for interface consistency, so that methods like Field.type() can return an object and the caller can be confident that a name property will be present.

ident

Return the identifier to be used in templates.

class gapic.schema.wrappers.RetryInfo(max_attempts: int, initial_backoff: float, max_backoff: float, backoff_multiplier: float, retryable_exceptions: FrozenSet[google.api_core.exceptions.GoogleAPICallError])

Representation of the method’s retry behavior.

class gapic.schema.wrappers.Service(service_pb: google.protobuf.descriptor_pb2.ServiceDescriptorProto, methods: Mapping[str, gapic.schema.wrappers.Method], visible_resources: Mapping[str, gapic.schema.wrappers.MessageType], meta: gapic.schema.metadata.Metadata = <factory>)

Description of a service (defined with the service keyword).

async_client_name

Returns the name of the generated AsyncIO client class

client_name

Returns the name of the generated client class

has_lro

Return whether the service has a long-running method.

has_pagers

Return whether the service has paged methods.

host

Return the hostname for this service, if specified.

Returns:The hostname, with no protocol and no trailing /. Return type:str

module_name

Return the appropriate module name for this service.

Returns:The service name, in snake case. Return type:str

names

Return a set of names used in this service.

This is used for detecting naming collisions in the module names used for imports.

oauth_scopes

Return a sequence of oauth scopes, if applicable.

Returns:A sequence of OAuth scopes. Return type:Sequence[str]

resource_messages

Returns all the resource message types used in all request and response fields in the service.

resource_messages_dict

Returns a dict from resource reference to the message type. This includes the common resource messages.

Returns:

A mapping from resource path

string to the corresponding MessageType. {“locations.googleapis.com/Location”: MessageType(…)}

Return type:Dict[str, MessageType]

shortname

Return the API short name. DRIFT uses this to identify APIs.

Returns:The api shortname. Return type:str

with_context(*, collisions: FrozenSet[str]) → gapic.schema.wrappers.Service

Return a derivative of this service with the provided context.

This method is used to address naming collisions. The returned Service object aliases module names to avoid naming collisions in the file being written.

utils

gapic.utils.case.to_camel_case(s: str) → str

Convert any string to camel case.

This is provided to templates as the camel_case filter.

Parameters:s (str) – The input string, provided in any sane case system without spaces. Returns:The string in lower camel case. Return type:str

gapic.utils.case.to_snake_case(s: str) → str

Convert any string to snake case.

This is provided to templates as the snake_case filter.

Parameters:s (str) – The input string, provided in any sane case system without spaces. Returns:The string in snake case (and all lower-cased). Return type:str

gapic.utils.lines.sort_lines(text: str, dedupe: bool = True) → str

Sort the individual lines of a block of text.

Parameters:dedupe (bool) – Remove duplicate lines with the same text. Useful for dealing with import statements in templates.

gapic.utils.lines.wrap(text: str, width: int, *, offset: int = None, indent: int = 0) → str

Wrap the given string to the given width.

This uses textwrap.fill() under the hood, but provides useful offset functionality for Jinja templates.

This is provided to all templates as the wrap filter.

Parameters:

• text (str) – The initial text string.
• width (int) – The width at which to wrap the text. If offset is provided, these are automatically counted against this.
• offset (int) – The offset for the first line of text. This value is subtracted from width for the first line only, and is intended to represent the vertical position of the first line as already present in the template. Defaults to the value of indent.
• indent (int) – The number of spaces to indent all lines after the first one.

Returns:

The wrapped string.

Return type:

str