aip-dev/google.aip.dev

API Improvement Proposals. https://aip.dev/

API Improvement Proposals

TL;DR: AIPs are lots of documents on how Google does APIs.

Overview

AIP stands for API Improvement Proposal, which is a design document providing high-level, concise documentation for API development. The goal is for these documents to serve as the source of truth for API-related documentation at Google and the way API teams discuss and come to consensus on API guidance. The program is named and styled after Python's enhancement proposals (PEPs) which have seemed to work pretty well over the years.

Specific areas inside Google

While much of the API-related guidance is general and spans across all the different products at Google, we've found that some teams working in different areas may have different customs, styles, or guidance. To accommodate these historical differences, we've provided separate blocks of numbers for those areas where they might override or extend the more general guidance.

Getting started

New to AIPs?

If you're new to AIPs, check out the Frequently Asked Questions which answer some common questions about how AIPs work and what you need to know.

Want to use this in your company?

If you like what you see and want to adopt the general AIPs for your organization, check out our guide on Adopting AIPs in your company. This guide walks you through how to start using AIPs and write your own guidance specific to your organization.

Have an idea for an AIP?

If you have an idea for an AIP that isn't written yet (yes, there are plenty!) check out Contributing to the project to see how you can write AIPs for others to follow.

License

Except as otherwise noted, the content of this repository is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License.

For the full text of each license, see LICENSE.md. For additional details, see the developer.google.com Site Policies.

google.aip.dev's Issues

[AIP-156] Singletons

Write an AIP around singleton resources.

• Number: 156
• Source material: Design Patterns

[AIP-151] Long-running operations

We need an AIP for long-running operations.

• Number: 151
• Source material: Design patterns

aip.dev/2510 redirects to a 404...

Not sure why, but aip.dev/2510 doesn't redirect to aip.dev/cloud/2510 -- instead it goes to https://github.com/googleapis/aip/cloud/2510

Hyphens in names

From @jgeewax:

GCP is seeing some conflict between underscores in names or hyphens in names.

In all of GCP except BQ and ML, we use kebob-case (my-first-resource). In BQ we use underscores for table names because hyphen means subtraction in SQL, so this was a conscious choice (my_table, not my-table). In ML, we made a mistake and launched CMLE and others with scale tiers using underscores (complex_model_m rather than complex-model-m). This is wrong primarily because GCE already has very similar options and sticks with hyphens (n1-standard-4).

In general, anywhere we have a string field that accepts a list of options provided by the service (akin to GCE VM preset sizes), we should use hyphens as the separators.

This AIP should also include a "i messed up... how do I fix it?" section

Header bar disappears when you scroll down.

[AIP-123] Unified resource types

Write an AIP around unified resource types.

• Number: 123
• Source material exists but is internal

When to have an updating state

From @jgeewax:

Some folks are asking why some products have resources that can be in a state called UPDATING whereas others don't have this.

Primarily, the UI wants to show a message saying "hey, we're doing that work you asked us to do!"

I think that updating states are useful if it is something that has an effect on the resource itself such that we want to tell users listing the resource elsewhere that they should wait before trying to do anything. In other words, if the update is resizing a VM to have more CPUs, it might make sense to put the VM into the updating state if it may suffer from degraded performance. If we're just updating the display name or description, which happens right away, it wouldn't make sense to do this.

Roll this into AIP-216.

Use (M|G|T)B to mean (Mebi|Gibi|Tebi)bytes

From @jgeewax:

Throughout GCP, we use the common parlance abbreviations for sizes such as MB, GB, or TB, however our interpreted meaning of these in the underlying systems is not what the IEC/ISO standards say they mean.

Luckily we are consistent with this interpretation, so this means that customers aren't triggered by billing numbers to verify that they're getting the number of bytes that the IEC standards say they should.

As a result, we need to standardize on what exactly MB/GB/TB mean. For the purposes of GCP, we use the following abbreviations:

• MB = 2^20 bytes
• GB = 2^30 bytes
• TB = 2^40 bytes

The same abbreviation styles apply when used in terms of throughput or rates of bytes (e.g. MB/s == 2^20 bytes per second)

Field names should be defined using these abbreviations as the unit (e.g., size_gb) rather than the technically correct abbreviation (NOT size_gib).

Fields defined using these abbreviations MUST declare the number of bytes per unit of the field (e.g., size_gb should have a comment saying "Size in GB (2^30 bytes)".) The same goes for throughput (e.g., export_throughput_limit_gbps should have a comment saying "Maximum export throughput in GB/s (2^30 bytes per second).")

Linter should warn when...

• Any fields using the IEC standard abbreviations (e.g., size_gib)
• Any fields using mb/gb/tb don't have a comment saying 2^(20/30/40) bytes per second on the field.
• Any fields don't use the proper abbreviations (*b_per_second instead of *bps, so gb_per_second should be gbps)

TODO:

• Do we standardize on rate limits as "throughput" or "bandwidth" ? (My vote is throughput).

Decouple database and API schemata

We often hear that teams want their APIs proto to "match" their DB protos. We should explain why DB schemas and APIs need to evolve independently. This could be included in AIP-215, a dedicated AIP, or in a FAQ. This is currently an API-don't.

Create an "edit this page" link

[AIP-140] Field names

Write an AIP around field names.

• Number: 140
• Source material: Naming Conventions (and lots of tribal knowledge)

One-hot resources

It is sometimes tempting to have situations where there is a single field that is shared between multiple resources in some way, e.g.

message Thing {
  bool is_default_thing = 1;
  // Other stuff...
}
rpc SetDefaultThing(...)

This is an anti-pattern; there should be a parent to the resource that points to or acts as its default.

[AIP-141] Quantities

We need an AIP around representing quantities and limits.

• What to do:
  • {foo}_count
  • {bar}_limit (discuss minimums)
• Source material: Naming Conventions
• Number: 141

Clarify the meaning of required primitives in AIP-203.

From @garrettjonesgoogle:

e.g.

• Required for booleans means the request fails if the boolean is false (known use case: a force field)
• Required for enums means the request fails if the UNSPECIFIED value is provided

[AIP-153] Import / Export

From @jgeewax:

Currently we have some standards about how to import and export (and back-up and restore) but teams are still doing their own things because we haven't documented this all that well (we're going mostly off of precedent).

We need to document how this should be done and make sure it covers all of the use-cases that are currently in flight and planned for later next year.

[AIP-126] Enumerations

Write an AIP around enums.

• Upsides and downsides
• When to use vs. strings
• When to use vs. bool

I generally tell people:

• Enums are only for sets of values that will change infrequently.
• For enum vs. bool: Enums are valuable when you need an unspecified that is distinct from false, or if false is not the default.
• For enum-like values that are covered by a well-known standard (e.g. language code, currency code), always use strings. Even if you only support a small subset of available values (e.g. only a couple of languages), because chaining.

Number: 126

Lookup Pattern

The standard method for Get expects a resource name, but often API producers want to have an alternate lookup based on a very specific, never-going-to-be-expanded set of fields.

Our current guidance generally suggests using Search for this purpose, but this can be a heavy-weight approach for simple lookups based on an alternative field.

Write an AIP for a "lookup" pattern.

[AIP-160] Filtering

From @jgeewax:

Currently [internal link redacted] has a mostly well-defined syntax and we recommend it for everyone trying to filter things (or query for things, though under a different name).

We should draw up an AIP that:

1. Documents the syntax in a public-friendly way. External customers must have a way of understanding this format.
2. Ask @wora and co to come up with some infrastructure (or choose the recommended option of those already implemented) for each language.
3. Clarify why you should use string filter (or string query) versus a rich proto-defined structured querying/filtering system.

[AIP-142] Time and Duration

Write an AIP around representing time.

• Number: 142
• Source material: Standard Fields, Naming Conventions

Fix menu for mobile viewing

Add the AIP number somewhere more prominent

There have been a few times where I'm looking at an AIP and want to cite it and can't find the number.

At the very least, the <title> tag should have the number. We probably should also have it right under the heading, rather than just off to the right in the info box.

[AIP-146] Generic fields

Write an AIP around generic fields.
Cover when to prefer oneof, map<str, str>, Struct, bytes, etc.

• Number: 125 144 145 146
• Source material: Design Patterns, AIP-2717

[AIP-193] Errors

Write an AIP around errors.

Number: 193
Source material: Errors

[AIP-159] List/Get/Search across collections

We need a single AIP for the pattern for listing / getting / searching across collections.

Permalink for cloud AIPs includes cloud/

I think the default we want is aip.dev/2500 not aip.dev/cloud/2500

Batch methods

Write an AIP around batching (atomicity, handling errors, etc.)

Go back and fix headers the right way

We edited Hercules to make the header bar stop disappearing (commented out two transform directives).

We should go back and fix this the right way ...

Date and time ranges

From @jgeewax:

Currently we have Dates and Times, but not ranges of those. If we don't want to define a standard proto (e.g., DateRange, TimeRange), we should at least specify whether the range should be a start and end timestamp (or date) versus a start time/date and the duration to add to get that.

Also this depends on defining whether ranges should be inclusive/exclusive (#86).

Name / URL match on resources with polymorphic parent types

From @jgeewax:

What happens when a single resource can belong to multiple different types of parents. For example, what if we have a GCP instance that might belong in a folder, directly to an organization, or to a project?

When it comes to the URL pattern matching on these resources we have a couple of options:

1. Use additional bindings (folders/*/..., organizations/*/..., projects/*/...)
   This raises the question of which of these should be the primary binding and which the additional.

2. Use a wild-card which is further enforced by the back-end as a 404 with an incorrect name (e.g., */*/...)
   This makes things a bit less clear as you must rely on the comment to figure out the naming options.

3. Add support for extra groupings like standard regexes (e.g., "(folders|organizations|projects)/*/...")
   This requires work on the infrastructure side to support this specific scenario.

It's unclear which is right, but worth discussing and making a decision.

[AIP-152] Jobs

Write an AIP around jobs.

• Number: 152
• No existing source material to speak of.

[AIP-180] Backwards compatibility

Write an AIP around backwards compatibility as a whole. (The 180-189 block will be for compatibility and versioning.)

• Number: 180
• Useful source material: Versioning, Compatibility

Ranges

From @jgeewax:

Currently we don't specify whether the endpoints of a range are inclusive or exclusive.

We have four different options:

1. [start, end]
2. (start, end)
3. [start, end)
4. (start, end]

For many ranges (like time Durations), this isn't super critical because they provide for microsecond granularity, and rarely do we care all that much about microsecond-level precision, but when the granularity lines up exactly with the precision we care about we actually do need to think about whether the starts and ends are included in or excluded from the range as specified.

This also tends to vary depending on the environment, for example:

• When talking about times, you typically want to be inclusive at the start and exclusive at the end: A meeting scheduled for "3-4 PM" typically starts at 3:00 PM and must end before 4 PM (so the next meeting from 4-5 PM can start).

• When talking about date ranges, you typically want to be inclusive: A work event scheduled for "Feb 1-2" is typically equivalent to Feb 1 AND Feb 2, from 9 AM to 5 PM on each day.

• When talking about items in an array or list, you're typically inclusive at the start and exclusive at the end: Items at indexes "0 to 2" is 2 items (Items 0, and 1, but not 2).

We should collect more examples and come up with some guidance.

Create a friendly landing page (aka, different from the AIP listing)

I think the AIP listing is great, but it doesn't do a good job of summarizing the problem we're tackling with the project generally, and doesn't introduce people to the whole idea.

I think we should split this out into two pages:

1. The landing page should be a simple and friendly introduction to what it is we're actually doing.
2. The AIP listing should focus mainly on listing the AIPs (not offering a second introduction).

Define and document process for adopting AIPs in other orgs

The current thinking is to have a separate GitHub repository that is forkable and can redirect for everything that is not aip-9xxx and serve the 9xxx AIPs directly from the repo

What to do when you do not support partial update

From @jgeewax:

A few APIs either can't or won't support partial update for a variety of reasons:

• For security reasons, they cannot have the situation where a customer updates something and has leftover values that they weren't aware of on the client side.
• They need the ability to update inside maps and go/advanced-field-masks isn't implemented yet.
• They have some other technical limitation that only allows them to update the entire resource.

What do you do? Currently we've seen several options:

1. Use Update that relies on the PUT verb.
2. Use Update with PATCH and a FieldMask update_mask that is simply ignored.
3. Use Update with PATCH and omit a FieldMask update_mask
4. Use Update with PATCH and a FieldMask update_mask that fails with a bad request if the value is set to anything other than null, '', or '*'.
5. Use Replace that relies on the PUT verb

My stance is that we should go with option 4 generally, and in rare exceptions (such as the security requirement in the examples above), go with option 5.

[AIP-191] File and directory structure

Write an AIP around file and directory structure.

• Number: 191
• Source material: File Structure, Directory Structure

Parsing and serializing timestamps

From @jgeewax:

strftime is standard in *nix, but not across all languages. Further, there are some things missing from the *nix standard (e.g., micros and nanos, both absolute and relative to the second itself).

We should come up with a standard for all Google APIs to use when taking a google.protobuf.Timestamp and converting it into a string as well as when taking a string and turning it into a google.protobuf.Timestamp.

From another employee on an internal forum (name withheld as I do not know the person well):

FTR, absl::FormatTime() already supports strftime-like specifiers for subsecond formatting. See the docs here: time.h

I suggest that it may be good to reuse those.

[AIP-125] Message design

Write an AIP around how to design messages well.
(A decent amount of this is likely to be new material.)

• Number: 124
• Source material: Naming Conventions

Unavailable Locations

From @jgeewax:

It's unclear to me what we are supposed to do when someone asks for "all the resources of type X across locations" when some locations are unreachable. Is there a recommended way to handle this scenario?

Compute / Flatten / Lookup Effective Method

From @jgeewax:

We've had a few different teams have an idea of different customization applied in a hierarchy that is then flatted to be evaluated. We need a standard method that covers this use case so that we all do it the same.

Let's assume we have some sort of Config resource that can apply to any resource, as well as orgs, projects, and folders. We want a way of saying "What config applies to my GCE VM (my-instance)?"

First, what do we call the method? Options:

• LookupEffectiveConfig
• CalculateEffectiveConfig
• ComputeEffectiveConfig
• GetEffectiveConfig
• FlattenConfig
• ResolveEffectiveConfig

Second, we need to decide the order in which to apply the hierarchy ( at least for GCP).

I think this would be Org -> Folder -> Project -> Resource, right?

Notification hooks

We need an AIP for the standard pattern for event notifications from APIs. It should cover the general architectural pattern (Pub/Sub topic...anything else?), API surface guidance, and so on.

Also, how do we document what the surface of the notification content is? Can the user control the content type, or is it always JSON? And so on and so forth.

Display names

From @jgeewax:

Extra special note about them not being unique.

[AIP-154] Request validation

Write an AIP around request validation.

• Number: 154
• Source material: Design Patterns

Upsert

From @jgeewax:

Do we allow Upsert? If so, how should it be implemented?

Proposal: yes, let's allow it. No, it should not be part of Update. No it should nto be part of Create. It should be it's own custom verb.

rpc UpsertBook(UpsertBookRequest) returns (Book) {
  option (google.api).http = {
    post: "/v1alpha1/{name=shelves/*/books/*}:upsert"
  };
}

This requires both update and create permissions. If you want to update something and only have Update perms, use Update. Same if you want to create and have Create permissions.

Mobile nav bar has some broken links from inside nested AIPs

To reproduce:

• Open aip.dev/4222 on mobile
• Click the left nav to view other Client Lib AIPs
• See 404

I think the mobile side navigation is using relative links rather than absolute links

Location-bound resource naming

From @jgeewax:

We often see folks put in APIs that have no location in the resource name at all. Sometimes this makes sense (the resource is actually global), but often it's an oversight and teams make the decision without understanding the complete consequences of their choice.

We should clarify when is OK to scope things at a global level, and how that should be done.

Some notes and questions:

• If you are at the global level, you must ensure that you actually make global availability guarantees to customers. In other words, if your ID space is spread across multiple locations, and us-central1-a is down, you can't guarantee that ID 1234 is available since it could already exist in the location that is down.

• Some people have used zones/* or regions/*, when they should use locations/*. There are scenarios when it makes sense to use the more specific option, but that is relatively rare and we should list out those specific case.

• When you have a global resource, should you just leave "locations/*" out of the name? Or use "locations/global" ? Or leave it out with the plan of adding it in later via an additional binding if you decide to support location-bound resources?

• Anything involving storage of customer data should very likely be in specific locations. This is due to data privacy and homing issues (e.g., data on EU people must be in the EU).

• Anything involving transfer of significant amounts of data (measured in 1GB+ ?) should be location-bound. This is because data will likely first be uploaded to GCS and then imported into the service. If you don't know where the data is going, customers don't know where to create their bucket to minimize the egress bandwidth costs.

• One big goal of putting services in specific locations is to minimize the failure domains. That is, if there is an outage in us-central1-a, and all of a customer's VMs and data is in asia-east1-a, it will be very frustrating to this customer who only ever does stuff in Asia to see an outage because something went wrong half-way around the world in Virginia. It's very important that we isolate far-away failures so that customers don't feel as though they're dragged through the muck of any mistake anywhere on the planet.

• We should also consider latency for these services, particularly in far-away places like Sydney, Australia. If someone spins up VMs in Sydney to serve their customers but we have a "global" service that is deployed in Taiwan to handle traffic in the APAC region, that's actually not all that close to Sydney: Asking folks to serve Sydney traffic out of Taipei (~140ms) is a bit like asking them to serve SF traffic out of London (~150ms) which is obviously not something we'd ask customers to do.

Search page with the sidebar has the search button overflowing onto the next line

[AIP-143] Standardized codes

Write an AIP around representing concepts with commonly-codified strings (e.g. language code, mime type, etc.)

This needs a better title. Thanks @noahdietz for helping me come up with a title.

• Number: 143
• Source material: Standard Fields, Naming Conventions

[AIP-192] Documentation

Write an AIP around documentation.

• Number: 192
• Source material: Documentation

Field masks

We need better documentation for field masks:

• Do we use the merge or replace semantics for messages?
• Do we append to lists or replace them?
• Several other advanced situations.