interagent / http-api-design Goto Github PK
View Code? Open in Web Editor NEWHTTP API design guide extracted from work on the Heroku Platform API
Home Page: https://geemus.gitbooks.io/http-api-design/content/en/
License: Other
HTTP API design guide extracted from work on the Heroku Platform API
Home Page: https://geemus.gitbooks.io/http-api-design/content/en/
License: Other
Hello,
I'm having trouble finding a way to use some PLSON functionality that helps me validate a string and escape characters that do not conform to the following:
any-Unicode-character-
except-"-or--or-
control-character
I would like something that if in my string find something other than that I substitute for something.
Tks
Would be nice to just enabled the Gitpage/website.
Why?
The user want to search/find this resource and read on mobile.
I think there should be a list of common error statuses and when to use them. Right now there's only a prescribed 401 for unauthenticated users, and 406 for users without permissions.
It'd be nice to have something like:
429 Too many requests
- for rate limiting APIs422 Unprocessable entity
- for POST/PUT validation errors500 Internal server error
- for errors that are your backend's faultI tried to implement some of these principles and found that it involved certain other resources. For ASP.net for example it seems like CacheCow is the best available way to implement ETags for Caching. Maybe adding some links to similar resources for specific server/client languages would serve as a good jumping off point for people who find this useful.
it'd be great to mention that POST is typically used for creating things (POST /articles
), and PUT/PATCH for editing (PUT /articles/123
).
The current advice implicitly in this doc and explicitly in the Heroku API reference breaks large result sets into ranges, and uses field values as constraints to be applied to the query for the rest of the result set. It isn't hard to stumble upon scenarios where a result set is sorted by a non-unique field (ratings, popularity, counts, times, etc), wherein ranges requested by the specified constraints would yield overlapping result sets.
Unfortunately, the Content-Range
scheme doesn't account for these scenarios (likely, because we haven't yet had such a scenario in the Heroku API). As this is document aims to serve as general API design advice and non-unique sort+pagination is not an uncommon situation, some thought towards this problem may be merited by the document authors. Additionally, our team (at Heroku) has come across some scenarios we'd like to introduce which would benefit from an answer.
@geemus @brandur have either of you thought about this much before?
@mathias and I have discussed the idea of using "composite keys" which include at least one unique field as a component in search to achieve something similar in spirit to the current advice, but nothing clearly simple emerges.
The given timestamp format is RFC5424, a subset of ISO8601
https://tools.ietf.org/html/rfc5424#section-6.2.3
A more liberal approach - though more restrictive than iso8601 - is RFC3339
Would simplify as presently we duplicate content and will have to update separately.
Having things at the top level in most cases improves consistency and helps avoid (at least in some cases) n+1 issues. That said, nested things make since to provide a scope for returning lists in particular (and some things can not easily be de-nested because they do not have globally unique identifiers, but this should probably be avoided also). Anyway... There are some gotchas to this approach, but overall we have tried it a few times and have been finding it useful, so I'd like to expound upon the virtues and usage within the guide (and wanted to start by adding some notes here).
Looks that I'm not the only one who wants to know why it's recommended to do something in that way or another.
So it'll be super helpful to have explanation under each paragraph, cause it helps to understand things and not only blindly follow someone's tips. And we don't need right explanation (I think there's no one), only what the author thinks, why he does so.
I agree that "sensitive data will already have been exposed during the first call". But sensitive data also have been exposed even if respond 403.
For example, I post some data via non-TLS requests. And server respond 403. The sensitive data in request body have been exposed during the request.
202: Request succeeded for a POST, DELETE, or PATCH call that will complete asynchronously
According to the http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html 202
The request has been accepted for processing, but the processing has not been completed.
So that doesn't means Request succeeded. The Request has successfully reached the server, and accepted for processing.
I was also looking at https://developer.yahoo.com/social/rest_api_guide/http-response-codes.html
I'm asking mostly because Swagger is being mentioned here. It seems you were aware of Swagger but you took a different path. It would be interesting to know the thought process that made you arrive to JSON Schemas instead.
Hi,
I wasn't clear how you would handle a case where you want to pull in X amount of results, for example in response to a search query without worrying about field names.
The Range header requires a field, so what if you just want to retrieve results regardless of fields. e.g. using a wildcard
Range: * 1..10
Could you elaborate?
https://github.com/interagent/http-api-design#version-with-accepts-header
What benefits does this have over having the version in the URL? I'm just curious, I'm not a huge advocate of using the version in the URL.
But with the URL, it's easier to see (logs etc) which version is being used.
Give each resource an id attribute by default. Use UUIDs unless you have a very good reason not to.
Don’t use IDs that won’t be globally unique across instances of the service or other resources in the service, especially auto-incrementing IDs.
Why is that? Why not use an auto-incrementing ID if the resource have to be identified by a resource name in the URL? The resource is identified by its path, I believe. So /orgs/1 it is complete unique and completely different from the /users/1
With RESTful & microservice architecture, having an unique ID for a resource is too harsh because I can have several distinct databases, fine tuned to each kind of service, and to guarantee uniqueness in this kind of situation within the scope of the entire system... It just doesn't seems to payoff. There is a specific reason for this?
I thought it might help to add a Github page for the Guide, so I took the liberty of generating a super simple one, with one of Github's default page templates.
Does there exist any way to test a server for its compliance with the recommendations (and specifications?) within this repository?
I think when we work with REST API, we have to handling authentication (session) data. OAuth and others based on JWT.
For example: https://github.com/vanioinformatika/poc-angularjs-jwt-rest
I have started looking into another project that I am supposed to add some features to. This project is using 409 Conflict
to convey that a unique constraint has failed. In past I have used 400 BadRequest
and now 422 Unprocessable entity
to indicate failures of unique constraint. I am not sure if use of 409 Conflict
for this purpose is right?
It's JavaScript Object Notation after all...
ie vs summaries for list views and/or mechanisms to specify desired fields.
pulling this in as a distinct issue from #8
/cc @creynders
Mobaily
I translated this guide into Traditional Chinese, and made the structure GitBook compatible. Since we already have translations in two (or more) languages, maybe we can add a section to list these works. Like the followings:
JSON allows for both escaped or non-escaped non-ascii characters.
It'd be useful for this document to include guidance on which style is preferred, or if there is no preference.
For example, the following is valid JSON:
{
"unicode black star": "\u2605"
}
As is the unescaped variant:
{
"unicode black star": "★"
}
I could see valid arguments for either case.
Happy of course if you consider this out-of-scope, but I know it's something I'd value knowing another team's design preferences.
For example, what is the best restful url for the following purpose?
* sub collection, eg. users in a specific country.
* GET /countries/:country/users #seems too hacky, since no country resource indeed.
* GET /users/by-country/:country #may conflict with /users/:name where name=by-country
* GET /users/country/:country
* GET /users/collections/country/:country # borrow ideas from resource/actions/:action
* GET /users-by-country/:country #no conflict, by seems not belongs to users resource.
* GET /user/by-country/:country #no confilict, use user instead of users but strange
* GET /users;country=:country # use matrix params
* GET /users/:country
* query by field, eg. find user by unique auth token
* GET /users/by-auth-token/:token
* GET /users-by-auth-token/:token
sure we can use url params, but when /users/by-country/:country is heavily used, a specific url is nicer.
To better communicate the purpose/intention of attributes it can be useful to use semantic hints (similar in a sense to type hints). They generally appear as suffixes to attribute names, a couple examples we use consistently:
#{past_tense_verb}_at
for timestamps#{past_tense_verb}_on
for dates (time can be ignored and/or should be midnight)Another case we have discussed, but that I'm not quite as convinced about yet. I'm not sure how universal it is, but I think it does make it more human-readable and a bit easier to reason about:
#{noun}_enabled
for boolean values. Historically we have a mix of these and just #{noun}
for boolean attributes, it seems as though choosing one or the other at least would be better.In addition to id/message, consider dev vs user errors, etc.
see: https://github.com/WhiteHouse/api-standards#error-handling
To quote the existing README
Use the plural version of a resource name unless the resource in question is a singleton within the system (for example, in most systems a given user would only ever have one account). This keeps it consistent in the way you refer to particular resources.
This IMO is a bad example - as a system has many users, which could make the entire example difficult to interpret. Perhaps a clearer example of a singleton API is /status
, or /search
.
As mentioned in the gitbook, Returning response with uuid is a better practice. But backend developer usually use database to store data.As we know,It is not recommended to use uuid in the database,So how to solve the problem or haveing good suggestion.
Section 1.6 and the Heroku API reference both recommend using Range
for pagination.
Using the Range
/Next-Range
mechanism from HTTP accidentally exposes in the API that the data is (likely) obtained by computing the requested data within a selected window; for instance, for a query that exposes data from a SQL database:
SELECT * FROM table WHERE condition ORDER BY field LIMIT (#page * size), size
Moreover, this implementation pattern cannot provide a consistent view of the data when concurrent actions can introduce new elements at arbitrary indexes. Consider the following sequence of events:
Next-Range: 20 ..
; let's call the elements e0
to e9
;x
and y
at indexes 9 and 14;e10 = e9
, e11
... e19
: e9
has been seen twice by the client (unless you use id-based pagination), and it saw y
but not x
(despite them having been added simultaneously, regardless of whether you use id-based pagination).The right solution is to provide the API client with a consistent view of the data; for APIs returning results of DB queries, this is easily done using either cursors or materialized views, the second having the additional advantage of supporting later refinement queries.
At the API level, this should be materialized using the Link
header, with (at least) a reference tagged rel=next
. Using Link
allows the API implementer to store there whatever is required to designate the right query result (for instance, a DB cursor id).
The guide doesn't preclude this, but IMHO it's worth including. RateLimit-Remaining is stated as returning the remaining number of request tokens, but it doesn't say how to specify how much time is left in the current 'window', which I think is worth including as a number of seconds.
I think Twitter's approach here is good, even if codifying this into the guide is considered a bad idea, it might be worth a link as a potential approach?
https://dev.twitter.com/docs/rate-limiting/1.1
I wondered why one should not return a 204: No content in response of a successful Patch or Delete request.
Since when you delete a resource, you eighter already have the full resource, or could easilly get it before deleting. The same goes for a patch request.
By doing this, the response can easily be a few kb lighter, which is always good, no?
I'm missing a paragraph on how authentication should be done. The paragraph on "Provide executable examples" mentions using a token with the comment "acquire from dashboard".
Thanks!
We have models using SmallInt
fields to store references to their more verbose string counterparts. Lets take a look at our User
model for example:
# django `models.py` file -- apologies for the framework specific example
class User(models.Model):
"""A representation of a user account."""
GENDER_UNSPECIFIED = 0
GENDER_MALE = 1
GENDER_FEMALE = 2
GENDER_CHOICES = (
(GENDER_UNSPECIFIED, 'unspecified'),
(GENDER_MALE, 'male'),
(GENDER_FEMALE, 'female')
)
gender = models.SmallIntegerField(choices=GENDER_CHOICES, default=GENDER_UNSPECIFIED)
# Create a new user and specify its gender
user = User.objects.create(gender=User.GENDER_MALE)
user.gender == 1
# Django has utility that implements `get_FOO_display` to return the
# more verbose string counterpart of the currently set value
user.get_gender_display() == 'male'
Behind the scenes the data for the gender
field is stored as an integer (for a multitude of reasons). When we render this field, we tend to expose the more "human readable" string counterpart, but when we manipulate the data, we do so by using its "key", like so:
user.gender = 'male'
user.save() # would raise a ValueError
user.gender = User.GENDER_FEMALE
user.save()
user.gender == 2
We're trying to figure out the best way to expose "choice" fields like this over our API and had a few questions:
gender
as an Integer
in our requests -- or should we be using the more human readable string counterpart instead?gender
options to a consumer?Squid
?Currently, a GET
request to our endpoint looks like this:
GET /api/users/
HTTP 200 OK
Content-Type: application/json ;utf-8[ { "url": "http://project.com/api/users/1/", "gender": 2 } ]
If someone posts well formed JSON that is invalid, what http error status is returned and what is the format of the error response?
For purposes of an example, assume that app_name does not allow spaces, what would be the response and status to this?
POST /apps
{"app_name": "foo bar":}
I started this conversation with @geemus via an email. We decided to make our discussion public for feedback.
I've been developing some API Design notes internally for our team at KISSmetrics. I wrote out our base design patterns, research notes, as well as resources for further information.
I had a question for you. In the section Next foreign key relations, you mention utilizing nesting versus owner_id
.
In my experience building API clients I have appreciated both approaches:
owner_id
approach.This way the client doesn't have to inspect the relation envelope to see if it has the full relation or if it needs to look it up via an other request.
I still make the object available in the links
envelope. This way a client can manage expansion or firing off another request if necessary. The option is left up to the client for performance.
Here is an example that may better describe my question.
@geemus responded with:
Good question. I err on the side of consistency where I can, so I'm in favor of always nesting (since it works in both cases). It does provide some possible ambiguity between foreign key and full representations. In our case they are (presently) only ever the foreign key(s), we have a rough plan to allow for users to ask for expanded representations in to the existing envelope in this case (probably via the user passing a header or query string). Since you would need to explicitly ask for the expanded version, I think the ambiguity of foreign keys vs full representation should be somewhat reduced at least. Does that help/make sense? I'm certainly up for having further discussion and would certainly be interested to here more about your guide as it develops.
My (@nateklaiber) response:
I agree with the desire for consistency. I also agree that if you are asking for expanded versions that the ambiguity should be reduced. In practice I have found this a little tough, as many times I wrap the JSON
responses in an object model. Here's an example when working with Stripe.
As I mentioned there, there may be better approaches which I am open to.
I have copied parts of our guide - it's a work in progress.
@geemus then responded with:
Yeah, I guess since we expect you would always have to explicitly expand, that it isn't so bad. In our case most of our serializations are fairly streamlined, so most of the overhead is in DB lookup anyway, so just returning the full thing seemed easiest (and helps prevent inadvertent n+1 stuff from naive clients). I guess my inclination would be toward defaulting to this easier to consume/harder to do wrong version and then perhaps allowing you to narrow the returned value as an optimization or something.
That said, I do think expanding a nested thing from foreign key to full serialization is also a good thing to allow opting-in to for optimization purposes.
It definitely depends a lot on specifics though, I'm not totally convinced we are universally correct there or anything, but it is working well for us and I think could work well for others (but we have other assumptions rolled in there I'm guessing).
I am about to implement a business process which can be more or less modeled as a state machine. In abstract terms, here is how the state machine would look like
[Entity1] --process1--> [state1]--process2-->[state2]--process3-->[final state]
Every state transition is atomic operation. The reason we want to model it as a state machine is that complete operation can take long time to run and we do not want the caller to be blocked. So idea is that we would accept the request from caller and return 202 Accepted
is request looks ok.
After that, a scheduled process would pick up the request from database and trigger process1
by calling a REST endpoint. Same for process2
and process3
.
There are two ways that this can be modeled. First is implement a single REST endpoint like below
PUT http://api.com/entity/{id}/process
Because we know the current state of the entity, we can determine which process to execute next. But then I feel this API is not expressive and following would look better
PUT http://api.com/entity/{id}/process1
PUT http://api.com/entity/{id}/process2
PUT http://api.com/entity/{id}/process3
In the above, there is a distinct endpoint for every process that can be triggered on the entity. If you trigger process1
on an entity which is in final state
then you would get back an error.
I am not sure which one is right or if both of these approaches are wrong. Anyone has any experience of doing something like this in past?
Hi Heroku and other folks,
The link to the Heroku platform docs helps to a certain degree, but it doesn't explain what the benefits are to using this approach to pagination. Given that there's not much tooling available for using the Range header this way, it might be good to describe why it's better than just, for example, embedding page details and next/previous URLs in the JSON response.
Also, some additional detail on how pagination would work in different scenarios would be useful. The platform docs only mention how to sort and specify a maximum value (and what does ".." mean in this context?). It'd be good to see how to specify a range bounded on both ends, and also how to respond when the returned range isn't the same as what was requested (either too many results were requested or there's not enough data to fill the request range).
I expect those docs were written with the expectation that we'd play with the Heroku platform API, but for a broader-purpose doc like this, it'd be good to see some real-world use cases without having to resort to experimenting with a specific implementation.
We commonly include an envelope containing foreign keys for parent objects. To simplify and avoid n+1 queries, it can often be very useful to ask for those envelopes to be expanded to serialize the entire parent object. We've started experimenting with this at Heroku and I wanted to at least note it here and link to a raw doc around it. I want to let the experiment run a bit longer before I settle on it and include it, but wanted to give some early ideas.
Copy of rough spec here: https://gist.github.com/geemus/6b0d59b5813d765c5267
From https://github.com/interagent/http-api-design
"See Summary for Table of Contents." links to:
https://github.com/interagent/http-api-design/blob/master/en/SUMMARY.md
it's OK;
but from Gitbook https://geemus.gitbooks.io/http-api-design/content/en/
this Summary link rewrites to https://geemus.gitbooks.io/http-api-design/content/en/SUMMARY.md
and it's a 404.
Same for the "We welcome contributions to this guide." link:
https://geemus.gitbooks.io/http-api-design/content/CONTRIBUTING.md (404)
A great example is Version with Accepts header. I wouldn't say it's obvious why putting the version in the accept header is valuable as opposed to versioning in the URL.
I perennially run into difficulty trying to draw a line around which properties it's reasonable for a resource to return. I'd love it if you could address that in your docs, or talk about how you settled these issues when building your UI on top of the REST API.
Consider a system with users and orgs. I'd assert that it's reasonable for the UI to present a table of all the orgs, with some vital stats about each. If the table must show the user count for each org, there are two obvious solutions for acquiring the user count:
userCount
to the org resourcecurl https://service.com/orgs
[
{
"id": "01234567-89ab-cdef-0123-456789abcdef"
"name": "SPECTRE",
"userCount": 21
},
// ... additional org descriptors
]
// repeat for each record returned by the org resource
curl https://service.com/orgs/{org_id}/users
[
{
id: "7e876304-bcd9-4ce6-9119-126be52f4486",
username: "eblofeld",
"created_at": "1908-05-25T12:00:00Z",
"updated_at": "1981-06-24T13:00:00Z",]
},
// ... plus 20 more user descriptors
]
Approach 1 can lead to death by a thousand cuts: the server code backing each resource grows with every new property required by UI, increasing CPU cost but reducing API traffic. And per your versioning strategy, the only way to withdraw support for any such property is to cut a new version (while maintaining the old version).
Approach 2 results in both client and server doing lots of work that neither is really interested in; in this case, the content of each user descriptor is discarded, because the only data needed is the length of the array. For collections with many members or whose members have many properties, you also end up with heavier payloads. It also results very quickly in a huge number of requests: 1 + n * c
, where n is the number of rows returned and c is the number of "vital statistics" that aren't part of the base response.
I'm reluctant to draw the line based on the underlying database schema, since that feels like an implementation detail that ought to be abstracted at or by this layer.
Your comments are very much appreciated.
from @joe-mojo #16 (comment)
other role (no update) like sending a message to a resource (handler) or annotating en existing resource.
this raises a point i've been meaning to bring up... using URLs for actions (like /runs/{run_id}/actions/stop) is something that doesn't seem like the most correct thing to do. would it be more appropriate to recommend using something like JSON-RPC and POST /runs/{run_id} where the JSON-RPC request indicates the action to be performed?
I am currently working on enhancing user service which would be consumed by several other services. Other than offering basic CRUD functionality around users this service is also supposed to act as a SAML enabled id provider and so it needs to be able to authenticate users somehow. The current implementation looks like below
PUT /authenticate
{
Userame: {username},
Passowrd: {password}
}
This returns a true
or false
depending on success or failure of authentication. I am thinking of changing this to be more RESTful. I am thinking on these lines.
Authentication is a process so there is real resource involved here. Should I define a new virtual resource? What should this resource be? Something that gets created when authentication succeeds? A LoggedInSession? A SecurityContext? SecurityContext sounds better. So let's design the API as below
POST /securitycontext
{
Userame: {username},
Passowrd: {password}
}
Not sure if I am thinking in the right direction.
ie https://github.com/WhiteHouse/api-standards does a good job of listing examples (and counter-examples) for a number of points, which may help clarify.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.