openservicebrokerapi/servicebroker

Open Service Broker API Specification

servicebroker's Issues

cost structure in existing spec is ill-defined

This is all currently optional, so I'm not sure how important it is to nail it down as part of a spec. Discussion did seem rather concerned with uses that allowed charging. I was trying to write down a swagger definition and having trouble with this part of the definition of a plan object.

cost is defined on a plan.

api.md defines it as 'cost'
catalog-metadata.md defines it as 'costs'

the structure of the 'cost' object is defined in terms of usd, which is obviously somewhat centered on a specific currency. As well as using that currency as a key in an object, which is not good.

It is unclear about how multiple costs combine together if they exist.

Use native REST calls into Kube instead of kubectl

We need some guidance on how to get the proper auth token.

Support bindable at the plan level as well as the service

Context

cloudfoundry/cloud_controller_ng#567

https://www.pivotaltracker.com/n/projects/966314/stories/115985379

As a service author I want to be able to flag bindable at the plan level, so I can have granular control

---

The bindable attribute should work by selecting the most specific value. For example:

Service level: bindable=true
Plan level: bindable=false
Result: the plan is not bindable

Service level: bindable=false
Plan level: bindable=true
Result: the plan is bindable

Bindable at the plan level should be optional and default to the service level attributes value.

{
  "services": [{
    "name": "fake-service",
    "id": "acb56d7c-XXXX-XXXX-XXXX-feb140a59a66",
    "description": "fake service",
    "tags": ["no-sql", "relational"],
    "requires": ["route_forwarding"],
    "max_db_per_node": 5,
    "bindable": true,
    "metadata": {
      "provider": {
        "name": "The name"
      },
      "listing": {
        "imageUrl": "http://example.com/cat.gif",
        "blurb": "Add a blurb here",
        "longDescription": "A long time ago, in a galaxy far far away..."
      },
      "displayName": "The Fake Broker"
    },
    "dashboard_client": {
      "id": "398e2f8e-XXXX-XXXX-XXXX-19a71ecbcf64",
      "secret": "277cabb0-XXXX-XXXX-XXXX-7822c0a90e5d",
      "redirect_uri": "http://localhost:1234"
    },
    "plan_updateable": true,
    "plans": [{
      "name": "fake-plan",
      "id": "d3031751-XXXX-XXXX-XXXX-a42377d3320e",
      "description": "Shared fake Server, 5tb persistent disk, 40 max concurrent connections",
      "max_storage_tb": 5,
      "bindable": true, #optional
      "metadata": {
        "cost": 0,
        "bullets": [{
          "content": "Shared fake server"
        }, {
          "content": "5 TB storage"
        }, {
          "content": "40 concurrent connections"
        }]
      }
    }]
      }
    }]
  }]
}

Verify that the registry->service instance APIs align with CF's definition

https://docs.cloudfoundry.org/services/api.html
Basically, the service registry->service broker calls related to #24
https://docs.cloudfoundry.org/services/api.html#provisioning

Verify the service instance APIs aligns with CF's definition

This is on the calls from a CLI to a service registry.

The cf calls:

   service                                Show service instance info
   create-service                         Create a service instance
   update-service                         Update a service instance
   delete-service                         Delete a service instance
   rename-service                         Rename a service instance

Support service-to-service bindings

In order for bindings to be of any value, they must be able to model relationships between two service instances. The current CF model for bindings is a CF app to a service instance, so this must be extended.

The initial naive proposal for PoC purposes is:

1. Binding happens between a from service instance and a to service instance
2. From service instance is instantiated only after all its bindings have been created
3. Instantiating the from service acts as a late binding for all the previously created bindings
4. Service broker understands binding information as part of a service instance creation request

Concretely, this is:

1. Replace app_guid with from_service_instance_name in binding request
2. Store all bindings in a structure for the uninstantiated from service instance
3. Pass binding information as part of service instantiation to service broker

Open questions:

1. What is the format/syntax of binding information in service broker request?
2. Late binding has many cons, e.g., no access to any identity information for the from service instance like IP address, etc.

Verify that the cf bind CLI ops align with CF's definition

cf actions:

   bind-service                           Bind a service instance to an app
   unbind-service                         Unbind a service instance from an app

followup tracking issues from #95

• The broker API has since been versioned to 2.10. Rebase or submit an additional PR with 2.10 changes?

• Why were the sections on the version header and authentication removed? Don't these deserve some explanation? They appear in example curls, but there is no other explanation. How should the broker author know what their responsibilities are?

• With the exception of the table of contents, none of the links work.

• The example response body for Provision contains "dashboard_url". The backslash should be removed. This may have been in an older version of github.com/cloudfoundry/docs-services but has since been fixed there.

• The first paragraph of the intro for Updating a Service Instance is out of date. I recall rewriting that some time ago.

• The whole section titled "Manage Application Requests with Route Services" can be removed. This link appears in the index of the current docs, but I've just sent a PR to remove it. This topic is for marketplace end-users (app developers), not for service broker authors. cloudfoundry/docs-services#145

• Remove the whole section titled "Volume Services (Experimental/Obsolete)" and fix the link in TOB for "Volume Services (Experimental)"

• You can remove the section "Using cf curl" under Access Control, as well as instructions to get the api version. I've just submitted a PR to docs-services to do the same. cloudfoundry/docs-services#147

Fix up links in spec

Now that multiple pages have been squished into one, all the links must be updated.

Verify the cf cli APIs for user-defined services aligns with CF's defintion

• Service Brokers
• Service Instances
• Service Bindings
• Services
• Service Plans

Service broker can declare supported schema for parameters field

Service broker will provide schema information for both service instances and bindings for each plan. The schema information is for both parameter inputs and output properties. Currently the proposal is that the schema format will be JSON schema v4.

Proposal: https://docs.google.com/document/d/1-IKI-PwPnhtbK0su1UzWA_UoyaCjL8S2d6ky3Xmuwvg/edit?usp=sharing

Rebase spec against 2.10

The spec is out of date. Rebase against latest changes in https://github.com/cloudfoundry/docs-services, including changes for v2.10 of the spec.

Ability to declare schema of binding credential output

Binding credentials are a generic map of data, and as such are difficult to reason about or program against in a meaningful way. Similar to issue #59, we would like to support the ability to declare the schema of credential outputs from a binding.

Proposal: https://docs.google.com/document/d/1JbsJgqgNtqthcfYwK_KbS6C8sjElrZNgoLhu40dUPAs/edit?usp=sharing

Service Instance update question

I understand the spec offers service plan updates (https://github.com/openservicebrokerapi/servicebroker/blob/master/_spec.md#updating_service_instance) but I'm concerned with a scenario where the service instance itself has changed.

Consider a case where a service provider offers a service which is "single tenant provisioned postgresql" and at the time a user provisions an instance of that service, they get their own postgresql v9.4 server stood up.

Later, the service provider decides that they are going to offer postgresql v9.5 instead. So users who provision from that day forward get a v9.5 server. Is there a workflow for legacy users to request their existing v9.4 instances get updated? Is the assumption that the user will have to deprovision their instance and provision a new one? Or that the service provider is responsible for handling updating provisioned instances outside the scope of a service catalog/service broker flow? Or should the v9.5 offering be registered as a new "plan"? (Is it an abuse of "plan" to use plan to handling the versioning of the catalog offering?)

@pmorie @jim-minter

Metadata spec appears to be missing from the spec

Likely removed inadvertently in #131

Option for an imageClass instead of an imageURL

We often try to use font icons (such as http://fontawesome.io/icons/ ) for images in UIs to reduce the need to load a large number of unique images. It would be good to have the option to define a CSS class for the image for a Service or Plan instead of requiring a unique URL for an image so that font icons can be used for these. UI's should be able to handle the existence of either an imageClass or an imageURL. This allows brokers that are knowledgeable about the frontends they are being deployed on to use known classes to minimize unique images that need to be hosted, but still allows brokers that are expected to be deployed across many environments to specify URLs for their images.

Add support for backup & restore to the service broker api

Problem

As a service user I want to have confidence that my service instance is backed up and know that if something bad happens that I can restore from a backup.

As a service operator I want to offer service users a self-service mechanism to perform backup and restores of the service instance, so I don't have to manage this process.

Workflows

As a service user I can trigger a backup for a given service instance

As a service user I can list all backups for a given service instance

As a service user I can restore to an existing service instance

As a service operator I can choose not to offer self-service backups

As a service operator I can choose not to offer self-service restores

---

Next steps

1. Community feedback on this feature
2. Create proposal google document

Improved authentication for security and federated provisioning

Given that a standardized API will be implemented and consumed by more parties, possibly accessed over the open Internet, does it make sense to allow and/or standardize on other auth mechanisms (i.e. such as OAuth?). The Spec currently specifies Basic Auth as the only mechanism for authentication.

EDIT by @avade

Working document: https://docs.google.com/document/d/1XLPMLJbf_lRXIdMg1XUyKNZUd2--fDROGvdEm25tMjc/edit

Remove K8 development from this repo

This github repo should be for development of the broker api specification. Example brokers could be in separate repos.

I would suggest that the only things that should be in this repo now are a README, license file, and the spec (w/images)

broken links

The spec contains some links that result in a 404 not found

• "Application Log Streaming" app-log-streaming.md
• "Route Services" route-services.md
• "Volume Services" volume-services.md
• "Service Metadata" catalog-metadata.md
• "Dashboard SSO feature" dashboard-sso.md
• "Binding Credential" binding-credentials.md
• "Manage Application Requests with Route Services" route-binding.md
• "service broker v2.10 format for volume mounts" volume-services-v2.10.md
• "Quota Plans" external link to docs.cloudfoundry.org
• "Example Service Brokers" examples.md
• "v1 broker API" api-v1.md
• "Service Broker API" api.md

Some of the links are part of a paragraph that looks like it is referencing itself or an incomplete paragraph.
E.g. #application-log-streaming

Application Log Streaming
...
For details, see Application Log Streaming.

Differentiate endpoint information in binding credentials

Bindings return several pieces of information as part of the credential response. One key set of data is endpoint information necessary for connecting to the service. There is currently no way of differentiating this information, but it is critical to be able to access from an automated system for establishing network connectivity in various ways as part of the binding consumption process.

This problem is exacerbated by the fact that many different brokers expose endpoint information under different field names, e.g., ip, host, hostname, address.

We need the ability to understand endpoint information about a binding, including both address/hostname and port, whether it is by annotation in a schema (requires #116), or by making these concrete fields.

Proposal is introduced as part of the proposal for #116: https://docs.google.com/document/d/1JbsJgqgNtqthcfYwK_KbS6C8sjElrZNgoLhu40dUPAs/edit?usp=sharing

Verify that the GET /v2/catalog call is aligned with CF's definition

https://docs.cloudfoundry.org/services/api.html

this is on the call from the service registry to a service broker

Inconsistent specification of accepts_incomplete

https://github.com/openservicebrokerapi/servicebroker/blob/master/_spec.md#asynchronous-operations

• specifies that accepts_incomplete is a query parameter for provision, update and deprovision

https://github.com/openservicebrokerapi/servicebroker/blob/master/_spec.md#request-2

• specifies that accepts_incomplete is part of the request body

k8s backend

Tracking issue

To prove our abstraction works beyond local storage.

• create the storage resource on k8s
• service brokers
  • add broker #29
  • list brokers #29 & adjusted in #45
  • get broker #45
  • delete broker #47
  • update broker #92
• service definition
  • add #84
  • list #85
  • get #85
  • delete
  • update #88
• service plans
  • add #84
  • list
  • get #85
  • delete #91
  • update #92
• service instances
  • add #88
  • list #88
  • get #88
  • delete #91
  • update #89
• service bindings
  • add #89
  • list #89
  • get #89
  • delete #91
  • update #92

Support search in the /catalog API

Following on from #126.
It was discussed that pagination is required since the catalog may respond with 10k+ entries. This would also mean that the paged response could include hundreds of pages or more. The scenario that I wanted to use to demonstrate a need for search is like so.

A user on the client platform is building an application and wants to look for NoSQL services that they can leverage. If the catalog API responds with all the entries and hundreds of pages, it becomes very difficult for someone to search through the catalog itself to locate what they may want to use.

I was suggesting the need to support search on the catalog API itself. The broker would be responsible for filtering entries accordingly and then return a paged response.

To seed discussion it might be a good start to allow searches on tags and possibly description in the service object. I also think it might be helpful to allow someone to filter services that have or do not have free plans.

These could be query parameters on the catalog api either by allowing for the properties themselves ?tags=nosql or allowing a fixed parameter like ?query=tags:nosql. The second might allow more operators like AND, OR etc.

support pagination in the /catalog API

Proposal

The catalog API should support pagination in the cases where there are large
numbers of services supported by a broker. We see a potential for 10k-15k
catalog entries which could take a bit of time to render to the caller.

Effectively the initial catalog call should expect the Link header or the
metadata in the body indicating that the broker should be invoked with
pagination.

The API should take the following optional parameters:

parameter name	description
per_page	number of items to show per page
page	page number requested, depends on page size. if > last page, then last page is returned

cURL

   $ curl -H "X-Broker-API-Version: 2.9" http://username:password@broker-url/v2/catalog?page=2&per_page=20

Body (changes in bold)

Response field	Type	Description
services*	array-of-service-objects	Schema of service objects defined below.
pagination	metadata about pagination	Schema for pagination defined below.

Response

The pagination block would be added to the catalog response at the same
level as services. In the case of requesting the first page, the response
will contain a next and a last item.

{
  "services": [{
     ...
     }],
  "pagination": {
    "next": "https://broker/v2/catalog?page=2",
    "last": "https://broker/v2/catalog?page=15333"
  }
}

Alternatively we could use the Link header RFC 5988. In this case there would be NO changes to the response body just the headers.

Link: <https://broker/v2/catalog?page=2>; rel="next",
      <https://broker/v2/catalog?page=15333>; rel="last"

Subsequent calls will yield 2 new options, first and prev. For example, if you request page 10000 the following response will be returned:

{
  "services": [{
     ...
     }],
  "pagination": {
    "next": "https://broker/v2/catalog?page=10001",
    "last": "https://broker/v2/catalog?page=15333",
    "first": "https://broker/v2/catalog?page=1",
    "prev":  "https://broker/v2/catalog?page=9999"
  }
}

Alternatively we could use the Link header RFC 5988

Link: <https://broker/v2/catalog?page=10001>; rel="next",
      <https://broker/v2/catalog?page=15333>; rel="last",
      <https://broker/v2/catalog?page=1>; rel="first",
      <https://broker/v2/catalog?page=9999>; rel="prev"

quite a few links are broken

looks like there are missing docs

These are all links on https://github.com/openservicebrokerapi/servicebroker/blob/master/spec.md that 404 on github for me (basically any link that isn't self referencing spec.md):
https://github.com/openservicebrokerapi/servicebroker/blob/devguide/services/index.html
https://github.com/openservicebrokerapi/servicebroker/blob/master/v2-api-changelog.md
https://github.com/openservicebrokerapi/servicebroker/blob/master/app-log-streaming.md
https://github.com/openservicebrokerapi/servicebroker/blob/master/route-services.md
https://github.com/openservicebrokerapi/servicebroker/blob/master/volume-services.md
https://github.com/openservicebrokerapi/servicebroker/blob/master/catalog-metadata.md
https://github.com/openservicebrokerapi/servicebroker/blob/master/dashboard-sso.md
https://github.com/openservicebrokerapi/servicebroker/blob/master/managing-service-brokers.md#register-broker
https://github.com/openservicebrokerapi/servicebroker/blob/master/api.md#broker-errors
https://github.com/openservicebrokerapi/servicebroker/blob/master/examples.md
https://github.com/openservicebrokerapi/servicebroker/blob/devguide/services/managing-services.html#update_service

this is a 404 on cloudfoundry.org
http://docs.cloudfoundry.org/running/managing-cf/quota-plans.md

I only made it about 20% though spec.md before creating this issue, but it looks like the supporting docs might not have been migrated.

basic notes from 2016-11-15 meeting

courtesy of @duglin. not sure if these all require action, but I put them into a checklist

• action: @duglin to remove non-spec stuff from repo
• topic: spec in our repo vs cf repo
• proposal: generate HTML docs from our repo’s src (md). Platforms can then decide whether to point to our HTML or to pull it into their docs
• proposal: convert existing CF docs into single page format ; move that single page to our repo ; CF then issues PRs against our repo instead of CF’s for v2.11 changes.
• action: shannon/CF-folks to see how to get the single page format in CF and remove CF-isms.

https://github.com/cloudfoundry/docs-services this is the current source of truth

Convert spec to be more RFC2119 and less CF-specific

Right now the spec is written like its a subset of the CF docs, which is it :-) but we need to make it more stand-alone and so a re-org is probably needed. While doing that we should make it more normative by adding the appropriate RFC2119 keywords to make it clear what's required/optional for people wanting to implement the spec.

Deprovision does not clarify how to handle active bindings

While reading the specification it was not clear to me what happens to bindings during a deprovision call. Are the bindings orphaned? or are they actively deleted by the broker? or should deprovision be disallowed if there are active bindings?

Support async binding operations

Binding to an instance may encapsulate a long running workflow, similar to creating an instance. This might include provisioning new user accounts in a service, setting up firewalls, etc. It seems like the Bind API would benefit from the same last_operation model that Provision uses, and I'd like to get feedback from others who may have more context around this.

Latest proposal: https://github.com/mattmcneeney/servicebroker/blob/async-bindings/spec.md#binding

broker requires user identity to perform operations on users behalf

Proposal from @gberche-orange:
https://docs.google.com/document/d/1DoAbJa_YiGIJbOZ_zPzakh7sc4TB9Tmadq41cfSX0dw/edit?usp=sharing. Discussion of that proposal: http://cf-dev.70369.x6.nabble.com/cf-dev-service-broker-user-delegation-beyond-service-dashboard-tt2658.html

Proposal from @fraenkel: https://docs.google.com/document/d/1w4swx_5erf_NmPtfcCW2fn2jbEBfRHf2xLVhWpc1PLM/edit

Identify pattern for *experimental* flags/extentions

From #95 (comment)

In v2.10 volume services are no longer experimental.

Note that there was a breaking change for volume services from v2.9 to 2.10
when the "experimental" flag was removed.

As part of this WG I would love to discuss the use of "experimental" API
specs. Not sure that it worked well in this case.

How to handle metadata/community driven standards

I'm not sure if this has been discussed yet but I tough it is interesting to define what to do with the current metadata and community driven standards in OpenSB.

Currently Cloud Foundry SB API has a metadata field. As part of the docs, some community driven standards are included.

http://docs.cloudfoundry.org/services/catalog-metadata.html

This is positive because:

• It defines data that is generally interesting for users.
• Since it is outside the spec, providers are not forced to support them.

Issue #94 raises the point that this data may need a pass of refinement at the moment. But then that raises the bigger question about what to do with this spec in the migration to OpenSB.

Should it also be migrated and kept as an accessory document?

Create a CLI

We probably can't use the CF cli for all operations so we should just create a cli for just the user facing service related operations we're going to support.

Named bindings support?

Currently bindings don't have names, I think they should. That would make it easier to keep track of bindings, would allow multiple bindings between serviceA and serviceB, say a single database, but two connections to it? @bmelville you had another use case for it too, I don't recall...

Differentiate secret vs. non-secret fields in binding credentials

We have come across this use-case several times in various places. Given the output of a Bind() operation, we would like to be able to understand which fields within the credential are secret and must be treated so, and which are non-secret and may be treated less securely.

For example, if the output of a binding is:

credential:
  username: admin
  password: mypassword
  hostname: localhost
  port: 8080

username and password may be treated as a secret in UI or in credential handling by the controller, while hostname and port may be treated as general configuration data.

Proposal is introduced as part of the proposal for #116: https://docs.google.com/document/d/1JbsJgqgNtqthcfYwK_KbS6C8sjElrZNgoLhu40dUPAs/edit?usp=sharing

Generic Extensions (Actions) Proposal

Following the discussion in #107 this issue aims to create a proposal for "Service Broker Actions".

---

Problem

As a service broker author, I want to be able to optionally offer operations beyond "provision, update, bind, unbind, delete" so that I can provide new features to users of my service via the service broker API

Examples

• Backup and restore functionality (previous discussion #107)
• Schedule things via the service broker

Workflows

Service clients can:

• Discover the actions available from a service broker
• Trigger a specific action synchronously
• Trigger a specific action asynchronously
• Pass input parameters to the action
• Discover and validate input parameters (input parameter schema)

Service authors can:

• Optionally define actions at service broker registration time

History

#107
https://docs.google.com/document/d/1Y5vwWjvaUIwHI76XU63cAS8xEOJvN69-cNoCQRqLPqU/edit#heading=h.kzfeyua6md59

Next steps

1. Proposal document
2. Review on OSBAPI WG call

instance_id should be precisely specified

The spec leaves open what service broker authors have to accept as a valid instance_id.
When we look at the following request:

PUT /v2/service_instances/:instance_id

• which charset is allowed?
• which encoding is used in the request url?
• how lang is the instance_id allowed to be?
  E.g. all of that is relevant when persisting a new instance in a relational database.

On some places the specs explicitly uses a guid .

/v2/service_instances/:guid/...

In my opinion the spec could more precise when we reduce the instance_id to only guid values allowed and a specific representation (dashes, lowercase letters, ...)

That's also true for some other id fields where the spec currently says a "GUID is recommended".

Service Object `requires` property uses permissions that seem CF-specific

https://github.com/openservicebrokerapi/servicebroker/blob/master/spec.md#-service-objects-

The requires property lists (currently limited) set of syslog_drain, route_forwarding and volume_mount. The permissions appear CF-specific. Is there plan to either extend this property to allow more general set of permissions, even implementation-specific ones, or is this field being removed as part of making the API more generic?

missing billing reference

The service catalog can provide plans with costs. When a service is provided from a third party there is no specification or recommendation how to pass billing or account information to the service provider.

A possible solution could be to extend the plan metadata by required parameters for the service creation

Plan Metadata Field:

"metadata":{
   ...
   "requires": [ "accountNumber" ],
  ...
}

When creating a service instance:

{
 ....
  "parameters":        {
    "accountNumber": 123456789
  }
}

Reference implementation & Test suite

Hi Guys,

Really like seeing this effort taking place. I was wondering if there is or should be a reference implementation and a test suite that can be run against brokers implementing the spec?

Support update on Bindings

There should be a way to update a binding object. Once a binding is created, one can only delete it. But there are some use cases where being able to support update on Binding would be nice to have. For example, a user of a service could apply quota adjustment that would be on per binding. The model proposed for Service Instance actions could be one way to achieve this.
Other use cases might include credential rotation as well as route modifications.

Remove some CF-isms

Problem

As a service broker author I can choose to ignore CF-isms in the Service Broker API specification and see that these are deprecated, but still available, just not as a top level attribute

History

#83

Next steps

1. Create proposal document
2. Review on OSBAPI WG call

Details

Branch with spec changes: https://github.com/duglin/servicebroker/tree/CFisms

• Including "context profile" doc: https://github.com/duglin/servicebroker/blob/CFisms/profile.md

Diff of changes: master...duglin:CFisms

OLD: Google doc for the original proposal profile: https://docs.google.com/document/d/15X9LWzZw8EF5TSufqsdrBcdlALr-6rWdMo3FVCbjrfs/edit#heading=h.9gcbm1qmz9wc

Make it clear that auth/creds for brokes are optional

Verify that the service broker CLI API aligns with CF's defintion

This is for the client->service registry flow.

I'm referring to the following CF cf calls:

service-brokers List service brokers
create-service-broker Create a service broker
update-service-broker Update a service broker
delete-service-broker Delete a service broker
rename-service-broker Rename a service broker
purge-service-offering Recursively remove a service and child objects from Cloud Foundry database without making requests to a service broker

more detail on API version

It would be useful if there were a little more detail in the spec around API versioning. How is versioning defined here and for what intention? What behaviour is expected of a client and server (e.g. if their versions don't match)?

Examples:

• you might want to state that X-Broker-API-Version header is required (e.g. client MUST include it in requests; server MUST include it in responses, etc.)
• you might want to declare semantic versioning / or behaviour like MUST deny requests with invalid API versions as follows, etc.

What is a plan?

The term plan is mentioned several times in the spec but it isn't clearly defined. It would be good to have it explained in the spec.

make it clear that services must have at least one plan

the spec implies it because planID is a required field in instance.create() but we should be explicit about it.

Verify that the registry->broker APIs for bindings align with CF's defintions

https://docs.cloudfoundry.org/services/api.html#binding
This is for the registry->SB/instance side of #26