fair-research/concierge

concierge's Issues

Add option for specifying the transfer label when staging bags

Add "transfer label" option when posting new bag to stage_bag/

Define new Globus Scope for this service

Currently, the Concirge service does not define a scope for the service it provides, which is much less than ideal. The service should define a scope with two dependent scopes for the following:

• Minid Service -- for registering minids
• Transfer Service -- for staging bdbags

This will be a slightly backward incompatible change. Users will need to request this scope and send the corresponding token instead of the previous ones they are currently using.

Rename search2bag createbag

The term search doesn't have any context in the current Concierge services. Better to rename or alias the search2bag view to createbag.

Bug when staging bags with similar content

Occasionally, staging a bag similar to another bag with result in an error. I have only seen this with copies of some bags where the protocol for fetching files has changed from Globus to https, which results in the concierge server throwing an error when attempting a third party transfer of https files.

If I remember right, there may be some logic for caching similar bags if they have already been unpacked once on the server.

Create Smart API Schema Descriptions

Create a schema for the Concierge Services based on the SmartAPI schema. Try to automate this as much as possible and store the scheme in the repo, or the means to generate it as needed.

Propogate "Invalid Schema" error when user attempts to stage HTTPS bag with CLI

If a user tries to stage a BDBag which has HTTPS files, the server returns a 500 and doesn't return the reason why the transfer failed to the Concierge Client.

Example server logs:

raise InvalidSchema("No connection adapters were found for '%s'" % url)

At the very least, the Server should return this message as part of the error message. In the future, it may be a nice feature to add support for partial stages if some of the files in a bag have Globus URLs or if one of multiple bags specified had Globus URLs. It would also be handy if cbag info returned information on the fetch.txt file to specify whether these files were Globus or not.

Add option to stage bags according to dir structure in fetch file

Currently, bags are staged according to the directory structure they are on the remote endpoint. This is due to the stage_bags/ endpoint allowing multiple bags to be staged at once, and using the original directory structure ensures no files will overwrite one another when files are from the same endpoint.

The above behavior ignores the structure of bags in the fetch file. Additionally, staging them according to the fetch file would make a lot more sense from a user perspective. We may want this as the default option.

Add feature for staging additional metadata in addition to files

Currently, the concierge server can only stage data. Add a feature for staging metadata associated with bags. This will require making the concierge service a GCS endpoint. A managed endpoint if we want to do sharing.

Support BDBag metadata on creation

Support BDBag metadata, which includes both general bag info and the newer ro_metadata still in development here

Concierge Service v2 API

Version 2 of the API will add better serializers, unifying both Remote File Manifests and Globus Manifests into a singular internal model which can be expressed in either format. The service may switch from using Bags as the internal format for storing remote file manifest information, and use simple JSON for storing manifests instead. This is due to Globus Manifests allowing for files to be specified without a checksum, which is in conflict with the BDBag spec requiring each file to have one or more checksums from the four ['md5', 'sha1', 'sha256', 'sha512'].

Additionally, the current way models are defined is very messy with its handling of JSON data. This is mostly due to the Models not being well normalized, which results in much of the JSON being stored in text fields. This isn't really a problem in the classical sense of needing to do SQL joins on specific fields, but it results in dirty serialization/deserialization from explicitly translating data to and from JSON strings.

Another fairly major change will be switching away from automatically registering each manifest with a Minid. This results in a lot of 'throwaway' identifiers where a new published datasets have errors and need to be resubmitted. New manifests will be created and with a UUID, where the user may decide to publish a manifest with an identifier at a later time.

V2 will include support for Globus Automate as a Globus Automate action provider, for both creating and transferring manifests.

The new basic API spec is here (only transfer action provider spec shown):

Add bag validation service

Determine if the bdbag is a valid structure and optionally if it contains valid data.