concierge's People
concierge's Issues
Add option for specifying the transfer label when staging bags
Add "transfer label" option when posting new bag to stage_bag/
Define new Globus Scope for this service
Currently, the Concirge service does not define a scope for the service it provides, which is much less than ideal. The service should define a scope with two dependent scopes for the following:
- Minid Service -- for registering minids
- Transfer Service -- for staging bdbags
This will be a slightly backward incompatible change. Users will need to request this scope and send the corresponding token instead of the previous ones they are currently using.
Rename search2bag createbag
The term search
doesn't have any context in the current Concierge services. Better to rename or alias the search2bag
view to createbag
.
Bug when staging bags with similar content
Occasionally, staging a bag similar to another bag with result in an error. I have only seen this with copies of some bags where the protocol for fetching files has changed from Globus to https, which results in the concierge server throwing an error when attempting a third party transfer of https files.
If I remember right, there may be some logic for caching similar bags if they have already been unpacked once on the server.
Create Smart API Schema Descriptions
Create a schema for the Concierge Services based on the SmartAPI schema. Try to automate this as much as possible and store the scheme in the repo, or the means to generate it as needed.
Propogate "Invalid Schema" error when user attempts to stage HTTPS bag with CLI
If a user tries to stage a BDBag which has HTTPS files, the server returns a 500 and doesn't return the reason why the transfer failed to the Concierge Client.
Example server logs:
raise InvalidSchema("No connection adapters were found for '%s'" % url)
At the very least, the Server should return this message as part of the error message. In the future, it may be a nice feature to add support for partial stages if some of the files in a bag have Globus URLs or if one of multiple bags specified had Globus URLs. It would also be handy if cbag info
returned information on the fetch.txt
file to specify whether these files were Globus or not.
Add option to stage bags according to dir structure in fetch file
Currently, bags are staged according to the directory structure they are on the remote endpoint. This is due to the stage_bags/ endpoint allowing multiple bags to be staged at once, and using the original directory structure ensures no files will overwrite one another when files are from the same endpoint.
The above behavior ignores the structure of bags in the fetch file. Additionally, staging them according to the fetch file would make a lot more sense from a user perspective. We may want this as the default option.
Add feature for staging additional metadata in addition to files
Currently, the concierge server can only stage data. Add a feature for staging metadata associated with bags. This will require making the concierge service a GCS endpoint. A managed endpoint if we want to do sharing.
Support BDBag metadata on creation
Support BDBag metadata, which includes both general bag info and the newer ro_metadata
still in development here
Concierge Service v2 API
Version 2 of the API will add better serializers, unifying both Remote File Manifests and Globus Manifests into a singular internal model which can be expressed in either format. The service may switch from using Bags as the internal format for storing remote file manifest information, and use simple JSON for storing manifests instead. This is due to Globus Manifests allowing for files to be specified without a checksum, which is in conflict with the BDBag spec requiring each file to have one or more checksums from the four ['md5', 'sha1', 'sha256', 'sha512'].
Additionally, the current way models are defined is very messy with its handling of JSON data. This is mostly due to the Models not being well normalized, which results in much of the JSON being stored in text fields. This isn't really a problem in the classical sense of needing to do SQL joins on specific fields, but it results in dirty serialization/deserialization from explicitly translating data to and from JSON strings.
Another fairly major change will be switching away from automatically registering each manifest with a Minid. This results in a lot of 'throwaway' identifiers where a new published datasets have errors and need to be resubmitted. New manifests will be created and with a UUID, where the user may decide to publish a manifest with an identifier at a later time.
V2 will include support for Globus Automate as a Globus Automate action provider, for both creating and transferring manifests.
The new basic API spec is here (only transfer action provider spec shown):
Add bag validation service
Determine if the bdbag is a valid structure and optionally if it contains valid data.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.