scitran / core Goto Github PK
View Code? Open in Web Editor NEWRESTful API
Home Page: https://scitran.github.io
License: MIT License
RESTful API
Home Page: https://scitran.github.io
License: MIT License
Internims p2p request signature should be created from information about the request itself, such as request method, request parameters and request headers, in addition to the request body.
We are looking to create a simple, yet effective way to launch the scitran API for developers. Besides the API itself, the only other launch requirement is mongod
. A single entry point, such as ./scitran.sh run
and a single config file are desired. Ideally, no further dependencies are introduced. Implementation candidates are bash and python. Options for app launching and live reloading are Reflex and possibly Paste Script.
Below is a starting point for discussion, intended to be refined in place before commencing work.
Requirements:
Targets:
prepare
- recent bash may be all we needrun
- setup
+ bootstrap
+ launch
setup
- install reflex, mongo, venv, etc.launch
- launch API and mongoupdate
- update this repocmd
- run arbitrary commands in the application contextConvenience targets (likely wrappers around cmd
):
bootstrap
- bootstrap users or datareset
- reset database (via boostrap.py clean
)mongo-log
- tail the mongod
logpython
- ipython shell with db
variable setlint
- run a linterDirectory structure:
scitran.sh
or scitran.py
- entry pointapi
- API source codebin
- api.wsgi
and bootstrap.py
test
- unit and integration testsruntime
- disposable environment for python dependencies and mongo binariestemplates
- bootstrap.json
, config.toml
persistent
- data
, db
This get
method in FileListHandler has high cyclomatic complexity.
https://github.com/scitran/api/blob/master/api/handlers/listhandler.py#L290
A simple refactoring should improve readability and maintainability.
Final goal should be to write consistent unit tests.
Also, propagate functions under Permission and Tag ListHandlers should be moved to keep all DB access confined in dao files.
Was blocking @mrDarcyMurphy, but I disabled the uwsgi cron job as a workaround.
Attach a file to any Project, then have rules.create_jobs called via uwsgi cron job or similar.
Tue Jan 5 17:04:52 2016 - error managing signal 0 on worker 1
Traceback (most recent call last):
File "bin/api.wsgi", line 88, in job_creation_timer
job_creation(signum)
File "bin/api.wsgi", line 43, in job_creation
rules.create_jobs(config.db, c, c_type, f)
File "./api/rules.py", line 139, in create_jobs
project = get_project_for_container(db, container)
File "./api/rules.py", line 166, in get_project_for_container
raise Exception('Hierarchy walking not implemented for container ' + str(con
tainer['_id']))
Exception: Hierarchy walking not implemented for container 568be6a7a104c7a4b4a89
567
Tue Jan 5 17:05:22 2016 - error managing signal 0 on worker 1
Add single file download capability to GET /nimsapi
, specifying level, id, and a unique file description in a JSON request body.
Currently the config .toml file is used as the source of truth. Reverse the convention to be more intuitive.
This impacts api.wsgi today, and bootstrap.py soon.
uwsgi_1 | [pid: 71|app: 0|req: 1/1] 192.168.99.1 () {34 vars in 399 bytes} [Wed Dec 30 19:37:47 2015] GET /api/jobs/next => generated 864 bytes in 35 msecs (HTTP/1.1 200) 3 headers in 112 bytes (2 switches on core 0)
uwsgi_1 | Traceback (most recent call last):
uwsgi_1 | File "bin/api.wsgi", line 87, in job_creation_timer
uwsgi_1 | job_creation(signum)
uwsgi_1 | File "bin/api.wsgi", line 76, in job_creation
uwsgi_1 | jobs.retry_job(config.db, j)
uwsgi_1 | NameError: global name 'jobs' is not defined
uwsgi_1 | Wed Dec 30 19:39:47 2015 - error managing signal 0 on worker 4
baseUrl="https://localhost:8443/api/jobs"
alias sCurl="curl -sk -H 'User-Agent: SciTran Drone Engine' -H 'X-SciTran-Auth: change-me'"
sCurl $baseUrl/next
If this error happens on uwsgi startup rather than cron execution, all subsequent api requests will error with http 500 response codes until uwsgi is restarted enough times to reset all stale "running" jobs.
TL;DR: an update request is returning 404 when it maybe should be 304.
When changing the permissions of a user on a project things work fine the first time. If I send the same request to change the permissions a second time, ie. trying to change {access:'rw'}
to {access:'rw'}
, then the response is 404 Element not updated...
.
Request:
http://localhost:9000/api/projects/abc123/permissions/local/[email protected]
{"access":"rw"}
1st Response:
200
{"modified": 1}
2nd Response:
404
{"code":404,"uid":"[email protected]","detail":"Element not updated in list permissions of container projects abc123"}
I'm looking through the module, and it looks like the organization has some non-standard conventions, which is making it a little more challenging than usual for me to understand:
import apps
) instead of relative (from . import apps
) so it's tough to tell what is from the local package vs built-in or external importsbootstrap.py
are in the same location as source modules -- typically these live in a bin
directory or so, to make it clear what is part of the module, and what is a callable scriptDo these need to be that way? I'm happy to work on reorganizing these aspects of package if it makes sense to do so at some point. I think it would make it easier to maintain and for other devs to understand the layout, and in the meantime would teach me a bit about the module. But if this seems overkill, it doesn't need to be done.
Add localpaths
option to project, session, and acquisition details.
Request URL:https://localhost/nimsapi/experiments?user=gsfr
At least for the local site, we should only allow new roles and permissions for actual user IDs.
Recently, my understanding is that the container_type
of a file object in the API did not have an s
prefix, at least in the case of an acquisition
.
Thus, to generate a download URI (as I do in jobs.py
) you add the s suffix:
'/' + i['container_type'] + 's/' + i['container_id'] + '/files/' + i['filename']
Surprisingly, this was not the case, as shown by these download URLs:
/acquisitionss/56a0159b6d7638bbbf9160d3/files/8892_nifti.nii.gz
/acquisitions/56a0159e6d7638bbbf9160f6/files/149_1_1_localizer.zip
It looks like spawning jobs from listhandler post will call rules.create_jobs
with acquisitions
.
I added code to jobs that throws an exception when called with plurals as a stopgap to limit the problem; here's a trace that hopefully illuminates the issue:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "/usr/local/lib/python2.7/dist-packages/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "./api/api.py", line 120, in dispatcher
rv = router.default_dispatcher(request, response)
File "/usr/local/lib/python2.7/dist-packages/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/usr/local/lib/python2.7/dist-packages/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "./api/base.py", line 128, in dispatch
return super(RequestHandler, self).dispatch()
File "/usr/local/lib/python2.7/dist-packages/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/usr/local/lib/python2.7/dist-packages/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "./api/handlers/listhandler.py", line 401, in post
rules.create_jobs(config.db, container, cont_name, file_properties)
File "./api/rules.py", line 140, in create_jobs
jobs.queue_job(db, alg_name, input)
File "./api/jobs.py", line 108, in queue_job
raise Exception('Container type cannot be plural :|')
the collection also doesn't get created
I'm guessing maybe we need to pass the access token to the api when we do the post?
Also we need to create a collection with delete candidates with this schema:
{
hash: "...",
timestamp: "...",
_id: "..."
}```
The API will check that the files have been deleted more than 24 hours ago and that the files are not used anywhere in the database.
Today when kevin and I went to create a collection, even though the post was successful and everything looked good in the log, it didn't create a new collection
As a general rule, classes should fall into two distinct categories:
Rather than self.abort, raise a variety of typed exceptions. This would be a better, drop-in replacement for what self.abort does anyway. At the very top, and/or whichever layers are appropriate, catch typed exceptions and handle a JSON error return there.
Some of the recent changes appear to have effected file downloads. This was previously working. The issue occurs when attempting to retrieve the ticket for the download.
Endpoint: /api/download
Method: POST
Payload:
{"optional":true,"nodes":[{"level":"session","_id":"5660680b9737e71a69dc8c4b"},{"level":"session","_id":"5660680c9737e71a69dc8c4f"}]}
Response Code: 500
Response:
<html>
<head>
<title>500 Internal Server Error</title>
</head>
<body>
<h1>500 Internal Server Error</h1>
The server has either erred or is incapable of performing the requested operation.<br /><br />
</body>
</html>
Log entry:
2015-12-08 17:40:49 root webapp2.py 1552:ERRO 'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
File "/scitran/persistent/venv/local/lib/python2.7/site-packages/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "/scitran/persistent/venv/local/lib/python2.7/site-packages/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "./api/api.py", line 126, in dispatcher
rv = router.default_dispatcher(request, response)
File "/scitran/persistent/venv/local/lib/python2.7/site-packages/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/scitran/persistent/venv/local/lib/python2.7/site-packages/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "./api/base.py", line 125, in dispatch
return super(RequestHandler, self).dispatch()
File "/scitran/persistent/venv/local/lib/python2.7/site-packages/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/scitran/persistent/venv/local/lib/python2.7/site-packages/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "./api/core.py", line 249, in download
return self._preflight_archivestream(req_spec)
File "./api/core.py", line 184, in _preflight_archivestream
prefix = project['group'] + '/' + project['label'] + '/' + session.get('label', 'untitled')
TypeError: 'NoneType' object has no attribute '__getitem__'
[pid: 14428|app: 0|req: 60/137] 10.0.2.2 () {52 vars in 918 bytes} [Tue Dec 8 17:40:49 2015] POST /api/download => generated 228 bytes in 31 msecs (HTTP/1.1 500) 2 headers in 99 bytes (1 switches on core 0)
#86 closed the loop on security vulns, but untrusted input could still perform a variety of annoying tasks:
os.path.basename
since I didn't validateClosing this requires at minimum:
We are currently silently skipping missing or unreadable files on download. As @kofalt points out, this is fundamentally opposed to our data management mission and must be corrected asap.
However, considerable edge cases exist. For example, how do we handle files being legitimately deleted, in parallel to a long-running download operation?
Also, we don't want to inform the user of a (potentially irrelevant) problem 3 hours into a large download and proceed to abort and purge all downloaded data.
We used to check for files during the download preflight, but that was deemed to slow. Let's again explore our options here. This pattern would allow us to mark files as being part of a download and delay physical removal until the download completes.
@kofalt, @ryansanford: Happy to hear your thoughts.
@rentzso: Let's take this on together next week.
Single exception: GET /api
will return HTML for human consumption.
For errors, emit a predictable error struct.
At the moment, bootstrap.py data
if run multiple times will generate duplicate data.
Avoid this by checking for identical files.
Begin to add support for oauth2.
Permissions, storages and validations management could be centralized in a manager class.
Investigate if this solution would improve code flexibility and maintainability.
@gsfr @rentzso Let me know if I can provide any additional detail.
# Headers & settings
baseUrl="https://inv-ryan-03.invenshure.com:8443/api/sessions"
alias sCurl="curl -sk -H 'User-Agent: SciTran Drone Engine' -H 'X-SciTran-Auth: change-me'"
# Get a session as an upload target
SessionId=$(sCurl $baseUrl | jq -r '.[0]._id')
# Generate a useless file
uname -a > example.bin
# Succeeds
sCurl $baseUrl/$SessionId/files -F "[email protected]"
2016-01-04 16:03:04 scitran.api base.py 127:DEBU from None None POST /api/sessions/568a8893e460bcea7c3fdd2a/files {}
2016-01-04 16:03:04 scitran.api liststorage.py 41:DEBU query {'_id': ObjectId('568a8893e460bcea7c3fdd2a')}
2016-01-04 16:03:04 scitran.api validators.py 125:DEBU {u'key_fields': [u'name'], u'required': [u'name', u'created', u'modified', u'size', u'hash', u'unprocessed'], u'additionalProperties': False, u'$schema': u'http://json-schema.org/draft-04/schema#', u'type': u'object', u'properties': {u'hash': {u'type': u'string'}, u'name': {u'type': u'string'}, u'created': {}, u'unprocessed': {u'type': u'boolean'}, u'measurements': {u'uniqueItems': True, u'items': {u'type': u'string'}, u'type': u'array'}, u'modified': {}, u'instrument': {u'type': u'string'}, u'metadata': {u'type': u'object'}, u'type': {u'type': u'string'}, u'tags': {u'uniqueItems': True, u'items': {u'type': u'string'}, u'type': u'array'}, u'size': {u'type': u'integer'}}}
2016-01-04 16:03:04 scitran.api validators.py 75:DEBU {'hash': '128a2aa7f6167cb09592541c97073b2bdc6e2a89b4a39a8cfad81f476e43eaa10b844f34811154632000050bd8b782de', 'name': 'example.bin', 'created': datetime.datetime(2016, 1, 4, 16, 3, 4, 667687), 'unprocessed': True, 'modified': datetime.datetime(2016, 1, 4, 16, 3, 4, 667687), 'size': 136}
2016-01-04 16:03:04 scitran.api liststorage.py 65:DEBU payload {'hash': '128a2aa7f6167cb09592541c97073b2bdc6e2a89b4a39a8cfad81f476e43eaa10b844f34811154632000050bd8b782de', 'name': 'example.bin', 'created': datetime.datetime(2016, 1, 4, 16, 3, 4, 667687), 'unprocessed': True, 'modified': datetime.datetime(2016, 1, 4, 16, 3, 4, 667687), 'size': 136}
2016-01-04 16:03:04 scitran.api liststorage.py 70:DEBU query {'files': {'$not': {'$elemMatch': {u'name': 'example.bin'}}}, '_id': ObjectId('568a8893e460bcea7c3fdd2a')}
2016-01-04 16:03:04 scitran.api liststorage.py 71:DEBU update {'$push': {'files': {'hash': '128a2aa7f6167cb09592541c97073b2bdc6e2a89b4a39a8cfad81f476e43eaa10b844f34811154632000050bd8b782de', 'name': 'example.bin', 'created': datetime.datetime(2016, 1, 4, 16, 3, 4, 667687), 'unprocessed': True, 'modified': datetime.datetime(2016, 1, 4, 16, 3, 4, 667687), 'size': 136}}}
2016-01-04 16:03:04 root webapp2.py 1552:ERRO [Errno 13] Permission denied: '/var/scitran/data/1/2'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "/usr/local/lib/python2.7/dist-packages/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "./api/api.py", line 118, in dispatcher
rv = router.default_dispatcher(request, response)
File "/usr/local/lib/python2.7/dist-packages/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/usr/local/lib/python2.7/dist-packages/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "./api/base.py", line 128, in dispatch
return super(RequestHandler, self).dispatch()
File "/usr/local/lib/python2.7/dist-packages/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/usr/local/lib/python2.7/dist-packages/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "./api/handlers/listhandler.py", line 400, in post
file_store.move_file(dest_path)
File "./api/files.py", line 112, in move_file
os.makedirs(target_dir)
File "/usr/lib/python2.7/os.py", line 150, in makedirs
makedirs(head, mode)
File "/usr/lib/python2.7/os.py", line 150, in makedirs
makedirs(head, mode)
File "/usr/lib/python2.7/os.py", line 150, in makedirs
makedirs(head, mode)
File "/usr/lib/python2.7/os.py", line 150, in makedirs
makedirs(head, mode)
File "/usr/lib/python2.7/os.py", line 150, in makedirs
makedirs(head, mode)
File "/usr/lib/python2.7/os.py", line 150, in makedirs
makedirs(head, mode)
File "/usr/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/var/scitran/data/1/2'
[pid: 15|app: 0|req: 29/52] 192.168.3.177 () {40 vars in 680 bytes} [Mon Jan 4 16:03:04 2016] POST /api/sessions/568a8893e460bcea7c3fdd2a/files => generated 228 bytes in 88 msecs (HTTP/1.1 500) 2 headers in 99 bytes (2 switches on core 1)
2016-01-04 16:03:09 scitran.api.jobs rules.py 59:WARN file example.bin in container 568a8893e460bcea7c3fdd2a has no type key
Updates to code and schema needed.
Acceptance Criteria:
Close when all routes from api.py are fully documented in api.raml
EDIT by Colton: Compare api.raml and routes in api.py to find api resources which have not yet been added to the raml spec
eg \<hash\>
should become sha384:\<hash\>
As mentioned in #25, we're using some pymongo functions which are deprecated; see diff there for more details. Eventually it might be wise to upgrade those.
Looks like this was caused by renaming some elements of the file document inside the acquisitions collection within mongodb.
dirty->unprocessed
filename->name
filehash->hash
filetype->type
One example of the mismatch
api.wsgi Line 136
for c in application.db[c_type].find({'files.dirty': True}, ['files']):
Proposed by @ryansanford as a preemptive production support feature.
Add a boolean now
flag to job objects.
Toggling this flag will cause the job to run as soon as possible.
Change the job modification (post/put) route to only allow superusers to manipulate the field.
Change the next
route to make two queries - first, check for any jobs with a now
flag, second, the normal query that exists today. Critically, the now
query should sort by the modified
timestamp in the opposite order the normal query uses - that is, return a job that has most recently been modified and has a now
flag. This contrasts with the normal query which is FIFO.
Should be set on endpoint tracked by #319.
SSL already provides integrity; all modern clients will most likely use HMAC-SHA256 at the transport level.
In order of desire:
Tagging @ryansanford he probably has the most experience in this area.
Uploading redundant files are rejected with 409, even if the force
flag is passed.
Expected behaviour is to overwrite the existing file and return 200.
To reproduce:
# Headers & settings
baseUrl="https://localhost:9000/api/projects"
alias sCurl="curl -sk -H 'User-Agent: SciTran Drone Engine' -H 'X-SciTran-Auth: change-me'"
# Get a project as an upload target
ProjectId=$(sCurl https://localhost:9000/api/projects | jq -r '.[0]._id')
# Generate a useless file
uname -a > example.bin
# Succeeds
sCurl $baseUrl/$ProjectId/files?force=true -F "[email protected]"
# Fails
sCurl $baseUrl/$ProjectId/files?force=true -F "[email protected]"
Example uses jq, an excellent CLI tool for JSON manipulation.
Obviously, your instance will need at least one project for this to work.
No permissions checking is currently done for attachment uploads. Use Container._get()
to do it:
self._get(_id, ‘rw’)
Also, file uploads currently require admin permission, but that should probably be changed to read-write as well.
At the moment, job creation only runs via uwsgi cron. Without uwsgi, these tasks will not run.
#93 will reduce but not eliminate our need for cron-like functionality. The current best proposal is to use APScheduler, which has several advantages over our old approach and in general looks pretty great.
In a batch download, if the label is missing in a session or an acquisition, in the structure of the extracted tar, we are naming the corresponding folder with the default "untitled".
Instead, we should use the timestamp or the uid of the session/acquisition.
Each API route should have a schema route that returns the corresponding json schema
master
Error is not consistent. About half the time I start uwsgi. Believe it's related to https://jira.mongodb.org/browse/PYTHON-986, as adding connect=False to pymongo.MongoClient() in config.py I'm unable to reproduce.
I'll create a pull request for this change.
Mongodb is in a standalone replicaset config
[uwsgi]
wsgi-file = bin/api.wsgi
chdir=code/api
pythonpath=code/data
master = True
die-on-term = True
socket = [::]:9000
processes = 4
threads = 2
Actual error
2016-01-12T18:38:53.641920306Z [pid: 15|app: 0|req: 4/27] 192.168.3.177 () {44 vars in 809 bytes} [Tue Jan 12 18:38:53 2016] GET /api/sessions => generated 6120 bytes in 47 msecs (HTTP/1.1 200) 3 headers in 113 bytes (2 switches on core 0)
2016-01-12T18:38:53.651092720Z 2016-01-12 18:38:53 scitran.api config.py 86:INFO Initializing database
2016-01-12T18:38:54.354535093Z 2016-01-12 18:38:54 root webapp2.py 1552:ERRO No replica set members found yet
2016-01-12T18:38:54.354623330Z Traceback (most recent call last):
2016-01-12T18:38:54.354646766Z File "/usr/local/lib/python2.7/dist-packages/webapp2.py", line 1535, in __call__
2016-01-12T18:38:54.354665146Z rv = self.handle_exception(request, response, e)
2016-01-12T18:38:54.354681000Z File "/usr/local/lib/python2.7/dist-packages/webapp2.py", line 1529, in __call__
2016-01-12T18:38:54.354697060Z rv = self.router.dispatch(request, response)
2016-01-12T18:38:54.354712743Z File "./api/api.py", line 120, in dispatcher
2016-01-12T18:38:54.354737566Z rv = router.default_dispatcher(request, response)
2016-01-12T18:38:54.354793196Z File "/usr/local/lib/python2.7/dist-packages/webapp2.py", line 1278, in default_dispatcher
2016-01-12T18:38:54.354811500Z return route.handler_adapter(request, response)
2016-01-12T18:38:54.354826626Z File "/usr/local/lib/python2.7/dist-packages/webapp2.py", line 1101, in __call__
2016-01-12T18:38:54.354842113Z handler = self.handler(request, response)
2016-01-12T18:38:54.354860460Z File "./api/handlers/userhandler.py", line 17, in __init__
2016-01-12T18:38:54.354876060Z super(UserHandler, self).__init__(request, response)
2016-01-12T18:38:54.354890716Z File "./api/base.py", line 21, in __init__
2016-01-12T18:38:54.354905476Z self.debug = config.get_item('core', 'insecure')
2016-01-12T18:38:54.354919783Z File "./api/config.py", line 133, in get_item
2016-01-12T18:38:54.354934540Z return get_config()[outer][inner]
2016-01-12T18:38:54.354949000Z File "./api/config.py", line 110, in get_config
2016-01-12T18:38:54.354963780Z initialize_db()
2016-01-12T18:38:54.354978013Z File "./api/config.py", line 87, in initialize_db
2016-01-12T18:38:54.354992570Z if not db.system.indexes.find_one():
2016-01-12T18:38:54.355007233Z File "/usr/local/lib/python2.7/dist-packages/pymongo/collection.py", line 798, in find_one
2016-01-12T18:38:54.355038073Z for result in cursor.limit(-1):
2016-01-12T18:38:54.355052866Z File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 977, in next
2016-01-12T18:38:54.355068156Z if len(self.__data) or self._refresh():
2016-01-12T18:38:54.355082543Z File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 902, in _refresh
2016-01-12T18:38:54.355097533Z self.__read_preference))
2016-01-12T18:38:54.355111673Z File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 813, in __send_message
2016-01-12T18:38:54.355126883Z **kwargs)
2016-01-12T18:38:54.355141630Z File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 728, in _send_message_with_response
2016-01-12T18:38:54.355156873Z server = topology.select_server(selector)
2016-01-12T18:38:54.355171403Z File "/usr/local/lib/python2.7/dist-packages/pymongo/topology.py", line 121, in select_server
2016-01-12T18:38:54.355186350Z address))
2016-01-12T18:38:54.355200403Z File "/usr/local/lib/python2.7/dist-packages/pymongo/topology.py", line 97, in select_servers
2016-01-12T18:38:54.355215306Z self._error_message(selector))
2016-01-12T18:38:54.355229466Z ServerSelectionTimeoutError: No replica set members found yet
related to cni/nimsweb#66
internimsclient is to report on remote users with experiment (project) access, collection (virtual project) access, and group membership.
Checking a new oauth token with the provider is currently unconditional. This results in more upstream provider calls than are necessary if more than one API request occurs in a short interval with the new token.
Example: when our web app receives a token error, we will contact Google for a new token, then re-make any failed requests to the scitran API. In practice, this is perhaps ~5 requests made simultaneously with the new token.
The API could coalesce oAuth token provider checks by sharing context between workers such that only one in-flight request for a given token occurs at a time. Expected result is making the correct & minimal request count on the oAuth provider.
Add handling for targeted uploads (user or processor-specified container level and ID) to PUT /nimsdata
. Such uploads will come in as multipart/form-data
and need to be differentiated from sortable uploads (such as those from the reaper), which carry no additional information.
Determine new location/home for /run.sh and /bin/* files. /scripts could be a place to do this.
Uploading a text file throws this error:
Traceback (most recent call last):
File "/scitran/persistent/venv/local/lib/python2.7/site-packages/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "/scitran/persistent/venv/local/lib/python2.7/site-packages/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "./api/api.py", line 126, in dispatcher
rv = router.default_dispatcher(request, response)
File "/scitran/persistent/venv/local/lib/python2.7/site-packages/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/scitran/persistent/venv/local/lib/python2.7/site-packages/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "./api/base.py", line 125, in dispatch
return super(RequestHandler, self).dispatch()
File "/scitran/persistent/venv/local/lib/python2.7/site-packages/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/scitran/persistent/venv/local/lib/python2.7/site-packages/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "./api/handlers/listhandler.py", line 367, in post
file_store = files.FileStore(self.request, tempdir_path, filename=kwargs.get('name'))
File "./api/files.py", line 77, in __init__
self.hash = self.received_file.get_hash()
AttributeError: 'cStringIO.StringO' object has no attribute 'get_hash'
Image (.png) files work just fine.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.