nipreps/migas-py

A ready-to-go python package for communicating with a migas server

NiPreps Python module

This repository was created to reserve the top-level nipreps module. This opens the door for various submodules to exist under the nipreps name.

migas-py's Issues

Make multiprocessing safe

Currently, the configuration file may load/save on initialization. If multiple processes are invoked (ala nipype with V1 of the client), this may run into race conditions.

Do not blow up if no network is available

In cases where we simulate no network connection, we should fail gracefully (and silently)

File "/opt/conda/bin/nibabies", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.9/site-packages/nibabies/cli/run.py", line 28, in main
    ping_migas()
  File "/opt/conda/lib/python3.9/site-packages/nibabies/utils/misc.py", line 135, in ping_migas
    res = migas.add_project(
  File "/opt/conda/lib/python3.9/site-packages/migas/config.py", line 44, in can_send
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/migas/operations.py", line 135, in add_project
    _, response = request(Config.endpoint, query)
  File "/opt/conda/lib/python3.9/site-packages/migas/request.py", line 34, in request
    conn.request("POST", purl.path, body, headers)
  File "/opt/conda/lib/python3.9/http/client.py", line 1285, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/opt/conda/lib/python3.9/http/client.py", line 1331, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/opt/conda/lib/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/opt/conda/lib/python3.9/http/client.py", line 1040, in _send_output
    self.send(msg)
  File "/opt/conda/lib/python3.9/http/client.py", line 980, in send
    self.connect()
  File "/opt/conda/lib/python3.9/http/client.py", line 1447, in connect
    super().connect()
  File "/opt/conda/lib/python3.9/http/client.py", line 946, in connect
    self.sock = self._create_connection(
  File "/opt/conda/lib/python3.9/socket.py", line 823, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/opt/conda/lib/python3.9/socket.py", line 954, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

Improve handling of server errors

GraphQL responses should return a dictionary consisting of {data: ..., (and optionally errors: ...)}

Check if errors is present, and if so, show it and exit gracefully.

Reconsider configuration

Current flow

• setup() optional, used to generate user-level configuration, saved at $HOME/.cache/migas/config.json
• add_project() & get_usage() will call setup() under the hood, which will do nothing if already called.

Pros	Cons
multi-process compatible	competing processes can clobber
same user id across projects	same user id across projects
	some information not cached

Alternative:

• setup() required, creates process specific configuration, saved at <tmp>/migas-<pid>.json
• add_project() fails if setup() is not called, get_usage doesn't need it (add endpoint parameter though)
• If child processes wish to use config, os.getppid() might be able to find a config

Pros	Cons
process specific config	harder to reuse config in child processes
can cache more information

Move away from versioneer

What would you like to see added in this software?

and embrace hatch/flit

Do you have any interest in helping implement the feature?

Yes

Additional information / screenshots

No response

Update testing

Tests are now failing since they are still dependent on the previously hosted test server, which has since been decommissioned.

Moving forward, it would likely suffice to spin up a minimal migas server locally and use that to test the python API.

Allow package to silently fail

Any failures with data collection should not result in end user application errors.

Standardize responses

3 different scenarios with 3 different response bodies.

>>> add_project(project='nipreps/fmriprep', project_version='21.0.2')

{'data': {'add_project': {'bad_versions': [],
    'cached': False,
    'latest_version': '21.0.2',
    'message': '',
    'success': True}}}

>>> add_project(project='nipreps/fmriprep', project_version='21.0.2')  # server unavailable

 {'success': False,
  'errors': [{'message': 'Connection to server timed out.'}]}

>>> add_project(project='nipreps/fmriprep', project_version='21.0.2')  # requests overload

{'data': None, 'errors': [{'message': 'Too many requests, wait a minute.'}]}

These should all converge to something similar, to allow easy filtering

Add non-blocking pings

What would you like to see added in this software?

Currently, the common way to communicate with the server is to call add_project(), which blocks and waits for a server response. In (rare) cases where the server is unavailable, this can cause a delay equal to the value of the request timeout.

I think in the long run, a better solution would be:

• Separate requests where server response is expected from those that it is not (related to nipreps/migas-server#66)
• Refactor the request to be done within a concurrent.futures.ThreadPoolExecutor to avoid blocking the main thread.

Do you have any interest in helping implement the feature?

Yes

Additional information / screenshots

No response

Client mock up (from server)

A rough implementation is created in https://github.com/mgxd/fastapi-telemetry/blob/master/examples/htp.py

Essentially, we want to address some shortcomings exposed by the original

• If the server is down, do not keep clobbering requests. Probably can generate something in $HOME/.cache/etelemetry to track in the case of multiple processes (aka nipype multiproc)
• Make it easy to change the endpoint (I already worked on this in sensein/etelemetry-client#39)

Syntax errors sporadically appearing

What happened?

Rarely, we're seeing Syntax Error: Unterminated string.

I'm not sure if this is a server or client problem, but logging here for the moment.

What command did you use?

mutation{add_project(p:{project:"nipreps/mriqc",project_version:"23.1.0",language:"python",language_version:"3.9.12",is_ci:false,status:F,status_desc:"Finished with error(s)",error_type:"<class 'RuntimeError'>",error_desc:"Traceback (most recent call last):"

What version of the software are you running?

0.3.0

How are you running this software?

Docker

Is your data BIDS valid?

No

Are you reusing any previously computed results?

No

Please copy and paste any relevant log output.

No response

Additional information / screenshots

No response

Add server compatibility check

Parse the server version (located at the root endpoint), and verify the client is compatible