imagingdatacommons/throttleproxy

(CORE REPO)

ThrottleProxy

A basic throttling proxy, implemented in Flask. Enforces a configurable daily byte limit per IP address. Also enforces a global daily byte limit for the whole server. Prior to hitting the daily limit, the server begins to degrade performance by inserting sleep requests between each chunk of relayed bytes. This degradation occurs in two steps. Once the daily limit is reached, the proxy returns HTTP status 429.

You need to deploy a REDIS server with Serverless VPC Access.

Using a redis server:

1. Set up the server. Enable the Memorystore API. Create the instance.
2. For App Enging Standard, have to set up Serverless VPC Access.
3. Enable the Serverless VPC Access API
4. Create a connector, e.g.: redis-connector default us-central1 10.8.180.0/28 200 300

Force Redeploy

throttleproxy's Issues

Requests not being passed to App Engine instance by load balancer chain; 500 returned instead

This has been reported to Google as case 25638287. During a demo, the viewer immediately black-screened after receiving two 500 errors that were thrown not by the throttle proxy, but by the Google load balancer sitting in front of it. Awaiting response.

extra ^M characters in multipart-related series download

When downloading a series using dicomweb wado-rs to get multipart content of type application/dicom, the contents of the download from the proxy is different from accessing the Google dicomweb directly.

In this image the left side is from the proxy and the right side is from google.

This is the proxy URL:

https://proxy-dot-idc-dev.appspot.com:443/v1beta1/projects/idc-dev-etl/locations/us-central1/datasets/idc_tcia_mvp_wave0/dicomStores/idc_tcia/dicomWeb/studies/1.3.6.1.4.1.14519.5.2.1.2744.7002.150059977302243314164020079415/series/1.2.276.0.7230010.3.1.3.8323329.18234.1440001306.224430

And this is to access the corresponding series from Google.

export PROJECT_ID=chc-tcia
export REGION=us-central1
export DATASET_ID=qin-headneck
export DICOM_STORE_ID=qin-headneck

export URL=https://healthcare.googleapis.com/v1beta1/projects/$PROJECT_ID/locations/$REGION/datasets/$DATASET_ID/dicomStores/$DICOM_STORE_ID/dicomWeb/studies/1.3.6.1.4.1.14519.5.2.1.2744.7002.150059977302243314164020079415/series/1.2.276.0.7230010.3.1.3.8323329.18234.1440001306.224430
echo $URL

curl -X GET --silent \
     -H "Authorization: Bearer "$(gcloud auth print-access-token) \
     $URL \
     -o /tmp/series.content2

Need to add the license

Connection reset by peer from REDIS

Gettng 500 errors from the server. Per the issue at the bottom, Redis in Google cloud might be a bit flaky in App Engine Standard. Wrap the call to catch the exception.
File "/srv/main.py", line 359, in root
curr_use_per_ip_str = redis_client.get(client_ip)
File "/env/lib/python3.7/site-packages/redis/client.py", line 1606, in get
return self.execute_command('GET', name)
File "/env/lib/python3.7/site-packages/redis/client.py", line 901, in execute_command
return self.parse_response(conn, command_name, **options)
File "/env/lib/python3.7/site-packages/redis/client.py", line 915, in parse_response
response = connection.read_response()
File "/env/lib/python3.7/site-packages/redis/connection.py", line 739, in read_response
response = self._parser.read_response()
File "/env/lib/python3.7/site-packages/redis/connection.py", line 324, in read_response
raw = self._buffer.readline()
File "/env/lib/python3.7/site-packages/redis/connection.py", line 256, in readline
self._read_from_socket()
File "/env/lib/python3.7/site-packages/redis/connection.py", line 223, in _read_from_socket
(ex.args,))
redis.exceptions.ConnectionError: Error while reading from socket: (104, 'Connection reset by peer')"

See: redis/redis-py#1186

Check for content-length

Once the proxy was revised to pass through gzip'd content untouched, the OHIF loading counter started to just hold at 0% during a load. Previously, when proxy was uncompressing, the counter progressed from 0 to 100. Investigate content-length header suppression.

500 internal server error accessing large series

Accessing a series using wado-rs series retrieve leads to server error if the series has more than a few items in it.

This URL leads to the crash for instance:

https://proxy-dot-idc-dev.appspot.com/v1beta1/projects/idc-dev-etl/locations/us-central1/datasets/idc_tcia_mvp_wave0/dicomStores/idc_tcia/dicomWeb/studies/1.3.6.1.4.1.14519.5.2.1.2744.7002.150059977302243314164020079415/series/1.3.6.1.4.1.14519.5.2.1.2744.7002.248974378224961074547541151175

It should return a 95MB multipart message of content-type application/dicom. The correct result is provided by accessing the Google dicom store directly.

Probably the proxy is running out of memory or something similar.

endpoint to query current status

OHIF is adding the ability to handle http errors like those generated when an IP hits the threshold by this proxy (OHIF/Viewers#1598).

But we want to be able to give users some advance warning, like a notification that they are close to the limit or maybe an area that turns yellow then red as you get close to the limit.

Can we have an api endpoint to query the current usage compared to the cap?