raenonx/nas2http

Bridge for serving and downloading the FTP content via HTTP.

nas2http's Issues

Download the whole folder as a zipped file

Possibly add an icon link beside the folder entry.

May need to use ajax for displaying the current zipping progress.

May send multiple file download requests instead to save the system resource consumption.

Logging system

Log the file transfers.

Also to log any output and errors. during transfer.

Things to log:

• File size
• File path
• Source IP
• Time consumption (?)

Add an endpoint to deliberately trigger the error to log.

Remove the intermediate bytesIO

Content at FTP will be downloaded and stored to the memory then gives to the user.

https://github.com/RaenonX/Ftp2Http/blob/d974ed2855587322d49944eca2fb8718abe2f7b9/ftp.py#L157

The aim is to directly provide a pipeline from FTP to the user end so that the application resource consumption will not be massive if downloading a big file.

Downloading process at v1.0.0

Download the whole file to the application memory, then forward the entire file content to the end user's client.

This causes the RAM consumption will be a lot for large files (> 1 GB).
Also, for these large files, users need to wait for the file to be downloaded to the RAM.

Downloading process at v1.1.0+

The server acts as a forwarder, forwarding the content get from the server to the end user's client.

The request can almost initiate instantly. Initiation time for the large file requests will be as same as the small files because it's just a handshake process.
Download time will also be reduced by 50% because the server no longer needs to download the file to its local first, then forward it to the end user's client. The content is directly handed to the end-users now.

CI tests / Code linting

Style & Stabililty improvements.

Use pytest, pylint, respectively.

Also to add load test.

I/O still being consumed even after canceling the download request

Possible Cause

The application is still processing the data, even if the request is being canceled.

Note

Not happening on development PC, but happening on server PC.

Temp Solution

Set a speed limit on NAS for the account designated for this application, so that if a request is canceled, the rate of dumping will be less (dumping still happening).

Revolutionize the underlying file acquiring mechanism

From NAS via FTP then HTTP, to NAS mapped to local then HTTP.

Sudden stop on downloading large files

Data transfer may suddenly stop for some unknown reasons.

MIME-type may be the reason. Seems not.

Check the implementation of ftp.retrbinary().

Result

This may be caused by wFastCGI via IIS.

Also, for #6, the application ran under IIS cannot find the network mapped drive. Current workaround is to use URL Rewrite on IIS to proxy the request to the application running in the background.