saeed349/microservices-based-algorithmic-trading-system

MBATS is a docker based platform for developing, testing and deploying Algorthmic Trading strategies with a focus on Machine Learning based algorithms.

microservices-based-algorithmic-trading-system's Issues

Amazing project. seems those 3 scheme files in starter_script.bat

.\starter_script.bat

python: can't open file '/home/jovyan/work/q_pack/db_pack/schema/secmaster_db_schema_builder.py': [Errno 2] No such file or directory python: can't open file '/home/jovyan/work/q_pack/db_pack/schema/secmaster_db_symbol_loader.py': [Errno 2] No such file or directory python: can't open file '/home/jovyan/work/q_pack/db_pack/schema/risk_db_schema_builder.py': [Errno 2] No such file or directory ...

starter_script.bat not working

C:\Users\max\Desktop\market\algorith_trading_docker\Microservices-Based-Algorithmic-Trading-System>docker exec -it jupyter-image /bin/sh -c "python /home/jovyan/work/q_pack/db_pack/schema/risk_db_schema_builder.py"
python: can't open file '/home/jovyan/work/q_pack/db_pack/schema/risk_db_schema_builder.py': [Errno 2] No such file or directory

PgAdmin - The container does not start

Docker Desktop
Expose daemon on tcp: // .... - enabled
WSL integration - Ubuntu 20.04

PyCharm with Docker, docker Compose

Logs from the container

`[2021-02-12 09:15:16 +0000] [1] [INFO] Starting gunicorn 19.9.0

[2021-02-12 09:15:16 +0000] [1] [INFO] Listening at: http://[::]:80 (1)

[2021-02-12 09:15:16 +0000] [1] [INFO] Using worker: threads

/usr/local/lib/python3.9/os.py:1023: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used

return io.open(fd, *args, **kwargs)

[2021-02-12 09:15:16 +0000] [89] [INFO] Booting worker with pid: 89

WARNING: Failed to set ACL on the directory containing the configuration database:

[Errno 1] Operation not permitted: '/var/lib/pgadmin'

HINT : You may need to manually set the permissions on

/var/lib/pgadmin to allow pgadmin to write to it.

[2021-02-12 09:15:17 +0000] [89] [ERROR] Exception in worker process

Traceback (most recent call last):

File "/usr/local/lib/python3.9/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker

worker.init_process()

File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/gthread.py", line 104, in init_process

super(ThreadWorker, self).init_process()

File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 129, in init_process

self.load_wsgi()

File "/usr/local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi

self.wsgi = self.app.wsgi()

File "/usr/local/lib/python3.9/site-packages/gunicorn/app/base.py", line 67, in wsgi

self.callable = self.load()

File "/usr/local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 52, in load

return self.load_wsgiapp()

File "/usr/local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp

return util.import_app(self.app_uri)

File "/usr/local/lib/python3.9/site-packages/gunicorn/util.py", line 350, in import_app

import(module)

File "/pgadmin4/run_pgadmin.py", line 4, in

from pgAdmin4 import app

File "/pgadmin4/pgAdmin4.py", line 94, in

app = create_app()

File "/pgadmin4/pgadmin/init.py", line 256, in create_app

create_app_data_directory(config)

File "/pgadmin4/pgadmin/setup/data_directory.py", line 90, in create_app_data_directory

os.chmod(config.SESSION_DB_PATH, 0o700)

PermissionError: [Errno 1] Operation not permitted: '/var/lib/pgadmin/sessions'

[2021-02-12 09:15:17 +0000] [89] [INFO] Worker exiting (pid: 89)

[2021-02-12 09:15:17 +0000] [1] [INFO] Shutting down: Master

[2021-02-12 09:15:17 +0000] [1] [INFO] Reason: Worker failed to boot.`

Docker compose error

Docker version 19.03.8, build afacb8b

After command docker-compose -f "docker-compose.yml" up -d --build I got this error:

`WARNING: The MAPBOX_API_KEY variable is not set. Defaulting to a blank string.
Creating network "microservices-based-algorithmic-trading-system_default" with the default driver
Creating volume "pg_data" with default driver
Creating volume "pg_data_airflow" with default driver
Creating volume "redis" with default driver
Building minio-image
Traceback (most recent call last):
File "site-packages\docker\credentials\store.py", line 80, in _execute
File "subprocess.py", line 395, in check_output
File "subprocess.py", line 487, in run
subprocess.CalledProcessError: Command '['C:\Program Files\Docker\Docker\resources\bin\docker-credential-desktop.EXE', 'list']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "docker-compose", line 6, in
File "compose\cli\main.py", line 72, in main
File "compose\cli\main.py", line 128, in perform_command
File "compose\cli\main.py", line 1077, in up
File "compose\cli\main.py", line 1073, in up
File "compose\project.py", line 548, in up
File "compose\service.py", line 351, in ensure_image_exists
File "compose\service.py", line 1106, in build
File "site-packages\docker\api\build.py", line 261, in build
File "site-packages\docker\api\build.py", line 308, in _set_auth_headers
File "site-packages\docker\auth.py", line 302, in get_all_credentials
File "site-packages\docker\credentials\store.py", line 71, in list
File "site-packages\docker\credentials\store.py", line 93, in _execute
docker.credentials.errors.StoreError: Credentials store docker-credential-desktop exited with "error listing credentials - err: exec: "docker-credential-wincred.exe": executable file not found in %PATH%, out: ``".
[4468] Failed to execute script docker-compose`

WD=/C/Users/****/Desktop/Microservices-Based-Algorithmic-Trading-System/Storage

Windows 10

`wget: command not found` error while docker compose

docker-compose up -d — build having error

 => [minio-image auth] minio/minio:pull token for registry-1.docker.io                                                                                            0.0s
 => [mlflow-image 2/3] COPY requirements.txt /tmp/                                                                                                               15.9s
 => [mlflow-image 3/3] RUN pip install --requirement /tmp/requirements.txt                                                                                      239.0s
 => [jupyter-image auth] jupyter/scipy-notebook:pull token for registry-1.docker.io                                                                               0.0s
 => ERROR [minio-image 2/2] RUN wget https://dl.min.io/client/mc/release/linux-amd64/mc &&     chmod +x mc                                                       55.3s
------
 > [minio-image 2/2] RUN wget https://dl.min.io/client/mc/release/linux-amd64/mc &&     chmod +x mc:
32.15 /bin/sh: wget: command not found
------
failed to solve: process "/bin/sh -c wget https://dl.min.io/client/mc/release/linux-amd64/mc &&     chmod +x mc" did not complete successfully: exit code: 127
[ubbe@e45kattegat Microservices-Based-Algorithmic-Trading-System]$ docker compose up -d --build
WARN[0000] The "MAPBOX_API_KEY" variable is not set. Defaulting to a blank string. 
[+] Building 33.5s (29/33)                       

secmaster_db_symbol_loader fails with insufficient auth

Built using Ubuntu 16.04.7 LTS
Docker version 20.10.1, build 831ebea
docker-compose version 1.27.4, build 40524192

Traceback (most recent call last):
File "/home/jovyan/work/q_pack/db_pack/schema/secmaster_db_symbol_loader.py", line 131, in
main()
File "/home/jovyan/work/q_pack/db_pack/schema/secmaster_db_symbol_loader.py", line 118, in main
symbols = parse_wiki_forex()
File "/home/jovyan/work/q_pack/db_pack/schema/secmaster_db_symbol_loader.py", line 38, in parse_wiki_forex
rv = client.request(r)
File "/opt/conda/lib/python3.8/site-packages/oandapyV20/oandapyV20.py", line 305, in request
response = self.__request(method, url,
File "/opt/conda/lib/python3.8/site-packages/oandapyV20/oandapyV20.py", line 242, in __request
raise V20Error(response.status_code,
oandapyV20.exceptions.V20Error: {"errorMessage":"Insufficient authorization to perform request."}

Amazing project. seems those 3 scheme files in starter_script.bat

.\starter_script.bat

python: can't open file '/home/jovyan/work/q_pack/db_pack/schema/secmaster_db_schema_builder.py': [Errno 2] No such file or directory python: can't open file '/home/jovyan/work/q_pack/db_pack/schema/secmaster_db_symbol_loader.py': [Errno 2] No such file or directory python: can't open file '/home/jovyan/work/q_pack/db_pack/schema/risk_db_schema_builder.py': [Errno 2] No such file or directory ...

Docker compose wget: command not found

When running docker compose it gives me this error.

PS C:\wsl\Microservices-Based-Algorithmic-Trading-System> docker-compose up -d --build
WARNING: The MAPBOX_API_KEY variable is not set. Defaulting to a blank string.
Building minio-image
Step 1/2 : FROM minio/minio
---> 88a529ae9ab6
Step 2/2 : RUN wget https://dl.min.io/client/mc/release/linux-amd64/mc && chmod +x mc
---> Running in 6ab6c6f42d7e
/bin/sh: wget: command not found
ERROR: Service 'minio-image' failed to build : The command '/bin/sh -c wget https://dl.min.io/client/mc/release/linux-amd64/mc && chmod +x mc' returned a non-zero code: 127

I also tried removing that line, which made the compose successful, however when running starter_script.bat I get the error bellow.

C:\wsl\Microservices-Based-Algorithmic-Trading-System>docker exec -it jupyter-image /bin/sh -c "python /home/jovyan/work/q_pack/db_pack/schema/secmaster_db_symbol_loader.py"
Traceback (most recent call last):
File "/home/jovyan/work/q_pack/db_pack/schema/secmaster_db_symbol_loader.py", line 131, in
main()
File "/home/jovyan/work/q_pack/db_pack/schema/secmaster_db_symbol_loader.py", line 118, in main
symbols = parse_wiki_forex()
File "/home/jovyan/work/q_pack/db_pack/schema/secmaster_db_symbol_loader.py", line 38, in parse_wiki_forex
rv = client.request(r)
File "/opt/conda/lib/python3.8/site-packages/oandapyV20/oandapyV20.py", line 305, in request
response = self.__request(method, url,
File "/opt/conda/lib/python3.8/site-packages/oandapyV20/oandapyV20.py", line 242, in __request
raise V20Error(response.status_code,
oandapyV20.exceptions.V20Error: {"errorMessage":"Insufficient authorization to perform request."}

db_pack for Binance

Has anyone worked on the development of db_pask for the Binance exchange?

Got error during starter_script.bat execution

After building docker i tried start the script "starter_script.bat" and got this error. It looks like I have a problem with secmaster_db_symbol_loader.py.

How can I solve this?

PS C:\Users\***\desktop\Microservices-Based-Algorithmic-Trading-System> .\starter_script.bat

C:\Users\***\desktop\Microservices-Based-Algorithmic-Trading-System>echo "scripts on jupyter notebook"
"scripts on jupyter notebook"

C:\Users\***\desktop\Microservices-Based-Algorithmic-Trading-System>docker exec -it jupyter-image /bin/sh -c "python /home/jovyan/work/q_pack/db_pack/schema/secmaster_db_schema_builder.py"
Database does not exist.
Creating new database.
Database exists.
Building tables.
Building tables.
Building tables.
Building tables.
Building tables.

C:\Users\***\desktop\Microservices-Based-Algorithmic-Trading-System>docker exec -it jupyter-image /bin/sh -c "python /home/jovyan/work/q_pack/db_pack/schema/secmaster_db_symbol_loader.py"
Traceback (most recent call last):
  File "/home/jovyan/work/q_pack/db_pack/schema/secmaster_db_symbol_loader.py", line 131, in <module>
    main()
  File "/home/jovyan/work/q_pack/db_pack/schema/secmaster_db_symbol_loader.py", line 118, in main
    symbols = parse_wiki_forex()
  File "/home/jovyan/work/q_pack/db_pack/schema/secmaster_db_symbol_loader.py", line 38, in parse_wiki_forex
    rv = client.request(r)
  File "/opt/conda/lib/python3.7/site-packages/oandapyV20/oandapyV20.py", line 306, in request
    request_args, headers=headers)
  File "/opt/conda/lib/python3.7/site-packages/oandapyV20/oandapyV20.py", line 243, in __request
    response.content.decode('utf-8'))
oandapyV20.exceptions.V20Error: {"errorMessage":"Insufficient authorization to perform request."}

C:\Users\***\desktop\Microservices-Based-Algorithmic-Trading-System>docker exec -it jupyter-image /bin/sh -c "python /home/jovyan/work/q_pack/db_pack/schema/risk_db_schema_builder.py"
Database does not exist.
Creating new database.
Database exists.
Building tables.
Building tables.
Building tables.
Building tables.

C:\Users\***\desktop\Microservices-Based-Algorithmic-Trading-System>echo "..."
"..."