florianbrede-ayet/retropilot-server

replacement for comma.ai backend and useradmin dashboard. comes bundled with a modified cabana instance for seamless drive reviewing.

retropilot-server

Replacement for comma.ai backend and useradmin dashboard. Bundled with a modified version of comma's cabana to allow viewing & analyzing drives.

If you don't want to host your own instance, check out https://api.retropilot.org/useradmin for a hosted version of the backend, useradmin and cabana.

[Server] Summary

The server consists of 2 node scripts.

server.js is using expressjs and runs the backend (file upload / communication with openpilot) and the useradmin dashboard to manage / view / download drives & logs. worker.js is a background worker that is processing drives (analyzing video files & logs) to prepare drives for playback in cabana and to gather statistics. It automatically terminates itself after 60 minutes to make sure the video/log libraries do not cause memory leaks.

Both scripts can be started with a cronjob each minute, they use locking to make sure they run exclusively.

Attention: Minimum required node version is node 10.

[Server] Installation

npm install
cp config.sample.js config.js
cp database.empty.sqlite database.sqlite
> EDIT config.js

[Server] Running

node -r esm server.js

node -r esm worker.js

[Server] CABANA Support

A compiled version of a custom cabana fork (https://github.com/florianbrede-ayet/retropilot-cabana) is directly bundled in the cabana/ subdirectory and will be served by the express app. After starting server.js, cabana is ready to use.

---

[Device] Preparation / Enable Custom Server

On the device or in your fork's code, replace all API endpoints with your own server endpoint. This could be executed directly on the device in the shell to use https://api.retropilot.org as backend:

find /data/openpilot -type f -exec sed -i 's/https:\/\/api.commadotai.com/https:\/\/api.retropilot.org/g' {} +

[Device] Swapping Servers (Back)

To switch a device between different servers, you have to remove the old DongleId and reboot:

rm /data/params/d/DongleId
reboot

There is no need to backup the DongleId, as the new server will identify your device based on its imei, serial and public key.

[Device] Raw Drives Not Uploading (fcamera & rlog)

1. Raw data is only uploaded if the device is sufficiently charged, not connected to an active panda (offroad) and there are no immediate files (boot, crash, qcamera, qlog) remaining.
   
2. Your branch might have raw uploads disabled, check Device Settings > Upload Raw Logs.

If that doesn't help or the option is not available, try:

echo "1" > /data/params/d/IsUploadRawEnabled
echo "1" > /data/params/d/UploadRaw
reboot

---

Current Limitations

OpenPilot before 0.8.3 will not display any statistics or pairing status in the dashboard. The reason is that pre 0.8.3, the offroad.apk with react and comma-api would require recompilation to accept the new endpoints.

The athena websockets interface is not implemented yet, so the comma app and athena specific remote control commands (including "upload on demand") are not functional as of now.

Screenshots

retropilot-server's Issues

Connect to RetroPilot

but cant connect to retropilot device id always N/A
changed liek the wrote at github here below:
find /data/openpilot -type f -exec sed -i 's/https://api.commadotai.com/https://api.retropilot.org/g' {} +

also how can connect your branch to : https://github.com/florianbrede-ayet/retropilot-server ?

tryed with DragonPilot 0.8.9 and 0.8.13 and with a Custom Fork Branch fir Audi

Get no Dongle ID - Device ID always N/A

also set at DragonPilot Register at Server etc
thank you

Diskstation docker error

Error in worker

Bzip error and RangeError when running worker.

start_worker.sh is "/usr/bin/node -r esm worker.js".
Node is v10.19.0. bzip2 is v1.0.8.

$ ./start_worker.sh 
bzip2, a block-sorting file compressor.  Version 1.0.8, 13-Jul-2019.

   usage: bunzip2 [flags and input files in any order]

   -h --help           print this message
   -d --decompress     force decompression
   -z --compress       force compression
   -k --keep           keep (don't delete) input files
   -f --force          overwrite existing output files
   -t --test           test compressed file integrity
   -c --stdout         output to standard out
   -q --quiet          suppress noncritical error messages
   -v --verbose        be verbose (a 2nd -v gives more)
   -L --license        display software version & license
   -V --version        display software version & license
   -s --small          use less memory (at most 2500k)
   -1 .. -9            set block size to 100k .. 900k
   --fast              alias for -1
   --best              alias for -9

   If invoked as `bzip2', default action is to compress.
              as `bunzip2',  default action is to decompress.
              as `bzcat', default action is to decompress to stdout.

   If no file names are given, bzip2 compresses or decompresses
   from standard input to standard output.  You can combine
   short flags, so `-v -4' means the same as -v4 or -4v, &c.

[2022-01-05T09:39:44.867] [INFO] default - STARTING WORKER...
[2022-01-05T09:39:44.871] [INFO] default - Verified storage path /footage/retropilot/realdata
[2022-01-05T09:39:49.908] [INFO] default - processSegmentsRecursive f3b1d6e8 2021-12-21--11-33-39 7
[2022-01-05T09:39:50.955] [INFO] default - processSegmentRLog GpsLocationExternal @ 506702182045: 
[2022-01-05T09:39:50.988] [INFO] default - processSegmentVideo duration: 59.999167s
[2022-01-05T09:39:54.114] [INFO] default - processSegmentRLog GpsLocationExternal @ 524691373913: 
[2022-01-05T09:39:54.277] [INFO] default - processSegmentRLog GpsLocationExternal @ 525693860319: 
[2022-01-05T09:39:54.472] [INFO] default - processSegmentRLog GpsLocationExternal @ 526791222974: 
[2022-01-05T09:39:54.653] [INFO] default - processSegmentRLog GpsLocationExternal @ 527791061776: 
[2022-01-05T09:39:54.849] [INFO] default - processSegmentRLog GpsLocationExternal @ 528790965578: 
[2022-01-05T09:39:55.020] [INFO] default - processSegmentRLog GpsLocationExternal @ 529791151671: 
[2022-01-05T09:39:55.203] [INFO] default - processSegmentRLog GpsLocationExternal @ 530791714014: 
[2022-01-05T09:39:55.370] [INFO] default - processSegmentRLog GpsLocationExternal @ 531791248077: 
[2022-01-05T09:39:55.543] [INFO] default - processSegmentRLog GpsLocationExternal @ 532791311253: 
[2022-01-05T09:39:55.698] [INFO] default - processSegmentRLog GpsLocationExternal @ 533790909326: 
[2022-01-05T09:39:55.864] [INFO] default - processSegmentRLog GpsLocationExternal @ 534790931982: 
[2022-01-05T09:39:56.027] [INFO] default - processSegmentRLog GpsLocationExternal @ 535790971044: 
[2022-01-05T09:39:56.205] [INFO] default - processSegmentRLog GpsLocationExternal @ 536790962710: 
[2022-01-05T09:39:56.373] [INFO] default - processSegmentRLog GpsLocationExternal @ 537790205783: 
[2022-01-05T09:39:56.553] [INFO] default - processSegmentRLog GpsLocationExternal @ 538790947397: 
[2022-01-05T09:39:56.727] [INFO] default - processSegmentRLog GpsLocationExternal @ 539791373021: 
[2022-01-05T09:39:56.897] [INFO] default - processSegmentRLog GpsLocationExternal @ 540794499167: 
[2022-01-05T09:39:57.057] [INFO] default - processSegmentRLog GpsLocationExternal @ 541789592604: 
RangeError [ERR_OUT_OF_RANGE]: The value of "offset" is out of range. It must be >= 0 and <= 12. Received 16
    at boundsError (internal/buffer.js:49:9)
    at Buffer.readUInt32LE (internal/buffer.js:105:5)
    at Object.readSize (/footage/retropilot/node_modules/capnp-split/buffer.js:19:23)
    at CapnpStream.readNextMessage (/footage/retropilot/node_modules/capnp-split/stream.js:18:23)
    at CapnpStream._write (/footage/retropilot/node_modules/capnp-split/stream.js:34:15)
    at doWrite (_stream_writable.js:415:12)
    at writeOrBuffer (_stream_writable.js:399:5)
    at CapnpStream.Writable.write (_stream_writable.js:299:11)
    at DestroyableTransform.ondata (/footage/retropilot/node_modules/readable-stream/lib/_stream_readable.js:619:20)
    at DestroyableTransform.emit (events.js:198:13)

Docker build error

Hi,

I'm getting an error building an image with the Dockerfile.

Any ideas ?

Thanks

Cloning into 'retropilot-server'...
npm WARN deprecated [email protected]: Please upgrade to latest, formidable@v2 or formidable@v3! Check these notes: https://bit.ly/2ZEqIau
npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: This package is no longer supported. It's now a built-in Node module. If you've depended on crypto, you should switch to the one that's built-in.
npm WARN deprecated [email protected]: request has been deprecated, see request/request#3142
npm WARN deprecated [email protected]: Please upgrade to @mapbox/node-pre-gyp: the non-scoped node-pre-gyp package is deprecated and only the @mapbox scoped package will recieve updates in the future
npm WARN deprecated [email protected]: Please upgrade to v7.0.2+ of superagent. We have fixed numerous issues with streams, form-data, attach(), filesystem errors not bubbling up (ENOENT on attach()), and all tests are now passing. See the releases tab for more information at https://github.com/visionmedia/superagent/releases.
npm WARN deprecated [email protected]: This version of tar is no longer supported, and will not receive security updates. Please upgrade asap.
npm ERR! code 1
npm ERR! path /retropilot-server/node_modules/sqlite3
npm ERR! command failed
npm ERR! command sh -c node-pre-gyp install --fallback-to-build
npm ERR! Failed to execute '/usr/bin/node /usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js configure --fallback-to-build --module=/retropilot-server/node_modules/sqlite3/lib/binding/napi-v3-linux-arm64/node_sqlite3.node --module_name=node_sqlite3 --module_path=/retropilot-server/node_modules/sqlite3/lib/binding/napi-v3-linux-arm64 --napi_version=8 --node_abi_napi=napi --napi_build_version=3 --node_napi_label=napi-v3 --python=/usr/bin/python3' (1)
npm ERR! node-pre-gyp info it worked if it ends with ok
npm ERR! node-pre-gyp info using [email protected]
npm ERR! node-pre-gyp info using [email protected] | linux | arm64
npm ERR! node-pre-gyp WARN Using request for node-pre-gyp https download
npm ERR! node-pre-gyp info check checked for "/retropilot-server/node_modules/sqlite3/lib/binding/napi-v3-linux-arm64/node_sqlite3.node" (not found)
npm ERR! node-pre-gyp http GET https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v5.0.2/napi-v3-linux-arm64.tar.gz
npm ERR! node-pre-gyp http 403 https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v5.0.2/napi-v3-linux-arm64.tar.gz
npm ERR! node-pre-gyp WARN Tried to download(403): https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v5.0.2/napi-v3-linux-arm64.tar.gz
npm ERR! node-pre-gyp WARN Pre-built binaries not found for [email protected] and [email protected] (node-v93 ABI, musl) (falling back to source compile with node-gyp)
npm ERR! node-pre-gyp http 403 status code downloading tarball https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v5.0.2/napi-v3-linux-arm64.tar.gz
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using [email protected]
npm ERR! gyp info using [email protected] | linux | arm64
npm ERR! gyp info ok
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using [email protected]
npm ERR! gyp info using [email protected] | linux | arm64
npm ERR! gyp ERR! find Python
npm ERR! gyp ERR! find Python checking Python explicitly set from command line or npm configuration
npm ERR! gyp ERR! find Python - "--python=" or "npm config get python" is "/usr/bin/python3"
npm ERR! gyp ERR! find Python - "/usr/bin/python3" is not in PATH or produced an error
npm ERR! gyp ERR! find Python Python is not set from environment variable PYTHON
npm ERR! gyp ERR! find Python checking if "python3" can be used
npm ERR! gyp ERR! find Python - "python3" is not in PATH or produced an error
npm ERR! gyp ERR! find Python checking if "python" can be used
npm ERR! gyp ERR! find Python - "python" is not in PATH or produced an error
npm ERR! gyp ERR! find Python
npm ERR! gyp ERR! find Python **********************************************************
npm ERR! gyp ERR! find Python You need to install the latest version of Python.
npm ERR! gyp ERR! find Python Node-gyp should be able to find and use Python. If not,
npm ERR! gyp ERR! find Python you can try one of the following options:
npm ERR! gyp ERR! find Python - Use the switch --python="/path/to/pythonexecutable"
npm ERR! gyp ERR! find Python (accepted by both node-gyp and npm)
npm ERR! gyp ERR! find Python - Set the environment variable PYTHON
npm ERR! gyp ERR! find Python - Set the npm configuration variable python:
npm ERR! gyp ERR! find Python npm config set python "/path/to/pythonexecutable"
npm ERR! gyp ERR! find Python For more information consult the documentation at:
npm ERR! gyp ERR! find Python https://github.com/nodejs/node-gyp#installation
npm ERR! gyp ERR! find Python **********************************************************
npm ERR! gyp ERR! find Python
npm ERR! gyp ERR! configure error
npm ERR! gyp ERR! stack Error: Could not find any Python installation to use
npm ERR! gyp ERR! stack at PythonFinder.fail (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:330:47)
npm ERR! gyp ERR! stack at PythonFinder.runChecks (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:159:21)
npm ERR! gyp ERR! stack at PythonFinder. (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:202:16)
npm ERR! gyp ERR! stack at PythonFinder.execFileCallback (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:294:16)
npm ERR! gyp ERR! stack at exithandler (node:child_process:406:5)
npm ERR! gyp ERR! stack at ChildProcess.errorhandler (node:child_process:418:5)
npm ERR! gyp ERR! stack at ChildProcess.emit (node:events:520:28)
npm ERR! gyp ERR! stack at Process.ChildProcess._handle.onexit (node:internal/child_process:289:12)
npm ERR! gyp ERR! stack at onErrorNT (node:internal/child_process:478:16)
npm ERR! gyp ERR! stack at processTicksAndRejections (node:internal/process/task_queues:83:21)
npm ERR! gyp ERR! System Linux 5.10.17-v8+
npm ERR! gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "configure" "--fallback-to-build" "--module=/retropilot-server/node_modules/sqlite3/lib/binding/napi-v3-linux-arm64/node_sqlite3.node" "--module_name=node_sqlite3" "--module_path=/retropilot-server/node_modules/sqlite3/lib/binding/napi-v3-linux-arm64" "--napi_version=8" "--node_abi_napi=napi" "--napi_build_version=3" "--node_napi_label=napi-v3" "--python=/usr/bin/python3"
npm ERR! gyp ERR! cwd /retropilot-server/node_modules/sqlite3
npm ERR! gyp ERR! node -v v16.14.0
npm ERR! gyp ERR! node-gyp -v v8.3.0
npm ERR! gyp ERR! not ok
npm ERR! node-pre-gyp ERR! build error
npm ERR! node-pre-gyp ERR! stack Error: Failed to execute '/usr/bin/node /usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js configure --fallback-to-build --module=/retropilot-server/node_modules/sqlite3/lib/binding/napi-v3-linux-arm64/node_sqlite3.node --module_name=node_sqlite3 --module_path=/retropilot-server/node_modules/sqlite3/lib/binding/napi-v3-linux-arm64 --napi_version=8 --node_abi_napi=napi --napi_build_version=3 --node_napi_label=napi-v3 --python=/usr/bin/python3' (1)
npm ERR! node-pre-gyp ERR! stack at ChildProcess. (/retropilot-server/node_modules/node-pre-gyp/lib/util/compile.js:83:29)
npm ERR! node-pre-gyp ERR! stack at ChildProcess.emit (node:events:520:28)
npm ERR! node-pre-gyp ERR! stack at maybeClose (node:internal/child_process:1092:16)
npm ERR! node-pre-gyp ERR! stack at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)
npm ERR! node-pre-gyp ERR! System Linux 5.10.17-v8+
npm ERR! node-pre-gyp ERR! command "/usr/bin/node" "/retropilot-server/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build"
npm ERR! node-pre-gyp ERR! cwd /retropilot-server/node_modules/sqlite3
npm ERR! node-pre-gyp ERR! node -v v16.14.0
npm ERR! node-pre-gyp ERR! node-pre-gyp -v v0.11.0
npm ERR! node-pre-gyp ERR! not ok

npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2022-03-15T13_29_23_887Z-debug.log

How to view qlog ?

Hi,

How can I view qlog files ? When downloading they are not human readable.

Thanks

Not Implemented

Hi
both these links:

http://IP:3000/backend/post_upload
http://IP:3000/realdata

returns:

Not Implemented

Am I missing the obvious?

Docker version ?

Hi,

Any plans to make a docker image of the server ?

Thanks