saltierl/distributedreplays

👨‍💻 Rocket League Replay Collection and Analysis Server that powers calculated.gg

Calculated.gg Backend

Setup

• Install Python 3.6/pip (do not use Anaconda)
• pip3 install -r requirements.txt
• Install and run Redis with default port + settings (Windows is included, Ubuntu)
• Install postgreSQL (Windows, Ubuntu, Mac)
• Make sure that postgreSQL is in the path
• If only for local development change the password in Ubuntu to postgres
• Ensure you have the latest LTS version of node and npm installed
• Run cd webapp, npm install

Running

Ubuntu

• Run RLBotServer.py.
• cd webapp, npm start

Windows

• Run win_run.bat
• You can optionally kill the python process and start it in an IDE
• You can log into psql command line with psql postgresql://postgres:postgres@localhost
• If the included redis does not work here is install directions
• DO NOT USE IDLE, either use command line or pycharm (pycharm is recommended for development of backend)

Mac

• Run mac_run.sh

You need to run in python3. Mac comes with python2 by default so do not run any commands without the 3

Structure

The structure of the server is split into different directories:

• blueprints - Backend Python code, split into subsections
• database - Database objects and queries
• helpers - Various scripts to help in maintaining the server
• static - Static website files (JS/CSS/images)
• tasks - Contains celery code user for processing the queue of replays
• templates - Dynamic website files (rendered HTML templates)

---

Setup and Running with Docker (Docs WIP)

Basic Dependencies

• Docker Community Edition (Stable)

  Docker will run Postgres, Redis, Flask, Celery, and Node inside linux based "Containers" on most platforms. Download and install Docker and Docker Compose for your platform:

  • Mac/Windows: Docker Desktop
  • Ubuntu/Debian-ish: Docker CLI

Run Everything

# Start containers. Should be in project root directory (where docker-compose.yml is)
docker-compose up

# See your containers running
docker ps

Now go to localhost:3000 and the site should be running.

Testing

We run with pytest and use some features from it. If you are using an ide (pycharm/intellij) you must change the default runner from unit tests to pytests You can make this change by going to settings and searching for "unittest" Python unit test configuration

distributedreplays's Issues

Automate adding new stats to database (from proto)

Win detection is wrong

It seems to be detecting a win if the player is on the Blue team, and a loss if they are on the Orange team.

Get status of replay in the queue

Number in queue, etc

Uploading files with model hash

we need a way to know what the model hash of the file is.
And a way for the server to associate an eval score with the hash of that model.

Add settings based on Steam login

Can include:

• Showing replay viewer
• Night theme (?)
• Auto uploader settings / key

Add serverside flags based on group

Have 3 global methods that can return true if you are in this group

isDev()
isAlpha()
isBeta()

Then we can flag features based on these values and don't have to worry about pushing bad features to the server.

Vague stat names

Explain what some of the more obscure stat names are (like >ball)

Put download button on the main stats page

General improvements need to be made.

Basic stats, replay viewer, advanced stats need a higher font weight.

Data output

Maybe to CSV, replay and player stats

Use redis instead of RabbitMQ

Logo Antialiasing

Antialiasing of the logo in the top left makes it look blurry

Keep a historical redis key that gets updated with latest rank information

If the rank API is down and the current key expired, use that information instead of fetching anew.

Left padding on card headers

Increase left padding on card headers

Prioritize uploaded replays over replays already on disk in processing queue

Some replays do not contain any stat data

http://localhost:5000/replays/parsed/view/5CAFB07745EBB494E66758B2ABC1E464

Some players are borked

http://saltie.tk/players/overview/76561198183261010
http://saltie.tk/players/overview/76561198325372380

Good looking WIP in advanced stats

Put a good looking WIP in the advanced stats (and in the tab)

Map out stat dependencies

Determine what stats rely on other stats to calculate.

Make global stats (histograms) a celery worker

It takes like 30+s now so it times out usually.

Ball Stats

Ball stats should have more to do with position the ball spent on the pitch

graph of wins and losses vs certain stats.

https://www.reddit.com/r/RocketLeague/comments/9hu2lo/introducing_calculatedgg_your_one_stop_shop_for/e6ep16h/

Car's appearance

They look really bad.

Bar charts are short

Why are all these bar charts so short.

Authentication state not present on nav bar / inner pages

On inner pages, current navbar does not contain an account/auth section. Convention is for this to live in top right.

Can't tell if logged in or not, and have to return to homepage to link to profile.

Becomes more important when more actions requiring auth (editing replay names, tagging replays, etc) become more prevalent.

Fix small font next to section titles

Fix that

Chart Title Font

Can we style the chart titles - that font really doesn't fit our site.

Sometimes global_std / global_mean are null

This is due to having no replays in a certain rank.

Firefox does not upload replays

OpenID login fails for steam profile names longer than 10 characters

Figured out why I was 500ing on steam login. My steam profile/persona name: "AlpacaFlightSim" is longer than the players.platformname column which is limited to 10 characters.

[SQL: 'INSERT INTO players (platformid, platformname, avatar, ranks) VALUES (%(platformid)s, %(platformname)s, %(avatar)s, %(ranks)s)'] [parameters: {'platformid': '76561197960785799', 'platformname': 'AlpacaFlightSim', 'avatar': 'https://steamcdn-a.akamaihd.net/steamcommunity/public/images/avatars/d9/d9af01e851ba44a965b178a59447a92228915281_full.jpg', 'ranks': None}]

Steam's API docs don't give a specified character limit, the input field for changing the name maxes out at 30.

<input class="dynInput" type="text" name="personaName" id="personaName" size="30" value="AlpacaFlightSim">

but there's always the possibility of legacy names or who knows. 50 is fine, and we'll be able to monitor once we get sentry setup and all that. No reason to be conservative with varchar sizes. Postgres doesn't care.

PR coming shortly :)

Remove unused fonts

Remove FiraMono (and other unused fonts)

if a user has no replays it does a 404 instead of informing them

We should inform the user we have no replays for them

Docker for services for easy setup & avoiding platform specific files

Setup my local environment with a docker-compose.yml file so I didn't need to worry about installing postgres, redis, rabbitmq (needed at the time).

The docker-compose.yml file currently using

version: "3"
services:
  redis:
    image: redis:4-alpine
    container_name: saltie-redis4
    restart: always
    volumes:
      - "./.data/redis:/data"
    ports:
      - 6379:6379
  # Don't think this is used anymore, but using docker saves headache of installing erlang and all that
  rabbitmq:
    image: rabbitmq:3-alpine
    container_name: saltie-rabbitmq
    restart: always
    volumes:
      - "./.data/rabbitmq:/data"
    ports:
      - 5672:5672

  postgres:
    image: postgres:10-alpine
    environment:
      POSTGRES_PASSWORD: postgres

Starting and stopping:

# Start service dependencies
$ docker-compose up
# Stop service dependencies
$ docker-compose down

Removes the needs for having most of the platform specific files that exist now. Great for local environments - optionally useful for future production deployments, but takes more consideration.

rattletrap could also be trivially be added to a docker image and ran in a container without much of a hit to total parsing performance - but thats for another issue

Player page stat: Pie chart for time high/low/ground

User should be able to rename "owned" replays

For people who occasionally rage-name replays.

https://calculated.gg/replays/parsed/view/453D0CF645FA86F77DF9D49DB5D33C83

Car Onhover Area

Car onhover's active area are smaller than the cars themselves?

Graphs on player overview messed up

http://saltie.tk/players/overview/76561198022548280 < open final game (ABADA4354BF2BEA39D600D9C68DA16B9).

Filter stats by date for spider charts

Avoids early replays affecting the player (2015?)

maxTickCount

maxTickCount? or something like that = 5

Create GitHub issues from website

https://gist.github.com/JeffPaine/3145490

Color switches for players between top graph and other stats

The color for players get switched.
Only have looked at doubles.

"Won turnovers" should be called "Takeaways"

That's the common terminology in sports

Some stats should be renames

Use Redis for Steam profile info

Common Legend

If possible, we could try to have a common legend.

Bar charts do not start at zero

Some bar charts need to start at zero instead of what visibly makes the biggest difference

Add an about page

We need an about page

Upload Replay Page touch up

The upload replays page can do with a quick touchup to use material buttons

Tag replays and aggregate all stats for that group

Club stat tracking

This will be difficult since clubs can change on a per-replay basis. However I think we can treat them like parties and do it on a per-replay basis. So if someone was in the club at the time of the replay, we count the stats towards the club.