milesmcc/shynet

Modern, privacy-friendly, and detailed web analytics that works without cookies or JS.

Modern, privacy-friendly, and cookie-free web analytics.
Getting started »

Screenshots • Features • Office Hours

Motivation

There are a lot of web analytics tools. Unfortunately, most of them come with the following caveats:

• They require handing all of your visitors' info to a third-party company
• They use cookies to track visitors across sessions, so you need to have those annoying cookie notices
• They collect so much personal data that even the NSA is jealous
• They are closed source and/or expensive, often with limited data portability
• They are hard to use

Shynet has none of these caveats. You host it yourself, so the data is yours. It works without cookies, so you don't need any intrusive cookie notices. It collects just enough data to be useful, but not enough to be creepy. It's open source and intended to be self-hosted. And you may even find the interface easy to use.

Shynet is a portmanteau of "Skynet" and "shy." The idea is that it gives you loads of useful information (Skynet) while also respecting your visitors' privacy (shy).

Screenshots

Note: These screenshots have been edited to hide sensitive data. The "real" Shynet has a lot more pages and information available, but hopefully this gives you an idea of the general look and feel of the tool.

Shynet's homepage, where you can see all of your services at a glance.

A real service page, where you can see higher-level details about a site.

Not shown: management view, session view, full service view. (You'll need to install Shynet for yourself to see those!)

Shynet is built using a17t, an atomic design library. Customization and extension is simple; learn more about a17t.

Features

Architecture

• Runs on a single machine — Because it's so small, Shynet can easily run as a single docker container on a single small VPS
• ...or across a giant Kubernetes cluster — For higher traffic installations, Shynet can be deployed with as many parallelized ingress nodes as needed, with Redis caching and separate backend workers for database IO
• Built using Django — Shynet is built using Django, so deploying, updating, and migrating can be done without headaches
• Multiple users and sites — A single Shynet instance can support multiple users, each tracking multiple different sites

Tracking

• JavaScript not required — It will fallback to using a 1x1 transparent tracking pixel if JavaScript isn't available
• Lightweight — The tracking script weighs less than a kilobyte (and doesn't look like your typical tracking script)
• Generally not blocked — Because you host Shynet yourself, it tends not to be on ad block lists
• Primary-key integration — You can easily associate visitors in Shynet with their user accounts on your site (if that's something you want)

Metrics

Here's the information Shynet can give you about your visitors:

• Hits — how many pages on your site were opened/viewed
• Sessions — how many times your site was visited (essentially a collection of hits)
• Page load time — how long the pages on your site look to load
• Bounce rate — the percentage of visitors who left after just one page
• Duration — how long visitors stayed on the site
• Referrers — the links visitors followed to get to your site
• Locations — the relative popularity of all the pages on your site
• Operating system — your visitors' OS (from user agent)
• Browser — your visitors' browser (from user agent)
• Geographic location & network — general location of your visitors (from IP)
• Device type — whether your visitors are using a desktop, tablet, or phone (from user agent)

Workflow

• Collaboration built-in — Administrators can easily share services with other users, as well
• Accounts (or not) — Shynet has a fully featured account management workflow (powered by Django Allauth)

Recommendations

Shynet isn't for everyone. It's great for personal projects and small to medium size websites, but hasn't been tested with ultra-high traffic sites. It also requires a fair amount of technical know-how to deploy and maintain, so if you need a one-click solution, you're best served with other tools.

Concepts

Shynet is pretty simple, but there are a few key terms you need to know in order to use it effectively:

Services are the properties on the web you'd like to track. These generally correspond to websites or single top-level domains. Shynet generates one tracking embed per service.

Hits are a single page-load on one of your services.

Sessions are a collection of hits (or just one) that are made by the same browser in a short period of time.

Installation

You can find instructions on getting started and usage in the Usage Guide. Out of the box, we support deploying via a simple Docker container, docker-compose, Heroku, or Kubernetes (see kubernetes).

FAQ

Does Shynet respond to Do Not Track (DNT) signals? Yes. While there isn't any standardized way to handle DNT requests, Shynet allows you to specify whether you want to collect any data from users with DNT enabled on a per-service basis. (By default, Shynet will not collect any data from users who specify DNT.)

Is this GDPR compliant? It depends on how you use it. If you're worried about GDPR, you should talk to a lawyer about your particular data collection practices. I'm not a lawyer. (And this isn't legal advice.)

Troubleshooting

Having trouble with Shynet? Check out the troubleshooting guide, or create an issue if you think you found a bug in Shynet itself (or have a feature suggestion).

Roadmap

To see the upcoming planned features, check out the repository's roadmap project. Upcoming features include data aggregation through rollups, anomaly detection, detailed data exports, two-factor authentication, and a data deletion tool.

In the Wild

These sites use Shynet to monitor usage without violating visitors' privacy: PolitiTweet, Miles' personal site, a17t, Lensant, WhoAreMyRepresentatives.org, and more. (Want to add your site to this list? Send a PR.)

Contributing

Are you interested in contributing to Shynet? Just send a pull request! Maybe once the project matures there will be more detailed contribution guidelines, but for now just send the code this way and we'll make sure it meets our standards together. Just know that by contributing, you agree to share all of your contributions under the same license as the project (see LICENSE). And always be sure to follow the Code of Conduct.

License

Shynet is made available under the Apache License, version 2.0.

---

Shynet was created by Miles McCain (@MilesMcCain) at the Recurse Center using a17t.

shynet's Issues

Ability to track individual events

Hi! Loving shynet so far. Really like the interface. Testing it out on my personal blog and hoping to ditch Google Analytics soon. Quick question: Is it possible to track individual events on pages? For example I want to check how many times someone clicks on a particular div. If this isn't already a feature, do you foresee this a possible feature for 1.0?

Error 500 when sending mail

Hello,
I'm using Shynet with docker compose on Ubuntu 20.04 and I have an error when I try to send the verification email.
The browser show me a "Server Error (500)" message.
Here are the logs.
Thanks for your help !

Possible to add to Dockerhub?

I dunno if that's a thing or not - I see it can be run in a Docker container.

DB Configuration for Heroku deployment

Currently, DB configuration for shynet uses the envvars DB_HOST, DB_USER, etc. Deployments to Heroku require your app to accept the environment variable DATABASE_URL, the value of which can change at any time. Currently, I can manually set the environment variables accepted by shynet based on the value of DATABASE_URL, but I will have to update the variables whenever the value of DATABASE_URL changes.

I would like the envvar DATABASE_URL to be supported in addition to the current database configuration options. I know it's less than ideal to have two means of configuring the same property, and not everyone may share this use case, so I'd like to hear what other users and contributors think. Happy to make a PR with these changes.

Public sign-ups are not allowed at this time. on signup page

I am deployed Shynet on heroku.
it will opens login page and i don't have a registred account.
i am opened signup tab and it will shows
"Public sign-ups are not allowed at this time."
How to create an account ?

Rollups

After awhile, storing individual hit and session data becomes less useful; it starts to take up more space in the database than it's worth. It would be helpful to add optional rollups that aggregate old information.

API Access

Is there a possibility an API could be added?

automatically ignore IP of shynet user/admin

I am just trying shynet on some small personal websites.
Since there is not much traffic in these site, I create most of it when working on content or writing new posts. That kind of distorts the overview.

Would it be possible to add a toggle to the advanced settings of a service to exclude the IP-address of the shynet user for eg. 24h. Most ISPs change the IP-adresses once a day.

possible workarounds right now:
I could add every IP adress I get from my ISP to the ignore list, but this is a lot of work.
I also could ignore a broader range of IP-adresses like 127.0.0.1/32, but this would possibly block other users and/or sometimes it wont match my real IP.

So I think the best option would be to check the shynet users IP-adress and exclude it for the day.
Another possibility would be to check a ddns Domain A-record for the current IP-adress.

What do you think?
Best regards!

uPlot (apex charts alternative)

hey @milesmcc

i came from https://news.ycombinator.com/item?id=23806037

allow me to shamelessly plug uPlot in case you were interested in evaluating alternatives to slow/bloated Apex:

https://github.com/leeoniya/uPlot#performance

uPlot's origins are simular to this project (diy, minimal, analytics). you can read more of the story here: https://news.ycombinator.com/item?id=23045207

cheers!

UI Improvements

There is a lot of room for improvement in the UI. I am going to list some ideas here.

• Merge referrals from the same domain in a drop down

I am thinking about something like what Wordpress.com does. They put all popular search engines under one list item and put all different referral paths from one domain under one list item. This is useful to get more information at one glance.

In addition to this, I am not sure why Shynet is forcing me to vertically scroll all the different tables. It clearly crops the URL to make space for the actual count right next to it. For some reason it is spacing the count more than it should and it is causing an overflow.

• Auto refresh the online user count

This is self explanatory. Right now I manually refresh the page to see new value. Unless I am missing something we don't have this feature. Instead of enabling this by default we can enable this on a per-site basis. Adding this as a toggle setting will make sure we don't force the server do extra work when it isn't really required by the user.

• Add icons behind country names

Again, wordpress does this beautifully. There is a finite list of countries and I am sure we can get a free list from somewhere. The cacheability and the size of these icons will make sure the server doesn't work over-time.

The same feature request goes for device types.

• Add a favicon to the shynet instance

This one is fairly simple but low priority. Right now there is no favicon and it doesn't let Shynet stand out from the crowd of tabs I usually have open. A favicon will make it easy to spot which tab is Shynet when the number of open tabs get large enough.

• Reduce UI friction

Currently, whenever a user creates a new service the dashboard for that service is show as the next step. I would love Shynet to show the dashboard but also open up a popup with the installation snipped. We know that the next logical step for any user is going to be to copy the snippet so let's make it easier for them to do that :)

Document health-check endpoint

IP version mismatch when processing ignored IPs

As I can guess this is something with IP version mismatch?

ozq4l  2a02:2168:8abf:ad00:e85d:2f13:b1f:c00/128 and 192.168.1.1/32 are not of the same version
ozq4l  Traceback (most recent call last):
ozq4l    File "/usr/src/shynet/analytics/tasks.py", line 67, in ingress_request
ozq4l      if ignored_network.supernet_of(remote_ip):
ozq4l    File "/usr/local/lib/python3.8/ipaddress.py", line 1004, in supernet_of
ozq4l      return self._is_subnet_of(other, self)
ozq4l    File "/usr/local/lib/python3.8/ipaddress.py", line 991, in _is_subnet_of
ozq4l      raise TypeError(f"{a} and {b} are not of the same version")
ozq4l  TypeError: 2a02:2168:8abf:ad00:e85d:2f13:b1f:c00/128 and 192.168.1.1/32 are not of the same version
ozq4l  2a02:2168:8abf:ad00:e85d:2f13:b1f:c00/128 and 192.168.1.1/32 are not of the same version
ozq4l  Traceback (most recent call last):
ozq4l    File "/usr/src/shynet/analytics/tasks.py", line 67, in ingress_request
ozq4l      if ignored_network.supernet_of(remote_ip):
ozq4l    File "/usr/local/lib/python3.8/ipaddress.py", line 1004, in supernet_of
ozq4l      return self._is_subnet_of(other, self)
ozq4l    File "/usr/local/lib/python3.8/ipaddress.py", line 991, in _is_subnet_of
ozq4l      raise TypeError(f"{a} and {b} are not of the same version")
ozq4l  TypeError: 2a02:2168:8abf:ad00:e85d:2f13:b1f:c00/128 and 192.168.1.1/32 are not of the same version
ozq4l  2a02:2168:8abf:ad00:e85d:2f13:b1f:c00/128 and 192.168.1.1/32 are not of the same version
ozq4l  Traceback (most recent call last):
ozq4l    File "/usr/src/shynet/analytics/tasks.py", line 67, in ingress_request
ozq4l      if ignored_network.supernet_of(remote_ip):
ozq4l    File "/usr/local/lib/python3.8/ipaddress.py", line 1004, in supernet_of
ozq4l      return self._is_subnet_of(other, self)
ozq4l    File "/usr/local/lib/python3.8/ipaddress.py", line 991, in _is_subnet_of
ozq4l      raise TypeError(f"{a} and {b} are not of the same version")
ozq4l  TypeError: 2a02:2168:8abf:ad00:e85d:2f13:b1f:c00/128 and 192.168.1.1/32 are not of the same version
ozq4l  2a02:2168:8abf:ad00:e85d:2f13:b1f:c00/128 and 192.168.1.1/32 are not of the same version
ozq4l  Traceback (most recent call last):
ozq4l    File "/usr/src/shynet/analytics/tasks.py", line 67, in ingress_request
ozq4l      if ignored_network.supernet_of(remote_ip):
ozq4l    File "/usr/local/lib/python3.8/ipaddress.py", line 1004, in supernet_of
ozq4l      return self._is_subnet_of(other, self)
ozq4l    File "/usr/local/lib/python3.8/ipaddress.py", line 991, in _is_subnet_of
ozq4l      raise TypeError(f"{a} and {b} are not of the same version")
ozq4l  TypeError: 2a02:2168:8abf:ad00:e85d:2f13:b1f:c00/128 and 192.168.1.1/32 are not of the same version

Getting Fatal Error: Cannot connect to database

Been trying for hours to get docker to start but at this point I'm going to start looking into running shynet directly instead of with docker since I can't seem to get the container to initialize properly and nothing's better than a good ol' sudo service shynet start. To be clear, I am using the latest docker image and can't get past a first run.

First I was getting a lot of:

django.db.utils.OperationalError: could not connect to server: Connection refused
	Is the server running on host "localhost" (127.0.0.1) and accepting
	TCP/IP connections on port 5432?
could not connect to server: Address not available
	Is the server running on host "localhost" (::1) and accepting
	TCP/IP connections on port 5432?

Migrations failed, exiting
Launching Shynet web server...
[2020-05-19 05:29:42 +0000] [11] [INFO] Starting gunicorn 20.0.4
[2020-05-19 05:29:42 +0000] [11] [INFO] Listening at: http://0.0.0.0:8080 (11)
[2020-05-19 05:29:42 +0000] [11] [INFO] Using worker: sync
[2020-05-19 05:29:42 +0000] [13] [INFO] Booting worker with pid: 13

...which would throw my terminal into a frozen state, forcing me to have to open a new session on my VPS just to get my prompt back by shutting down the docker container and then try again. After a bit, this comment gave me the idea that the container couldn't see my database at all since 'localhost' to it was itself and not the server it was on. I then tried using my VPS's public IP which didn't work:

File "/usr/local/lib/python3.8/site-packages/psycopg2/__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
	Is the server running on host "<my_ip_address>" and accepting
	TCP/IP connections on port 5432?

Tried adjusting postgresql settings to watch that IP instead of localhost, even thought that dashes in my db name were throwing it off, or that postgres might be looking for a socket connection instead— nothing.

Unless I can get help diagnosing the problem, I'm going to start looking into running shynet as a linux service since I think that will probably help uncomplicate a lot of the connection issues I'm having. Tool looks promising though!

Support for MFA

I've been trying Shynet out as an analytics platform, and I really love it! However, I'm a bit disappointed that there is no option to have MFA, especially on the admin account. Do you have any plans to support MFA (like Google Authenticator/Authy or Webauthn)?

Googlebot is generating hits

Is there an option to filter out bots or is something wrong with my configuration?
Anyway, thanks for this project. Looking promising so far!

Please add a requirements.txt

I presume you already have the requirements in a different format (Pipfile?) but could you just do a pip freeze > requirements.txt for us people that use good old pip?

Put version number in footer

Many services do this, it's useful to quickly see which version is deployed.

Expose metrics to prometheus

I believe it would be beneficial to instrument shynet with prometheus metrics as well as expose data collected by shynet in prometheus format. This would provide two benefits:

• with an instrumented application, a monitoring system can act on issues with shynet itself
• exposing data collected by shynet would allow to correlate it with other data in prometheus and create meaningful alerts about website.

I propose prometheus exposition format as there is already an official python library and format is on the path to becoming an RFC standard (under the name of OpenMetrics).

"Allowed hostnames" field only allows one hostname

The Access-Control-Allow-Origin header is simply set to the value of this field, meaning you can't allow multiple origins for one service (e.g. https://example.com,https://www.example.com), as Access-Control-Allow-Origin only allows you to specify one origin.

The server needs to check if the request Origin header matches any hostname in this field, and send the corresponding origin back to the browser as Access-Control-Allow-Origin.

No outer padding on certain viewport widths

• Browser: Firefox 75
• Shynet: v0.2.1

Probably caused by px-4 md:px-0 on the outermost <section> element.

This seems to affect all Shynet pages.

Below and above that it's fine:

Step by step guide with docker-compose

This is not an issue.

I just wanted to thank you all for Shynet and let you know that I have just published https://github.com/lucasrla/shynet-docker-lightsail

I took the basic instructions from the Usage Guide (i.e. HTTP-only install + Cloudflare as reverse proxy) and tweaked them to use docker-compose instead. I also added a few helper scripts to quicjly spin up a docker-ready Amazon Lightsail instance.

I hope it makes it even easier for newcomers to adopt Shynet.

Thanks again!

Document primary-key integration

Readme page says that I can associate visitors in Shynet with their user accounts on my site. But how? I can't find any clue to this.

File type mismatch on the script.js

Not exactly sure when this started happening, but when trying to load the script on my webpage, Firefox gives me the following error:

The resource from “https://<my-domain>/ingress/<site-id>/script.js” was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff).

This is due to my script.js giving a 500 Error with the following traceback:

ERROR Internal Server Error: /ingress/<site-id>/script.js
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 552, in connect
    sock = self._connect()
  File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 578, in _connect
    for res in socket.getaddrinfo(self.host, self.port, self.socket_type,
  File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/utils/decorators.py", line 43, in _wrapper
    return bound_method(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/src/shynet/analytics/views/ingress.py", line 64, in dispatch
    origins = cache.get(f"service_origins_{service_uuid}")
  File "/usr/local/lib/python3.8/site-packages/redis_cache/backends/base.py", line 30, in wrapped
    return method(self, client, key, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/redis_cache/backends/base.py", line 267, in get
    return self._get(client, key, default)
  File "/usr/local/lib/python3.8/site-packages/redis_cache/backends/base.py", line 255, in _get
    value = client.get(key)
  File "/usr/local/lib/python3.8/site-packages/redis/client.py", line 1579, in get
    return self.execute_command('GET', name)
  File "/usr/local/lib/python3.8/site-packages/redis/client.py", line 875, in execute_command
    conn = self.connection or pool.get_connection(command_name, **options)
  File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 1185, in get_connection
    connection.connect()
  File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 557, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error -2 connecting to redis.default.svc.cluster.local:6379. Name or service not known.

This is the case for both the code I pulled yesterday and the code I just pulled before writing this.

Option to show session IP address in session details

As we already have an IP address for each session (viewable at /admin/analytics/session/) it would be useful to show that in the session details interface.

Option to not collect personal information (IP addresses, user agent, etc)

Currently, Shynet will record the IP address and user agent of all tracked requests (provided the request doesn't specify DNT). There needs to be an option to not collect this data, as it's incompatible with some organizations' privacy policies.

Currently, cross-hit session linking is done by correlating IP addresses and user agents, so using hashed data for session linking (potentially stored in the cache, not the DB) will need to come first.

Links on same domain being counted as referrers

The referrer page lists pages that are hosted under the same domain that is being tracked, which I consider a bug as self-referrers aren't typically included in analytics.

Example:

The "blog" URL and all subpages shouldn't show up as they are pages being tracked, and therefore not really a referrer.

Data deletion tool

It's often necessary to delete data from Shynet in order to comply with legal restrictions and removal requests. It'd be nice if Shynet had a simple interface for querying and deleting sessions (instead of having to use the admin panel).

Worker Timeout

Hi there,

I may well be doing something wrong, but having issues with timeouts.. System works fine for a couple of minutes, then randomly stops - this halts fetching the file from the server and makes web pages hang.
I am running an unusual setup, using two VMs. One is my main ingress via a Caddy reverse proxy, which then sends the request internally to the ShyNet server which is on a different machine.
I don't think this should affect operation - any thoughts?

Launching Shynet web server...
[2020-04-28 21:45:16 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2020-04-28 21:45:16 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
[2020-04-28 21:45:16 +0000] [1] [INFO] Using worker: sync
[2020-04-28 21:45:16 +0000] [8] [INFO] Booting worker with pid: 8
[2020-04-28 21:45:16 +0000] [9] [INFO] Booting worker with pid: 9
[2020-04-28 21:45:16 +0000] [10] [INFO] Booting worker with pid: 10
[2020-04-28 21:47:16 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:8)
[2020-04-28 22:47:16 +0100] [8] [INFO] Worker exiting (pid: 8)
[2020-04-28 21:47:16 +0000] [11] [INFO] Booting worker with pid: 11
[2020-04-28 21:47:18 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:10)
[2020-04-28 22:47:18 +0100] [10] [INFO] Worker exiting (pid: 10)
[2020-04-28 21:47:19 +0000] [12] [INFO] Booting worker with pid: 12

Thanks in advance - this project looks really promising !

Possible duplicate sessions

Opening an issue I actually won't be able to work on because I don't know most the underlying tech.

I've installed shynet but I see some "weird" hit counts and session counts.
In short inspecting the activity for the sites shows mostly three recordings for each person, with the same IP, most of the same hits but wiht minor differences.

I'm unsure if this is actually people loading the site three times, which I find doubtful in the amount of times I see it, or if t's some conflict in the way that sessions are grouped together.
Example.

Ability to exclude IP addresses/ranges from being counted

I think this is self-explanatory - in some cases it's undesirable to have your office or home location counted in the statistics. I don't know if this hinges on tracking and storing IP addresses; this filtering would happen before the data would make it to the database, but I don't know if the IP address removal is done at the client or at the server.

Health check endpoint

It would great if there was a /healthz endpoint that "pinged" the database (usually done with a "SELECT 1") on the SQL DB and returns a 200 OK on successful DB connection.

Kubernetes would leverage this to know which shynet pods to remove from serving user traffic as they couldn't get to the database anyway.

• https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

Dockerfile build hangs when downloading geoip

Hi,

I'm trying to build the docker image but the step curl "https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-ASN&license_key=G4Lm0C60yJsnkdPi&suffix=tar.gz" hangs (w/o a timeout, at least not for 30 minutes). Downloading this file locally won't work either.

This probably does not depend directly on you, but I'd still like to be able to build the image ;-)

Any help appreciated!

Referer checking failed

I set everything up as specified in the GUIDE but on logging in I get this error:

WARNING Forbidden (Referer checking failed - https://sub.example.com/accounts/login/?next=/dashboard/ does not match any trusted origins.): /accounts/login/

Any help would be appreciated.

This only happens over HTTPS. HTTP works fine.

My NGINX config:

server {
    server_name sub.example.com;
    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_pass http://127.0.0.1:8080;
    }
    listen 80; # for some reason if I don't set this, it gives 501 (Cloudflare) error
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/sub.example.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/sub.example.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
}

Command used to start the image:

sudo docker run -p 8080:8080 --env-file=shynet.env --network='host' milesmcc/shynet:latest

Shynet.env file:

# This file shows all of the environment variables you can
# set to configure Shynet, as well as information about their
# effects. Make a copy of this file to configure your deployment.

# Database settings (PostgreSQL)
DB_NAME=shynet_db
DB_USER=user
DB_PASSWORD=pass
DB_HOST=127.0.0.1
DB_PORT=5432

# General Django settings
DJANGO_SECRET_KEY=secret_key

# For better security, set this to your deployment's domain. Comma separated.
ALLOWED_HOSTS=127.0.0.1

# Set to True (capitalized) if you want people to be able to sign up for your Shynet instance (not recommended)
SIGNUPS_ENABLED=False

# Should user email addresses be verified? Only set this to `required` if you've setup the email settings and allow
# public sign-ups; otherwise, it's unnecessary.
ACCOUNT_EMAIL_VERIFICATION=none

# The timezone of the admin panel. Affects how dates are displayed.
TIME_ZONE=America/New_York

# Set to "False" if you will not be serving content over HTTPS
SCRIPT_USE_HTTPS=True

# How frequently should the monitoring script "phone home" (in ms)?
SCRIPT_HEARTBEAT_FREQUENCY=5000

# How much time can elapse between requests from the same user before a new
# session is created, in seconds?
SESSION_MEMORY_TIMEOUT=1800

# Should only superusers (admins) be able to create services? This is helpful
# when you'd like to invite others to your Shynet instance but don't want
# them to be able to create services of their own.
ONLY_SUPERUSERS_CREATE=True

# Whether to perform checks and setup at startup, including applying unapplied
# migrations. For most setups, the recommended value is True. Defaults to True.
# Will skip only if value is False.
PERFORM_CHECKS_AND_SETUP=True

# The port that Shynet should bind to. Don't set this if you're deploying on Heroku.
PORT=8080

# Redis, queue, and parellization settings; not necessary for single-instance deployments.
# Don't uncomment these unless you know what you are doing!
# NUM_WORKERS=1
# Make sure you set a REDIS_CACHE_LOCATION if you have more than one frontend worker/instance.
# REDIS_CACHE_LOCATION=redis://redis.default.svc.cluster.local/0 
# If CELERY_BROKER_URL is set, make sure CELERY_TASK_ALWAYS_EAGER is False and
# that you have a separate queue consumer running somewhere via `celeryworker.sh`.
# CELERY_TASK_ALWAYS_EAGER=False
# CELERY_BROKER_URL=redis://redis.default.svc.cluster.local/1

Single-command Docker-compose deploy

It'd be great to have a way to run shynet from scratch based on a single docker-compose file.
I finished most of the (minor) adaptations that are needed in order to get that running, so I'll PR when I'm done.

For transparency, modifications include:

• exposing current function Arguments as Environement variables (https://github.com/milesmcc/shynet/tree/master/shynet/core/management/commands)
• Adding a check to Webserver.sh to query postgre on startup and check:
  • if the db is running (otherwise exit container)
  • if the database tables exist, otherwise run migrate
  • if the admin user is set, otherwise run createadmin with the env passed as parameter
  • if the hostname is set, otherwise run hostname with with the env passed as parameter
  • if the whitelabel is set, otherwise run whitelabel with the env passed as parameter

the Compose file itself is pretty much ready but I suck at postgre so it' staking me a while to write valid checks in the bash.

`script.js` being requested often during app usage

So I've applied this to a few of my react applications, and i've noticed that even during non-routing based clicks / actions, script.js gets queried all the time.

The request payload is identical in all of those requests.

Is there anyway to reduce the amount of requests being made? Especially if no navigation is taking place.

Analytics not showing up

I have succesfully installed Shynet behind a reverse proxy. I have created a new service, added the snippet to my site but I don't see any analytics showing up on the interface.

I do see the snippet being loaded on my site, and I see requests being made every 5000ms (hearthbeat value).

Expose "sendUpdate" function for external use.

It'd be nice if the sendUpdate function in script.js was exposed. In SPAs, Shynet doesn't properly detect page transitions because the browser page is never reloaded. Hence you have to wait for the next heartbeat for anything to be tracked.

If the sendUpdate function was exposed, it could be called in your router transition event handler and would allow for these events to be tracked. For example with Fathom and Next.js you can do something like this:

Router.events.on("routeChangeComplete", () => {
  fathom.trackPageview();
});

Anomaly alerts

It'd be great if Shynet could send service owners an email when anomalies occur—for example, when traffic spikes or dips.

Disable all email features option

I have an instance of Shynet deployed on a Kubernetes cluster. After I created my super user account - I tried to log in with the credentials and was told that I needed to verify my email.

However this instance has no email servers available and never will, it would be useful if there was a single option I could set to disable all email services.

Ability to exclude bots from being counted

I could see this going two ways: either a switch in the frontj end to toggle the bots in the tables and graphs, and/or a switch to prevent them from being counted at all. Personally, my preference would be the former, as it gives you a bit more flexibility.

Fastest way to deploy shynet

This is a really great project, the simple image tracker script looks awesome. I will run it for my personal websites.
I was wondering if Shynet could be deployed as a Dockerized Heroku App? And if so, could be possible to create a quick deploy button?

So I did It. And I think it is working pretty well.

Deploy Shynet on Heroku

You may try it in your account right now. Just go to my shynet fork and press the "Deploy to Heroku" button.

After deploy your app it will not work. You need to manually fill the generated database config like so:

heroku config:get DATABASE_URL --app=my-app

heroku config:set DB_NAME "myDbName" --app=my-app
heroku config:set DB_USER "myDbUser" --app=my-app
heroku config:set DB_PASSWORD "myDbPassword" --app=my-app
heroku config:set DB_HOST "myDbHost" --app=my-app
heroku config:set DB_PORT "myDbPort" --app=my-app

heroku restart --app my-app

Gmail SMTP (quick'n dirty)

To use your gmail account as smtp you need to:

Allow less secure apps
Display Unlock Captcha

heroku config:set EMAIL_HOST "smtp.gmail.com" --app=my-app
heroku config:set EMAIL_HOST_PASSWORD "mypassword" --app=my-app
heroku config:set EMAIL_HOST_USER "[email protected]" --app=my-app
heroku config:set EMAIL_PORT "465" --app=my-app

heroku restart --app my-app

I think it would also be possible to recognize the presence of heroku's environment variable DATABASE_URL, parse it and then overwrite the others (uppon the application boot).

Does anyone have an interest in making skynet compatible with heroku?

Docker Image optimization

Opening an issue for myself to track this.

I believe the current docker image can be slimmed down by roughly 800Mo by switching from python:3 to python:3-alpine, purging apt caches and limiting the number of individual layers in the final image.

Sessions have local IP address, country and location is unknown

I installed the app with docker-container and I use Nginx to reverse proxy it. The problem is that every session seems to be coming from a local IP address (192.168.16.1) therefore the country and the location is unknown. If I check this IP it belongs to the shynet_internal docker network.

$ sudo docker network ls --no-trunc
NETWORK ID          NAME                DRIVER              SCOPE
<...OMITTED...>
4fbb80aa345c        shynet_internal     bridge              local

$ sudo ifconfig 
<...OMITTED...>
br-4fbb80aa345c: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.16.1  netmask 255.255.240.0  broadcast 192.168.31.255

Has anyone experienced something similar?

Detailed data exports

Right now, there isn't an easy way to export data from Shynet into a spreadsheet or any other format. This isn't a huge issue because everyone has access to the underlying database (we're self hosting Shynet, after all), but there should be a way to easily export things with a click.

Button to return to main page on session page is the wrong color

Deploy as a service on Debian/Ubuntu

This project looks really great and I'm interested in running it for my personal websites. I'm wondering if Shynet can be deployed as a service on a Debian/Ubuntu instance (e.g. using LXC) instead of using Docker. Do you have any plans to add documentation for this?

[FEATURE REQUEST] Support ClickHouse instead of PostgreSQL for storing data

please look at
https://clickhouse.tech
it's a blazing fast COLUMNAR database
your script can be store data to clickhouse as long term storage via https://github.com/carrotquest/django-clickhouse

Any way to track utm parameters?

Hey, I am loving Shynet so far. Well done! However, I would like to track users who come from different sources via utm parameters. Is there any way this can be done?

Data not showing up

Hi there, I am in a similar position to #24 but have tried the suggestions from there and cannot get data to be logged.

I have also tried the troubleshooting ...

I am using an nginx reverse proxy and can access the admin area of shynet.

What is weird is that if I go to the script created by shynet for my site and paste in into a browser, I get a 404, which is odd!

It uses an 'ingress' directory which isn't present on my server, so am not sure if shynet should be redirecting that in some way....as I am not using Kubernetes.

Thanks for any help.

Cheers

Explore implementing differential privacy

This issue will house discussion of adding differential privacy support to Shynet.