vesparny/fair-analytics

📊 An analytics server that doesn't undermine user's privacy

fair-analytics's Issues

Error: listen EADDRNOTAVAIL: address not available

When I try to scan my site with this command:

fair-analytics -H v1rtl.site -p 80

I get this error:

Error: listen EADDRNOTAVAIL: address not available 35.228.53.122:80
    at Server.setupListenHandle [as _listen2] (net.js:1292:21)
    at listenInCluster (net.js:1357:12)
    at GetAddrInfoReqWrap.doListen [as callback] (net.js:1496:7)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:69:10)
Emitted 'error' event on Server instance at:
    at emitErrorNT (net.js:1336:8)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  code: 'EADDRNOTAVAIL',
  errno: -99,
  syscall: 'listen',
  address: '35.228.53.122',
  port: 80
}

My website loads fine, so it is available:

Looks like it checks an IP of now.sh instead of my website and throws an error. Even though, the address (IP address) is available:

SyntaxError: Unexpected token function

OS: 4.4.0-78-generic x86_64 GNU/Linux 
nvm: 0.29.0 
node: v6.10.3 
npm: 4.6.1

npm install -g fair-analytics

Click to expand the log

/home/user/.nvm/versions/node/v6.10.3/bin/fair-analytics -> /home/user/.nvm/versions/node/v6.10.3/lib/node_modules/fair-analytics/bin/index.js

[email protected] install /home/user/.nvm/versions/node/v6.10.3/lib/node_modules/fair-analytics/node_modules/sodium-native

node-gyp-build "node preinstall.js" "node postinstall.js"


[email protected] install /home/user/.nvm/versions/node/v6.10.3/lib/node_modules/fair-analytics/node_modules/utp-native

node-gyp-build

/home/user/.nvm/versions/node/v6.10.3/lib

└─┬ [email protected]

├─┬ [email protected]

│ ├── [email protected]

│ ├─┬ [email protected]

│ │ ├── [email protected]

│ │ ├── [email protected]

│ │ ├─┬ [email protected]

│ │ │ └── [email protected]

│ │ ├─┬ [email protected]

│ │ │ └── [email protected]  deduped

│ │ └── [email protected]

│ ├── [email protected]

│ ├── [email protected]

│ └─┬ [email protected]

│   └── [email protected]  deduped

├── [email protected]

├── [email protected]

├─┬ [email protected]

│ └── [email protected]

├─┬ [email protected]

│ └── [email protected]

├─┬ [email protected]

│ ├── [email protected]

│ ├── [email protected]

│ ├─┬ [email protected]

│ │ └── [email protected]

│ ├── [email protected]

│ ├─┬ [email protected]

│ │ ├── [email protected]  deduped

│ │ └─┬ [email protected]

│ │   ├── [email protected]

│ │   ├── [email protected]

│ │   ├── [email protected]  deduped

│ │   ├── [email protected]

│ │   ├── [email protected]  deduped

│ │   ├─┬ [email protected]

│ │   │ └── [email protected]  deduped

│ │   └── [email protected]

│ ├── [email protected]

│ ├── [email protected]

│ ├─┬ [email protected]

│ │ ├── [email protected]  deduped

│ │ └── [email protected]  deduped

│ ├─┬ [email protected]

│ │ ├─┬ [email protected]

│ │ │ ├─┬ [email protected]

│ │ │ │ ├── [email protected]

│ │ │ │ ├── [email protected]  deduped

│ │ │ │ └── [email protected]  deduped

│ │ │ ├─┬ [email protected]

│ │ │ │ └── [email protected]

│ │ │ ├─┬ [email protected]

│ │ │ │ ├─┬ [email protected]

│ │ │ │ │ ├── [email protected]  deduped

│ │ │ │ │ ├── [email protected]  deduped

│ │ │ │ │ └── [email protected]

│ │ │ │ ├─┬ [email protected]

│ │ │ │ │ └─┬ [email protected]

│ │ │ │ │   ├── [email protected]  deduped

│ │ │ │ │   ├── [email protected]  deduped

│ │ │ │ │   ├── [email protected]

│ │ │ │ │   └── [email protected]

│ │ │ │ ├─┬ [email protected]

│ │ │ │ │ ├── [email protected]

│ │ │ │ │ ├── [email protected]

│ │ │ │ │ ├── [email protected]

│ │ │ │ │ └─┬ [email protected]

│ │ │ │ │   └── [email protected]

│ │ │ │ ├─┬ [email protected]

│ │ │ │ │ ├── [email protected]

│ │ │ │ │ ├── [email protected]

│ │ │ │ │ ├── [email protected]

│ │ │ │ │ └── [email protected]

│ │ │ │ ├─┬ [email protected]

│ │ │ │ │ └── [email protected]

│ │ │ │ ├── [email protected]

│ │ │ │ ├─┬ [email protected]

│ │ │ │ │ ├── [email protected]

│ │ │ │ │ └── [email protected]  deduped

│ │ │ │ ├─┬ [email protected]

│ │ │ │ │ ├── [email protected]  deduped

│ │ │ │ │ ├── [email protected]  deduped

│ │ │ │ │ ├── [email protected]

│ │ │ │ │ └── [email protected]

│ │ │ │ ├── [email protected]

│ │ │ │ ├─┬ [email protected]

│ │ │ │ │ └─┬ [email protected]

│ │ │ │ │   ├── [email protected]

│ │ │ │ │   ├── [email protected]

│ │ │ │ │   └── [email protected]  deduped

│ │ │ │ └─┬ [email protected]

│ │ │ │   ├── [email protected]  deduped

│ │ │ │   └─┬ [email protected]

│ │ │ │     └── [email protected]

│ │ │ └─┬ [email protected]

│ │ │   ├── [email protected]  deduped

│ │ │   └── [email protected]  deduped

│ │ ├── [email protected]  deduped

│ │ ├─┬ [email protected]

│ │ │ ├── [email protected]  deduped

│ │ │ ├── [email protected]

│ │ │ ├─┬ [email protected]

│ │ │ │ └── [email protected]

│ │ │ ├── [email protected]

│ │ │ ├─┬ [email protected]

│ │ │ │ └── [email protected]

│ │ │ └── [email protected]

│ │ ├── [email protected]  deduped

│ │ ├── [email protected]  deduped

│ │ └── [email protected]

│ ├── [email protected]

│ ├── [email protected]

│ ├── [email protected]

│ ├─┬ [email protected]

│ │ ├── [email protected]  deduped

│ │ └── [email protected]  deduped

│ ├─┬ [email protected]

│ │ ├── [email protected]

│ │ ├─┬ [email protected]

│ │ │ └── [email protected]

│ │ ├── [email protected]  deduped

│ │ ├─┬ [email protected]

│ │ │ └── [email protected]

│ │ └── [email protected]  deduped

│ ├─┬ [email protected]

│ │ ├─┬ [email protected]

│ │ │ ├── [email protected]

│ │ │ └── [email protected]

│ │ └── [email protected]

│ ├─┬ [email protected]

│ │ ├─┬ [email protected]

│ │ │ └── [email protected]

│ │ └── [email protected]  deduped

│ ├─┬ [email protected]

│ │ └── [email protected]  deduped

│ ├── [email protected]

│ ├── [email protected]

│ ├── [email protected]

│ └── [email protected]

├─┬ [email protected]

│ ├─┬ [email protected]

│ │ └── [email protected]  deduped

│ ├─┬ [email protected]

│ │ ├── [email protected]  deduped

│ │ ├── [email protected]

│ │ ├── [email protected]  deduped

│ │ ├─┬ [email protected]

│ │ │ ├─┬ [email protected]

│ │ │ │ ├── [email protected]

│ │ │ │ ├── [email protected]  deduped

│ │ │ │ ├── [email protected]  deduped

│ │ │ │ ├── [email protected]  deduped

│ │ │ │ ├─┬ [email protected]

│ │ │ │ │ ├── [email protected]  deduped

│ │ │ │ │ ├── [email protected]  deduped

│ │ │ │ │ └── [email protected]

│ │ │ │ ├─┬ [email protected]

│ │ │ │ │ ├── [email protected]  deduped

│ │ │ │ │ ├── [email protected]  deduped

│ │ │ │ │ └─┬ [email protected]

│ │ │ │ │   └── [email protected]  deduped

│ │ │ │ ├─┬ [email protected]

│ │ │ │ │ └── [email protected]  deduped

│ │ │ │ └── [email protected]  deduped

│ │ │ ├── [email protected]  deduped

│ │ │ ├─┬ [email protected]

│ │ │ │ ├── [email protected]  deduped

│ │ │ │ ├─┬ [email protected]

│ │ │ │ │ └─┬ [email protected]

│ │ │ │ │   ├── [email protected]  deduped

│ │ │ │ │   └── [email protected]  deduped

│ │ │ │ ├─┬ [email protected]

│ │ │ │ │ └── [email protected]

│ │ │ │ ├─┬ [email protected]

│ │ │ │ │ └── [email protected]  deduped

│ │ │ │ ├── [email protected]  deduped

│ │ │ │ ├─┬ [email protected]

│ │ │ │ │ ├── [email protected]  deduped

│ │ │ │ │ └── [email protected]

│ │ │ │ ├── [email protected]

│ │ │ │ └── [email protected]

│ │ │ ├── [email protected]

│ │ │ └── [email protected]

│ │ ├─┬ [email protected]

│ │ │ └── [email protected]

│ │ ├─┬ [email protected]

│ │ │ ├─┬ [email protected]

│ │ │ │ └── [email protected]  deduped

│ │ │ └─┬ [email protected]

│ │ │   └── [email protected]

│ │ ├── [email protected]

│ │ └─┬ [email protected]

│ │   ├── [email protected]  deduped

│ │   ├── [email protected]  deduped

│ │   └── [email protected]  deduped

│ └── [email protected]

├── [email protected]

├─┬ [email protected]

│ ├── [email protected]

│ ├── [email protected]

│ ├── [email protected]

│ └─┬ [email protected]

│   └── [email protected]  deduped

├─┬ [email protected]

│ ├─┬ [email protected]

│ │ ├── [email protected]  deduped

│ │ ├── [email protected]  deduped

│ │ ├── [email protected]  deduped

│ │ ├── [email protected]  deduped

│ │ └── [email protected]  deduped

│ ├─┬ [email protected]

│ │ ├── [email protected]

│ │ └─┬ [email protected]

│ │   └── [email protected]

│ ├── [email protected]

│ ├─┬ [email protected]

│ │ ├─┬ [email protected]

│ │ │ └─┬ [email protected]

│ │ │   ├── [email protected]

│ │ │   ├─┬ [email protected]

│ │ │   │ └── [email protected]

│ │ │   └── [email protected]  deduped

│ │ ├── [email protected]  deduped

│ │ ├── [email protected]  deduped

│ │ ├── [email protected]

│ │ ├─┬ [email protected]

│ │ │ ├── [email protected]

│ │ │ └── [email protected]  deduped

│ │ ├─┬ [email protected]

│ │ │ └─┬ [email protected]

│ │ │   ├─┬ [email protected]

│ │ │   │ ├── [email protected]  deduped

│ │ │   │ └── [email protected]  deduped

│ │ │   ├── [email protected]  deduped

│ │ │   ├─┬ [email protected]

│ │ │   │ └── [email protected]  deduped

│ │ │   ├── [email protected]

│ │ │   ├── [email protected]

│ │ │   └── [email protected]  deduped

│ │ └─┬ [email protected]

│ │   └─┬ [email protected]

│ │     ├── [email protected]  deduped

│ │     ├── [email protected]  deduped

│ │     └── [email protected]  deduped

│ ├── [email protected]  deduped

│ ├─┬ [email protected]

│ │ └─┬ [email protected]

│ │   ├─┬ [email protected]

│ │   │ ├─┬ [email protected]

│ │   │ │ ├── [email protected]

│ │   │ │ └── [email protected]

│ │   │ ├─┬ [email protected]

│ │   │ │ └── [email protected]

│ │   │ └─┬ [email protected]

│ │   │   └── [email protected]

│ │   ├── [email protected]

│ │   ├── [email protected]

│ │   ├─┬ [email protected]

│ │   │ └── [email protected]

│ │   ├── [email protected]

│ │   ├── [email protected]

│ │   └── [email protected]

│ ├── [email protected]

│ ├── [email protected]

│ ├── [email protected]  deduped

│ ├── [email protected]

│ ├── [email protected]

│ ├── [email protected]  deduped

│ ├─┬ [email protected]

│ │ ├── [email protected]

│ │ ├── [email protected]

│ │ └── [email protected]

│ └── [email protected]  deduped

├─┬ [email protected]

│ ├─┬ [email protected]

│ │ ├── [email protected]

│ │ ├── [email protected]

│ │ ├── [email protected]

│ │ ├─┬ [email protected]

│ │ │ └── [email protected]

│ │ ├── [email protected]

│ │ ├── [email protected]

│ │ ├─┬ [email protected]

│ │ │ ├── [email protected]

│ │ │ ├── [email protected]  deduped

│ │ │ └── [email protected]  deduped

│ │ ├─┬ [email protected]

│ │ │ ├─┬ [email protected]

│ │ │ │ ├── [email protected]

│ │ │ │ └─┬ [email protected]

│ │ │ │   └── [email protected]

│ │ │ └── [email protected]

│ │ ├─┬ [email protected]

│ │ │ ├─┬ [email protected]

│ │ │ │ └── [email protected]  deduped

│ │ │ ├─┬ [email protected]

│ │ │ │ └── [email protected]  deduped

│ │ │ ├── [email protected]

│ │ │ └─┬ [email protected]

│ │ │   └── [email protected]  deduped

│ │ ├─┬ [email protected]

│ │ │ ├── [email protected]

│ │ │ ├─┬ [email protected]

│ │ │ │ ├── [email protected]

│ │ │ │ ├── [email protected]

│ │ │ │ ├── [email protected]

│ │ │ │ └─┬ [email protected]

│ │ │ │   └── [email protected]  deduped

│ │ │ └─┬ [email protected]

│ │ │   ├── [email protected]

│ │ │   ├── [email protected]

│ │ │   ├─┬ [email protected]

│ │ │   │ └── [email protected]  deduped

│ │ │   ├─┬ [email protected]

│ │ │   │ └── [email protected]

│ │ │   ├─┬ [email protected]

│ │ │   │ └── [email protected]  deduped

│ │ │   ├─┬ [email protected]

│ │ │   │ └── [email protected]

│ │ │   ├─┬ [email protected]

│ │ │   │ └── [email protected]  deduped

│ │ │   ├── [email protected]

│ │ │   └── [email protected]  deduped

│ │ ├── [email protected]

│ │ ├── [email protected]  deduped

│ │ ├── [email protected]

│ │ ├─┬ [email protected]

│ │ │ └── [email protected]

│ │ ├── [email protected]

│ │ ├── [email protected]

│ │ ├── [email protected]

│ │ ├── [email protected]

│ │ ├── [email protected]

│ │ ├─┬ [email protected]

│ │ │ └── [email protected]

│ │ ├─┬ [email protected]

│ │ │ └── [email protected]  deduped

│ │ └── [email protected]

│ └─┬ [email protected]

│   ├── [email protected]  deduped

│   ├─┬ [email protected]

│   │ └── [email protected]  deduped

│   ├── [email protected]

│   └── [email protected]  deduped

├── [email protected]

├─┬ [email protected]

│ └── [email protected]

├─┬ [email protected]

│ ├─┬ [email protected]

│ │ ├── [email protected]

│ │ ├─┬ [email protected]

│ │ │ └── [email protected]

│ │ └── [email protected]

│ ├── [email protected]

│ └── [email protected]  deduped

└─┬ [email protected]

├── [email protected]  deduped

├── [email protected]  deduped

├─┬ [email protected]

│ ├─┬ [email protected]

│ │ └── [email protected]

│ ├── [email protected]  deduped

│ ├─┬ [email protected]

│ │ └── [email protected]

│ ├─┬ [email protected]

│ │ └── [email protected]

│ ├─┬ [email protected]

│ │ ├── [email protected]  deduped

│ │ ├── [email protected]

│ │ └── [email protected]

│ └── [email protected]  deduped

├── [email protected]

├─┬ [email protected]

│ └─┬ [email protected]

│   ├─┬ [email protected]

│   │ ├─┬ [email protected]

│   │ │ └── [email protected]

│   │ ├── [email protected]

│   │ ├── [email protected]  deduped

│   │ ├── [email protected]

│   │ ├── [email protected]

│   │ ├── [email protected]  deduped

│   │ ├── [email protected]

│   │ ├── [email protected]  deduped

│   │ ├── [email protected]

│   │ ├── [email protected]

│   │ └─┬ [email protected]

│   │   └── [email protected]

│   ├─┬ [email protected]

│   │ ├─┬ [email protected]

│   │ │ ├── [email protected]

│   │ │ ├── [email protected]

│   │ │ ├── [email protected]  deduped

│   │ │ └── [email protected]

│   │ └── [email protected]  deduped

│   ├─┬ [email protected]

│   │ └── [email protected]  deduped

│   └── [email protected]  deduped

├── [email protected]

├─┬ [email protected]

│ └── [email protected]

└── [email protected]

fair-analytics
/home/user/.nvm/versions/node/v6.10.3/lib/node_modules/fair-analytics/lib/server.js:17
  async function storeLog (rawData) {
        ^^^^^^^^
SyntaxError: Unexpected token function
    at createScript (vm.js:56:10)
    at Object.runInThisContext (vm.js:97:10)
    at Module._compile (module.js:542:28)
    at Module._extensions..js (module.js:579:10)
    at Object.require.extensions.(anonymous function) [as .js] (/home/ilyar/.nvm/versions/node/v6.10.3/lib/node_modules/fair-analytics/node_modules/async-to-gen/register.js:42:5)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)

Origin check for POST requests seems wrong

I noticed when setting up the origin configuration for the fair analytics server that the server side logic doesn't make much sense. It's not possible for the origin configuration to equal both the host and hostname at the same time, and it shouldn't equal either of those since the origin header actually needs to include the network protocol (http:// or https://).

I gave a go at fixing it in my fork and I could make a PR if you'd like: benwiley4000@15eb9b6

CORS for get endpoints

Unless I'm missing something, it seems like there's no supported way to access the endpoints like /_stats and /_live outside of the analytics domain in a browser. It seems relatively harmless for these to be access-control-allow-origin: *. However that should be different configuration than cors for post requests, since while it makes sense to make the auditing info available wherever, I'd likewise want to be able to restrict analytics post requests to a single domain.

I'd be happy to make a PR if we can agree on what the API would look like. What do you think? I'm wondering if it makes sense to add a new flag called endpointsOrigin which is the same as origin but only for the non-/ GET endpoints?

update to [email protected]

deployment instructions

• nginx
• dockerfile

Consider putting some screenshots

It's only fair for people to get a visual look at what they'll deploy. Please consider some screenshots?

Throws a Syntax Error during startup

Following the instructions for programmatic integration, using Node v4.8.2 on Windows.

import appRoot from 'app-root-path';
import fa from 'fair-analytics';

const server = fa({
  storageDirectory: `${appRoot}/.analytics`,
});

const { feed } = server;

feed.on('ready', () => {
  server.listen(8080);
});

W20170510-17:28:59.780(-4)? (STDERR) const { json, send } = micro
W20170510-17:28:59.781(-4)? (STDERR)       ^
W20170510-17:28:59.781(-4)? (STDERR)     -
W20170510-17:28:59.782(-4)? (STDERR) SyntaxError: Unexpected token {
W20170510-17:28:59.782(-4)? (STDERR)     at exports.runInThisContext (vm.js:53:16)
W20170510-17:28:59.782(-4)? (STDERR)     at Module._compile (module.js:373:25)
W20170510-17:28:59.782(-4)? (STDERR)     at Object.Module._extensions..js (module.js:416:10)
W20170510-17:28:59.783(-4)? (STDERR)     at Module.load (module.js:343:32)
W20170510-17:28:59.783(-4)? (STDERR)     at Function.Module._load (module.js:300:12)
W20170510-17:28:59.783(-4)? (STDERR)     at Module.require (module.js:353:17) 
W20170510-17:28:59.783(-4)? (STDERR)     at require (internal/module.js:12:17)
W20170510-17:28:59.784(-4)? (STDERR)     at Object.<anonymous> (.\node_modules\fair-analytics\lib\fair-analytics.js:2:22)
W20170510-17:28:59.784(-4)? (STDERR)     at Module._compile (module.js:409:26)ject metadata
W20170510-17:28:59.784(-4)? (STDERR)     at Object.Module._extensions..js (module.js:416:10)

Enable hyperdiscovery to use custom ports

You were added to awesome-humane-tech

This is just a FYI issue to notify that you were added to the curated awesome-humane-tech in the 'Analytics' category, and - if you like that - are now entitled to wear our badge:

By adding this to the README:

[![Awesome Humane Tech](https://raw.githubusercontent.com/humanetech-community/awesome-humane-tech/main/humane-tech-badge.svg?sanitize=true)](https://github.com/humanetech-community/awesome-humane-tech)

https://github.com/humanetech-community/awesome-humane-tech

Add/ignore favicon

Logs are a bit noisy because of the missing favicon route. It'd be nice to either have one or ignore that route:

/favicon.ico
Error: Not Found
    at createError (/usr/src/app/node_modules/fair-analytics/lib/server.js:9:15)
    at app.use (/usr/src/app/node_modules/fair-analytics/lib/server.js:83:11)
    at Layer.handle [as handle_request] (/usr/src/app/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/usr/src/app/node_modules/express/lib/router/index.js:317:13)
    at /usr/src/app/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/usr/src/app/node_modules/express/lib/router/index.js:335:12)
    at next (/usr/src/app/node_modules/express/lib/router/index.js:275:10)
    at textParser (/usr/src/app/node_modules/body-parser/lib/types/text.js:69:7)
    at Layer.handle [as handle_request] (/usr/src/app/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/usr/src/app/node_modules/express/lib/router/index.js:317:13)

use tape instead of ava

add tests for text events

Initial discussion

Open Web Traffic

This project is inspired by an awesome @staltz's article

Goal

Create an open and fair google analytics alternative.

How

My idea is to use hypercore to store events in a log, that is also shareable using feed.discoveryKey with other peers.

I imagine an event to be something like:

{
  type: 'pageVisit',
  when: '2017-04-26T08:43:00.088Z',
  path: '/open-analytics.html'
}

This event will be stored in the feed in its raw format.

Then

The hypercore's feed is just a raw data container.
We'd probably want to feed a database with incoming events in order to visualize them properly in a nice dashboard.

function processNewEntry (entry) {
  // store the entry in a database
}

feed.on('append', () => {
  feed.get(feed.length, processNewEntry)
})

Open points

This is an open discussion, and there are a lot of open points:

• how do we store information like unique page views without abusing user's privacy?
• same for average visit duration
• where do we store events for making them available to a dashboard? I was thinking about PouchDb. Perhaps better alternatives exist?
• If a website gets a fair amount of traffic, the hypercore's log might get quite big soon. I am wondering if storing events there is a viable approach.

Thanks to everyone willing to give their input/opinion/help here 😀

remove micro