francescostl/site-sonar-dashboard

A Dashboard + RESTful API for the Site Sonar web-extension.

site-sonar-dashboard's Issues

Change radio button to dropdown in /search

• Triage
• Debate

Site Crashes When DB is Empty or Down

2016-08-26T21:27:24.630802+00:00 app[web.1]:     process.nextTick(function() { throw err; });
2016-08-26T21:27:24.630798+00:00 app[web.1]: /app/node_modules/mongodb/lib/utils.js:98
2016-08-26T21:27:24.630810+00:00 app[web.1]:     at /app/node_modules/mongodb-core/lib/connection/pool.js:446:18
2016-08-26T21:27:24.630805+00:00 app[web.1]:     at /app/routes/index.js:50:58
2016-08-26T21:27:24.630803+00:00 app[web.1]:                                   ^
2016-08-26T21:27:24.630809+00:00 app[web.1]:     at /app/node_modules/mongodb-core/lib/cursor.js:579:16
2016-08-26T21:27:24.630804+00:00 app[web.1]: TypeError: Cannot read property 'count' of undefined
2016-08-26T21:27:24.630811+00:00 app[web.1]:     at authenticateStragglers (/app/node_modules/mongodb-core/lib/connection/pool.js:402:16)
2016-08-26T21:27:24.412850+00:00 app[web.1]: GET /dashboard 302 2.810 ms - 82
2016-08-26T21:27:24.630811+00:00 app[web.1]:     at null.messageHandler (/app/node_modules/mongodb-core/lib/connection/pool.js:424:5)
2016-08-26T21:27:24.630807+00:00 app[web.1]:     at handleCallback (/app/node_modules/mongodb-core/lib/cursor.js:156:5)
2016-08-26T21:27:24.630806+00:00 app[web.1]:     at handleCallback (/app/node_modules/mongodb/lib/utils.js:96:12)
2016-08-26T21:27:24.630806+00:00 app[web.1]:     at /app/node_modules/mongodb/lib/cursor.js:858:16
2016-08-26T21:27:24.630809+00:00 app[web.1]:     at Object.queryCallback [as cb] (/app/node_modules/mongodb-core/lib/cursor.js:217:18)
2016-08-26T21:27:24.630808+00:00 app[web.1]:     at setCursorNotified (/app/node_modules/mongodb-core/lib/cursor.js:506:3)
2016-08-26T21:27:24.652819+00:00 app[web.1]: 
2016-08-26T21:27:24.664484+00:00 app[web.1]: npm ERR! argv "/app/.heroku/node/bin/node" "/app/.heroku/node/bin/npm" "start"
2016-08-26T21:27:24.664745+00:00 app[web.1]: npm ERR! node v5.11.1
2016-08-26T21:27:24.665627+00:00 app[web.1]: npm ERR! code ELIFECYCLE
2016-08-26T21:27:24.666414+00:00 app[web.1]: npm ERR! Failed at the [email protected] start script 'node ./bin/www'.
2016-08-26T21:27:24.665842+00:00 app[web.1]: npm ERR! [email protected] start: `node ./bin/www`
2016-08-26T21:27:24.663990+00:00 app[web.1]: npm ERR! Linux 3.13.0-93-generic
2016-08-26T21:27:24.666022+00:00 app[web.1]: npm ERR! Exit status 1
2016-08-26T21:27:24.667682+00:00 app[web.1]: npm ERR! You can get information on how to open an issue for this project with:
2016-08-26T21:27:24.666214+00:00 app[web.1]: npm ERR! 
2016-08-26T21:27:24.666605+00:00 app[web.1]: npm ERR! Make sure you have the latest version of node.js and npm installed.
2016-08-26T21:27:24.667073+00:00 app[web.1]: npm ERR! not with npm itself.
2016-08-26T21:27:24.668273+00:00 app[web.1]: npm ERR!     npm owner ls site-SonarDashboard
2016-08-26T21:27:24.667906+00:00 app[web.1]: npm ERR!     npm bugs site-SonarDashboard
2016-08-26T21:27:24.667283+00:00 app[web.1]: npm ERR! Tell the author that this fails on your system:
2016-08-26T21:27:24.667479+00:00 app[web.1]: npm ERR!     node ./bin/www
2016-08-26T21:27:24.675090+00:00 app[web.1]: 
2016-08-26T21:27:24.668070+00:00 app[web.1]: npm ERR! Or if that isn't available, you can get their info via:
2016-08-26T21:27:24.666807+00:00 app[web.1]: npm ERR! If you do, this is most likely a problem with the site-SonarDashboard package,
2016-08-26T21:27:24.665266+00:00 app[web.1]: npm ERR! npm  v3.8.6
2016-08-26T21:27:24.675393+00:00 app[web.1]: npm ERR! Please include the following file with any support request:
2016-08-26T21:27:24.675611+00:00 app[web.1]: npm ERR!     /app/npm-debug.log
2016-08-26T21:27:24.668481+00:00 app[web.1]: npm ERR! There is likely additional logging output above.
2016-08-26T21:27:24.774622+00:00 heroku[web.1]: Process exited with status 1
2016-08-26T21:27:24.644253+00:00 heroku[router]: at=error code=H13 desc="Connection closed without response" method=GET path="/networksbyloadtime" host=www.site-sonar.com request_id=1144baef-65f8-463f-8c6e-dd9997c7b203 fwd="71.91.201.231" dyno=web.1 connect=1ms service=180ms status=503 bytes=0
2016-08-26T21:27:24.786858+00:00 heroku[web.1]: State changed from up to crashed
2016-08-26T21:27:24.787682+00:00 heroku[web.1]: State changed from crashed to starting
2016-08-26T21:27:26.038937+00:00 heroku[web.1]: Starting process with command `npm start`
2016-08-26T21:27:27.993223+00:00 app[web.1]: > [email protected] start /app
2016-08-26T21:27:27.993224+00:00 app[web.1]: > node ./bin/www
2016-08-26T21:27:27.993225+00:00 app[web.1]: 
2016-08-26T21:27:27.993207+00:00 app[web.1]: 
2016-08-26T21:27:28.475227+00:00 app[web.1]: Fri, 26 Aug 2016 21:27:28 GMT body-parser deprecated bodyParser: use individual json/urlencoded middlewares at app.js:24:9
2016-08-26T21:27:28.493602+00:00 app[web.1]: Fri, 26 Aug 2016 21:27:28 GMT body-parser deprecated undefined extended: provide extended option at node_modules/body-parser/index.js:105:29
2016-08-26T21:27:28.921920+00:00 heroku[web.1]: State changed from starting to up
2016-08-26T21:27:29.600325+00:00 app[web.1]: /app/node_modules/mongodb/lib/utils.js:98
2016-08-26T21:27:29.600334+00:00 app[web.1]:     process.nextTick(function() { throw err; });
2016-08-26T21:27:29.600334+00:00 app[web.1]:                                   ^
2016-08-26T21:27:29.600337+00:00 app[web.1]: 
2016-08-26T21:27:29.600338+00:00 app[web.1]:     at /app/routes/index.js:50:58
2016-08-26T21:27:29.600339+00:00 app[web.1]:     at /app/node_modules/mongodb/lib/cursor.js:858:16
2016-08-26T21:27:29.600339+00:00 app[web.1]:     at handleCallback (/app/node_modules/mongodb/lib/utils.js:96:12)
2016-08-26T21:27:29.600338+00:00 app[web.1]: TypeError: Cannot read property 'count' of undefined
2016-08-26T21:27:29.600340+00:00 app[web.1]:     at handleCallback (/app/node_modules/mongodb-core/lib/cursor.js:156:5)
2016-08-26T21:27:29.600340+00:00 app[web.1]:     at setCursorNotified (/app/node_modules/mongodb-core/lib/cursor.js:506:3)
2016-08-26T21:27:29.600341+00:00 app[web.1]:     at /app/node_modules/mongodb-core/lib/cursor.js:579:16
2016-08-26T21:27:29.600342+00:00 app[web.1]:     at Object.queryCallback [as cb] (/app/node_modules/mongodb-core/lib/cursor.js:217:18)
2016-08-26T21:27:29.600342+00:00 app[web.1]:     at /app/node_modules/mongodb-core/lib/connection/pool.js:446:18
2016-08-26T21:27:29.600343+00:00 app[web.1]:     at authenticateStragglers (/app/node_modules/mongodb-core/lib/connection/pool.js:402:16)
2016-08-26T21:27:29.600343+00:00 app[web.1]:     at null.messageHandler (/app/node_modules/mongodb-core/lib/connection/pool.js:424:5)
2016-08-26T21:27:29.610124+00:00 app[web.1]: 
2016-08-26T21:27:29.617662+00:00 app[web.1]: npm ERR! Linux 3.13.0-91-generic
2016-08-26T21:27:29.617981+00:00 app[web.1]: npm ERR! argv "/app/.heroku/node/bin/node" "/app/.heroku/node/bin/npm" "start"
2016-08-26T21:27:29.618181+00:00 app[web.1]: npm ERR! node v5.11.1
2016-08-26T21:27:29.618618+00:00 app[web.1]: npm ERR! npm  v3.8.6
2016-08-26T21:27:29.618871+00:00 app[web.1]: npm ERR! code ELIFECYCLE
2016-08-26T21:27:29.618997+00:00 app[web.1]: npm ERR! [email protected] start: `node ./bin/www`
2016-08-26T21:27:29.619110+00:00 app[web.1]: npm ERR! Exit status 1
2016-08-26T21:27:29.619274+00:00 app[web.1]: npm ERR! 
2016-08-26T21:27:29.619434+00:00 app[web.1]: npm ERR! Failed at the [email protected] start script 'node ./bin/www'.
2016-08-26T21:27:29.619603+00:00 app[web.1]: npm ERR! Make sure you have the latest version of node.js and npm installed.
2016-08-26T21:27:29.619760+00:00 app[web.1]: npm ERR! If you do, this is most likely a problem with the site-SonarDashboard package,
2016-08-26T21:27:29.619915+00:00 app[web.1]: npm ERR! not with npm itself.
2016-08-26T21:27:29.620070+00:00 app[web.1]: npm ERR! Tell the author that this fails on your system:
2016-08-26T21:27:29.620240+00:00 app[web.1]: npm ERR!     node ./bin/www
2016-08-26T21:27:29.620394+00:00 app[web.1]: npm ERR! You can get information on how to open an issue for this project with:
2016-08-26T21:27:29.620552+00:00 app[web.1]: npm ERR!     npm bugs site-SonarDashboard
2016-08-26T21:27:29.620706+00:00 app[web.1]: npm ERR! Or if that isn't available, you can get their info via:
2016-08-26T21:27:29.620861+00:00 app[web.1]: npm ERR!     npm owner ls site-SonarDashboard
2016-08-26T21:27:29.621015+00:00 app[web.1]: npm ERR! There is likely additional logging output above.
2016-08-26T21:27:29.624284+00:00 app[web.1]: 
2016-08-26T21:27:29.624499+00:00 app[web.1]: npm ERR! Please include the following file with any support request:
2016-08-26T21:27:29.624655+00:00 app[web.1]: npm ERR!     /app/npm-debug.log
2016-08-26T21:27:29.720569+00:00 heroku[web.1]: State changed from up to crashed
2016-08-26T21:27:29.604169+00:00 heroku[router]: at=error code=H13 desc="Connection closed without response" method=GET path="/networksbyloadtime" host=www.site-sonar.com request_id=fb311403-d8d1-4c26-9253-1f77c569e0eb fwd="71.91.201.231" dyno=web.1 connect=0ms service=224ms status=503 bytes=0
2016-08-26T21:27:29.679117+00:00 heroku[web.1]: Process exited with status 1
2016-08-26T21:27:30.005601+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/networksbyloadtime" host=www.site-sonar.com request_id=54044eb8-64b7-47e2-8fa0-3f7318d7e76d fwd="71.91.201.231" dyno= connect= service= status=503 bytes=
2016-08-26T21:27:32.724499+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=www.site-sonar.com request_id=45792c33-c1f2-4904-b922-13c912f52d2d fwd="71.91.201.231" dyno= connect= service= status=503 bytes=
2016-08-26T21:27:33.708415+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=www.site-sonar.com request_id=61b214f4-a977-4c12-a869-ff9f5b4e72e8 fwd="72.222.254.2" dyno= connect= service= status=503 bytes=
2016-08-26T21:27:34.667307+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=www.site-sonar.com request_id=81380d20-a7df-4b84-8982-e0bc95ac98f9 fwd="71.91.201.231" dyno= connect= service= status=503 bytes=
2016-08-26T21:27:36.086353+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=www.site-sonar.com request_id=4bccc3c2-a0a9-45d0-9fa5-fc0477b311f5 fwd="71.91.201.231" dyno= connect= service= status=503 bytes=
2016-08-26T21:27:36.380616+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=www.site-sonar.com request_id=ad5f23ee-59e8-4c58-a765-05baac723fdc fwd="71.91.201.231" dyno= connect= service= status=503 bytes=
2016-08-26T21:27:37.006309+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=www.site-sonar.com request_id=bfbaf21e-5da0-4243-abdd-e40f2e76a527 fwd="71.91.201.231" dyno= connect= service= status=503 bytes=
2016-08-26T21:28:11.020726+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=www.site-sonar.com request_id=2a45f0cb-5410-4659-a32f-b2feb9560720 fwd="71.91.201.231" dyno= connect= service= status=503 bytes=
2016-08-26T21:28:12.306674+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=www.site-sonar.com request_id=e2ec149d-003a-454e-81b4-ba59a7dd11e6 fwd="71.91.201.231" dyno= connect= service= status=503 bytes=
2016-08-26T21:28:16.426725+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=POST path="/log/" host=ultra-lightbeam.herokuapp.com request_id=00be13af-8c12-483a-bf40-9e95e59acd18 fwd="199.73.114.67" dyno= connect= service= status=503 bytes=

$limit instead of array.splice in all queries

Use mongo optimization of $sort | $limit instead of splice

Add contributing.md

I think we can have a helping hand in :

1. Front end UX/UI
2. Ad metrics knowledge
3. Porting addon to other browsers
   We can throw this out there for open source enthusiasts to help us out.

Remove Site Dashboards + Latest 100

We'd like to remove site dashboards and the lastest 100 results pages.

Data-vis Upgrade

This issue will be for tracking our dashboard data-vis upgrades.

Some ideas:

1. Grab top ~25 records rather than 100 so we can show data-vis in the top half of the dashboard
2. Distribution displayed as sparkline
3. Distribution displayed as ECDF's
4. Individual Ad-Network/Website metrics dashboards via click

Remove MongoDB Connection String

Convert mongodb connection string to a environmental variable on heroku side so that people don't try to connect and do bad things to it. Have localhost mongoDB for development/testing branches.

Counts on dashboard differ between networksbyfilesize and networksbyloadtime endpoints

SiteScout has count=5 on networks by load time site but =1 on networks by file size

Dashboard Upgrade

Its time for a dashboard upgrade. We need to determine what sort of data-vis will be most useful for this project, along with adding some styling on the dashboards because they look pretty bland atm.

Replace Origin w/ Host

We actually want to be aggregating by ad asset hostUrl rather than originUrl because the former is the true top level host while the latter can also be ad assets. See FrancescoSTL/Site-Sonar#35

Investigate Mongo+Heroku Scalability

In order to release this to the world, we probably want to make sure that our stack will scale. We should determine whether heroku+mongo is a scalable solution, and how we might scale most effectively when we release.

Individual Reports for Each Network/Site

We'd like to have individual reports for each network+website so that you can break down metrics further than what we're currently reporting.

Check for security issues

Run w3af.org

Refactor mongodb queries + defensive programming

Refactor :

1. MongoDB queries -> extract out common form, use var for networks, sites specific fields
2. Do defensive programming checks like:
3. check if collection exists
4. check if query was successful
5. check if array/fields for null, empty, type check

Signing data from users - stopping trolls

Related to #5

Dashboard slowing down

Investigate. Probably related to #57

Round load time more reasonably

Per @digitarald FrancescoSTL/Site-Sonar#45:

Right now it displays 3 digits (5083.666). One should enough and makes it easier to read.

Purush- nit fixes on dashboard

Dashboard Issues

1. On histograms - wording stuff
   1. counts -> assets benchmarked
   2. fileSizes -> file size
2. Network/site overview :
   1. Summary like dashboard home - centers and tiling
   2. site/network summary

      Project Wonderful network summary

      Google.com website summary

3. Overview page
   1. make numbers bigger and everything else sized accordingly
   2. increase Community Summary size
   3. margin between header and assets benchmarked
4. Footer - current page bold
5. Header - download -> drop down -> firefox, chrome links

Add Privacy Policy Page

We need to have the privacy policy on our website.

Can't Access Resources

Looks like there may have been some errors introduced in the last PR. We're getting Internal Server Error's (500 response) from various bootstrap assets.

Group ad load time average by ad networks

Average load times and group by ad networks in a separate route /groupBy

Adding distributions, counts to all 4 endpoints

Using this separate issue to work on adding distributions, counts

Add Clarification for Metrics Collected

We need to add clarification to what makes each metric important and what the metrics tell the users. Right now it isn't very explicit what the metrics mean or why they are relevant. This should remove any ambiguity in the data we're reporting.

Search bar for networks and sites

Adding a search bar for networks and sites.

Change "group by ad hosts" to Slowest Ads by Network

Ad host doesn't tell us much. Now that we're grabbing the ad-network, we should break the data down using that grouping.

Add queries for /networksbyfilesize endpoint

/networksbyfilesize is now empty. Needs filling with data ✨

Adding sort switch on tables

Making the current table display more interactive.

Implement SEO Best Practices

We need to provide meta tags on all our pages so we can get indexed. Also we should figure out what sort of data we need to provide to fb, twitter, etc to get nice meta description when we're linking inbound.

Rename Network Load Time to TTFB

I believe the measurement we're actually grabbing when we say "network response time" is TTFB or, the time it takes for the network to respond with the first byte of information back from the server.

So far as I understand it, TTFB can be calculated by taking (onHeaderSent time - onHeaderRecieved time), which is what we're currently doing. Is this correct @digitarald?

Make Dashboard Mobile Responsive

Currently, our dashboard doesn't display any menus when viewed on mobile.

Investigate -215 on facebook load time stat

Isolate dashboard from /log endpoints

Separate python flask REST API for /log and inserting into mongo

Move app to AWS

Infrastructure change - v2.0 possibly

Change plots to MetricsGraphics.js from Plot.ly

MetricsGraphics.js is a Mozilla library for plots using D3 JS. We should transition to that post v1.0 release

Implement Pagination

If we can page our results, we don't have to restrict to top n results

Moving MongoDB to AWS EC2

#30

Fix outliers in networks/sites individual pages

Creating this as separate issue to have different PRs for the two issues.

Show distribution as standard deviation (not range)

Per @digitarald: From a statistical perspective, SD is easier to understand and apply to the average that also gets shown.

Sort by "Best First" for Each Dashboard

So we'd like to sort by best results first (inverse of what we currently have) so that there isn't a negative vibe with the project.

Update Homepage

Create download buttons for Chrome and Firefox rather than just Firefox.

Also we need to be more descriptive as to what is happening and clarify what Site-Sonar is.

Add Main Dashboard Page

We'd like a dashboard overview page with similar numbers to the overview in the add-on:

1. Cumulative network time
2. Sum of all file sizes
3. Count of all ad-assets benchmarked

Use light-server to publish dashboard changes instead of endpoints to run aggregate queries

• Move aggregate queries to batch jobs
• Have light-server read json files and publish to dashboard on file update
• Dashboards update every few minutes/hours based on traffic from clients

Data validation -Stopping trolls at the gate

We have to build a wall and have the trolls pay for it. Schema validation! Voila!

Add queries for "/sitesbyfilesize" endpoint

/recordsbyfilesize needs data ✨

Adding new relic APM on heroku

Relevant results on each n/w sites by ttfb/file size

Cut down results by a median/average asset count.