gdg-x/hub

[DEPRECATED] API Data Hub for the Global GDG Community

hub's Issues

Create an endpoint to get user's chosen home chapter.

Currently we have a PUT request to /frisbee/user/home which stores users home chapter. I doesn't actually work. Even if it works, we currently don't use it.

• First we should remove frisbee name from the endpoint.
• Second, we should have a GET request to read the value.

Add web page with map of all devfests

This is part of the migration from developers.google.com

Enable SPDY

Add support for drive folder of event series

Organizers should be able to add organizers to a drive folder by selecting an event tag.

All organizers of a chapter that is hosting an event with the given tag are automatically added to the selected folder.

Steps:

1. text input for tag
2. pick drive folder (https://developers.google.com/picker/docs/)
3. get events with tags (https://hub.gdgx.io/api/v1/events/tag/:tagname)
4. for each event get chapter
5. for each chapter get organizers (Chapter.organizers)
6. for each organizer add permission (https://developers.google.com/drive/v2/reference/permissions/insert)
7. As a result list number of users that have permission of this folder an a link to the folder

Full day event missing from GDG-X hub

WTM Istanbul 2015 event that will occur on March 15th is missing from the GDG-X Hub and it is not showing on GDG Android App.

The tags seem to be OK.

Event is shown on developers.google.com calendars
https://developers.google.com/events/4580370115723264/

but not on the GDG-X Hub API and calendars.

How do I use the /events/:start/:end API?

http://hub.gdgx.io/api/v1/chapters/103959793061819610212/events/1384021175/1415557175

The above is trying to get the events in the last year, but it is returning 0 items.

Hub API Analytics is no longer recording usage

Prior to September, the Google Analytics for the Hub was very active with around 4-6k hits per month. A lot of the other metadata that you would get with website hits was not there, but we could at least see general API usage numbers.

We should get this working again. Unfortunately, there was no documentation left behind about how this was configured.

hub.gdgx.io appears to be down

https://hub.gdgx.io/
http://hub.gdgx.io/developers/api

Unknown Host
Description: Unable to locate the server named "hub.gdgx.io" --- the server does not have a DNS entry. Perhaps there is a misspelling in the server name, or the server no longer exists. Double-check the name and try again.

Resource leak that brings down the Hub

Last night, the Hub stopped responding to all requests. Antonio Bertucci let us know about it in the Gitter chat about 5 hours ago.

When I logged into the console for the Hub and attached to the tmux session, I found the following issue:
nodejs/node-v0.x-archive#9409

There is a work around in that thread, but I haven't had time to try it. Also we've only seen this issue once, so it may be a fluke. If it happens again, and the work around fixes it, we can be a bit more confident.

There is a PR for the issue but it was never merged and now appears to be targetted at an abandoned repo of Node.js.

Update SSL Certs for 2015

Get new Certs from StartSSL and install them via the OpenShift Console

Events endpoint returns unsorted list of events.

In Frisbee, we realized that the events endpoint returns the list unsorted. I think it is the order that they are stored in the db.

For example in this endpoint, you can see that the location of the 6th entry is wrong.

https://hub.gdgx.io/api/v1/chapters/115635340252753847571/events/2016-03-01T13:05:41.526+01:00/2017-05-31T13:05:41.526+02:00

If Redis can't be contacted, the Hub stops serving content and goes down

Redis disconnect issues appear to have taken down the hub at 6:56pm ET today:

[30384] Express server listening on port 3000 in production mode
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: Redis connection to 130.211.142.195:6379 failed - connect ECONNREFUSED
    at RedisClient.on_error (/opt/hub/node_modules/redis/index.js:196:24)
    at Socket.<anonymous> (/opt/hub/node_modules/redis/index.js:106:14)
    at Socket.emit (events.js:107:17)
    at net.js:459:14
    at process._tickCallback (node.js:355:11)
[17884] worker 30384 died. Code: 1, Signal: null
[17884] worker 30389 born.

Use npm-scripts to make onboarding easier

• Use a postinstall script to install grunt-cli and bower.
• Add a prestart script to build before serving.
• Update README to be more simple.

Add API for list of active tags

Should return a list of tags similar to /tags with values of number of upcoming events using this tag.

For API call
https://hub.gdgx.io/api/v1/tags/active?callback=JSON_CALLBACK

Space between paragraphs are huge.

The space between paragraphs seems to be lot bigger than the ones in Google+ and Google Developers page for the events. Here is one screenshot.

date/time formatting issue

Date and time should be displayed with proper formatting. https://hub.gdgx.io/statistics

FYI, Currently it is getting displayed as 2015-11-01T11:00:00.000Z

Include time span on returned list of events

When querying events, like https://hub.gdgx.io/api/v1/events/tag/globalandroiddevcamp the response should contain start and duration of the time span covered by the request

Deploy Hub to GCE instead of OpenShift

A lot of this discussion has already happened on Gitter.

Mongo Cluster (to be shared with Firefly)

• Arbiter Node
  • f1-micro (1 vCPU, 0.6 GB memory)
• DB Node x 2
  • n1-highmem-4 (4 vCPUs, 26 GB memory)
  • 100 GB SSD data disk.

Hub API VM

• n1-highcpu-2 (2 vCPUs, 1.8 GB memory)

Redis Cluster x 3

• n1-standard-1 (1 vCPU, 3.75 GB memory)

/chapters/:chapterId/events/upcoming API is listing deleted events

This is the event list in the Developers site for my chapter: https://developers.google.com/groups/chapter/117368911868983524073/

This is the event list in the Hub for my chapter:
https://hub.gdgx.io/chapters/117368911868983524073

Notice that https://hub.gdgx.io/events/5064898009104384 (GameFest) is a deleted event, which does not show in the Dev site.

We're using this API for our community event listing as well: http://events.sabah.io/ and this same issue appears.

Time formatting issue in events calendar

On events calendar page https://hub.gdgx.io/events, exact hours are not getting displayed as 12:00 but only 12.

Migrate static event type description/color metadata from OpenShift Hub deployment

As you can see from the above screenshot, the new Hub does not have a lot of the static data that it needs. This includes the colors of different event tags, geo location field on events, description text about what a devfest, android, or other event type.

This data needs to be migrated over to the new Hub. Is the only/best way of doing this via POST requests or some kind of Mongo tool like Robomongo? Other ideas?

Update: The geocoder / map marker issue is open in a separate issue here: #47

VM startup script not working

This script should run the web server on VM start so that login and manual execution is not required. This isn't currently working due to the following errors:

Feb 14 16:12:04 hub-backup startupscript: Running startup script /var/run/google.startup.script
Feb 14 16:12:06 hub-backup startupscript: 
Feb 14 16:12:06 hub-backup startupscript: > [email protected] configProd /opt/hub
Feb 14 16:12:06 hub-backup startupscript: > sudo iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 3000;export PORT=3000;
Feb 14 16:12:06 hub-backup startupscript: 
Feb 14 16:12:06 hub-backup kernel: [   16.219301] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
Feb 14 16:12:07 hub-backup startupscript: 
Feb 14 16:12:07 hub-backup startupscript: > [email protected] startProd /opt/hub
Feb 14 16:12:07 hub-backup startupscript: > grunt serve:dist
Feb 14 16:12:07 hub-backup startupscript: 
Feb 14 16:12:09 hub-backup ntpdate[709]: adjust time server x.x.x.x offset 0.399524 sec
Feb 14 16:12:09 hub-backup systemd[1]: Starting LSB: Start NTP daemon...
Feb 14 16:12:09 hub-backup ntp[1051]: * Starting NTP server ntpd
Feb 14 16:12:09 hub-backup ntpd[1059]: ntpd [email protected] Wed Nov 11 18:20:12 UTC 2015 (1)
Feb 14 16:12:09 hub-backup ntp[1051]: ...done.
Feb 14 16:12:09 hub-backup systemd[1]: Started LSB: Start NTP daemon.
Feb 14 16:12:09 hub-backup ntpd[1060]: proto: precision = 0.109 usec
Feb 14 16:12:09 hub-backup ntpd[1060]: ntp_io: estimated max descriptors: 1024, initial socket boundary: 16
Feb 14 16:12:09 hub-backup ntpd[1060]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
Feb 14 16:12:09 hub-backup ntpd[1060]: Listen and drop on 1 v6wildcard :: UDP 123
Feb 14 16:12:09 hub-backup ntpd[1060]: Listen normally on 2 lo 127.0.0.1 UDP 123
Feb 14 16:12:09 hub-backup ntpd[1060]: Listen normally on 3 eth0 10.0.0.2 UDP 123
Feb 14 16:12:09 hub-backup ntpd[1060]: peers refreshed
Feb 14 16:12:09 hub-backup ntpd[1060]: Listening on routing socket on fd #20 for interface updates

Ubuntu 15.04 hub-backup ttyS0

hub-backup login: Feb 14 16:12:11 hub-backup startupscript: Loading "cdnify.js" tasks...#033[31mERROR#033[39m
Feb 14 16:12:11 hub-backup startupscript: #033[31m>> #033[39mError: Unable to parse global .bowerrc file: Arguments to path.join must be strings
Feb 14 16:12:12 hub-backup startupscript: 
Feb 14 16:12:12 hub-backup startupscript: #033[4mRunning "serve:dist" (serve) task#033[24m
Feb 14 16:12:12 hub-backup startupscript: #033[33mWarning: Task "cdnify" not found.#007 Use --force to continue.#033[39m
Feb 14 16:12:12 hub-backup startupscript: 
Feb 14 16:12:12 hub-backup startupscript: #033[31mAborted due to warnings.#033[39m
Feb 14 16:12:12 hub-backup startupscript: 
Feb 14 16:12:12 hub-backup startupscript: 
Feb 14 16:12:12 hub-backup startupscript: Execution Time (2016-02-14 16:12:12 UTC)
Feb 14 16:12:12 hub-backup startupscript: loading tasks  42ms  ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 88%
Feb 14 16:12:12 hub-backup startupscript: serve:dist      2ms  ▇▇ 4%
Feb 14 16:12:12 hub-backup startupscript: build           2ms  ▇▇ 4%
Feb 14 16:12:12 hub-backup startupscript: Total 48ms
Feb 14 16:12:12 hub-backup startupscript: 
Feb 14 16:12:12 hub-backup startupscript: npm ERR! Linux 3.19.0-49-generic
Feb 14 16:12:12 hub-backup startupscript: npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "runScript" "startProd"
Feb 14 16:12:12 hub-backup startupscript: npm ERR! node v0.12.10
Feb 14 16:12:12 hub-backup startupscript: npm ERR! npm  v2.14.9
Feb 14 16:12:12 hub-backup startupscript: npm ERR! code ELIFECYCLE
Feb 14 16:12:12 hub-backup startupscript: npm ERR! [email protected] startProd: `grunt serve:dist`
Feb 14 16:12:12 hub-backup startupscript: npm ERR! Exit status 3
Feb 14 16:12:12 hub-backup startupscript: npm ERR!
Feb 14 16:12:12 hub-backup startupscript: npm ERR! Failed at the [email protected] startProd script 'grunt serve:dist'.
Feb 14 16:12:12 hub-backup startupscript: npm ERR! This is most likely a problem with the gdgx-hub package,
Feb 14 16:12:12 hub-backup startupscript: npm ERR! not with npm itself.
Feb 14 16:12:12 hub-backup startupscript: npm ERR! Tell the author that this fails on your system:
Feb 14 16:12:12 hub-backup startupscript: npm ERR!     grunt serve:dist
Feb 14 16:12:12 hub-backup startupscript: npm ERR! You can get their info via:
Feb 14 16:12:12 hub-backup startupscript: npm ERR!     npm owner ls gdgx-hub
Feb 14 16:12:12 hub-backup startupscript: npm ERR! There is likely additional logging output above.
Feb 14 16:12:12 hub-backup startupscript: 
Feb 14 16:12:12 hub-backup startupscript: npm ERR! Please include the following file with any support request:
Feb 14 16:12:12 hub-backup startupscript: npm ERR!     /opt/hub/npm-debug.log
Feb 14 16:12:12 hub-backup startupscript: /tmp/tmp.uBOvwuQDBp: line 5: EOF: command not found
Feb 14 16:12:12 hub-backup startupscript: Finished running startup script /var/run/google.startup.script
Feb 14 16:12:12 hub-backup systemd[1]: Started Google Compute Engine user scripts.

Configure an uptime monitoring tool like StatusCake

StatusCake

See an example here.

Remove deprecated open

Replace with opn.

Out Of Memory errors

There are a number of reasons that the Hub stops responding to requests (OOM, exceptions, hangs, etc). The goal isn't to solve all of these problems because there will likely be more introduced in the future via open source contributions and limited automated testing.

We need to setup a system to improve the Hub's HA, ideally via pm2 and possibly other packages. This is a common problem with Node.js projects and there are many examples and guides for handling this. We just need someone to set it up, test it, and finally work with me on deployment.

---

On Oct 6th at 6am the Hub started responding to all requests with a 502 error and the console just logged the request and timed out processing it at 2 seconds. This appears to be different than the previous issue with a resource leak which left a clear exception.

I've restarted the Hub and it's back online.

We need to spin up another Hub node and connect it to the load balancer so that if one goes down, we don't loose service. Then we probably also need to enable Stackdriver Monitoring and alerts so that we get emailed when the Health Checks fail for a node under the load balancer. We currently get no such notification.

Redis suddenly failing Auth

Now Redis is failing for another reason. It was working just fine for a couple of days, nothing changed... and then this happened...

/opt/hub/node_modules/redis/index.js:575
                throw callback_err;
                      ^
Error: Ready check failed: NOAUTH Authentication required.
    at RedisClient.on_info_cmd (/opt/hub/node_modules/redis/index.js:380:35)
    at Command.callback (/opt/hub/node_modules/redis/index.js:430:14)
    at RedisClient.return_error (/opt/hub/node_modules/redis/index.js:571:25)
    at ReplyParser.<anonymous> (/opt/hub/node_modules/redis/index.js:317:18)
    at ReplyParser.emit (events.js:107:17)
    at ReplyParser.send_error (/opt/hub/node_modules/redis/lib/parser/javascript.js:296:10)
    at ReplyParser.execute (/opt/hub/node_modules/redis/lib/parser/javascript.js:181:22)
    at RedisClient.on_data (/opt/hub/node_modules/redis/index.js:547:27)
    at Socket.<anonymous> (/opt/hub/node_modules/redis/index.js:102:14)
    at Socket.emit (events.js:107:17)

Refactor config to read from a local properties file that is not source controlled

This is similar to what Frisbee and Firefly do now instead of depending on a wide range of environment variables to be set properly.

This work is needed as part of the new Hub deployment off of OpenShift.

Chapters page not loading in hub

https://hub.gdgx.io/chapters does not load

TypeError: Cannot read property 'name' of null
at https://hub.gdgx.io/scripts/ba177e92.scripts.js:5:26288
at https://hub.gdgx.io/scripts/3091e993.vendor.js:4:26843
at j (https://hub.gdgx.io/scripts/3091e993.vendor.js:5:8568)
at j (https://hub.gdgx.io/scripts/3091e993.vendor.js:5:8568)
at https://hub.gdgx.io/scripts/3091e993.vendor.js:5:9245
at k.$eval (https://hub.gdgx.io/scripts/3091e993.vendor.js:5:13454)
at k.$digest (https://hub.gdgx.io/scripts/3091e993.vendor.js:5:12158)
at k.$apply (https://hub.gdgx.io/scripts/3091e993.vendor.js:5:13773)
at f (https://hub.gdgx.io/scripts/3091e993.vendor.js:4:27269)
at r (https://hub.gdgx.io/scripts/3091e993.vendor.js:4:29093)

Functionality request: Organizers for particular Chapter

A GET request for organizers for specific Chapter.
Also if possible add organizers IDs (G+) to event info.

Country page is broken

What is the schedule for Hub data updates? And Attendance Google Sheet

It appears that the gdg-x hub data is not updated with recent data.

(For the attendance google sheet see https://j.mp/gdgattendance and below.)

Do we know what the update schedule is? Does it read from developers.google.com once a day?

Here's an example:

$ curl "https://hub.gdgx.io/api/v1/events/5307708566667264" | json_pp
 {
   "iconUrl" : "/_static/images/gdg-icon.png",
   "__v" : 3,
   "title" : "IWD 2015: Women Techmakers GDG New York",
   "_id" : "5307708566667264",
   "allDay" : false,
   "location" : "New York, NY, United States",
   "geo" : {
      "lng" : -74.0059413,
      "lat" : 40.7127837
   },
   "about" : "Placeholder for a Women TechMakers event organized by GDG New York. The date/time and venue are TBD but we will update this event as things become more concrete.",
   "updated_at" : "2015-02-08T05:00:44.037Z",
   "created_at" : "2015-02-08T05:00:42.282Z",
   "end" : "2015-03-22T21:30:00.000Z",
   "timezone" : "America/New_York",
   "tags" : [
      "iwd",
      "wtm",
      "gdgwomen"
   ],
   "start" : "2015-03-22T13:30:00.000Z",
   "chapter" : "102033742326416578905"
}

You can see there is no value for "participants".

But dev site (https://developers.google.com) has a value and it was set around March 22, 2015 (it appears)

See

% curl   "https://developers.google.com/events/feed/json?group=102033742326416578905&start=0&end=1846075200" | json_pp
...   
{
      "end" : "22 Mar 2015 17:30 -0400",
      "id" : "5307708566667264",
      "start" : "22 Mar 2015 09:30 -0400",
      "participantsCount" : 101,
      "timezoneName" : "America/New_York",
      "group" : "102033742326416578905",
      "percentWomen" : 80,
      "location" : "New York, NY, United States",
      "link" : "/events/5307708566667264/",
      "description" : "Placeholder for a Women TechMakers event organized by GDG New York. The date/time and venue are TBD but we will update this event as things become more concrete.",
      "temporalRelation" : "past",
      "iconUrl" : "/_static/images/gdg-icon.png",
      "title" : "IWD 2015: Women Techmakers GDG New York"
   },
...

Dev site has a value for "participantsCount" : 101

But the gdg-x hub does not.

It appears that the gdg-x hub is not updating.

The reason I noticed this is because I am generating a report for missing attendance numbers for all GDGs and here is the report: https://j.mp/gdgattendance

The sheet is very simple, it shows every GDG event throughout the world and the attendance or the lack of attendance numbers. Plus a link to update the attendance numbers.

How do I ask the server to send me the next page or page 2?

http://hub.gdgx.io/api/v1/chapters/103959793061819610212/events/past

This returns 34 items, 25 per page. Thus I'm only getting 25 items as I don't know the API (query param?) to use for asking for page 2 of the results. I looked in the Wiki and on the API reference page but didn't see any information about paging.

Hub GeoCoder not working

Ingestion is not geo tagging chapters or events. This is handled in https://github.com/gdg-x/hub/blob/develop/lib/clients/devsite.js#L9

Remind Organizers to enter their Attendee stats

Notify Organizers if they haven't provided Attendee stats for an Event that happened more than 2 weeks ago.

DevFest Western Kenya 2015 not showing on Hub feed

The DevFest Western Kenya is registered on DevSite, as you can see.
But, for some reason, the Hub is not indexing and showing on the feed.

Document API features

The docs are helpful, but they are missing a lot of information.

Add information on

• sorting (field to sort and asc (number?))
• fields to return in response
• perpage limit
• page requested

Chapters should be displayed in alphabetical order

Chapters list (for the particular country) on the https://hub.gdgx.io/chapters should be displayed in alphabetical order.

Remove deprecated ngmin

Replace with ng-annotate.

Events API description/about field parsing

It looks like the API is doubling my <br/> tags by adding a <br /> tag for each of them. This leads to my formatting being messed up on the boomerang events page.

See the output below. Every <br/> in the data has an additional <br /> added to it.

"about": "Tech Talk: Testing your AngularJS application with Karma and Jasmine<br/><br />Code Lab: Go: Build a RESTful Back-end on App Engine (60-90 mins)<br /><br/><br/><br />Using Go on App Engine, you will create a fully working todo list manager. The provided front-end is implemented using AngularJS and Restangular, and connects to the back-end via a REST API that you will implement.<br /><br/><br/><br />#angularjs #golang #googlecloudplatform #appengine #karma #jasmine",

Also was there a reason to change this field from description to about?

Filter on multiple countries

https://github.com/gdg-x/hub/blob/master/lib/controllers/api/v1/chapters.js

Would be nice to be able to filter on multiple countries by passing CSV
example : /chapters/country/us,ca

Improve README or CONTRIBUTING docs for new contributors

Add a link to MongoDB installation instructions.
Add details about API keys.
Add information about how to run the test server locally.
Add details on how to run the unit tests.

Content-Encoding UTF-8 for all responses

There seems to be a bug in the Cache Layer preventing the delivery of proper Content Encoding headers when loaded from Cache.

Add API and UI for admins to define special event tag for Frisbee

Admins of Hub should be able to define

• tag
• title + icon (in navigation drawer)
• tag line/short description (above list of events)
• icon for action bar

A simple API should return these values

Add API for event series

Frisbee uses a list of event series. These should be maintained on the hub.

See also gdg-x/frisbee#256

Add support for Gitter Sidecar

More details here: https://sidecar.gitter.im/

This would be great since the Hub is a developer focused site. We should make it easy for developers to chat with us.

Add API for chapters to manage 'apps influenced'

One of the new pushes for GDG in 2015 is influencing real published apps (web, Chrome, Android, etc). We should make it easy for logged in members of the chapters and organizers to add those links to the database. Then it should be possible for Boomerang to query that API and display a new page listing apps influenced/created by that chapter.

Of course the app listing would need to include at least a name, date submitted, date modified, and URL. Then it could possibly include some kind of small image/icon.

Upcoming events API doesn't list all upcoming events

/chapters/:chapterId/events/upcoming
Returns a list of all upcoming Events scheduled by the specified Chapter

http://hub.gdgx.io/api/v1/chapters/103959793061819610212/events/upcoming

Should show me the August and October event, but only lists the August event. Ideally it should show events up to 6 months in advance at least (if it can't do 'all upcoming' like it says).

Remove deprecated express-annotations library

This is blocking us from upgrading to Express 4.x.

Content covers the navigation on mobile.

Here is how it looks like. I hope I don't open a duplicate issue.

Route root paths to documentation

When visiting URLs like
https://hub.gdgx.io/api/ or
https://hub.gdgx.io/api/v1/

Redirect or route to the docs
https://hub.gdgx.io/developers/api

Or improve 404 document to include link to documentation