mozilla/status.persona.org

an experiment

Status: Inactive

This repository is no longer active, because the Persona service has been shut down. It remains only for historical purposes.

---

Realtime Service Status for Mozilla Persona

This repository holds the status reporting system for Mozilla Persona to be hosted at http://status.persona.org. People who care about Persona can check the status of the service at that url. People working on persona can report incidents by adding files to this repository.

How it works

events/ holds a collection of incident reports. An incident report is a directory, named by convention YYYY.MM.DD-#, containing the following files:

• discovery.txt - A textual report of a discovered problem
• update_X.txt - any number (zero to infinity) of incremental status updates
• resolution.txt - A textual report of the resolution of the problem.

These files have a special first line containing the date of the report, and the rest of the file is plain text prose. For example:

when: Tue Sep 25 22:13:19 PDT 2012

We've isolated the problem, it's a bogus flux capacitor.  We're going to try to stop
in the Aquila System to buy one from the semi-hostile Midorians there.

NOTE: Date parsing is pretty flexible. When an event occurs, type date at your terminal, and then embed this in your report.

NOTE: The .txt suffixes are totally optional.

NOTE: you can name updates whatever you want. The only hardcoded file names are discovery(.txt)? and resolution(.txt)?.

How you report an incident

1. create a new directory for it in events/

2. write a discovery.txt file

3. commit it

4. push it up to github

5. build and publish the update to gh-pages:

    scripts/build-gh-pages.sh
    scripts/publish-gh-pages.sh
   

How you resolve an incident

1. write a resolution.txt file in the directory

2. commit it

3. push it up to github

4. build and publish the update to gh-pages:

    scripts/build-gh-pages.sh
    scripts/publish-gh-pages.sh
   

Some details

Scripts in scripts/ are responsible for validating the pile of files in events/, turning them into JSON that the webpage (under html/ can render)

hacking on it

1. clone it
2. npm install
3. mkdir html/data
4. scripts/to_json.js > html/data/1.json
5. scripts/run.js
6. open http://127.0.0.1:8080/
7. hack on html/*

status.persona.org's Issues

Include timezone in the HH:MM of each update

I haven't looked at the code and perhaps the front end is showing it in localtime. But even so, it should name the TZ for e.g., 2:19 PM.

Allow for categorizing events and visualizing the differing uptimes for each category

Previously personastatus.org shows a single percentage uptime based on the number of hours events took up. This ended up representing downtime much higher than was realistic. Many of the events identified on personastatus.org are not outages but either partial outages, or maintenance windows.

I'd like to enable adding a category field to events. Currently discovery, update and resolution files have two fields.

Format

The first line is the date for the record, followed by the message ( here's how this is parsed currenlty )

I propose updating the file spec to be

• Line 1 : <Date>
• Line 2 :<Category> <Percentage>
• Remaining lines : <The Message>

It would look like

when: Sat Jul 5 19:00:00 PDT 2014
category: db-write 50%

Our Phoenix datacenter is inaccessible resulting in writes to Persona being blocked. New user account signups and password changes are not working.

This could either be done as it is now, ignoring what comes before the : character and just using the position to identify if it's date, category or message, or you could parse the content before the :

We can also either require a category line in the events (in which case I can go through and update all the events with a category line), or we can, in the absence of a category line, default to the site-offline category and 100%

Categories

I propose the following categories (which we may want to add to later)

Code	Name	Description
site-offline	Entire Site	The entire Persona site is down preventing new user signups, password changes, logins for users regardless of email address and browsing to the https://login.persona.org/ website
db-write	New user signups and password changes	New users are unable to create Persona accounts and existing users are unable to change their Persona password
db-sync	New user logins	New users who have recently created an account or existing users who have recently changed their password, are not be able to login
bridge-gmail	gmail.com user logins	Users with @gmail.com email addresses are unable to login
bridge-yahoo	yahoo.com user logins	Users with @yahoo.com email addresses are unable to login
primary-logins	Other domain user logins	Users who login to their email provider in order to access Persona are unable to login
email	Email delivery	Email delivery to users from Persona is delayed preventing new users from verifying their email addresses and existing users who have forgotten their password from resetting it

If it's not too hard it would be nice to allow an event to be associated with multiple categories. This could either be accomplished by parsing the header of each event file for all lines beginning with category : or you could take the single category line and load it with multiple category assertions like category : bridge-gmail 100% bridge-yahoo 100% primary-logins 100%

Visualization

Finally, I'd like to add to the top of the page 2 new visualization elements

Table

I'd like to put up a table at the top of the page, reminiscent of the Google Apps Status Dashboard. The X axis would be time, showing the last 7 days. The Y axis would be the Name field of the categories defined above. Mousing over the Name field would tooltip the Description (or some other method to reveal the more detailed Description

Each cell in the table would represent if there had been an event during that date of that given category, maybe coloring the cell red or green (or a red or green icon)

Uptime Percentage

I'd like to represent the percentage uptime for each category as well. I think this should be the uptime over the last month. To calculate the percentage uptime over a given window of time (e.g. the last month) for a given category :

For each event assigned to the given category, take the duration of the event and multiply it by the percentage field following the category code in the event file.
Add up these resulting durations and divide by the duration of the time window (e.g. one month)

You could put each of these uptimes in the same table as described above or call them out separately.

Above all of this should be the uptime percentage for the site-offline category as this is the most important number.

Here's the old uptime parsing method

site is not served over HTTPS

I cannot visit https://personastatus.org. :(

Update persona status site automatically

Either need to poll for updates or listen for a githook.

Otherwise, on-call folks won't be able to update the status page in real time.

Get rid of the (inaccurate) scary low uptime percentage

As @gene1wood has pointed out, this number is calculated in such a way that events that do not diminish uptime are actually tallied as though they did.

This could easily cause people curious about Persona to come to the conclusion that it is unreliable.

Dependency tree changed breaking personastatus due to a need for node >= 0.8.0

Some point since 4/16 the dependency tree changed for personastatus and connect version 2.7.9 is now being loaded which depends on a newer node version

remote: npm ERR! Unsupported
remote: npm ERR! Not compatible with your version of node/npm: [email protected]
remote: npm ERR! Required: {"node":">= 0.8.0"}
remote: npm ERR! Actual:   {"npm":"1.1.21","node":"0.6.17"}

Either assert an older version of whatever it is that depends on connect or upgrade node.