mozilla-services/autopush-rs

Push Server in Rust

Autopush-rs

Mozilla Push server built with Rust.

This is the fourth generation of the Mozilla Web Push server. It currently supports websocket connections and support for Megaphone broadcast.

Please consult the autopush documentation for information about how this server works, as well as any error messages you may see when sending push messages to our server.

MDN has information about how to use WebPush.

Note while rust-doc style comments are used prolifically through the source, only public structures are rendered automatically. For those curious about the inner workings, You may wish to read the code files directly.

Debugging on Mobile

Mobile devices can specify the Push Server URL via the "secret settings".

Do not use the mobile about:config menu settings. These are not read or used by the mobile browser.

The secret settings can be activatedby following these instructions. Once the secret menu is active, select Sync Debug from the the mobile Settings menu, and specify the Custom Push server URL.

NOTE: the default Push server url is wss://push.services.mozilla.com/

autopush-rs's Issues

1.54.3 docker fails with `OPENSSL_1_1_1' not found

I can't reproduce this w/ 1.54.2, yet it happens w/ 1.54.3. Surprisingly enough, not a lot changed in between these versions:

1.54.2...1.54.3

$ docker run --rm mozilla/autopushrs:1.54.3
/app/bin/autopush_rs: /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1: version `OPENSSL_1_1_1' not found (required by /app/bin/autopush_rs)

Update dependencies [Q1]

howto: mozilla-services/autopush#1170 (comment)

Likely avoiding the hyper 0.12 upgrade (again)

cleanup megaphone terminology to match spec doc

basically "Service" -> "Broadcast"

Investigate Push Mega-logging issue

Digging a bit further into the logging event from 9/17.

Setup a cadence error_handler

We now utilize Cadence's new quiet send method, so we should hook up the associated error handler to probably log a debug! message for when sends fail:

https://docs.rs/cadence/0.14.0/cadence/#quiet-metric-sending-and-error-handling

Fix Python integration test

Looks like its py.test dependency need updating, as seen on travis:

$ py.test -v
Traceback (most recent call last):
  File "/home/travis/virtualenv/python2.7.14/bin/py.test", line 11, in <module>
    sys.exit(main())
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/_pytest/config.py", line 50, in main
    config = _prepareconfig(args, plugins)
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/_pytest/config.py", line 160, in _prepareconfig
    pluginmanager=pluginmanager, args=args)
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/pluggy/__init__.py", line 617, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/pluggy/__init__.py", line 222, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/pluggy/__init__.py", line 216, in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/pluggy/callers.py", line 196, in _multicall
    gen.send(outcome)
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/_pytest/helpconfig.py", line 68, in pytest_cmdline_parse
    config = outcome.get_result()
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/pluggy/callers.py", line 77, in get_result
    _reraise(*ex)  # noqa
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/pluggy/callers.py", line 180, in _multicall
    res = hook_impl.function(*args)
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/_pytest/config.py", line 942, in pytest_cmdline_parse
    self.parse(args)
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/_pytest/config.py", line 1107, in parse
    self._preparse(args, addopts=addopts)
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/_pytest/config.py", line 1070, in _preparse
    self.pluginmanager.load_setuptools_entrypoints('pytest11')
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/pluggy/__init__.py", line 402, in load_setuptools_entrypoints
    "Plugin %r could not be loaded: %s!" % (ep.name, e))
pluggy.PluginValidationError: Plugin 'pytest_cov' could not be loaded: (pytest 3.3.0 (/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages), Requirement.parse('pytest>=3.6'))!
The command "py.test -v" exited with 1.

Error: unknown variant `hello mozilla`, expected one of `hello`, `register`, `unregister`, `broadcast_su...

https://sentry.prod.mozaws.net/operations/autopush-prod/issues/5010782/

Msg: invalid json text
Error: unknown variant `hello mozilla`, expected one of `hello`, `register`, `unregister`, `broadcast_subscribe`, `ack`, `nack`, `ping` at line 1 column 30

message_data metrics arent emitted

In python:

https://github.com/mozilla-services/autopush/blob/e155e34/autopush/websocket.py#L1018

For offline notifications, connection endpoint replies back 502 Bad Gateway response code to Router Endpoint instead of replying 404 Not Found

Connection endpoint replies back 502 Bad Gateway response code to Router Endpoint instead of replying 404 Not Found. When sending a notification while the browser is disconnected (offline), connection endpoint (or Rust websocket server), reports 502 to the Python router, a response code that is not expected by the router. In the original implementation, and in such scenario, router used to receive 404 from the Python connection node.

Handle error cases in await_session_complete

https://github.com/mozilla-services/autopush-rs/blob/1d5e718/src/client.rs#L454

Should at least emit an error! message

connected_at set to 0 in many cases

We seem to defaulting connected_at to 0 vs ms_since_epoch in some if not many cases.

Create example configuration files with all settings.

There should be example config files in this repo with all settings and what they do.

Cleanup documentation to be accurate

Docs in lib.rs and such still reference the Python functioning. All the docs should be rewritten/updated to reflect the current code architecture.

Add code tag to nack metric

Our nack metric should include the code the client sends if its in the range we allow per:
https://hg.mozilla.org/mozilla-unified/file/tip/dom/push/PushServiceWebSocket.jsm#l40

Update dependencies [Q3]

$ cargo install cargo-edit   # if you don't have it already, provides "cargo upgrade"
$ cargo upgrade  # upgrades Cargo.toml
$ cargo update  # updates Cargo.lock
$ cargo build
$ py.test  # ensure it still works..

Fix user lookup of valid months in hello

Followup to #44

We're still not correctly checking that the user has a valid month

Upgrade Sentry in Rust connection node to the newest version

https://blog.sentry.io/2018/05/01/sentry-for-rust announced 0.3, they're now past 0.5.

This does require a new Sentry in our deployment env, so its blocked right now.

Notify sentry on MegaphoneUpdater failures

Currently we write to debug log when MegaphoneUpdater attempts fail, with a TODO to notify sentry.

With debug logs being hidden by default on prod, we should begin notifying sentry about issues communicating with megaphone instead.

For the sake of diagnosing such issues: https://bugzilla.mozilla.org/show_bug.cgi?id=1585042

Emit metrics for notification delivery time

Notifications include a created at time (the timestamp field). Assuming the endpoint and connection nodes clocks are accurate/in sync, we should be able to track the total time taken from notification send to its delivery.

Should this be emitted when the notification is sent or when it's finally ACK'd?

(Remote Settings has begun tracking its delivery times on its own side as part of its pipeline checks, so this would be nice to have on our side).

No updates after subscribing to a broadcast that "doesn't exist yet"

To reproduce:

• Using wscat, issue a message such as: {"messageType": "broadcast_subscribe", "broadcasts": {"remote-settings/blahblahblah": "abcd"}}.
• No "broadcasts" are received (because the broadcast doesn't exist yet).
• Post an update to this broadcast ID using something like curl -X PUT -H "Authorization: Bearer XXX" -d "BREAKING NEWS: v4" https://megaphone-default.stage.mozaws.net/v1/broadcasts/remote-settings/blahblahblah.

Expected behavior:

• wscat eventually receives an update saying that the blahblahblah subscription has a new version.

Actual behavior:

• wscat only receives boring pings for a while (until I got bored).

@pjenvey observes that this bug may also affect any subscriptions enumerated as part of the "hello" message.

Quiet excessive logging

We lost the log crate dependency along with its Cargo features = ["max_level_info", "release_max_level_info"] when we split the code into 2 separate crates.

This causes excessive trace/debug logging to be on by default, so we'll need to add this back.

Cargo Update to address requests vuln

Github reporting requests needs to be updated.

CODE_OF_CONDUCT.md file missing

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please reach out to [email protected].

(Message COC001)

Invalid message types specified for "hello" being logged in Sentry

https://sentry.prod.mozaws.net/operations/autopush-prod/issues/5011322/?environment=release

https://sentry.prod.mozaws.net/operations/autopush-prod/issues/5010782/?environment=release

Update/fix rust documentation

Use similar automatic Rust doc generation to syncstorage-rs, and update doc strings where needed. Add new doc strings where they're missing.

Setup endpoint crate structure and required dependencies

Create the autoendpoint crate, copy/update main/settings from autopush and setup basic module structure with the actix framework.

Refactor code for autoendpoint port

The code is styled for a single app, autopush. It needs to be re-organized a bit as many of the db/notification handling portions are shared between implementations for adding autoendpoint.

check for port in hostname when resolving IP addresses

Someone might mistakenly specify a hostname to include the port number (e.g. "foo.bar.co:9000") It might be a nice thing to automatically parse out the port number from that.

Remove Secondary Index

With the Database TTL auto-removing old records, we shouldn't need to maintain the secondary index. This impacts the "last_connect" attribute

match crypto_key format w/ python autopush

being a list of keys, expect the comma separated list to be wrapped in a []

Ensure Docker uses Cargo.lock

As of rust 1.37 cargo install no longer uses Cargo.lock by default (noticed here mozilla-services/syncstorage-rs#378).

The Docker build should be reproducible by using Cargo.lock.

cleanup cargo clippy warnings

It's been a while since we went through the clippy warnings and since then they've piled up.

We should run cleanup all the incidents of cargo clippy -- -W clippy::clone_on_ref_ptr violations

Update dependencies [Q4]

howto: mozilla-services/autopush#1170 (comment)

I may attempt to finish my previous effort from last quarter to upgrade to hyper 0.12

Include debuginfo in release builds

We can include debuginfo in our release builds for minimal performance hit, but more meaningful errors.

Run cargo-audit in CI

This project does not appear to be using cargo-audit (https://github.com/RustSec/cargo-audit) - that’s strongly recommended in order to be warned about dependencies with security vulnerabilities.

Fix timer/histogram metrics

autopush-rs's metrics go through dogstatsd vs the direct datadog HTTP API like autopush-py.

For whatever reason:

Note: DogStatsD does NOT implement the following from StatsD:

    Gauge deltas (see this issue)
    Timers as a native metric type (though it does support them via histograms)

Even though Timers, which exist in StatsD, are a sub-set of histograms in DogStatsD.

I'm not sure why dogstatsd doesn't handle timers for us (I imagine that's what the python datadog is doing for us under the covers/via the HTTP API), so we'll need to switch to the histogram type ourselves

no lookup/notify of nodes after storing unacked messages on disconnects

I don't recall us intentionally omitting this from autopush-rs:

https://github.com/mozilla-services/autopush/blob/5a2c3d0/autopush/websocket.py#L591

When storing unacked messages due to a client disconnecting, we also check if they've already reconnected to another node and if so notify them to check storage

Limit sent messages to clients

We missed porting this over to rust: the MessageOverloadException related code in websocket.py that limits the maximum amount of messages sent to clients

Update dependencies [Q3]

Let's update the rust deps when we get to autopush's updates: mozilla-services/autopush#1339

TODO: should add "update the python/rust in the Dockerfile" to the dep update checklist

Support webpush API pings

We only support emitting websocket level ping frames. Clients also ping us with "older" style webpush API pings (empty json dicts "{}"), which they expect a response to.

Retry and don't report increment error

The db/commands.rs was already updated to retry internal server errors, the db/mod should also have these updates, per Sentry:

InternalServerError

Internal server error

Msg

Error incrementing storage

Add dockerflow

Ops and SecOps have adopted DockerFlow which specify common set of behaviors for products.

We should add those to autopush to make life easier for Ops.

Create extractor/ validators for autoendpoint API methods

autoendpoint has a fairly large amount of validation required, this should all be ported in as actix extractors and custom validation logic as needed (syncstorage-rs has some examples of this).

Depends on #102.

Detailed Sentry logging

Our details for sentry logging are a bit light compared to the python version. We should track more metadata per event about user-agent and the user id causing the events.

Extract Rust connection code from autopush repo

Ensure log output is reasonable

Our integration tests run autopush-rs, then fire off test_integration.py against it.

Let's add a check to test_integration ensuring autopush-rs's log output is reasonable: that it's not logging an excessive amount of statements during the test run. It should log probably less than a few lines during the test run (this might depend on whether it was built in debug vs release mode).

This would act as a fail-safe for the recent issue where the log trace level was mistakenly enabled.

Metric notification drops

The dynamodb notification loading has a few spots where messages will be dropped silently that fail deserialization. We should do basic metric counters on these just so that we understand how frequently it happens and whether we need more detailed logging of these occurrences.

Extend sentry event reporting and fix local tests

The sentry test should be fixed to actually pass, and reason diagnosed for why it isn't sending out data correctly. The sentry event capture should also include more data about the context (uaid hash, stats, etc).

Don't report invalid length errors

Per Sentry:

InvalidLength

invalid length: expected one of [36, 32], found 0

Msg

Invalid channelID

These should silently shut down the channel.

Find a workaround for HTTP/2 headers issue within GCP.

Filing this to track progress.

Support case with GCP