mozilla/release-services

Mozilla Release Engineering Services

Welcome to Release Services

This repository contains various Mozilla Release Engineering and Management Services used in our Continous Integration and Release Automation.

Documentation:	https://docs.mozilla-releng.net
Issues:	https://github.com/mozilla/release-services/issues
Code:	https://github.com/mozilla/release-services

release-services's Issues

command that creates taskcluster hooks tasks

• (hourly) shipit_dashboard and upload artifacts to s3
• (hourly) releng_clobberer and upload artifacts to s3
• (daily) self update (to run make update-all) and create PR
• (daily) vulnix and send email with results (if vulnix is ready)

frontend should send Authentication header to backend

look into switching from newrelic to prometeous

create update script which runs pypi2nix/node2nix and generates nix expressions

build continuosly with taskcluster-github hook

example integration: https://mykzilla.org/2016/08/02/continuous-integration-of-positron-with-taskcluster/

in debug mode run with log_level = debug

all apps should use the same version

relengapi_fronend should be packaged with nix

build app specific docker images using taskcluster

rename relengapi_common to releng

automatically update dependencies (python/elm/..) in master branch

create a nightly hook(s) for master branch that will run make update-apps and make update-tools

ask heroku admin to remove unused apps

apps to be removed:

• https://dashboard.heroku.com/apps/releng-stag-database
• https://dashboard.heroku.com/apps/relengapi-master
• https://dashboard.heroku.com/apps/relengapi-staging-clobberer
• https://dashboard.heroku.com/apps/relengapi-prod-clobberer
• https://dashboard.heroku.com/apps/relengapi-prod-postgres
• https://dashboard.heroku.com/apps/relengapi-staging-postgres

also this pipeline needs to be removed:

• https://dashboard.heroku.com/pipelines/6e02ffe0-7bc4-43ee-a402-2c0f7820e554

pushing docker images to heroku should be possible without docker command

example https://gist.github.com/matejc/582ddd9e0c6eaf5459838469f4bed82b

remove vendor.bundle.js from frontend

also apply to shipit_frontend

clobberer shoulnt fail when database is empty

tables should be created

Require neo for develop-run-FRONTEND

When I try to run make develop-run-FRONTEND, I get the following:

make develop-run-FRONTEND APP=releng_clobberer
nix-build nix/default.nix -A tools.createcert -o result-tool-createcert --fallback
/nix/store/p9q9dzslfwwip6f52vrdkvnl951psd1d-createcert
nix-shell nix/default.nix --pure -A releng_clobberer \
        --run " neo start --port 8002 --config webpack.config.js"
Setting "releng_common" in development mode ...
Setting "releng_clobberer" in development mode ...
/run/user/1000/nix-shell.YRGURn/rc: line 1: neo: command not found
Makefile:116: recipe for target 'develop-run-FRONTEND' failed
make: *** [develop-run-FRONTEND] Error 127

neo is not installed on my laptop.

Shipit: Show patches landing dates

A suggestion from Ritu

Show dates when the fix landed in central/aurora/beta?
When I am approving a riskier change for uplift to Beta, I would like to confirm this has been on Nightly for a few days and on Aurora as well.
You could show this on the right side of the page, under the status flags in a section called “Commit dates”

• Mozilla-central: mm/dd/yy
• Mozilla-aurora: mm/dd/yy
• Mozilla-beta: mm/dd/yy

Check the data is present in libmozdata

should monitoring with opbeat be better?

https://opbeat.com/flask/

check continuosly (nightly/weekly) for security vulnerabilities

there are multiple options:

• setup vulnix
• talk to infosec team if we could hook into their tools

rename S3 buckets to match style used heroku

script to sync database from current production to staging

create update helper for nixpkgs

move python+nix+docker document into releng_docs

https://docs.google.com/document/d/1d66PQmsFgJLn9Dck4e1d_2yvmIoDLUsws4vDMOh7RDA/edit

Run flake8 against all Python files

I just tried to run a freshly created app and got an the following error:

Traceback (most recent call last):
  File "shipit/__init__.py", line 38, in <module>
    app.run(**app.get_run_options())
  File "/home/rail/work/mozilla/git/services/lib/releng_common/releng_common/__init__.py", line 84, in get_run_options
    debug=DEBUG,
NameError: name 'DEBUG' is not defined

It'd be great to run flake8 against all files. This would also improve PEP8 compatibility. ;)

decorator to handle authenticated header and checks for required roles

Shipit: Highlight answers that indicate risks

A suggestion from Ritu:

Highlight answers that indicate risks (in red color)

• If the “string/UUID change” is set to anything but “No or None or N/A”.
• If test coverage question is empty or No or N/A
• If risk is tagged as “medium” or “high”

The analysis should be made in libmozdata, as usual (ping @marco-c).

build docker images for all applications

author of all modules should be Mozilla's RelEng team

update README

update relengapi_frontend to work with new api

use logbook to handle logs

*.mozilla-releng.org domains for releng apps

Production

mozilla-releng.org -> https://console.aws.amazon.com/cloudfront/home?region=us-west-2#distribution-settings:E7SMSK9P6UW3N
clobberer.mozilla-releng.org -> https://dashboard.heroku.com/apps/releng-production-clobberer
docs.mozilla-releng.org -> https://console.aws.amazon.com/cloudfront/home?region=us-west-2#distribution-settings:EPUEJ5MNC6OE

Staging

staging.mozilla-releng.org -> https://console.aws.amazon.com/cloudfront/home?region=us-west-2#distribution-settings:E21OUD0VFT2UQ6
clobberer.staging.mozilla-releng.org -> https://dashboard.heroku.com/apps/releng-staging-clobberer
docs.staging.mozilla-releng.org -> https://console.aws.amazon.com/cloudfront/home?region=us-west-2#distribution-settings:E1LW0QJF456NUG

package shipt_frontend (as it releng_frontend)

deploy project wide sphinx documentation (to s3)

Shipit: Update bug payload directly after an action

A bug data should always stay consistent, even after an action (approval/reject/edit) by a user.
On a successfull Bugzilla update, the changes made (Bz response) should be sent to our backend in order to update the bug.
The backend response must be the standard bug serialization so it can be parsed and updated directly in the stack.

automate creation/configuration stuff

• creation of s3 buckets
• hooking s3 buckets with clouldfront

app.run_options() is not a valid method

I tried to create an example app and failed to use the app.run(**app.run_options()) pattern. Looks like the proper call is app.run(**app.get_run_options())

deploy static websites to s3

create a cli to fetch taskcluster branches script to run via hooks service

deploy to heroku

command that creates binary cache and uploads it to s3 bucket

drop dependency on neo

... and build resources with livereload in development

releng_frontend should use correct releng_clobberer api endpoint

add basic flake8 linting to releng

rename relengapi_tools to relengapi_frontend

write help for root Makefile

can we avoid using docker command when deploying docker image to heroku

create base docker image to hook it up with taskcluster

we need docker image with nix commands and custom binary cache (pointing to s3, via some variable)

check on prod instances which api endpoints are used

remove cache support from clobberer settings.py