mozilla-releng/tooltool

Tasks in the RelEng infrastructure and make use of generic binary artifacts, which are stored in dedicated artifacts repositories (S3 buckets). ToolTool application provides an interface to those artifacts repositories.

ToolTool API, Frontend and Client

Basics

Some of the jobs in the releng infrastructure make use of generic binary artifacts, which are stored in dedicated artifacts repositories.

Tooltool is the client program, written in Python, which is used to retrieve these artifacts from the servers, and to perform integrity checks on them based on hashcode verification. Hence, the servers storing these artifacts are named "tooltool servers".

A set of files to be fetched by tooltool is specified via a tooltool manifest, usually a file in JSON format with tt extension.

The following is an example of a valid tooltool manifest:

[
{
"size": 139308,
"digest": "b2a463249bb3a9e7f2a3604697b000d2393db4f37b623fc099beb8456fbfdb332567013a3131ad138d8633cb19c50a8b77df3990d67500af896cada8b6f698b4",
"algorithm": "sha512",
"filename": "file2.pdf"
},
{
"size": 3017536,
"digest": "630d01a329c70aedb66ae7118d12ff7dc6fe06223d1c27b793e1bacc0ca84dd469ec1a6050184f8d9c35a0636546b0e2e5be08d9b51285e53eb1c9f959fef59d",
"algorithm": "sha512",
"filename": "file1.pdf"
},
{
"size": 3420686,
"digest": "931eb84f798dc9add1a10c7bbd4cc85fe08efda26cac473411638d1f856865524a517209d4c7184d838ee542c8ebc9909dc64ef60f8653a681270ce23524e8e4",
"algorithm": "sha512",
"filename": "file3.pdf"
}
]

The simplest usecase for tooltool is to run a "fetch" command to download the files mentioned in the manifest:

python tooltool.py fetch -m my-manifest.tt

Uploading to Tooltool

First, plan out your upload. Are all of the files you need to upload public? If not, tooltool also offers storage of "internal" files, which are not made publicly available but can be downloaded by those with proper permissions. Internal files might be under a non-redistribution license (e.g., Google SDKs, Microsoft DLLs), but must not include secrets such as passwords or private keys.

Use the tooltool.py client to build a manifest containing the files you would like to upload, annotating each file with its visibility level:

python tooltool.py add --visibility public gcc.tar.gz

Next, you will need credentials for the upload. Tooltool uses Taskcluster for authentication. To retrieve taskcluster credentials, run:

export TASKCLUSTER_ROOT_URL=https://firefox-ci-tc.services.mozilla.com/
taskcluster signin

...and then put the clientId and accessToken you obtained into a JSON file as follows:

{
    "clientId": "xxxxxxxxxxxxx",
    "accessToken": "xxxxxxxxxxxxxx"
}

Now, you're ready to upload with a command like:

python tooltool.py upload --authentication-file=~/.tooltool-token --message "Bug 1234567: add new frobnicator binaries"

Local Development

Run:

export TASKCLUSTER_ROOT_URL=https://firefox-ci-tc.services.mozilla.com/
# Use "taskcluster signing" if you don't have these
export TASKCLUSTER_CLIENT_ID=xxxxx
export TASKCLUSTER_ACCESS_TOKEN=xxxxx
# You will need to create your own bucket & AWS credentials
export S3_REGIONS=us-west-2:your-s3-bucket
export S3_REGIONS_ACCESS_KEY_ID=xxxxxx
export S3_REGIONS_SECRET_ACCESS_KEY=xxxxxx
docker-compose up

Tooltool should then be ready for use on https://localhost:8010.

Deployment process

To trigger the deployment you have to push the code to the branch with the same name as environment you want to deploy to.

This will start Taskcluster graph which will build and push docker image to docker hub (mozilla/releng-tooltool) with the same tag as is the environment.

Cloudops team Jenkins is listening for the change and will deploy it to GCP once it confirms that the docker images was build in a trusted environment. It usually takes around 5min for deployment to be done. For more how things are configures you can check cloudops infrastructure.:

You can check that the service was deployed correctly by visiting the /__version__ endpoint which should include the revision it was built from.

Deployed Environments

We have a number of deployed ToolTool environments.

• Staging

  URL

  https://stage.tooltool.mozilla-releng.net/

  Deploys from

  staging branch

• Production

  URL

  https://tooltool.mozilla-releng.net/

  Deploys from

  production branch

tooltool's Issues

frontend is busted

In at least two ways:

1. Uncaught ReferenceError: hawk is not defined after logging in
2. Error: hawk is not defined and Error: can't access property "is_restapi_request", response.config is undefined when trying to search

frontend is broken

All XHR requests fail. Most likely because the base URL is set wrong (it's trying localhost).

figure out what to do with logins

Even after #9 is fixed, login will be broken because the /auth0 endpoints don't exist. We probably need to change auth to talk directly to Taskcluster, although this needs a bit more investigation first.

worker is broken

Hitting these exceptions:

tooltool_api.lib.pulse: Reconnecting in 10 seconds (exception='Traceback (most recent call last):
 File "/appenv/lib/python3.7/site-packages/aioamqp/frame.py", line 78, in read
 data = await reader.readexactly(7)
 File "/usr/local/lib/python3.7/asyncio/streams.py", line 677, in readexactly
 raise IncompleteReadError(incomplete, n)
asyncio.streams.IncompleteReadError: 0 bytes read on a total of 7 expected bytes

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File "/app/src/tooltool_api/lib/pulse.py", line 107, in create_consumer
 return await _create_consumer(user, password, exchange, topic, callback)
 File "/app/src/tooltool_api/lib/pulse.py", line 46, in _create_consumer
 port=port,
 File "/appenv/lib/python3.7/site-packages/aioamqp/__init__.py", line 63, in connect
 insist=insist)
 File "/appenv/lib/python3.7/site-packages/aioamqp/protocol.py", line 216, in start_connection
 await self.dispatch_frame()
 File "/appenv/lib/python3.7/site-packages/aioamqp/protocol.py", line 263, in dispatch_frame
 frame_channel, frame = await self.get_frame()
 File "/appenv/lib/python3.7/site-packages/aioamqp/protocol.py", line 250, in get_frame
 return await amqp_frame.read(self._stream_reader)
 File "/appenv/lib/python3.7/site-packages/aioamqp/frame.py", line 80, in read
 raise exceptions.AmqpClosedConnection() from ex
aioamqp.exceptions.AmqpClosedConnection')

Also have aioamqp.protocol: only PLAIN login_method is supported, falling back to AMQPLAIN in the logs.

tooltool client location, duplication and usage

Hey, the doc says

If you want to use the client, just copy out tooltool.py -- it has no dependencies.

I think it's a problem.

Having people copy that script here and there has two major disadvantages:

• it leads to some issues of having deprecated tooltool.py instances in many places.
• it makes it hard to deploy a new improved version
• we don't know if the version we use is the right one

For example the version in this repo says version == '1' but the one in mozharness
has the same version but is a different module.

We bumped into the issue because we started to work on https://phabricator.services.mozilla.com/D54090 to add a small retry mechanism to try
to reduce intermittents on Raptor, and decided to use mozbuild's tooltool version as
the source of truth, but it turns out this repo is the one :)

there's also https://github.com/mozilla/build-relengapi ?

I would like to make this proposal:

• ask people not to copy around tooltool.py
• remove all tooltool.py instances in mozilla-central
• release on pypi and our internal pypi a tooltool client with no dep that can be used in all py versions
• make that package a commodity everywhere in our CI and tests environements

Update documentation on credentials.

I'm looking at https://docs.mozilla-releng.net/projects/tooltool.html#request-authentication-credentials-for-the-client-tooltool-py (which I don't know where is now maintained, if anywhere).

The instructions say that people should file a bug, and then we will generate taskcluster credentials for them.

This should be changed to indicate that we will grant their taskcluster role the appropriate scopes, and then they can create a client with the appropriate scopes to use tooltool.

It may also make sense to tooltool to use the standard taskcluster environment variables, rather than having its own file format for them.

use cloudfront for downloads instead of redirecting directly to s3

synchronize version.txt and __version__ in the client

Here we have 1.3.0: https://github.com/mozilla-releng/tooltool/blob/master/client/version.txt
and here, 1 : https://github.com/mozilla-releng/tooltool/blob/master/client/tooltool.py#L55

let's synchronize both, and change __version__ so it matches version.txt
(I guess in the long term we drop __version__)

UnicodeDecodeError: when running with python 3

I see this error when trying to download a manifest file from tooltool
manifest:
https://searchfox.org/mozilla-central/source/testing/mozbase/mozproxy/mozproxy/backends/mitm/mitmproxy-rel-bin-4.0.4-win.manifest

command line: python3.6 tooltool.py fetch -o -m <manifets path>

    /mozilla/tooltool/client$ python3.6 tooltool.py fetch -o -m /home/florinstrugariu/mozilla/mozilla-central/testing/mozbase/mozproxy/mozproxy/backends/mitm/mitmproxy-rel-bin-4.0.4-win.manifest 
    INFO - Attempting to fetch from 'https://tooltool.mozilla-releng.net/'...
    INFO - ...failed to fetch 'mitmproxy-4.0.4-windows.zip' from https://tooltool.mozilla-releng.net/
    Traceback (most recent call last):
      File "tooltool.py", line 741, in fetch_file
        indata = to_text(indata)
      File "tooltool.py", line 99, in to_text
        return val.decode('utf-8')
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xad in position 10: invalid start byte
    ERROR - The following files failed: 'mitmproxy-4.0.4-windows.zip'

file size limited to 2gb

The releng_tooltool_files.size column is an integer, meaning it takes values between -2147483648 and [phone].
It should probably be a bigint instead.

add test to verify that in-tree tooltool.py copies match this one

We have 3 different copies of the tooltool.py client vendored into the tree:

• python/mozbuild/mozbuild/action/tooltool.py
• testing/mozharness/external_tools/tooltool.py
• testing/web-platform/tests/tools/third_party/tooltool/tooltool.py

The last time we tried to update the in-tree copies, we discovered that two of them had differences not found in the other, nor the copy in this repo.

We should add a test here to verify that all copies are the same. Maybe a test in tree as well.

tox not running on pull request

I created a PR that should have failed a test case on tox, but all checks passed. It looks like tox is not being run.

TypeError: POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str.

Running the delete command with the client fails with this error:

Traceback (most recent call last):
  File "/tmp/tooltool/client/tooltool.py", line 1713, in <module>
    sys.exit(main(sys.argv))
             ^^^^^^^^^^^^^^
  File "/tmp/tooltool/client/tooltool.py", line 1709, in main
    return 0 if process_command(options, args) else 1
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tooltool/client/tooltool.py", line 1540, in process_command
    return delete_instances(
           ^^^^^^^^^^^^^^^^^
  File "/tmp/tooltool/client/tooltool.py", line 1473, in delete_instances
    return send_operation_on_file(data, base_urls, digest, auth_file)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tooltool/client/tooltool.py", line 1450, in send_operation_on_file
    urllib2.urlopen(req)
  File "/usr/lib/python3.11/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 516, in open
    req = meth(req)
          ^^^^^^^^^
  File "/usr/lib/python3.11/urllib/request.py", line 1279, in do_request_
    raise TypeError(msg)

That being said, even with that fixed up, delete doesn't seem to work.

TypeError: common_error_handler() missing 1 required positional argument: 'exception'

Sentry Issue: TOOLTOOL-1A

BadRequestProblem: 
  File "flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "connexion/decorators/decorator.py", line 68, in wrapper
    response = function(request)
  File "connexion/decorators/uri_parsing.py", line 149, in wrapper
    response = function(request)
  File "connexion/decorators/validation.py", line 377, in wrapper
    raise BadRequestProblem(detail=error)

TypeError: common_error_handler() missing 1 required positional argument: 'exception'
  File "flask/app.py", line 2529, in wsgi_app
    response = self.full_dispatch_request()
  File "flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "flask/app.py", line 1676, in handle_user_exception
    return self.ensure_sync(handler)(e)

TypeError: common_error_handler() missing 1 required positional argument: 'exception'

Update the README to indicate when tooltool should (or shouldn't) be used

Capture some of the discussion on https://bugzilla.mozilla.org/show_bug.cgi?id=1743179 to guide future tooltool travellers.