taskcluster/cloud-mirror

cloud mirror

Taskcluster

The task execution framework that supports Mozilla's continuous integration and release processes.

---

Usage

This repository is used to develop, build, and release the Taskcluster services.

Table of Contents

• Clients
  • Taskcluster Client for JS
  • Taskcluster Client for Go
  • Taskcluster Client for Python
  • Taskcluster Client for Rust
    • Taskcluster API Client
    • Taskcluster Download Support
    • Taskcluster Upload Support
  • Taskcluster Client for Shell
  • Node client test
  • Taskcluster Client for Web
• Taskcluster Database
  • Database Versions
• Development Documentation
• Infrastructure
  • Docker Images
  • References
  • Taskcluster Builder
• Internal Go Packages
  • Mock Services Design
• Libraries
  • API Library
  • App Library
  • Config Library
  • Iterate Library
  • Loader Library
  • Monitor Library
  • Postgres Library
  • Pulse Library
  • References Library
  • Testing Library
  • Validate Library
• Services
  • Auth Service
  • Built-In Workers Service
  • GitHub Service
  • Hooks Service
  • Index Service
  • Notify Service
  • Object Service
  • Purge-Cache Service
  • Queue Service
  • Secrets Service
  • Web-Server Service
  • Worker Manager Service
    • About the Keys in this Directory
    • Azure Trusted Certificates
• Tools
  • d2g
    • tools/d2g/d2gtest/testdata
  • jsonschema2go
  • livelog
  • taskcluster-proxy
  • Websocktunnel
  • Worker Runner
  • Runner / Worker Protocol
• Taskcluster UI
  • ui/src/components/CopyToClipboardListItem
  • ui/src/components/DateDistance
  • ui/src/components/Duration
  • ui/src/components/Search
  • ui/src/components/Snackbar
  • ui/src/components/SpeedDial
  • ui/src/components/StatusLabel
• Workers
  • Docker Worker
  • Generic Worker

Team Mentions

Do you need to reach a specific subset of the team? Use the team handles to mention us with GitHub's @mention feature.

Team Name	Use To...
@taskcluster/Core	ping members of the Taskcluster team at Mozilla
@taskcluster/services-reviewers	ping reviewers for changes to platform services and libraries
@taskcluster/frontend-reviewers	ping people who can review changes to frontend (and related) code in the services monorepo
@taskcluster/security-folks	ping people who do security things

Contributors

Thanks goes to these wonderful people (emoji key):

James Lal 💻 👋	Selena Deckelmann 💻 👋	Dustin J. Mitchell 💻 👋	Wander Lairson Costa 💻 👋	Greg Arndt 💻 👋	Pete Moore 💻 🔧	Hassan Ali 💻 👋
John Whitlock 💻 👋	Brian Stack 💻 👋	John Ford 💻 👋	Eli Perelman 💻 👋	Jonas Finnemann Jensen 💻 👋	owlishDeveloper 💻 👋	Miles Crabill 💻 👋
Chris Cooper 💻 👋	Mathieu Leplatre 💻 👋	Rob Thijssen 💻	Anup 💻	Hammad Akhtar 💻	Chinmay Kousik 💻	Anthony Miyaguchi 💻
Ana Rute Mendes 💻	Andrea Del Rio 💻	kristelteng 💻	Elena Solomon 💻	Xavier L. 💻	Yann Landry 💻	Ayub 💻
lteigrob 💻	Bastien Abadie 💻	Amjad Mashaal 💻	Tom Prince 💻	Samantha Yu 💻	Auni Ahsan 💻	alex 💻
Alisha Aneja 💻	Prachi Manchanda 💻	Simon Fraser 💻	Yashvardhan Didwania 💻	Cynthia Pereira 💻	Hashini Galappaththi 💻	Fienny Angelina 💻
Kanika Saini 💻	Biboswan Roy 💻	sudipt dabral 💻	Ojaswin 💻	Матрешка 💻	Alok Kumar 💻	Arshad Kazmi 💻
Jason Yang 💻	Shubham Gupta 💻	Arun Kumar Mohan 💻	Brian Pitts 💻	E. Dunham 💻	Shubham Chinda 💻	Patrick Kang 💻
Rishabh Budhiraja 💻	ededals 💻	Ajin Kabeer 💻	Catherine Chepkurui 💻	Jo 💻	vishakha 💻 📖	Noor Fatima 💻
Michael 💻	Mariana Zangrossi 💻	ANURADHAJHA99 💻	Edil 💻	Olympia 💻 📖	Michael Ozoemena 💻	lailahgrant 💻
km-js 💻	Carolina Machado 💻	reenesa 💻	Kelli Blalock 💻	naima shaikh 💻	Jiwoon Kim 💻	Michael Umanah 💻
Fahd Jamal A. 📖	shilpi verma 💻	somchi 💻	Anastasia 💻	Lubna 💻	Soundharya AM 💻	Mustafa Jebara 💻
Aryaman Puri 💻	Simon Sapin 💻	thoran 💻	Manish Giri 💻	Tiger Oakes 💻	Ricky Taylor 💻	Alex Lopez 💻
Michelle 🐛 🚇	Mrs. Velena 💻	Ahmed A. 💻	Matt Boris 💻 🔧	Yaraslau Kurmyza 💻 🔧	Bastien Orivel 💻	HamdiAmine 💻

This project follows the all-contributors specification. Contributions of any kind are welcome!

cloud-mirror's Issues

CODE_OF_CONDUCT.md isn't correct

Your required text does not appear to be correct

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please reach out to [email protected].

(Message COC003)

Use redis pub/sub instead of polling to monitor pending transfers

Currently in the /redirect/ endpoint, if we encounter a file in the pending state, we poll every second until we see that pending status change to either 'error' or 'present'. We should probably have a failsafe of polling every 10 seconds to make sure that pub/sub being flakey doesn't cause too much trouble. In this case, we should log something to make it clear that we have experienced this condition.

Ensure support for pull-through caches

A pull-though cache is just a simple HTTP cache... When working in data-center etc, we won't have cheap object storage available, so we have to do pull-through caches.

cloud-mirror can support this by taking the rawUrl choosing a cache server using something like: cacheServers[fnv(rawUrl) % cacheServers.length]...
So that if we have multiple cache servers, urls get distributed based on hash of the url...

The put operation for such a pull-through cache is pretty naive, as there is no copying, just choose a cache server and rewrite the url. This could be a special implementation CacheHandler as proposed in #9. Or we could do the redis and sqs messaging even if it is unnecessary and do a trivial implementation of StorageProvider.

Either way, when region detection based on ip ranges is moved to cloud-mirror (issue #6) it would be very nice to support backends that are essentially pull-through caches. Otherwise, support for data-center and providers like digital ocean or packet will be hard.

Move validateInputURL

It's okay to make functions, not everything has to be a method...
And duplicating configuration like allowedPatterns in an effort to make method is bad.

Pretty sure I gave this feedback before, validateInputURL should be moved.
Probably it belongs inside the API handler. As we should reject illegal URLs immediately.

Do region resolution in cloud-mirror

Move EC2RegionResolver from taskcluster-queue in here, see:
https://github.com/taskcluster/taskcluster-queue/blob/2ab0317576d9b7c6ca9b011a76ff36aaf7eaff3e/queue/ec2regionresolver.js

Write GCE storage backend

We're interested in having a storage backend for the GCE cloud environment. A great starting place for this is to look at src/s3-backend.js as an example of what needs to be done for a storage backend.

Reduce number of address concepts

Currently you have:

• rawUrl, that's the user input,
• storageAddress(rawUrl), function that takes a rawUrl and returns a storageAddress object, I assume?
• storageAddressToUrl, function that takes a storageAddress object and returns a url (concept of this url has no name, and is unnecessary afaik).

I suspect you might need 2 concepts, but 3 really?

I propose:

• rawUrl, obviously this is the given by user in the request (url to the source object)
• cachedObject, a dictionary consisting of {ttl, rawUrl, cacheUrl, ...} plus any number of backend-specific properties (as long as it has ttl, rawUrl and cacheUrl).

Clearly, what you are doing now is:
cacheUrl = storageAddressToUrl(storageAddress(rawUrl))

There is no reason that cacheUrl can't be a propert on the object returned from storageAddress(rawUrl).

I guess you can still call it storageAddress rather than cachedObect if you like... Just get rid of storageAddressToUrl as it introduces a new concept that we don't have a good name for... And worse, a concept we don't need.

Assuming I understood half the code of-course :)

Remove encoded url from end-point

Some clients can't handle urls encoded in other urls..
And we have support a lot of clients that are less than perfect...

We should refactor cloud-mirror so that:
https://cloud-mirror.taskcluster.net/v1/redirect/s3/us-east-1/<key>
mirrors <key> in a bucket hardcoded in configuration.

This will also remove the need for regular expressions, I think it's fine to make cloud-mirror less generic, we only want to use to mirror S3 buckets anyways.

Rather than passing a complete URL to cloud-mirror.

Set protected status on production branch

Hello! This is your neighborhood secops team looking out for you!

The production branch on this repository is not protected against force pushes. This setting is recommended as part of Mozilla's Guidelines for a Sensitive Repository.

Anyone with admin permissions for this repository can correct the setting using this URL.

If you have any questions, or believe this issue was opened in error, please contact us and mention SOGH001-0 and this repository.

Thank you for your prompt attention to this issue.
--Firefox Operations Security team

Write Azure storage backend

We're interested in having a storage backend for the Azure cloud environment. A great starting place for this is to look at src/s3-backend.js as an example of what needs to be done for a storage backend.

Split abstract implementation from concrete `StorageBackend`

The StorageBackend class is huge... And it's very hard to see what is abstract and what isn't.
It's even hard to comprehend all the methods.

I think this can be refactored into two classes:

• StorageProvider pure abstract class with no methods, (or just very short default implementations)
• CacheHandler wrapper around a StorageProvider object that deals with meta-data caching in redis, polling redis, sqs messaging, etc...

---

Currently StorageBackend says method starting with _ must be implemented (not a bad idea). But I would rather that we use _ to signify private methods. Especially, since StorageBackend has so many of them... It's really hard to see what gets called from API, etc...
So when making CacheHandler make sure all private methods starts with _, it makes it a lot easier to see what is an auxiliary method, and what is part of the public interface.
(I don't always do this, but when objects have lots of methods it really helps).

For StorageProvider I propose pure abstract methods, hence, the abstract ones wouldn't start with _.. Unless, for some reason you need to wrap them, but I don't see why that would be necessary.

parse REDIS environment variable instead of using REDIS_* vars

Because Heroku automatically changes the REDIS variable but not the ones we create.

Write a localhost storage backend and unit tests

Right now we don't have any sort of no-op storage backend for unit tests. We could mock out the AWS api, but that's not great because we really need a real HTTP backed service.

It would be great if we had a basic 'cloud' server which has a PUT, GET and DELETE end point. These are the routes I would see:

GET http://localhost/artifact/:key
PUT http://localhost/artifact/:key
DELETE http://localhost/artifact/:key

This server should be written in node and able to run concurrently in the same process as the unit tests.

Then a backend implementation which knows how to insert and retrieve files from this basic service would be great because we could write integration tests which require no credentials. We could also theoretically use this service to provide cloud-mirror support in on-premises data centers.