bouyguestelecom/a7

Enterprise-grade cloud-native solution for serving immutable static assets

A7

Enterprise-grade cloud-native solution for serving immutable static assets

🤿 Dive in!

---

Motivation

Static resources (aka assets) are prevalent in web projects. Yet one could say they have been sidelined for a long time, as handling these assets present particular challenges:

• Naming: For most, these resources are few- a bit of a css, fonts and images. For some, these resources are manifold, and teams have to leverage some cognitive ingeniosity to architect and name directories; and either drop files in ill-named folders 🙈 or reinvent the wheel by imagining new conventions.

  🔠 A7 enforces naming conventions, based on semantic versioning and industry standards.

• Serving: Serving these assets is currently handled by: (a) the application server itself; and/or (b) a static storage service; and/or (c) a CDN for nearby-server delivery.

  💁 A7 is a Docker-ready application server dedicated to static resources, that is portable and gives you the best of both worlds: use it locally or as a remote service, integrated in a cloud.

• Caching: Relying on static storage services or CDN services brings value to the table, part of which is fine-tuned caching mechanism; this is clearly not always the case with custom implementations.

  🐦 A7 serves assets with immutability as standard all through.

... to name a few. A7 also addresses redirections (with smart redirects), custom domains handling, CORS support, security, compressed directory download, resources discovery (with an API), centralized assets with a focus on reuse.

Getting started

1. Given a directory of assets <my-assets-dir> containing all the assets to serve:

   Directory guidelines & structure example

   
   

   Guidelines:

   1. An asset is a static file, whose MIME type is readable and operable by a web browser;

   2. An asset must be placed within a package directory at the root;

   3. A package directory can be of different flavors, with either:

      • <package-name>@<package-version>; or
      • @<package-scope>/<package-name>@<package-version>; or
      • <package-namespace>/<package-name>@<package-version>.

      In the case of a scoped or namespaced package, the package directory is composed of two folders.

   4. The organization of assets within the package directory is free (the asset files can be contained in file structures of any depth and hierarchy);

   5. There can be a default path that refers to one asset as the resource fallback, for when a request does not specify one;

   Directory structure example:

   The assets directory must be structured as follows:

   <my-assets-dir>/
     foo@1.3.0/            # standard package
       path/to/file.js
     foo@1.3.1/            # patched version
       path/to/file.js
     foo@1.4.0/            # more/most recent release
       path/to/file.js
     @myscope/
       bar@1.0.0/          # scoped package
         dist/file.css
         dist/font.woff2

2. Given an assets catalog file that references all the assets:

   Assets catalog example

   
   

   // <my-assets-dir>/.catalog.json
   [
     {
       "name": "[email protected]",
       "defaultPath": "dist/file.js"
     },{
       "name": "[email protected]",
       "defaultPath": "dist/file.js"
     },{
       "name": "[email protected]",
       "defaultPath": "dist/file.js"
     },{
       "name": "@myscope/[email protected]",
       "defaultPath": "dist/file.css"
     }
   ]

3. Start a service container:

   $ ASSETS_PATH=<my-assets-dir>
   $ docker run -it \
       -p 45537:45537 \
       -e A7_VOLUME_MOUNT_PATH=$ASSETS_PATH \
       -v $(pwd)/assets:$ASSETS_PATH \
       bouyguestelecom/a7

4. Access it: http://localhost:45537

See the integrations section for more use cases, such as with docker compose.

Use cases

• Host images, illustrations and logotypes;
• Serve a common CSS framework and its webfonts;
• Serve micro frontend static applications;
• ... and more.

Features

Assets are immutable

An asset is immutable: for a given version of the package its content won't change. Ever.

In order to enforce this trait, the Cache-Control header is used extensively, with the immutable value, indicating that the response body will not change over time.

Handle every request

When requesting unknown or uncertain content, A7 tries its best to handle requests by:

1. directly serving the requested content (for fully-qualified URLs);
2. indirectly serving the requested content (for URLs with special or missing parts) through a smart redirect or can be configured (via A7_PATH_AUTO_RESOLVE) to serve them instead;
3. serving a 404 status page by default.

Adopt smart routing conventions

Prerequisite: Set the A7_PATH_AUTO_EXPAND environment variable to true.

By relying on semantic versioning conventions, some industry standards, and by extending them a bit, A7 supports all the following routing resolutions to the table.

Also: Set the A7_PATH_AUTO_EXPAND_INIT environment variable to true or always in order to generate the metadata files in all the assets subdirectories (true will generate metadata files if a prior existing file; always will generate metadata files systematically and overwrite any prior existing file).

Set the A7_PATH_AUTO_RESOLVE environment variable to true in order to resolve the path part of the URI to the actual file path on disk, instead of relying on default client-side redirections. This is useful in performance-sensitive environments, where the client-side redirections are not desirable.

Fully qualified URI

/[email protected]/path/to/file.js
➔ Status: 200

Here, we request a URL with complete information about the requested asset (package name, version, and path).

URI with missing path

/[email protected]
➔ Status: 302; Location: /[email protected]/path/to/file.js
    ➔ Status: 200

Here, we request a URL with a package name, and a precise package version, but no path. In such a situation, the "defaultPath" of the matching catalog entry is used.

URI with incomplete semver

/[email protected]
➔ Status: 302; Location: /[email protected]/path/to/file.js
    ➔ Status: 200

/foo@1
➔ Status: 302; Location: /[email protected]/path/to/file.js
    ➔ Status: 200

/foo@1-snapshot
➔ Status: 302; Location: /[email protected]/path/to/file.js
    ➔ Status: 200

Here, we request a URL with a package name, and an imprecise package version, and no path. In such a situation, the "defaultPath" of the matching catalog entry is used to complete the path part. The version is also expanded into the latest version that best matches the request.

Latest version

/foo@latest
➔ Status: 302; Location: /[email protected]/path/to/file.js
    ➔ Status: 200

Here, we request the absolute latest version of a package.

⚠️ This is highly discouraged in production.

Scoped & namespaced package

/@myscope/[email protected]/dist/file.css
➔ Status: 200

/namespaced/[email protected]/dist/file.js
➔ Status: 200

As packages can be scoped or namespaced, these cases are also handled by the same rules of fully-qualified and incompleted URIs.

Unknown URI

/unexpected@1
➔ Status: 404

When requesting something strange, or a missing asset, a 404 HTTP response is returned.

Browsable

Prerequisite: Set the A7_AUTOINDEX environment variable to true.

/[email protected]/
/[email protected]/path/
/[email protected]/path/to/

Any request ending with the / character indexes its own resources and displays them in a browsable web page.

Any request with the ?catalog query string enables access to the corresponding assets catalog (in JSON mode) and makes exploring resources possible through a basic API endpoint, dedicated to listing all the current directory assets.

API mode

Prerequisite: Set the A7_META_QUERIES environment variable to true.

Any request containing the ?meta query string enables the API mode and makes exploring resources possible through a basic API.

Compressed directories

Prerequisite: Set the A7_ZIP_DIRECTORIES environment variable to true.

Directories can be downloaded in a singled zip file. Any request to a directory URI, with the .zip extension appended will start downloading the directory in a zip file, on-the-fly.

CORS

Prerequisite: Set the A7_CORS_ALL environment variable to true. Optionally: set the A7_META_QUERIES_CORS_ALL environment variable to true.

Make the resources available for use by any domain by relying on the following Cross-Origin Resource Sharing (CORS) configuration:

Access-Control-Allow-Origin: *

Personalization

Make the service yours with two white-label features:

• A7_TITLE lets you change the name of the service;
• A7_ICON lets you change the icon of the service;
• A7_BRAND_COLOR lets you change the color of the top hero banner.

These two properties will appear in the browsing UI.

Restrict HTTP methods

With A7_GET_REQUESTS_ONLY, restrict the usage of HTTP methods to only HEAD and GET.

Internal APIs

Prerequisite: Set the A7_AUTOINDEX environment variable to true. Optionally: set the A7_AUTOINDEX_CORS_ALL environment variable to true.

The A7_INTERNAL_API environment variable (true by default) lets you disable the write and search internal APIs used to enhance the browsing UI experience.

Note: writing files to disk requires write permissions on the mounted volume.

Integrations

with Docker from sources

docker build -t a7 .
docker run -p 45537:45537 -v $(pwd)/assets:/assets -it a7
open http://localhost:45537

with docker-compose

1. Prepare your compose.yml:

   version: "2.0"
   
   services:
     a7:
       build:
         context: .
       environment:
         - A7_VOLUME_MOUNT_PATH=<my-assets-dir>
         - A7_AUTOINDEX=true
         - A7_AUTOINDEX_CORS_ALL=true
         - A7_ZIP_DIRECTORIES=true
         - A7_META_QUERIES=true
         - A7_META_QUERIES_CORS_ALL=true
         - A7_CORS_ALL=true
         - A7_PATH_AUTO_EXPAND=true
         - A7_PATH_AUTO_EXPAND_INIT=false
         - A7_PATH_AUTO_RESOLVE=false
         - A7_RUN_SCRIPTS_ONLY=false
         - A7_GET_REQUESTS_ONLY=true
         - A7_INTERNAL_API=true
         - A7_TITLE=A7
         - A7_ICON=📦
         - A7_PUBLIC_ORIGIN=https://my.domain.tld
       ports:
         - 45537:45537
       volumes:
         - ./assets:<my-assets-dir>

2. Run it: docker-compose up

3. Access it: http://localhost:45537

Inspirations

These two services have been a source of inspiration in the conceptual design of the service:

• unpkg
• jsdelivr

Licenses

• A7 is released under Apache 2.0 licence

Dependencies

A7 relies on dependencies that are downloaded at build time. These dependencies might have their own licenses, notably:

• Nginx is released under BSD 2-clauses licence
• mod_zip is released under BSD 3-clauses licence

a7's Issues

Multiple max-age: directives in Cache-Control: response headers

Prerequisites

• Put an X between the brackets on this line if you have done all of the following:
  • Reproduced the problem
  • Checked that your issue isn't already filed: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Abouyguestelecom

Description

When requesting assets, we can see two Cache-Control headers which is valid.

Both of them declare a max-age: directives which is invalid as stated in RFC7234 § 4.2.1 :

When there is more than one value present for a given directive (e.g., two Expires header fields, multiple Cache-Control: max-age directives), the directive's value is considered invalid. Caches are encouraged to consider responses that have invalid freshness information to be stale.

Steps to Reproduce

1. make a GET request to an already deployed a7 service to download content
2. Inspect response headers

Expected behavior:

cache-control: public, max-age=31536000, immutable

or

cache-control: max-age=31536000 
cache-control: public, immutable

Actual behavior:

cache-control: max-age=31536000 
cache-control: public, max-age=31536000, immutable

Reproduces how often:

100%

Versions

all

Additional Information

Cached browsing page

Prerequisites

• Put an X between the brackets on this line if you have done all of the following:
  • Reproduced the problem
  • Checked that your issue isn't already filed: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Abouyguestelecom

Description

The browsing UI serves immutable cached content. It shouldn't.

Steps to Reproduce

1. Access the browsing UI; here goes the list of assets
2. Add an asset in the volume
3. Hit refresh
4. We see the previous list of assets, without the new one
5. Hit force-refresh
6. We now see the new asset

Expected behavior:

See the new asset upon refresh.

Actual behavior:

We're forced to force-refresh in order to see the new asset.

Reproduces how often:

Always

Versions

All

The presence of a "/>" prevents browsing an asset

Prerequisites

• Put an X between the brackets on this line if you have done all of the following:
  • Reproduced the problem
  • Checked that your issue isn't already filed: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Abouyguestelecom

Description

Whenever we try to browse an asset that contains the character <, the browsing page doesn't display the asset content.

Steps to Reproduce

1. Have a file containing <
2. Browse it

Expected behavior:

Display the asset content

Actual behavior:

Doesn't display the asset content

Reproduces how often:

Always the case

Versions

Not applicable

Additional Information

Troubleshooting led me to the following:

document.querySelector('#data').content.textContent
// => contains the actual asset content AND the rest of the browsing asset page (html, scripts)

Natural sorting order of assets in directory listings

Prerequisites

• Put an X between the brackets on this line if you have done all of the following:
  • Reproduced the problem
  • Checked that your issue isn't already filed: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Abouyguestelecom

Description

When listing a directory of assets, said assets are sorted alphabetically. Up to a point.

Steps to Reproduce

1. Browse an asset directory
2. Recall how to count in order
3. Compare with what you see

Expected behavior:

3, 4, 5, 26, 27, 28

Actual behavior:

26, 27, 28, 3, 4, 5

Reproduces how often:

Always

Versions

All

Additional Information

N/A

Template substitution is too loose

Description

The template substitution script replaces template blocks in a too loose way.

Steps to Reproduce

1. Use multiple if-env blocks in etc/nginx/conf.d/a7.conf
2. Make sure to put some custom nginx configuration between these blocks
3. Start the service by setting the corresponding env var to false
4. Which will disable these blocks
5. And also whatever was in the between (from step 2)

Reproduces how often:

Always

Versions

1.6.0

Additional Information

In the meantime, make sure not to use multiple template blocks for the same env var in etc/nginx/conf.d/a7.conf.

Compression of text/html response only

Prerequisites

• Put an X between the brackets on this line if you have done all of the following:
  • Reproduced the problem
  • Checked that your issue isn't already filed: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Abouyguestelecom

Description

No content-encoding: gzip header is present when requesting for an assets in response headers.
This result in a loss of "performance" score from Lighthouse reports when auditing our test envs.

Steps to Reproduce

1. open your favorite browser
2. open the network panel in the devtools section
3. make a request with "Accept-encoding: gzip" as a request header
4. look for "content-encoding: gzip" response header from the server
5. keep looking for eternity 'cause it won't be there

Expected behavior:

Every assets shipped by the service should be compressed by default in order to be as light as possible for response time matters.

Actual behavior:

Uncompressed content is actually served when the service is requested.
Our CDN adds the correct header on our production env.
This way it does not make this misbehaving a "production problem", but impacts every other env.

Reproduces how often:

100%

Versions

Additional Information

More info on default gzip_types conf : https://nginx.org/en/docs/http/ngx_http_gzip_module.html#gzip_types
Possible fix to add in conf :

gzip_types        text/html text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript image/x-icon image/bmp;

Clean debug traces

a7/etc/nginx/includes/internal_api.conf

Line 9 in a0eba35

add_header X-Debug "Christian Clavier" always;