andygrunwald/perseus

Local git mirror for your PHP (composer) project dependencies that works together with Satis.

---

Local git mirror for your PHP (composer) project dependencies that works together with Satis.

perseus is a successor out of and drop-in replacement for Medusa.

Table of contents

• Whats wrong with Medusa?
• Features
• Installation
  • From binary
  • From docker image
  • From source
• Usage
  • Add a new package
  • Mirror all packages
  • Update all mirrored packages
  • Show me the version of perseus
• Configuration
  • Command line flags
  • medusa.json configuration file
• Drop-in replacement
• Development
  • Build
  • Build the docker image
  • Unit tests
  • Release a new version
• Project background
  • The name "perseus"
• Credits

Whats wrong with Medusa?

Nothing. Really. Medusa is a great software. It works well for many people and companies. Thanks to Sebastien Armand, Instaclick Inc. and all others who have helped and contributed to this project. But it has its limitations, flaws and disadvantages like:

• Very poor documentation (as mentioned in the readme)
• Nearly no error handling (for API requests to Packagist, system commands like triggering git, etc.)
• Long mirror/update runs, due to sequential procedure and single threaded nature (long runtimes can ruin a fast development workflow)
• Stops the complete mirror/update run, if one package/url/composer.json is faulty and stops updating other packaging in the list
• Need to implement auxiliary processes to make it work in a bigger engineering team like self-service to add new or remove old packages, monitoring and reliabilities

perseus was born out of the motivation to eliminate these points.

Features

• Drop-in replacement for Medusa
• Fully documented
• Concurrency and usage of multiple threads for faster mirror/update runs
• Serious error handling
• Reporting of faulty packages or packages that can't be processed

Installation

From binary

We offer pre-compiled binaries of perseus for easy usage.

1. Checkout our Releases page and choose the latest release
2. Select your system architecture and operating system of your choice
3. Download and extract the archive
4. Switch to the extracted directory and fire a perseus version
5. Have fun

From docker image

perseus is available as Docker image at andygrunwald/perseus. To download the image, fire:

$ docker pull andygrunwald/perseus

Commands can be executed like a normal installation in the format:

$ docker run andygrunwald/perseus <Command-Name> [Flags] <Parameter>

E.g. the add command:

$ docker run andygrunwald/perseus add --with-deps symfony/console /var/config/medusa-small.json

Inside the container, example medusa and satis configurations from the .docker folder are available in the path /var/config. Those can be used to play around.

From source

To install perseus from source, a running Golang installation is required.

$ go get github.com/andygrunwald/perseus
$ cd $GOPATH/src/github.com/andygrunwald/perseus
$ go get ./...
$ make install
$ $GOPATH/bin/perseus

Usage

Add a new package

The add command will mirror the given down to disk (with dependencies if requested) and adds the package into the configured Satis.json file.

Usage:

$ perseus add <Package-Name> [Config-File]

Examples:

$ perseus add "twig/twig"
$ perseus add --with-deps "symfony/console"
$ perseus add --with-deps "guzzlehttp/guzzle" /var/config/medusa.json

Mirror all packages

The mirror command will mirror all configured packages from medusa.json down to disk (incl. dependencies) and adds all packages into the configured satis.json file.

Usage:

$ perseus mirror [Config-File]

Examples:

$ perseus mirror
$ perseus mirror /var/config/medusa.json

Update all mirrored packages

The update command will update all mirrored packages that are located at disk and update them to the latest state. To find all packages it will do a search in the path configured at repodir.

Usage:

$ perseus update [Config-File]

Examples:

$ perseus update
$ perseus update /var/config/medusa.json

Show me the version of perseus

Print the version number incl. build details of perseus.

Usage:

$ perseus version

Examples:

$ perseus version
perseus v0.1.0-Alpha-4C8098CE24FA56AC7DFD512EA756F95AD9D941EB darwin/amd64 BuildDate: 2017-05-09T16:35:17Z

Configuration

perseus has two different kinds of configurations:

1. Process settings (via command line flags)
2. medusa.json configuration file

Command line flags

Several settings can be set by command line flags:

• Flag --config: Path to the medusa.json configuration (default: medusa.json)
• Flag --numOfWorkers: Number of worker used, when a concurrent process is started (default: number of available CPUs)

medusa.json configuration file

Perseus is mainly configured with a JSON file (like Medusa). Here is a minimalistic example:

{
    "repositories": [
        {
            "name": "myvendor/package",
            "url": "git@othervcs:myvendor/package.git"
        },
        ...
    ],
    "require": [
        "symfony/symfony",
        "monolog/monolog",
        ...
    ],
    "repodir": "/tmp/perseus/git-mirror",
    "satisurl": "http://php.pkg.company.tld/git-mirror",
    "satisconfig": "./satis.json"
}

In the next sections an explaination of the single configuration parts can be found.

repositories

A list of custom packages that are not available on the configured https://packagist.org/. Per each repository, a name and a url must be given.

require

A list of repositories to mirror down to disk.

The packages will be searched on the given Packagist instance. Per default the standard instance https://packagist.org/ will be used.

repodir

Directory to write all repositories to.

This directory needs to be writable.

satisurl

URL of the future satis installation.

This URL will be used to prefix all package URLs in the final satis configuration.

satisconfig

At the end of the run, perseus write a valid satis configuration file. In this setting a valid path to a writeable satis configuration is expected. Further more the file needs to be exists before and it needs to be a valid satis configuration.

preseus itself will only touch and edit the repositories section in this satis configuration. All other parts of the file will be untouched.

Example satis.json

{
    "archive": {
        "directory": "dist",
        "format": "tar",
        "prefix-url": "http://php.pkg.company.tld/packages/",
        "skip-dev": true
    },
    "homepage": "http://php.pkg.company.tld/packages/",
    "name": "private php package repositories",
    "providers": true,
    "repositories": [
        {
            "type": "git",
            "url": "http://php.pkg.company.tld/git-mirror/symfony/debug.git"
        },
        ...
    ],
    "require-all": true
}

Drop-in replacement

We are a Drop-in replacement for Medusa. We have the same command structure and functionality.

But in one point we are not compatible: Logging. We log way more information during the process as the original Medusa.

Be aware: If you parse the logs of the original Medusa process, you might have to adjust your scripts.

Development

Build

To build the application, fire

$ make build

A binary, called perseus should appear in the same directory.

Build the docker image

To build the docker image on your own machine, fire

$ docker build -t andygrunwald/perseus .

Unit tests

A running go installation is required to execute unit tests. To execute them, run:

$ make test

Tip: If you plan to contribute via a Pull Request, the use of unit tests is encouraged.

Release a new version

We use goreleaser to build and ship our releases. Further more we follow semantic versioning. Here is how you create and ship a ner version:

1. Export a GITHUB_TOKEN environment variable with the repo scope selected. This will be used to deploy releases to your GitHub repository. Create yours here.

   $ export GITHUB_TOKEN="YOUR_TOKEN"

2. GoReleaser uses the latest Git tag of your repository. Create a tag:

   $ git tag -a v0.1.0 -m "First release"

   If you don't want to create a tag yet but instead simply create a package based on the latest commit, then you can also use the --snapshot flag.

3. Now you can run GoReleaser at the root of your repository:

   $ goreleaser

4. That's it! Check your GitHub project's release page.

Project background

The name "perseus"

Naming projects is hard. I often struggle with this. The name needs to be simple, "catchy" and easy to remember.

In this case it was easy. Medusa was part of the greek mythology. I started looking in this direction and found Perseus. Checkout Perseus with the Head of Medusa for more details..

Credits

The perseus logo was created by @mre.
The original Gopher was designed by Renee French.
Go Gopher vector illustration by Hugo Arganda (@argandas)
Hosted at the gopher-vector repository.
The Medusa vector art was adjusted from Amanda Downs work from the Noun Project.
The perseus font is called Dalek created by K-Type.

perseus's Issues

Provide a Benchmark of Medusa vs. Perseus

A benchmark between Medusa and Perseus should be a good idea :)

Add command fails on windows

When I try to add twig using

perseus add "twig/twig"

I get a warning

Failed to retrieve information about package "twig/twig" from Packagist. Called https://packagist.org/packages%5Ctwig%5Ctwig.json. Error: Expected a return code within 2xx for package "twig/twig". Got 404

It seems that the %5C escapes are not accepted by packagist anymore as

https://packagist.org/packages%5Ctwig%5Ctwig.json

returns the requested json

Expected Behavior

Should just work

Current Behavior

Fails to retrieve the package description.

Possible Solution

Steps to Reproduce (for bugs)

1. run perseus add "twig/twig" on windows.

Context

Your Environment

• Version used (perseus version):
• Operating System and version:
• Medusa configuration:
• Satis configuration:

More sophisticated logging?

Logging is important for us.
Should we implement a 3rd party package like logrus for more deeper logging?

https://github.com/sirupsen/logrus

Unstable behaviour of resolving dependencies

The dependency resolver of perseus seems to be unstable.
If you have a list of packages that need to resolve the dependencies, there is a high chance that a different number of dependencies will be resolved.
But this should be stable and return the same amount every time.

Expected Behavior

Resolving the same amount of dependencies every time.

Current Behavior

Different amount of resolved dependencies

Possible Solution

Not sure yet, what the detailed issue is or how to solve it.

Steps to Reproduce (for bugs)

perseus ships multiple configuration files with: ./.docker/medusa-small.json, ./.docker/medusa-medium.json and ./.docker/medusa-big.json

Those can be used for testing.
How to reproduce this issue:

1. Clone this repository
2. Fire make build && ./perseus mirror ./.docker/medusa-medium.json --numOfWorkers=98
3. Let it run, wait for the message Start concurrent download process and check the amountPackages=201-Output in the log message.
4. Do it multiple times for the same configuration file and compare the amount of resolved packages to download
5. Expected behavour should be that the amount is every time the same for the same configuration file

Context

How has this issue affected you?

This can lead to missing packages for local repositories.
If packagist is complete deactivated, it is not able to install a PHP application via composer, because not all deps can be resolved.

A few numbers

Numbers of today from my local network:

➜  perseus git:(master) ✗ make build && ./perseus mirror ./.docker/medusa-small.json --numOfWorkers=98
go build -ldflags "-X main.Version=`git rev-parse --abbrev-ref HEAD 2>/dev/null` -X main.CommitHash=`git rev-parse --short HEAD 2>/dev/null` -X main.BuildDate=`date +%FT%T%z`" github.com/andygrunwald/perseus/cmd/perseus
INFO[2017-05-15T13:06:22+02:00] Using configuration file                      path="./.docker/medusa-small.json"
INFO[2017-05-15T13:06:22+02:00] Running "mirror" command
INFO[2017-05-15T13:06:22+02:00] Configuration                                 error="No repositories defined/configured."
INFO[2017-05-15T13:06:23+02:00] Start concurrent download process             amountPackages=33 amountWorker=98
...

➜  perseus git:(master) ✗ make build && ./perseus mirror ./.docker/medusa-medium.json --numOfWorkers=98
go build -ldflags "-X main.Version=`git rev-parse --abbrev-ref HEAD 2>/dev/null` -X main.CommitHash=`git rev-parse --short HEAD 2>/dev/null` -X main.BuildDate=`date +%FT%T%z`" github.com/andygrunwald/perseus/cmd/perseus
INFO[2017-05-15T13:04:25+02:00] Using configuration file                      path="./.docker/medusa-medium.json"
INFO[2017-05-15T13:04:25+02:00] Running "mirror" command
INFO[2017-05-15T13:04:25+02:00] Configuration                                 error="No repositories defined/configured."
INFO[2017-05-15T13:04:26+02:00] Start concurrent download process             amountPackages=201 amountWorker=98
^C

➜  perseus git:(master) ✗ make build && ./perseus mirror ./.docker/medusa-big.json --numOfWorkers=78
go build -ldflags "-X main.Version=`git rev-parse --abbrev-ref HEAD 2>/dev/null` -X main.CommitHash=`git rev-parse --short HEAD 2>/dev/null` -X main.BuildDate=`date +%FT%T%z`" github.com/andygrunwald/perseus/cmd/perseus
INFO[2017-05-15T15:45:42+02:00] Using configuration file                      path="./.docker/medusa-big.json"
INFO[2017-05-15T15:45:42+02:00] Running "mirror" command
INFO[2017-05-15T15:45:42+02:00] Configuration                                 error="No repositories defined/configured."
INFO[2017-05-15T15:45:43+02:00] Error while resolving dependencies of package  error="API returned status code 404: Expected a return code within 2xx for package "thecodingmachine/nodejs-installer". Got 404" package="thecodingmachine/nodejs-installer" responseCode=404
INFO[2017-05-15T15:45:43+02:00] Error while resolving dependencies of package  error="API returned status code 404: Expected a return code within 2xx for package "psr/cache-implementation". Got 404" package="psr/cache-implementation" responseCode=404
INFO[2017-05-15T15:45:44+02:00] Start concurrent download process             amountPackages=373 amountWorker=78
^C

➜  perseus git:(master) ✗ make build && ./perseus mirror ./.docker/medusa-big.json --numOfWorkers=98
go build -ldflags "-X main.Version=`git rev-parse --abbrev-ref HEAD 2>/dev/null` -X main.CommitHash=`git rev-parse --short HEAD 2>/dev/null` -X main.BuildDate=`date +%FT%T%z`" github.com/andygrunwald/perseus/cmd/perseus
INFO[2017-05-15T14:34:41+02:00] Using configuration file                      path="./.docker/medusa-big.json"
INFO[2017-05-15T14:34:41+02:00] Running "mirror" command
INFO[2017-05-15T14:34:41+02:00] Configuration                                 error="No repositories defined/configured."
INFO[2017-05-15T14:34:42+02:00] Error while resolving dependencies of package  error="API returned status code 404: Expected a return code within 2xx for package "thecodingmachine/nodejs-installer". Got 404" package="thecodingmachine/nodejs-installer" responseCode=404
INFO[2017-05-15T14:34:42+02:00] Error while resolving dependencies of package  error="API returned status code 404: Expected a return code within 2xx for package "psr/cache-implementation". Got 404" package="psr/cache-implementation" responseCode=404
INFO[2017-05-15T14:34:42+02:00] Start concurrent download process             amountPackages=362 amountWorker=98
^C

If you have more test results, feel free to post your results here as well

Your Environment

• Version used (perseus version): master (rev 1dcc4e8)
• Operating System and version: macOS Sierra v10.12.4
• Medusa configuration: Located in ./docker/-Folder
• Satis configuration: Located in ./docker/-Folder

Logo for perseus

This project misses a logo. Anyone ready to help here?

The name perseus and the background of Medusa provides already ideas. Checkout the wiki pages of Perseus with the Head of Medusa and Medusa.

This project is written in Go(lang). One idea (maybe a crazy one) would be a gopher that hold a medusa head (in the same comic style) in his hand (like Perseus).

Images / graphics about gophers can be found here:

• golang-samples/gopher-vector
• egonelbre/gophers
• shalakhin/gophericons

But i am open for any other suggestion, too!