composer/mirror

Mirroring script for packagist.org and composer repositories metadata in general

Dependency Management for PHP

Composer helps you declare, manage, and install dependencies of PHP projects.

See https://getcomposer.org/ for more information and documentation.

Installation / Usage

Download and install Composer by following the official instructions.

For usage, see the documentation.

Packages

Find public packages on Packagist.org.

For private package hosting take a look at Private Packagist.

Community

Follow @packagist or @seldaek on Twitter for announcements, or check the #composerphp hashtag.

For support, Stack Overflow offers a good collection of Composer related questions, or you can use the GitHub discussions.

Please note that this project is released with a Contributor Code of Conduct. By participating in this project and its community you agree to abide by those terms.

Requirements

Latest Composer

PHP 7.2.5 or above for the latest version.

Composer 2.2 LTS (Long Term Support)

PHP versions 5.3.2 - 8.1 are still supported via the LTS releases of Composer (2.2.x). If you run the installer or the self-update command the appropriate Composer version for your PHP should be automatically selected.

Binary dependencies

• 7z (or 7zz)
• unzip (if 7z is missing)
• gzip
• tar
• unrar
• xz
• Git (git)
• Mercurial (hg)
• Fossil (fossil)
• Perforce (p4)
• Subversion (svn)

It's important to note that the need for these binary dependencies may vary depending on individual use cases. However, for most users, only 2 dependencies are essential for Composer: 7z (or 7zz or unzip), and git.

Authors

• Nils Adermann | GitHub | Twitter | [email protected] | naderman.de
• Jordi Boggiano | GitHub | Twitter | [email protected] | seld.be

See also the list of contributors who participated in this project.

Security Reports

Please send any sensitive issue to [email protected]. Thanks!

License

Composer is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• This project's Solver started out as a PHP port of openSUSE's Libzypp satsolver.

mirror's Issues

How to ensure mirror syncing has been done?

As title, I've set up a Composer v2 mirror syncing on my own server with supervisor process yesterday.

When I check my mirror storage size today, it seems that the size has about 83 MB.

It seems that this mirror syncing work is very slow and I think it depends on networking issues.

Is it possible to check the mirror syncing has been done or other approaches about checking the mirror syncing progress?

Thanks :).

Will it break the existing mirror if i migrate to this ?

Hi There

I am currently running a mirror in india (https://packagist.in/) and it uses https://github.com/webysther/packagist-mirror

and it runs mirror for V1. but i want to support V1 & V2

so i am planing on using this script. i can create my own home page but will this break existing users if i migrate to this script ?

Packages.json missing V2 update not working

./mirror.php --resync -v

...----------------------------------------------------RRRRRRRRRRRR??????
Downloaded 602900 files

After this I see there is only p2 folder created without packages.json.

If I try run --v2 it just fails.

root@packagist:/usr/local/www/mirror # ./mirror.php --v2
Mirror v2 job failed at 2021-05-04 18:38:54
[Exception] Too many retries, could not update /p2/rebing/graphql-laravel~dev.json as the origin server returns an older file (1620109991, expected 1620153524)

Fatal error: Uncaught Exception: Too many retries, could not update /p2/rebing/graphql-laravel~dev.json as the origin server returns an older file (1620109991, expected 1620153524) in /usr/local/www/mirror/mirror.php:278
Stack trace:
#0 /usr/local/www/mirror/mirror.php(329): Mirror->{closure}(Object(Symfony\Component\HttpClient\Response\CurlResponse), Array)
#1 /usr/local/www/mirror/mirror.php(151): Mirror->downloadV2Files(Array)
#2 /usr/local/www/mirror/mirror.php(796): Mirror->syncV2()
#3 {main}
  thrown in /usr/local/www/mirror/mirror.php on line 278

Config

return [
    // directory where metadata files will get saved
    'target_dir' => __DIR__.'/public'/* TODO */,
    // user agent describing your mirror node, if possible include domain name of mirror, and a contact email address
    'user_agent' => 'xxxxxxxx'/* TODO Mirror for foo.com ([email protected]) */,
    // source repository URL
    'repo_url' => 'https://repo.packagist.org',
    // source repository hostname (optional, will guess from repo_url)
    //'repo_hostname' => 'repo.packagist.org',
    // source API URL
    'api_url' => 'https://packagist.org',
    // how many times the script will run the mirroring step before exiting
    'iterations' => 120,
    // how many seconds to wait between mirror runs
    'iteration_interval' => 5,
    // set this to false if you do not run the --v1 mirror job, to ensure that the v2 will then take care of syncing packages.json
    'has_v1_mirror' => false,
];

System

root@packagist:/usr/local/www/mirror # php -v
PHP 7.4.18 (cli) (built: May  2 2021 01:15:47) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
------------------------------------------------------------
root@packagist:/usr/local/www/mirror # cat /etc/os-release
NAME=FreeBSD
VERSION=12.2-RELEASE
VERSION_ID=12.2
ID=freebsd
ANSI_COLOR="0;31"
PRETTY_NAME="FreeBSD 12.2-RELEASE"
CPE_NAME=cpe:/o:freebsd:freebsd:12.2

The mirror.php doesn't download the packages.json

After following the step to accomplish the Composer V2 mirror server, it seems that the mirror.php doesn't download the packages.json on specific targeted directory defined on mirror.config.php file.

It only has the p2 directory on above directory. And I also use the supervisor service to let mirror.php run on background process.

Could anyone know this problem? Thanks.

Checkup since a new package is unknown in the mirror

Hey,

I'm extensively using this code to run a private self hosted packagist mirror.

Today a new package has been added to packagist: https://packagist.org/packages/fakerphp/faker at 2020-10-27 13:44 UTC.
Unfortunately, I'm unable to download the package through composer when I run composer require fakerphp/faker

The error:

$ composer update --no-scripts
Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Root composer.json requires fakerphp/faker, it could not be found in any version, there may be a typo in the package name.

Potential causes:
 - A typo in the package name
 - The package is not available in a stable-enough version according to your minimum-stability setting
   see <https://getcomposer.org/doc/04-schema.md#minimum-stability> for more details.
 - It's a private package and you forgot to add a custom repository to find it

Read <https://getcomposer.org/doc/articles/troubleshooting.md> for further common problems.

As soon as I remove

    {
      "packagist": false
    },

from the composer.json, making a fallback to the public repository I can see from the outgoing network, the package can be installed succesfully.

My mirror.config.php:

<?php

return [
    // directory where metadata files will get saved
    'target_dir' => './public',

    // user agent describing your mirror node, if possible include domain name of mirror
    'user_agent' => 'Maximum Composer Mirror',

    // source repository URL
    'repo_url' => 'https://repo.packagist.org',

    // source repository hostname (optional, will guess from repo_url)
    //'repo_hostname' => 'repo.packagist.org',

    // source API URL
    'api_url' => 'https://packagist.org',

    // how many times the script will run the mirroring step before exiting
    'iterations' => 120,

    // how many seconds to wait between mirror runs
    'iteration_interval' => 30,

    // set this to false if you do not run the --v1 mirror job, to ensure that the v2 will then take care of syncing packages.json
    'has_v1_mirror' => false,
];

and in supervisor only --v2 is configured and seems to be okay-ish as ps aux | grep php shows:

maximum   7480 58.4 14.1 1661384 1157900 ?     R    20:00   9:03 php /var/www/packagist-mirror-official/mirror.php --v2

Last Sunday I disabled --v1.

ls -al public/p | wc -l shows 108853 directories and ls -al public/p2 | wc -l shows 99456 directories.
So there seems to be a discrepancy there although I don't know if p and p2 should yield the same result.

Is there anything that I'm missing here or does this mirroring-repo need an update after the composer 2 release? Or does the currently running --v2 mirror simply need more time since --v1 is disabled since Sunday?

The problem of php mirror.php --gc command

After looking at this code snippet, it seems that this will remove the files are on p folder.

That is, it will clean the outdated Composer V1 mirror files.

What about the outdated Composer v2 mirror files? (These files will be on p2 folder) Are they need to be removed?