hufnagel / uncvmfs Goto Github PK
View Code? Open in Web Editor NEWThis project forked from ic-hep/uncvmfs
License: Other
This project forked from ic-hep/uncvmfs
License: Other
unCVMFS ======= unCVMFS is a tool for unpacking an entire CVMFS repository into a local directory in a time efficient manner. Configuration Files ------------------- /etc/uncvmfs.conf - The main configuration file. /etc/sysconfig/uncvmfs - Settings for the cron job (uncvmfs_cron). /etc/cron.d/uncvmfs - The main cron entry. [email protected] - Systemd timer unit (CentOS7+) Basic Principles & Usage ------------------------ unCVMFS manages three directories, the repo, store & database dirs. The repo dir is the main output directory, this is populated with the standard contents of the target CVMFS repository (i.e. on completion this dir will look like /cvmfs/<repo_name> would on a standard CVMFS system). The store dir is where the main data files for the filesystem are kept. This is stored in a similar format to a CVMFS server repository and is generally of little interest. As the repo is poulated, files are downloaded into the store dir and if possible hardlinked into the repo dir. This allows for inherent deduplication, but requires that that the repo and store dir _must_ be on the same host file system. Finally the database (db) dir holds the current unCVMFS (catalog.db) and CVMFS (various <sha1_sum>.db) catalogs in sqlite3 format. unCVMFS uses a single config file to describe the repositories to download. The config file is in INI/ConfigParser format with each section represting a repository. See the provided default config file in /etc/uncvmfs.conf for the full config file documentation. unCVMFS would normally run regularly from a Cron job or systemd timer to keep the given repo directory up-to-date. The provided example crontab file has a default run time of every two hours for the repositories listed in /etc/sysconfig/uncvmfs. This file enables no repositories by default as most of the configuration will be site-specific. For systemd based platforms, the timer can be enabled with "systemctl enable uncvmfs@<repo>.timer", which by default will run uncvmfs every two hours for the given repo. On a clean installation, it's recommended that the initialisation of the repository is done manually until the repository is synced and then the cron job/timer can be enabled. The initial run on a large repository, such as ATLAS or CMS (.cern.ch) may take one to two days. It is also recommended that unCVMFS be run twice for the initialisation just to ensure that all files are correctly initialised (downloading millions of files over HTTP is seemingly prone to bursts of errors every so often). The syncronisation process can be interrupted & restarted by ctrl+c for any reason, it will also stop with a suitable message if an unrecoverable error (such as running out of disk space) occurs. Once a repository is successfully synced, the cron job/timer should be enabled to keep it up-to-date. An example of running unCVMFS to sync the CMS area using 16 threads is (assuming cms.cern.ch is a section in the config file): uncvmfs -vv -n4 /etc/uncvmfs.conf cms.cern.ch Use of a multiplexing terminal manager such as screen is advised for operations that may take a long time over remote connections. Maintenance & Problems ---------------------- A tool for doing repository maineance, uncvmfs_tool is included. Care should be taken to disable any cron jobs/timers that may access the database while using this, although the database will be locked to avoid any accidental damage. There are two main maintenance operations that can be done - fsck & tidy; other informational operations are also available (see uncvmfs_tool --help for a complete list). The uncvmfs_tool file system checker (fsck) checks through the repo directory and the catalog in parallel looking for any inconsistencies. Any differences found will be corrected or (in the case of missing files) marked for repair on the next main uncvmfs run. unCVMFS should be run on the repository after an uncvmfs_tool fsck operation completes (if errors were reported). In general there should be no need to run fsck unless it is suspected that the repo has been modified directly or if there are problems with repo files. The tidy operation scans through the store directory and removes any files which only have only one link. These occur when a file is superceded in the catalog and can't be deleted immediately for performance reasons. If any further repository corruption occurs unCVMFS may exit with an sqlite exception. This indicates a problem with the hardware running unCVMFS or a bug in the code. If a hardware problem is not suspected then please report it to the developers. Running uncvmfs with the debug level at maximum (-vv flags) should return the sha1 hash of the faulty catalog. Remove the <sha1_sum>.db file from the database directory and the matching entry from the catalogs table of catalogs.db to recover. If the problem persists then this indicates a faulty/unexpected upstream catalog.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.