idsia/sacred

Sacred is a tool to help you configure, organize, log and reproduce experiments developed at IDSIA.

Sacred

Every experiment is sacred

Every experiment is great

If an experiment is wasted

God gets quite irate

Sacred is a tool to help you configure, organize, log and reproduce experiments. It is designed to do all the tedious overhead work that you need to do around your actual experiment in order to:

• keep track of all the parameters of your experiment
• easily run your experiment for different settings
• save configurations for individual runs in a database
• reproduce your results

Sacred achieves this through the following main mechanisms:

• Config Scopes A very convenient way of the local variables in a function to define the parameters your experiment uses.
• Config Injection: You can access all parameters of your configuration from every function. They are automatically injected by name.
• Command-line interface: You get a powerful command-line interface for each experiment that you can use to change parameters and run different variants.
• Observers: Sacred provides Observers that log all kinds of information about your experiment, its dependencies, the configuration you used, the machine it is run on, and of course the result. These can be saved to a MongoDB, for easy access later.
• Automatic seeding helps controlling the randomness in your experiments, such that the results remain reproducible.

Example

Script to train an SVM on the iris dataset

The same script as a Sacred experiment

from numpy.random import permutation from sklearn import svm, datasets C = 1.0 gamma = 0.7 iris = datasets.load_iris() perm = permutation(iris.target.size) iris.data = iris.data[perm] iris.target = iris.target[perm] clf = svm.SVC(C=C, kernel='rbf', gamma=gamma) clf.fit(iris.data[:90], iris.target[:90]) print(clf.score(iris.data[90:], iris.target[90:]))

from numpy.random import permutation from sklearn import svm, datasets from sacred import Experiment ex = Experiment('iris_rbf_svm') @ex.config def cfg(): C = 1.0 gamma = 0.7 @ex.automain def run(C, gamma): iris = datasets.load_iris() per = permutation(iris.target.size) iris.data = iris.data[per] iris.target = iris.target[per] clf = svm.SVC(C=C, kernel='rbf', gamma=gamma) clf.fit(iris.data[:90], iris.target[:90]) return clf.score(iris.data[90:], iris.target[90:])

Documentation

The documentation is hosted at ReadTheDocs.

Installing

You can directly install it from the Python Package Index with pip:

pip install sacred

Or if you want to do it manually you can checkout the current version from git and install it yourself:

git clone https://github.com/IDSIA/sacred.git

cd sacred

python setup.py install

You might want to also install the numpy and the pymongo packages. They are optional dependencies but they offer some cool features:

pip install numpy pymongo

Tests

The tests for sacred use the pytest package. You can execute them by running pytest in the sacred directory like this:

pytest

There is also a config file for tox so you can automatically run the tests for various python versions like this:

tox

Update pytest version

If you update or change the pytest version, the following files need to be changed:

• dev-requirements.txt
• tox.ini
• test/test_utils.py
• setup.py

Contributing

If you find a bug, have a feature request or want to discuss something general you are welcome to open an issue. If you have a specific question related to the usage of sacred, please ask a question on StackOverflow under the python-sacred tag. We value documentation a lot. If you find something that should be included in the documentation please document it or let us know whats missing. If you are using Sacred in one of your projects and want to share your code with others, put your repo in the Projects using Sacred <docs/projects_using_sacred.rst>_ list. Pull requests are highly welcome!

Frontends

At this point there are three frontends to the database entries created by sacred (that I'm aware of). They are developed externally as separate projects.

Omniboard

Omniboard is a web dashboard that helps in visualizing the experiments and metrics / logs collected by sacred. Omniboard is written with React, Node.js, Express and Bootstrap.

Incense

Incense is a Python library to retrieve runs stored in a MongoDB and interactively display metrics and artifacts in Jupyter notebooks.

Sacredboard

Sacredboard is a web-based dashboard interface to the sacred runs stored in a MongoDB.

Neptune

Neptune is a metadata store for MLOps, built for teams that run a lot of experiments. It gives you a single place to log, store, display, organize, compare, and query all your model-building metadata via API available for both Python and R programming languages:

In order to log your sacred experiments to Neptune, all you need to do is add an observer:

from neptune.new.integrations.sacred import NeptuneObserver
ex.observers.append(NeptuneObserver(api_token='<YOUR_API_TOKEN>',
                                    project='<YOUR_WORKSPACE/YOUR_PROJECT>'))

For more info, check the Neptune + Sacred integration guide.

SacredBrowser

SacredBrowser is a PyQt4 application to browse the MongoDB entries created by sacred experiments. Features include custom queries, sorting of the results, access to the stored source-code, and many more. No installation is required and it can connect to a local database or over the network.

Prophet

Prophet is an early prototype of a webinterface to the MongoDB entries created by sacred experiments, that is discontinued. It requires you to run RestHeart to access the database.

Related Projects

Sumatra

Sumatra is a tool for managing and tracking projects based on numerical

simulation and/or analysis, with the aim of supporting reproducible research.

It can be thought of as an automated electronic lab notebook for

computational projects.

Sumatra takes a different approach by providing commandline tools to initialize a project and then run arbitrary code (not just python). It tracks information about all runs in a SQL database and even provides a nice browser tool. It integrates less tightly with the code to be run, which makes it easily applicable to non-python experiments. But that also means it requires more setup for each experiment and configuration needs to be done using files. Use this project if you need to run non-python experiments, or are ok with the additional setup/configuration overhead.

Future Gadget Laboratory

FGLab is a machine learning dashboard, designed to make prototyping

experiments easier. Experiment details and results are sent to a database,

which allows analytics to be performed after their completion. The server

is FGLab, and the clients are FGMachines.

Similar to Sumatra, FGLab is an external tool that can keep track of runs from any program. Projects are configured via a JSON schema and the program needs to accept these configurations via command-line options. FGLab also takes the role of a basic scheduler by distributing runs over several machines.

License

This project is released under the terms of the MIT license.

Citing Sacred

K. Greff, A. Klein, M. Chovanec, F. Hutter, and J. Schmidhuber, ‘The Sacred Infrastructure for Computational Research’, in Proceedings of the 15th Python in Science Conference (SciPy 2017), Austin, Texas, 2017, pp. 49–56.

sacred's Issues

remove TODO items and post issues instead

Mention the Experiment / Run division in the documentation

Warn about type-changes through updates

If a config element is changed by command line and its type changes we should issue a warning.

adding config entries does not work without config scope

If you try to add a config entry to ./hello_world.py it does not work, because it has no ConfigScope

capture function for subdict of config

maybe like this:

@ex.config
def cfg():
    dataset = {'path':'/', 'variant':'foo'}
    pass

@ex.capture('dataset')
def load(path, variant):
    pass

separate mongoDB entries for experiments and runs

This would:

• remove duplication for saving stuff in the DB
• make it easier to determine which runs are from the exact same experiment
• better fit with the internal structure of sacred
• make the run DB entries more comprehendable by shrinking them

Problems might be:

• need database "joins" to get full info for a run
• makes using the entries harder

Maybe also extract out host info?

Decide on how to properly access ``info``

Right now info belongs to Run but is accessed through Experiment. This is not very clean. We could maybe use the dependency injected run to store info?

Lazy strings in updates

So far we support lazy strings (without quotes) only in direct assignments. E.g.: name=bob. But it would be nice to also have that in lists and dicts:

a=[bob,lisa,hugo]
b={name:bob,age:23}

instead of

a=["bob","lisa","hugo"]
b={"name":"bob","age":23}

This would be a big gain because entering quotes on the commandline is painful.

fixing lists in ConfigScope has unexpected behaviour

The current behaviour is to override assignments, which in this most basic case is fine:

@ConfigScope
def cfg():
    a = [1, 2]

cfg(fixed={a:[7, 8, 9]})['a']   # returns [7, 8, 9]

But as soon as you start modifying your list it gets weird:

@ConfigScope
def cfg():
    a = [1, 2]
    a += [3, 4]

cfg(fixed={a:[7, 8, 9]})['a']   # returns [7, 8, 9, 3, 4]

Similar things happen with append, insert, extend, *=, pop, remove, del, sort, reverse. This can be fixed by injecting an immutable list, which then simply ignores all the changes.

This removes the weirdness from before it also completely removes the possibility of adding something to a list afterwards. Also it seems a little inconsistent to me to have blocking dictionary that is open to changes apart from the forced ones, but have a list that is completely immutable. But for a lack of better ideas this is still the path I'll take.

Commands cannot make use of captured functions

Because there is no Run the captured functions won't be set up, so dependency injection does not work for them.

print_config: coloring for nested adds/typechanges is wrong

adding/type-changing nested values leads to wrong coloring of parent

Figure out how to reuse Experiments as modules for Meta-Experiments

Maybe allow the transfer of non-JSON objects between config scopes?

In case of multiple ConfigScopes we could allow using the non-JSON locals from one cfg in the others. This would ease implementing the network architecture module providing inputLayer and outputLayer for later usage in config scopes.

But it will complicate the implementation. Is it worth the trouble?

Rename `Module`

The term module is used for pythons internal packaging system a lot and might lead to confusions. We should use a different term. Suggestions:

• Component
• Relic
• Element
• Unit
• Ingredient
• Part
• Parcel

Capture stdout and stderr

Capture the console output of a program and write it to the database

make pymongo dependency optional

support commands for modules

print_config: when changing values in submodules, submodule is not colored

Allow for custom values provided by Modules

Don't warn about minor typechanges

there are warnings about typechanges in the config that are not really helpful:

• changes from int64 to int (numpy type to python int)
• changes from string to unicode

experiments can't be run in REPL

Because loading the sourcefile fails

Feature: Reseed a running experiment

It would be useful be able to restart the seeding at any point even while running an experiment.

Warn about new items

If a config update from the command line added a new entry we should issue a warning.

Plot crashes on exit

Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/local/lib/python2.7/dist-packages/matplotlib/_pylab_helpers.py", line 86, in destroy_all
    manager.destroy()
  File "/usr/local/lib/python2.7/dist-packages/matplotlib/backends/backend_gtk3.py", line 427, in destroy
    self.canvas.destroy()
AttributeError: FigureManagerGTK3Agg instance has no attribute 'canvas'

Add special stacktrace printing for DB entries and run_commandline

For those cases we should format the stacktrace ourselves and remove the noise

missing values report can be wrong

If you execute a command with a missing value some parameters with default values will be reported as missing too.

print_config command

Implement the print_config command

Turn main into a regular command

main could maybe just be a regular command.
If so, we would have to define what is the default command, or do away with default commands altogether.
This should be decided soon, as it might break the API.

interface for adding files to an experiment

It would be very useful if one could add files to an experiment that can be then be stored by the observers alongside the information for the run.

I can think of several different types of files to add:

1. dependencies:
2. static files:
3. dynamic files:

Dependencies

This includes additional source-files etc.. They would be part of the info about the experiment. So they could potentially be shared among many runs.

• they exist as files already, so you want to specify them as a path
• they should be added as part of the setup before initialization and of course also before the run is started
• a possible interface could be a simple ex.add_dependency("/my/path/filname.foo")

Static Files

Files that are written once and never changed, but belong to the run not to the experiment. For example a report that is written after the run is finished would fall in this category. They might be a special case of Dynamic Files.

Dynamic Files

Files that are updated continuously. So they should be updated as part of the heartbeat. Examples include current weights or log files.

• they are part of the run, so the interface should be part of the run object.
• they might not exist yet, so it would be convenient to have an interface that provides a temporary named file to write to that is also tracked and deleted afterwards. Something like f = run.create_tracked_file(filename_in_db)
• optionally you might want to create the file yourself. Then a simple run.track_file(filename, filename_in_db=None) might be sufficient.
• Remembering the file handler in client code would be tedious. So maybe there should be a method f = run.get_tracked_file(filename_in_db) that returns the file if present or otherwise creates it?

remove dependency on blessings

The fewer dependencies the better

Document experiment

• How to import and instantiate
• what decorators you can use
• how to run
• how to add observers

post-mortem debugging

Have the experiment upon failure fire up a pdb post-mortem debugger.
This should only happen on automain trigger.
This should be deactivateable with a commandline option.

document randomness

command line option for loglevel

provide an option to change loglevel on the commandline.
Maybe
-l LEVEL --logging LEVEL
where LEVEL can be either an int or the level string debug, info, warn, error, ...

include info for modules into experiment info

right now only the mainfile is added to the DB entry. If modules are from different files they are ignored

implement the help command

document observers

Enable saving weights to the Database

We should use GridFS to save big numpy arrays to the database. That would enable saving the weights of the network alongside the experiment.

document command-line interface

improve print_config to highlight changes

stacktrace pruning removes wrong entries

document logging

track libraries used and their version

consistently allow dotted path notation

for example in

• config_updates

document captured functions

Running experiments with IPython parallel can result in missing mainfile

The experiment will then have a mainfile entry like this
u'/home/greff/<ipython-input-1-762a1b9c0e46>'

There should be a way of manually setting the mainfile.

Split running an experiment into 3 parts

Right now the Experiment class is used for the set-up, and then a Run is created for initialization and running. These two steps should be split.

Syntax Errors in commandline parsing are not caught

so for example this will fail:

python hello_config.py with message="Hello World"

support for experiment 'inheritance'

Allow to reuse code from a base-experiment with some configuration.

allow non-ConfigScope options

Some people might prefer using regular dictionaries