aisopous/checkpoints.jl

A package for dynamically checkpointing program state

Checkpoints

Checkpoints.jl allows packages to register checkpoints which can serialize objects to disk during the execution of an application program, if the application program configures them.

A minimal working example consists of the package:

module MyPackage

using Checkpoints

MODULE = "MyPackage"

__init__() = Checkpoints.register(MODULE, ["foo", ])

function foo(x)
    with_checkpoint_tags(:foo1 => 1, :foo2 => 2) do
        checkpoint(MODULE, "foo", :data => 2x)
    end
    return 2x
end

end

and the application program:

using Checkpoints

Checkpoints.config("MyPackage.foo", "./path/to/checkpoints")

for i in 1:2
    with_checkpoint_tags(:iteration => i) do
        MyPackage.foo(1.0)
    end
end

which results in recorded checkpoints at

./path/to/checkpoints/iteration=1/foo1=1/foo2=2/MyPackage/foo.jlso
./path/to/checkpoints/iteration=2/foo1=1/foo2=2/MyPackage/foo.jlso

You can use index_checkpoint_files to get an index of the files, which is a Tables.jl table and so can e.g. be passed to DataFrame (and then you can do things like groupby etc):

julia> using DataFrames

julia> DataFrame(index_checkpoint_files("./path/to/checkpoints/"))
2×6 DataFrame
 Row │ prefixes        checkpoint_name  iteration   foo1        foo2        checkpoint_path
     │ Tuple…          SubString…       SubString…  SubString…  SubString…  PosixPath…
─────┼────────────────────────────────────────────────────────────────────────────────────────────────────────
   1 │ ("MyPackage",)  foo              1           1           2           ./path/to/checkpoints/iteration=…
   2 │ ("MyPackage",)  foo              2           1           2           ./path/to/checkpoints/iteration=…

or worked with directly:

julia> [checkpoint_path(out) for out in index_checkpoint_files("./path/to/checkpoints/") if out.iteration=="1"]
1-element Array{FilePathsBase.PosixPath,1}:
 p"./path/to/checkpoints/iteration=1/foo1=1/foo2=2/MyPackage/foo.jlso"