ocurrent/solver-service

An OCurrent service for solving opam dependencies

OCurrent

OCurrent allows you to specify a workflow / pipeline for keeping things up-to-date.

For example, the pipeline shown above fetches the head of a GitHub repository's master branch, builds it, runs the tests, and deploys the binary if the tests pass. When a new commit is pushed, it runs the pipeline again.

Another use might be to keep the GitHub build status of each PR in your Git repository showing the result of fetching, building and testing the PR's head commit. If the head commit changes, the result must be recalculated.

An OCurrent pipeline is written using an OCaml eDSL. When OCurrent evaluates it, it records the inputs used (e.g. the current set of open PRs and the head of each one), monitors them, and automatically recalculates when an input changes.

Larger uses of OCurrent include the OCaml Docker base image builder and ocaml-ci, which is the CI that tests this repository itself.

Documentation

The OCurrent docs contains user documentation and examples. In particular, you might like to start by reading about the example pipelines or how to write your own plugins.

For technical docs, see the API Documentation.

Licensing

OCurrent is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

solver-service's Issues

Adapt solver-service for ocaml-ci and ocaml-docs-CI

This is the steps.

• Upgrading solver-service to ocaml-ci one.
  • the solver in solver-service is from ocaml-ci solver at some point. The last code of ocaml-ci's solver don't change much.
• Adapt the protocole of solver-service for more opam-repository.
• Split schema.capnp file to 2 two file solve.capnp and build.capnp
• Connect ocaml-multicore-ci using the protocol from solver-service-api library (vendored). ocurrent/ocaml-multicore-ci#37
• Connect ocaml-ci to submit solves ocurrent/ocaml-ci#634

Discussion around the improvement of solver-service

slight re-design/improvement

Actual limitations

• Solving different list of opam repositories at the same time
• Cancelling a job (the job remain until the end)
• No caching system, the same request is always resolved again and again.

Improvement

graph LR
    A[A.Opam commits\nupdate\nlock]--->|Resquest| B[Cache of \nResponse]
    B[Cache of \nResponse] --->|Response| A[Opam commits\nupdate\nlock]

    C[Verify \nto \nresolve] ---> |Response|B[Cache \nof \nResponse]
    B[Cache \nof \nResponse] --->|Resquest| C[Verify \nto \nresolve]

    C[Verify \nto \nresolve] -->|Resquest| D[Distribute\nby\nplatform\nto worker-processes]
    C[Verify \nto \nresolve] --> GG[Get the names of\nthe opam urls packages]
    GG[Get the names of\nthe opam urls packages] --> D[Distribute\nby\nplatform\nto worker-processes]
    D[Distribute\nby\nplatform\nto\n workers] --> |Response|C[Verify \nto \nresolve]
    
  
    D --> H[wker1]
    D --> E[wker2]
    D --> F[wker3]
    D -->.......
    D --> G[wker n]

stages

• The first stage use a lock to update all opam url commits in the request if there's a new hash commit.
• The second stage, try to get the a result from the cache. Each new response of a request will be stored.
• The third stage is important to improve the efficiency of the Solver-service. This stage is trying to verify if there's a change about the dependencies in the request that are related to the change in the opam urls commits. An example, if there is a new hash of opam-repository and the hash bring changes which are not related to the dependencies or request, so resolving is unnecessary for the request if there's already a response in the cache. It's remove, looking for the oldest commits after a resolve to prevent the cache of builds in the pipeline of a CI. To be little bit concrete here this command will be used git -C {clone-path of opam-repo} diff {old-hash} {new-hash} --- packages/{name} to know if the changes implies a package. To be more efficient, the stage can have caching also.
• The last stage is to get all the opam urls packages, when the request is distributed to workers, those packages also are sent to the workers. The packages is sent by their opam urls hash commit in order to facilitate a caching. A mechanism could be added to manage the workers like when a job(request) is canceled, the process could be killed at the time.

cancelling a job

It's possible when distributing the workers to manage their state. A worker could be in different state:

• Waiting (waiting for work)
• Cancelled (when a cancelled job/request that gives some work to do, kills the process behind the worker)
• Running

This is the plan :

• Solve the issue about cancelling job ( kill the internal-worker process as necessary).
  • Cancel a job when its switch is off. #54
• Different list of different types of opam-repository.
  • Replace epoch-lock by opam-repository update lock
  • Use a map for the different list of opam-repositories in the internal-workers
  • Sqlite cache of opam-repositories could be used by the controller of the internal-workers
• Cache and prevent resolving again.
  • build or rebuild as needed.

EDITED: to add the plan.

Tracking the file descriptor leaks issues

This is for tracking the current file descriptor leaks issues:

• the pipes are not close after a child process finished its job and solved by this PR #39.
• Using the git-unix during the update of opam-repository and that could be solved by this PR mirage/ocaml-git#590
• When we are getting the oldest opam-repository commit after doing a solve, we can end-up with lot pipes opened at same time. This is PR #47 that is supposed to solve it.
• close the all unused file descriptors opened by git-unix #49

cc @tmcgilchrist

Application and Operational Metrics

solver-service should report on application and operational level metrics via prometheus/grafana.

• Production dashboard in grafana on https://status.ci3.ocamllabs.io
• Instrument solver-worker to export application level stats
• Add clarke energy monitoring support (might not be supported on ARM64?)
• Setup scheduler connection to export stats (it already supports prometheus)
• Setup server export of stats from linux-arm64 machine

Non-exhaustive list of tasks to cover the monitoring of solver-service application.

README example fails

The example at the start of the README fails:

$ dune exec -- ./examples/main.exe --package=yaml --version=3.0.0 capnp://...
main.exe: [INFO] Connecting to tcp:127.0.0.1:7000...
main.exe: [INFO] Generating new private key...
main.exe: [INFO] Generated key with hash sha-256@ILYjkOefHoCXjRuGt6smyJ8LaGgR_iseUXnHKXziULA
main.exe: [INFO] Connecting to 127.0.0.1:7000...
main.exe: [INFO] Doing TLS client-side handshake...
main.exe: [INFO] Connected to tcp:127.0.0.1:7000
solve-local: internal error, uncaught exception:
             Failure("hd")

It looks like there are several bugs here:

1. The example client uses List.hd to get the result for the single requested platform. But if there is no solution then the list will be empty.
2. The default compiler version for the example solve is 4.14.0, but the default opam-repository hash is too old to contain that version.
3. The log from the solver isn't shown until the end, and then only if the solver itself fails. So in this case, there's just a long delay where it doesn't seem to be doing anything, then a crash.

It's also odd that the command is called solve-local, even though it's doing a remote solve.