sasa1977/ci

CI/CD toolkit as a library

ci's Issues

Managed docker containers

@Hanspagh

With the initial version of Sidekick in place I think we can start adding the basic support for managed containers. Here's a sketch of how I currently see the usage:

1. We'll stash everything inside the ManagedDocker namespace to avoid possible clashes with other libraries like excontainers.
2. We'll create a module called ManagedDocker.Container, which is used to start a process on the sidekick node. The client developer is responsible for starting the sidekick node and include this child:

   # on the main node
   Supervisor.start_link(
     [
       {Sidekick, {:my_sidekick_node, [ManagedDocker.Container, ...]}},
       ...
     ],
     ...
   )

3. The client developer can then start the container as ManagedDocker.Container.run(:my_sidekick_node, "docker_arg_1 docker_arg_2 ..."). We assume that docker args include the --detached option. This function should return {:ok, container_id} | {:error, output_of_docker_run}.
4. On the sidekick node, ManagedDocker.Container will maintain a logical link between the starter process and the container. If the starter process stops, it will take down the container by executing docker rm -f container_id.
5. It's also possible to manually stop the container with e.g. ManagedDocker.Container.stop(:my_sidekick_node, container_id)

We wouldn't support any other features. Developers can use other libs, such as the aforementioned excontainers to interact with their containers.

We can probably provide better UX by relying on the events API (https://docs.docker.com/engine/api/v1.41/#operation/SystemEvents). Among other things this would allow us to detect when the container stops and send some custom exit message to the owner process. But I propose we don't concern ourselves with this at the moment. Our initial goal would be to get the basic thing working, and then we can expand on this.

Thoughts?

mix ci.check error unexpected port exit

I found after run mix ci.check:

$ mix ci.check
Compiling 1 file (.ex)
starting mix compile --warnings-as-errors
CI checks took 0 seconds
** (Mix) mix compile --warnings-as-errors failed: unexpected port exit

After investigate, I found that test suite was fail in case run a command with pty: true

$ mix test
Compiling 1 file (.ex)
Excluding tags: [ci: true]

..........................................................

  1) test GenServer can use pty (OsCmdTest)
     test/os_cmd_test.exs:31
     ** (EXIT from #PID<0.507.0>) %OsCmd.Error{exit_status: nil, message: "echo 1 failed: unexpected port exit", output: nil}

....

Finished in 1.2 seconds
64 tests, 1 failure, 1 excluded

Randomized with seed 950894

After I realize that, I modify check.ex to set pty: false and it works:

$ mix ci.check
Compiling 1 file (.ex)
starting mix compile --warnings-as-errors
Compiling 1 file (.ex)
mix compile --warnings-as-errors took 0.6 seconds
starting mix dialyzer
starting mix test
starting mix format --check-formatted
starting mix docs
mix format --check-formatted took 0.5 seconds
Finding suitable PLTs
Checking PLT...
Generating docs...
[:compiler, :eex, :elixir, :kernel, :logger, :mix, :mox, :nimble_parsec, :parent, :stdlib, :telemetry]
Looking up modules in dialyxir_erlang-23.3_elixir-1.11.3_deps-test.plt
Looking up modules in dialyxir_erlang-23.3_elixir-1.11.3.plt
Looking up modules in dialyxir_erlang-23.3.plt
Finding applications for dialyxir_erlang-23.3.plt
Excluding tags: [ci: true]

Finding modules for dialyxir_erlang-23.3.plt
........View "html" docs at "doc/index.html"
...................View "epub" docs at "doc/ci.epub"
mix docs took 1.7 seconds
.Checking 18 modules in dialyxir_erlang-23.3.plt
.

  1) test GenServer can use pty (OsCmdTest)
     test/os_cmd_test.exs:31
     ** (EXIT from #PID<0.414.0>) %OsCmd.Error{exit_status: nil, message: "echo 1 failed: unexpected port exit", output: nil}

.Adding 181 modules to dialyxir_erlang-23.3.plt
................................

Finished in 1.4 seconds
64 tests, 1 failure, 1 excluded

Randomized with seed 422814
mix test took 2.5 seconds
...

I try running os_cmd command locally it works fine:

$ ./priv/os_cmd "-pty=true" "mix" "compile" "--warnings-as-errors"
started)hwoutputmCompiling 9 files (.ex)
"hwoutputmGenerated ci app
$ echo $status
0

mix test failed on macOS

Found 2 tests fail on macOS:

$ mix test
$ mix test
Excluding tags: [ci: true]

..............................

  1) test GenServer can use pty (OsCmdTest)
     test/os_cmd_test.exs:31
     ** (EXIT from #PID<0.416.0>) %OsCmd.Error{exit_status: nil, message: "echo 1 failed: unexpected port exit", output: nil}

.......

  2) test GenServer returns error on invalid directory (OsCmdTest)
     test/os_cmd_test.exs:19
     Assertion with =~ failed
     code:  assert error.message =~ "no such file or directory"
     left:  "dir failed: exec: \"dir\": executable file not found in $PATH"
     right: "no such file or directory"
     stacktrace:
       test/os_cmd_test.exs:22: (test)

........................

Finished in 1.2 seconds
64 tests, 2 failures, 1 excluded

Randomized with seed 875537

The macOS doesn't have dir command. I think we can replace with ls command to check this case. What do you think?

UPDATE: I open a new issue #8 for test GenServer can use pty case.

Does not compile from hex

Elixir: 1.11.3-otp-23
Erlang: 23.0.2

lawik@MacBook-Pro webring % mix deps.compile ci
==> ci
Compiling 9 files (.ex)
Generated ci app
spawn: Could not cd to ports/os_cmd

could not compile dependency :ci, "mix compile" failed. You can recompile this dependency with "mix deps.compile ci", update it with "mix deps.update ci" or clean it with "mix deps.clean ci"

Maybe this is something to do with Erlang 23.1 but I'm guessing not. Writing this issue while installing 23.1 so intend to try just building the library itself after that but want the right deps and Erlang installs take a bit of time. Figured I'd let you know before I forget.