Understanding the purpose of `files`

[zekenie]

Understanding the purpose of `files`

[nalgeon]

The docs are in the early stages, and I'm not sure about the final structure. So it's probably best if you ask questions as they come up, and I'll answer them and eventually come up with a doc :)

[nalgeon]

I'm afraid I don't get it, sorry :)

Anyway, if you want persistent volumes, you can prepare and run a persistent container with volumes using docker run with --volume argument, or using docker compose up with proper configuration.

Then, on the Codapi side, you can setup a sandbox to execute commands in this persistent container. Here is an example of a sandbox configuration file (e.g. configs/commands/python.json):

{
    "run": {
        "engine": "docker",
        "entry": "main.py",
        "steps": [
            {
                "box": "python",
                "action": "exec",
                "command": ["python", "/path/to/main.py"]
            }
        ]
    }
}

Where:

• python is the name of the running Docker container.
• /path/to/ is the path inside the container that you've mapped to the path on the host machine using Docker volumes.

[zekenie]

i see, that clarifies things, thank you!

[TravisRoad]

in internal/engine/docker.go:168, when the filename equals to "", it just assigns the Entry value (default main.sh or main.py ...) to the name.
https://github.com/nalgeon/codapi/blob/3fd59120f57cdf381b86fd1dc9f9c28b91679530/internal/engine/docker.go#L168C1-L179C2

// writeFiles writes request files to the temporary directory.
func (e *Docker) writeFiles(dir string, files Files) error {
	var err error
	files.Range(func(name, content string) bool {
		if name == "" {
			name = e.cmd.Entry
		}
		path := filepath.Join(dir, name)
		err = fileio.WriteFile(path, content, 0444)
		return err == nil
	})
	return err
}

---

In this way files: { "": "someCode()" } just equals to files: { "main.py": "someCode()" }. And I believe you can upload as many files as you want through the files field.

[nalgeon]

Sure, files can contain data. Here is an example of a data.txt file that accompanies a Python script main.py:

data.txt:

one
two
three

main.py:

for line in open("data.txt"):
    print(line)

API request:

{
    "sandbox": "python",
    "command": "run",
    "files": {
        "main.py": "for line in open(\"data.txt\"): print(line)",
        "data.txt": "one\ntwo\nthree"
    }
}

API response:

{
  "id": "python_run_c152676b",
  "ok": true,
  "duration": 687,
  "stdout": "one\n\ntwo\n\nthree\n",
  "stderr": ""
}

[zekenie]

Fantastic, thanks for the explanation. Do you want any pull requests on docs stuff as I discover things?

[zekenie]

is there any way to give a container access to volumes. (I know in many cases this would be a terrible idea)

[nalgeon]

Could you please describe the use case in more detail?

[zekenie]

It's a little hard to explain, but I'm building a prototype of this idea. It's a chat product and a repl. Every channel is supposed to be a directory on the filesystem. I want to be able to have chats like this. I also think this format will be great for LLMs.

person0: Hey @ person1 what do you think we should do first?
person1: let's fetch some data from an api
person0: (js) await fetch('...')
person0: now what?
person1: let's read the sqlite db in this dir and compare each result to the data from the api, then we can make a chart with vegalite?
person0: ...

Importantly, when i execute code, it's code I write on my computer. I'm thinking that every channel (or folder) could have a docker file which plugs into a codapi sandbox. The "standard lib" let's do a set of things but if you want to make a particular channel have special behavior you can just extend the docker file. I also want to be able to have the commands read/write from a shared volume if configured to do so

[zekenie]

I'm going to make the "standard" runtime have access to an empty sqlite db, vega lite, http, etc. but if someone wanted to create a channel with another sandbox that did something in python, that'd be fine

[zekenie]

i see! i think i thought that python was an image name, but it's a container name and codapi doesn't handle booting your containers. that makes sense

[nalgeon]

Note the action = exec parameter, it makes a big difference. It means that the step will be executed using docker exec, so in this case python is the name of the existing (and running) container.

The default value for action is different (action = run). It means that the step will be executed using docker run, so in this case python would be the name of the box defined in configs/boxes.json that maps to an image.