Allow overriding the output file for a target

[peacememories]

Allow overriding the output file for a target

[lydell]

Thank you for the thorough answers, and good luck on your setup!

I’m going to close these two issues for now, but feel free to comment more if you find something new and we can re-open or create a new issue in the future.

[lydell]

Hi!

I don’t quite understand what you’re trying to do. Could you provide a lot more detail? Feel free to talk about this and #69 here, I think that’ll be easier than having two conversations going on at the same time. I used to work with Django a couple of years ago, but I’m a bit rusty.

[peacememories]

I'll try, but it might be more confusing since there is a lot of weird details in our project. 😅

Our Project

We're developing a relatively ancient Django project with a lot of different apps and quite a few different ways of authoring front-end assets. Some coffee-script files are built during rendering, some Angular projects have their build results checked in - in short, it's a mess. Some time ago we added Elm to the mix, first with Vite, then we switched to elm-watch because of the excellent hot-reloading and its standalone nature.

How we build

Our projects are currently built using a Dockerfile which, among other things, runs elm-watch make. During development, the resulting Docker container is booted, and the project directory is mounted into the container, to allow reloading of the Python code when editing.

This comes with a few problems. The most important here is that, if we'd store the elm-watch output in the project directory, the result would be shadowed by the mounted directory. We also have a bunch of backend developers who do not code Elm and therefore are unlikely to run elm-watch locally. This means they might just not have any of the needed js files compiled, causing errors in the application.

Our current workaround is to specify some folder in /opt as the build target, which will not be overridden by our mount.

But this means that users now need to run elm-watch hot inside the container. This has been the state for a while. It's also generally not ideal since developers are not guaranteed to rebuild their image very often, leaving them with very outdated Elm artifacts...

Some changes - adopting Nix

This went okay for a while, but now we're trying to introduce Nix into our development environment to alllow people to develop without entering the Docker container, and to improve the stability and build speed of our project.

I am currently dogfooding this setup, and running into the problem that when starting elm-watch hot it fails because it - obviously - has no access to /opt. And even if it did, the server inside Docker wouldn't see it 😅

So now I am faced with the problem of matching all of this together. I want to emit my build files into the project directory so I can develop from outside Docker, but I want people to be guaranteed a compiled Elm app. Also, as mentioned above, the high chance of outdated artifacts is not ideal in general.

Guaranteeing the Elm build always runs

So the obvious question is - why do we not just guarantee the elm compiler is always running? The simplest option would be to start it together with our Django server. But I don't like that very much for a few reasons:

• It would be the first companion process in our project
• We would need to make sure it correctly restarts if it crashes (which happens from time to time when very gnarly elm errors are encountered, and sometimes because of weird elm-stuff corruption). Also, restarting it manually would be harder, which might also be necessary if some state gets messed up.
• Error reporting is made worse. Right now developers see elm build errors directly in the console. If we run the server in the background this output would be pretty hidden
• We could instead rely on the in-browser errors, but we also lose the "open in editor" functionality when running elm-watch in the container and the editor outside of it.

Building the JS files once on each request

This is my currently favored solution: I would still have developers run elm-watch hot on their machine normally, getting all the error messages and editor-opening functionality. But to guarantee that all developers always get a working, up to date, elm app, I would trigger an elm-watch make for the appropriate targets for each request made to the server. Somewhat like the on-demand Coffeescript compiler. Since Elm builds incredibly fast I think this should be feasible.

Doing this with the current capabilities of elm-watch does not seem feasible though. I can either insert the hot-reload code by using elm-watch hot, which also starts the whole server and blocks the port, or I can do a one-off build, but this does not give me a way to insert the hot-reload code.

Also I was thinking that I would like to either pipe the built js directly into Python, or store the output in a tmp directory for the one-off build, but I realize now that this might not make much of a difference, so this issue is definitely less important to me than #69

[lydell]

We would need to make sure it correctly restarts if it crashes (which happens from time to time when very gnarly elm errors are encountered, and sometimes because of weird elm-stuff corruption).

FYI, a goal of elm-watch is that you should never need to restart it. I have a bunch of code for dealing with elm crashing, and I even watch the elm-stuff/ folder being deleted and cause a re-compile then. So I’m sad to hear stuff get messed up for you where you need to restart 😞

---

At work, all of us are fullstack (more or less – some are leaning more towards backend, and some more towards frontend). Everyone runs one command to start all backend stuff, and in another terminal tab we run another command to start all frontend stuff (including elm-watch). So until now I haven’t considered the case where there’s backend-only people who refuse to install/start frontend stuff.

What I would try, is probably writing a Django thing that compiles on request, like you say. I would probably do that in Python and not involve elm-watch. You could read the elm-watch.json file to tell which input files you need to compile to create the requested output file. Then call elm make directly, and not bother with elm-watch make. To make it extra robust, make sure in your Python code that you don’t call elm make in parallel – that can cause errors during the Elm packages installation phase.

Then, for frontend people, make it possible to opt-in to running elm-watch hot on their own. Maybe have an environment variable that disables the compile-on-request stuff and just have Django serve static files and let elm-watch do its thing.

[lydell]

TIL about docker compose watch – maybe that can help in your situation, not sure.

[peacememories]

Sorry, I completely forgot that you posted an answer here - sadly, docker compose watch would probably not help for us for two reasons:

1. Rebuilding the container takes too long to make the feedback loop viable
2. We actually want to move away from relying on docker compose in the long term, and are moving to Arion (a nix wrapper of docker compose) in the short term.

FYI, a goal of elm-watch is that you should never need to install it. I have a bunch of code for dealing with elm crashing, and I even watch the elm-stuff/ folder being deleted and cause a re-compile then. So I’m sad to hear stuff get messed up for you where you need to restart 😞

The main reason this happened for us is when some part of elm/docker/vscode broke ownership in the elm-stuff folder because of the bind mount, and the elm builds started failing. This had to be manually fixed, and elm-watch had to be restarted there. I wouldn't beat myself up about this, overall the resilience of elm-watch is leagues ahead of any other tool I'm aware of.

At work, all of us are fullstack (more or less – some are leaning more towards backend, and some more towards frontend). Everyone runs one command to start all backend stuff, and in another terminal tab we run another command to start all frontend stuff (including elm-watch). So until now I haven’t considered the case where there’s backend-only people who refuse to install/start frontend stuff.

One thing I forgot that makes this even more complicated for us is that not only are there people not aware/willing to engage with the frontend pipeline, the same development stack might also be used for completely different applications (it's a very janky multiservice application), which depend on some of the frontend in some cases, but are in completely different repositories 😅 we really know how to construct a torment nexus here.

What I would try, is probably writing a Django thing that compiles on request, like you say. I would probably do that in Python and not involve elm-watch. You could read the elm-watch.json file to tell which input files you need to compile to create the requested output file. Then call elm make directly, and not bother with elm-watch make. To make it extra robust, make sure in your Python code that you don’t call elm make in parallel – that can cause errors during the Elm packages installation phase.

You're probably right... I'm not super happy with it, because that means you need to start the servers differently depending on whether you want to work on the frontend or not, but it's also better in the sense that you then don't have a hot reload build trying to find a nonexistant hot reload server and showing the failing status in the corner.

Maybe I can do some trickery to automatically disable on-render compiling when the file is new enough or something weird like that 😅

Thanks for taking the time to look into this and for your suggestions :)