Comments (10)
Where would cached files be stored if we don't use Docker caching?
That is what we have to figure out 😆
The problem with Docker caching is that it is linear. A changed env var before the build phase will break the cache for every command below (which is fairly common on Railway because we pass in the github commit sha to every build). It would also be nice to benefit from the cache even if the files change. For example you can still benefit from the .next
cache even if the source files you are building from change.
We will see though. Can optimize for the Docker cache now and see where the issues are and if/what we can do to improve it.
from nixpacks.
I'd love to work on this, I have some ideas about how this can be made extremely modular.
from nixpacks.
I was initially thinking of allowing providers to specify what files are necessary for the install step. This would allow us to take advantage of the Docker layer cache if these files do not change between builds.
fn install_files(&self, _app: &App) -> Result<Vec<String>> {
Ok(Vec::new())
}
Alternatively install_cmd
could be extended to return a list of (command, files)
pairs, but ideally if the provider does not specify any install files, all files are presumed to be needed.
For example, in the NodeJS case, only the package.json
file is necessary to run npm install
. If the NPM provider was extended to
impl Provider for NpmProvider {
// ...
fn install_files(&self, _app: &App) -> Result<Vec<String>> {
Ok(vec!["package.json".to_string()])
}
fn install_cmd(&self, _app: &App) -> Result<Option<String>> {
Ok(Some("npm install".to_string()))
}
// ...
}
Then when creating the Dockerfile we can first only copy over the install files (package.json
) file, run the install step (npm install
), and then copy over the remaining files. If only the source code changes between builds then the dependencies installed with npm can be pulled from the cache.
Obviously, there are more advanced use cases with different languages and frameworks. Please share them here or bring up any concerns you see with this approach.
from nixpacks.
How would this handle for example the .next
folder with NextJS? That isn't needed at install time but isn't (but can be used) at build time.
PS: This approach shouldn't error if the file doesn't exist.
from nixpacks.
Hmm at the moment it wouldn't handle that case and caching for specific frameworks (next, remix, etc) will require a bit more involved solution. Happy to hear thoughts/suggestions here.
from nixpacks.
I do think this should happen in multiple steps. First step to just get deps cached between builds (easy win). Next step can be to give providers and API for more advanced fine grain cache control.
from nixpacks.
Hmm at the moment it wouldn't handle that case and caching for specific frameworks (next, remix, etc) will require a bit more involved solution. Happy to hear thoughts/suggestions here.
I kinda struggled with this a tiny bit when I took a swing at the Rust and Zig providers
I think we need a way to have postbuild
actions which allow you to insert 0 - N "COPY" calls into the dockerfile to move them into the right place
from nixpacks.
I think we need a way to have postbuild actions which allow you to insert 0 - N "COPY" calls into the dockerfile to move them into the right place
I really like the idea of doing it like this. Not sure if COPY is the write call, more should be ADD but can be disclosed later.
from nixpacks.
The providers sort of have that ability now. They can define files that are necessary for each phase and only those files are copied over when running that phase so we can make ues of the build cache. However it doesn't work all the time and there are definitely some holes and I would like to see it fleshed out some more.
Honestly I think we will need some semi-custom caching solution in the not too distant future because relying on the Docker layer cache isn't great and it would be nice to have more control over what is cached.
from nixpacks.
Where would cached files be stored if we don't use Docker caching? IMO, just using Docker caching is a better idea.
from nixpacks.
Related Issues (20)
- Nixpacks config 'onlyIncludeFiles' should support custom destination paths HOT 2
- Issue building multi-arch platform images HOT 7
- Support specific node versions instead of just majors HOT 5
- Unable to get prisma 5 with nixpacks openssl HOT 21
- SIGINT not passed to application HOT 5
- "Error: Cannot find module './pnpm.cjs'" when building Nx monorepo with NIXPACKS_NX_APP_NAME set HOT 2
- Running commands in Python container throws errors for incompatible ldd version HOT 1
- Option to skip `COPY . /app` HOT 6
- PHP 8.0 default Laravel build failing HOT 5
- add asdf support to python + poetry HOT 7
- Stream did not contain valid UTF-8 HOT 3
- Able to set Docker network and custom hosts during build HOT 7
- error: undefined variable 'pnpm-8_x' HOT 1
- Output environment variables during build
- Add NIXPACKS_VERSION to environment
- Projects containing read-only files cannot be built HOT 4
- Elixir nixpack shouldn't run migrations in build step HOT 1
- Support running as non root user
- nixpacks cargo workspaces problem HOT 1
- Force nixpack. to use latest version of Bun? HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nixpacks.