Giter Site home page Giter Site logo

scalingo / nodejs-buildpack Goto Github PK

View Code? Open in Web Editor NEW

This project forked from heroku/heroku-buildpack-nodejs

8.0 8.0 23.0 124.85 MB

Buildpack for Node.js and Meteor apps.

Home Page: http://doc.scalingo.com/languages/javascript/nodejs/getting-started-with-meteor/

License: MIT License

Shell 86.82% Makefile 0.79% Ruby 7.97% JavaScript 1.16% TypeScript 3.25%

nodejs-buildpack's Introduction

Buildpack for Node.js, io.js And Meteor

nodejs

This buildpack has been developped to deploy any node.js application or an application based on the Meteor Framework (0.9+).

It is based on the original nodejs buildpack and has been updated to get an out of the box experience for Meteor.

How it Works

Apps are built via one of four paths:

  1. A regular npm install (first build; default scenario)
  2. Copy existing node_modules from cache, then npm prune, then npm install (subsequent builds)
  3. Skip dependencies (if package.json doesn't exist but server.js does)
  4. Skip cache, run npm rebuild before npm install (node_modules are checked into source control)

You should only use #3 (omitting package.json) for quick tests or experiments. You should never use #4 - it's included for backwards-compatibility and will generate warnings.

Adding node_modules directory in git repository is an antipattern.

Specify Node.js/NPM version

Node.js and NPM version are read from the engines section of the package.json file: https://doc.scalingo.com/languages/nodejs/start#specifying-a-nodejs-version

Common

  • Allows any recent version of node to be used, including pre-release versions.
  • Discourages use of dangerous semver ranges like * and >0.10.
  • Uses the version of npm that comes bundled with node.
  • Puts node and npm on the PATH so they can be executed with scalingo run.

Caching

  • Caches the node_modules directory across builds for fast deploys.
  • Meteor specific: we cache the meteor bundle to avoid spending time downloading it at each deployment
  • Doesn't use the cache if node_modules is checked into version control.
  • Runs npm rebuild if node_modules is checked into version control.
  • Always runs npm install to ensure npm script hooks are executed.
  • Always runs npm prune after restoring cached modules to ensure cleanup of unused dependencies.

For more technical details, see the compile script.

Documentation

For more information about using Node.js and buildpacks on Scalingo, see these Documentation pages:

Debug build

Meteor

By default, Meteor minify all your assets to stand in one single javascript file, if you want to make a 'debug' build (assets unminified), please defined the following environment variable:

scalingo -a app-name env-set METEOR_DEBUG_BUILD=true

Options for Meteor

If you're using the meteor framework, you just need to commit the .meteor directory at the root of your Meteor project, we'll detect and handle it to make your application work, out of the box.

If you are using Meteor β‰₯ 1.3, the flag --server-only will be used automatically to build your application for Meteor mobile integration.

Memory in the build process

Meteor build is handled by a Node.JS process which has a default limit: it can't use more than ~3.7GB of RAM. It can be too little for large Meteor applications and the build process will fail with the following error: Allocation failed.

The buildpack is fixing this issue by adding to the environment variable TOOL_NODE_FLAGS, the flag --max-old-space-size=8192 (except if this precise flag is already defined).

To update the value 8192, the variable BUILD_MAX_MEMORY can be overriden.

Starting flags for node process

For some reasons, you may want to use custom flags to run your application. The following environment variable let you customize this.

Example: (when meteor need to use 4GB of RAM)

scalingo -a app-name env-set NODE_BOOT_FLAGS="--max-old-space-size=4096"

Options for Node

Enable or disable node_modules caching

For a 'clean' build without using any cached node modules:

scalingo env-set NODE_MODULES_CACHE=false
git commit -am 'rebuild' --allow-empty
git push scalingo master
scalingo env-unset NODE_MODULES_CACHE

Caching node_modules between builds dramatically speeds up build times. However, npm install doesn't automatically update already-installed modules as long as they fall within acceptable semver ranges, which can lead to outdated modules.

Default: NODE_MODULES_CACHE defaults to true

Enable or disable devDependencies installation

During local development, npm install installs all dependencies and all devDependencies (test frameworks, build tools, etc). This is usually something you want to avoid in production, so npm has a 'production' config that can be set through the environment:

To install dependencies only:

scalingo -a app-name env-set NPM_CONFIG_PRODUCTION=true

To install dependencies and devDependencies:

scalingo -a app-name env-set NPM_CONFIG_PRODUCTION=false

Default: NPM_CONFIG_PRODUCTION defaults to true on Scalingo

Configure npm with .npmrc

Sometimes, a project needs custom npm behavior to set up proxies, use a different registry, etc. For such behavior, just include an .npmrc file in the root of your project:

# .npmrc
registry = 'https://custom-registry.com/'

Reasonable defaults for concurrency

This buildpack adds two environment variables: WEB_MEMORY and WEB_CONCURRENCY. You can set either of them, but if unset the buildpack will fill them with reasonable defaults.

  • WEB_MEMORY: expected memory use by each node process (in MB, default: 512)
  • WEB_CONCURRENCY: recommended number of processes to Cluster based on the current environment

Clustering is not done automatically; concurrency should be part of the app, usually via a library like throng. Apps without any clustering mechanism will remain unaffected by these variables.

This behavior allows your app to automatically take advantage of larger containers. The default settings will cluster 1 process on a S and M containers, 2 processes on L containers and 4 on XL containers.

For example, when your app starts:

app[web-1]: Detected 1024 MB available memory, 512 MB limit per process (WEB_MEMORY)
app[web-1]: Recommending WEB_CONCURRENCY=2
app[web-1]:
app[web-1]: > [email protected] start /app
app[web-1]: > node server.js
app[web-1]: Listening on 51118
app[web-1]: Listening on 51118

Chain Node with multiple buildpacks

This buildpack automatically exports node, npm, and any node_modules binaries into the $PATH for easy use in subsequent buildpacks.

Feedback or want to report an issue

Hacking

To make changes to this buildpack, fork it on Github. Push up changes to your fork, then create a new Scalingo app to test it, or configure an existing app to use your buildpack:

# Configure an existing Scalingo app to use your buildpack
scalingo -a app-name env-set BUILDPACK_URL=<your-github-url>

# You can also use a git branch!
scalingo -a app-name env-set BUILDPACK_URL=<your-github-url>#your-branch

Tests

The buildpack tests use Docker to simulate Scalingo environment.

To run the test suite:

make test

The tests are run via the vendored shunit2 test framework.

Debugging

To display the logged build outputs to assist with debugging, use the "echo" and "cat" commands. For example:

test() {
  local log_file var

  var="testtest"
  log_file=$(mktemp)
  echo "this is the log file" > "$log_file"
  echo "test log file" >> "$log_file"

  # use `echo` and `cat` for printing variables and reading files respectively
  echo $var
  cat $log_file

  # some cases when debugging is necessary
  assertEquals "$var" "testtest"
  assertFileContains "test log file" "$log_file"
}

Running the test above would produce:

testtest
this is the log file
test log file

The test output writes to $STD_OUT, so you can use cat $STD_OUT to read output.

nodejs-buildpack's People

Contributors

androbin avatar brandon-welsch avatar btubbs avatar colincasey avatar curzolapierre avatar danielleadams avatar ddollar avatar dependabot[bot] avatar dzuelke avatar edmorley avatar etiennem avatar frzk avatar geoffharcourt avatar github-actions[bot] avatar heroku-linguist[bot] avatar hone avatar hunterloftis avatar ipfaze avatar jkutner avatar jmorrell avatar john-scalingo avatar josegonzalez avatar joshwlewis avatar lillianzhang331 avatar mattgraham avatar oliverjash avatar schneems avatar sihamais avatar soulou avatar zeke avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

nodejs-buildpack's Issues

Make Node v20.10 available

The new NodeJS LTS version 20.10 is released. We would need to add this to our buildpack, for users to use it.

Inject NODE_EXTRA_CA_CERTS

Node use a statically compiled list of CA. See this issue: nodejs/node#4175

So we should inject:

NODE_EXTRA_CA_CERTS=/usr/share/ca-certificates/Scalingo/scalingo-database.pem

Doc during build and runtime to prevent users from manually configuring their drivers (which might not even work in some cases).

If engines.yarn is specified and engines.npm is unspecified, use yarn

Is your enhancement request related to a problem? Please describe.
Actually if there is no yarn.lock and no package-lock.json but engines.yarn is specified and engines.npm is unspecified, the buildpack use npm.

Describe the solution you'd like
If there is no yarn.lock, no package-lock.json but engines.yarn is specified and engines.npm is unspecified, the buildpack shoukd be yarn.

Describe alternatives you've considered
I maintain a fork of the project I new to deploy ( cal.com )

Additional context
https://github.com/calcom/cal.com don't use yarn.lock neither package-lock.json but require usage of yarn.

Build error: .dependencies.phantomjs: command not found

The following update of the Buildpack introduced a regression.

A customer have the following error in his deployment:

-----> Installing dependencies
       Installing node modules (package.json)
/tmp/buildpackWBcuO/lib/dependencies.sh: line 248: .dependencies.phantomjs: command not found
-----> Build failed
       
       We're sorry this build is failing!

The error is located here, the $JQ variable doesn't exist anymore:
image
In the file: /lib/dependencies.sh like the error of the deployment show

Looking at the PR of the update i saw that $JQ was changed to jq:
image

We have jq installed on all our available Stacks, so there is no need to make additional modification

We are not installing the latest stable version of node by default

Our buildpack still install v14 which is a Maintenance LTS currently.

The README of our buildpack and our documentation say that we install the latest stable version by default but it's not right:

We should install the latest stable version (v17) instead of v14 here:

local version=${1:-14.x}

https://nodejs.org/en/about/releases/

Deployment example:

-----> Installing binaries
engines.node (package.json):  unspecified
engines.npm (package.json):   unspecified (use default)

Resolving node version 14.x...
Downloading and installing node 14.18.3...
Using default npm version: 6.14.15

NPM workspaces not supported

Hello,

We have a mono-repository with multiple Node.js applications, using NPM packages: one package = one application

The structure is as follows:

/package.json β†’ defines common dependencies
/node_modules β†’ all dependencies will be downloaded here
/backend
/backend/package.json β†’ backend dependencies only
/frontend
/frontend/package.json β†’ frontend dependencies only

We have tried deploying a single workspace, by setting an NPM_CONFIG_WORKSPACE environment variable. The goal is to run all NPM commands with the option --workspace=$NPM_CONFIG_WORKSPACE, so that only the workspace dependencies are installed and cached.

However, if fails with the following error:

       Fetching source code
-----> Using buildpack 'nodejs'
       
-----> Creating runtime environment
       
       NPM_CONFIG_LOGLEVEL=error
       NPM_CONFIG_WORKSPACE=frontend
       NODE_EXTRA_CA_CERTS=/usr/share/ca-certificates/Scalingo/scalingo-database.pem
       NODE_VERBOSE=false
       NODE_ENV=production
       NODE_MODULES_CACHE=true
       
-----> Installing binaries
       engines.node (package.json):   20
       engines.npm (package.json):    10
       
       Resolving node version 20...
       Downloading and installing node 20.13.1...
       Bootstrapping npm 10 (replacing 10.5.2)...
npm ERR! Workspaces not supported for global packages
npm ERR! A complete log of this run can be found in: /tmp/npmcache.aaJTU/_logs/2024-05-28T16_10_04_024Z-debug-0.log
       Unable to install npm 10.  Does npm 10 exist?  Is npm 10 compatible with this Node.js version?
-----> Build failed
grep: Unmatched ( or \(
       
       We're sorry this build is failing!
       
       If you're stuck, please send us an email so we can help:
       [email protected]
       
       Keep coding,
       Scalingo
       
!     An error occurred during buildpack compilation
 !   Error deploying the application
 !   β†’ Invalid return code from buildpack 

It would be very helpful to have support for NPM workspaces, so I am available to help if needed.

Thank you


Some documentation:

Update upstream from v219 to v222

Is your enhancement request related to a problem? Please describe.
Similar to #85 please update the upstream heroku-buildpack-nodejs to a newer version (v222 see https://github.com/heroku/heroku-buildpack-nodejs/blob/main/CHANGELOG.md#v222-2023-09-25)

Describe the solution you'd like
Enables nodejs version 18.18.0 to be used & fixes issue where npm versions >=10 were being downgraded to version 5.x

Describe the solution you'd like

Describe alternatives you've considered

Additional context

devDependencies are installed by default

Describe the bug
The documentation (https://doc.scalingo.com/languages/nodejs/start#install-devdependencies) say that devDependencies are not installed by default unless the NPM_CONFIG_PRODUCTION environment variable is set to false. However, they seem to be installed anyway if the env var is not defined at all.

To Reproduce
Steps to reproduce the behavior:

  1. Create a new app on Scalingo
  2. Deploy a nodejs app containing devDependencies in their package.json
  3. Check the deploy log to see that devDependencies were installed
  4. Connect to the app with a one-off
  5. Notice that the devDependencies are present in the image

[Node/meteor] Not compatible with meteor 2.0

It seems that with meteor 2.0, the building process fail in the node-pre-gyp:

   npm ERR! argv "node" "/build/9329c3e7-a984-4f67-9c80-7869c955ee65/.scalingo/node/bin/npm" "install" "--production=false" "--unsafe-perm" "--userconfig" "/build/9329c3e7-a984-4f67-9c80-7869c955ee65/.npmrc"
       npm ERR! node v0.10.48
       npm ERR! npm  v3.10.10
       npm ERR! code ELIFECYCLE
       
       npm ERR! [email protected] install: `node-pre-gyp install --fallback-to-build`
       npm ERR! Exit status 8
       npm ERR! 
       npm ERR! Failed at the [email protected] install script 'node-pre-gyp install --fallback-to-build'.
       npm ERR! Make sure you have the latest version of node.js and npm installed.
       npm ERR! If you do, this is most likely a problem with the bcrypt package,
       npm ERR! not with npm itself.
       npm ERR! Tell the author that this fails on your system:
       npm ERR!     node-pre-gyp install --fallback-to-build
       npm ERR! You can get information on how to open an issue for this project with:
       npm ERR!     npm bugs bcrypt
       npm ERR! Or if that isn't available, you can get their info via:
       npm ERR!     npm owner ls bcrypt
       npm ERR! There is likely additional logging output above.

Add a prebuild hook

I'm looking for a way to customize git just before the build step by executing this kind of command: git config --global url."https://${GITHUB_TOKEN}@github.com/".insteadOf [email protected]:. So that the npm install step can point to private github repo.

heroku provides a mechanism to do so https://devcenter.heroku.com/articles/nodejs-support#heroku-specific-build-steps, and it seems to be added in the buildpack earlier this year https://github.com/heroku/heroku-buildpack-nodejs/search?utf8=%E2%9C%93&q=heroku-prebuild

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.