heartsucker/node-deb

Debian packaging for Node.js projects

node-deb

Debian packaging for Node.js projects written 100% in bash.

Simple.

Installation

npm install node-deb

or

git clone ${url} && cd node-deb && npm run node-deb && sudo dpkg -i $(find . -maxdepth 1 -type f -name '*.deb' | tail -n 1)

Usage

A simple project can be packaged with the following command.

node-deb -- index.js lib/

This command will add all of the above files and directories to a Debian package as well as generate the scripts necessary to install, uninstall, start, and stop your application. On installation, via dpkg -i $your_package_name, dedicated Unix users and groups will be created and your distribution's default init system will start and monitor the process.

node-deb uses sane defaults, so the only thing you need to add to your package.json is the app/cli entrypoint. However, if you don't like these, there are two options for overrides: command line options, or the JSON object node_deb at the top level of your package.json. A full explanation of the different options can be found by running node-deb --help.

By default, if any of the following files exist, they will be included in the Debian package: package.json, npm-shrinkwrap.json and package-lock.json.

For example, here are some sample node_deb overrides. The full list can be found by running node-deb --list-json-overrides.

{
  "name": "some-app",
  ...
  "node_deb": {
    "init": "systemd",
    "version": "1.2.3-beta",
    "entrypoints": {
      "daemon": "foo.js --config /etc/some-app/config.js"
    }
  }
}

Command line options always override values found in the node_deb object, and values found in the node_deb object always override the values found in the rest of package.json.

Examples

Ex. 1

package.json:

{
  "name": "some-app",
  "version": "1.2.3",
  "node_deb": {
    "entrypoints": {
      "daemon": "app.js arg1 arg2"
    }
  }
}

cmd: node-deb -- app.js lib/

You will get:

• A Debian package named some-app_1.2.3_all.deb
  • Containing the files app.js & package.json and the directory lib
  • Installed via
    • apt-get install some-app
    • apt-get install some-app=1.2.3

On install, you will get.

• An executable named some-app
  • That starts the app with the command app.js arg1 arg2 arg3
• An upstart init script installed to /etc/init/some-app.conf
• A systemd unit file installed to /lib/systemd/system/some-app.service
• A sysv int script installed to /etc/init.d/some-app
• A Unix user some-app
• A Unix group some-app

Ex. 2

package.json:

{
  "name": "some-other-app",
  "version": "5.0.2",
  "node_deb": {
    "entrypoints": {
      "daemon": "index.js --daemon"
    }
  }
}

cmd: node-deb -u foo -g bar -v 20150826 -- index.js lib/

You will get:

• A Debian package named some-other-app_20150826_all.deb
  • Containing the files index.js, package.json, & npm-shrinkwrap.json|package-lock.json and the directories lib & node_modules
  • Installed via
    • apt-get install some-other-app
    • apt-get install some-other-app=20150826

On install, you will get.

• An executable named some-other-app
  • That starts the app with the command index.js --daemon
• An upstart init script installed to /etc/init/some-other-app.conf
• A systemd unit file installed to /lib/systemd/system/some-other-app.service
• A sysv int script installed to /etc/init.d/some-other-app
• A Unix user foo
• A Unix group bar

Ex. 3

package.json:

{
  "name": "a-third-app",
  "version": "0.10.1",
  "node_deb": {
    "init": "none",
    "dependencies": "apparmor, tor",
    "templates": {
      "postinst": "my-teplates/my-postinst-template.txt"
    },
    "entrypoints": {
      "cli": "app.js"
    }
  }
}

cmd: node-deb -- app.js lib/

You will get:

• A Debian package named a-third-app_0.10.1_all.deb
  • Containing the files index.js, package.json, & npm-shrinkwrap.json|package-lock.json and the directories lib & node_modules
  • With additional dependencies on apparmor and tor
  • Installed via
    • apt-get install a-third-app
    • apt-get install a-third-app=0.10.1
  • With the postinst script rendered from the template my-postinst-template.txt

On install, you will get.

• An executable named a-third-app
  • That starts the app with the command app.js
• No upstart, systemd, or sysv scripts
• No Unix user or group

Ex. 4

package.json:

{
  "name": "a-forth-app",
  "version": "0.10.1",
  "node_deb": {
    "init": "none",
    "dependencies": "apparmor, tor, nodejs",
    "templates": {
      "postinst": "my-teplates/my-postinst-template.txt"
    },
    "entrypoints": {
      "cli": "app.js"
    }
  }
}

cmd: node-deb --no-default-package-dependencies -- app.js lib/

You will get:

• A Debian package named a-forth-app_0.10.1_all.deb
  • Containing the files index.js, package.json, & npm-shrinkwrap.json|package-lock.json and the directories lib & node_modules
  • With dependencies on apparmor, tor and nodejs only. No default dependencies added
  • Installed via
    • apt-get install a-forth-app
    • apt-get install a-forth-app=0.10.1
  • With the postinst script rendered from the template my-postinst-template.txt

On install, you will get.

• An executable named a-forth-app
  • That starts the app with the command app.js
• No upstart, systemd, or sysv scripts
• No Unix user or group

&c.

Note: Removal via apt-get purge will attempt to remove the user and group defined in the Debian package. This can have serious consequences if the user or group is shared by other applications!

node-deb can Debian-package itself. Just run npm run node-deb.

More complete examples can be found by looking at test.sh and the corresponding projects in the test directory.

Options

This section incldues addtional details about the more advanced functionality of node-deb

--install-strategy

The install strategy determines how dependencies in node_modules are included in the final Debian package.

• auto: This attempts to take a minimal subset of package from the node_modules director using npm ls --prod. If this is not possible, it falls back to the copy method. On install, if node_modules is present, it runs npm rebuild --prod. If node_modules is not present, it runs npm install --prod. If npm is not present, it issues a warning that dependencies may be missing and continues with the Debian package installation.
• copy: This runs a blind cp -rf on the node_modules directory and includes everything in the Debian package. No actions are taking during package installation.
• npm-install: This option does not include the node_module in the Debian package and runs npm install --production as part of the postinst maintainer script.

Requirements

• dpkg
• fakeroot
• jq

These are all available through apt and brew.

Dev Requirements

Tests are run via docker. This is also available through apt and brew.

Support

node-deb only officially supports the currently supported versions of Debian and Ubuntu (LTS). This includes both for building packages and deploying packages. At the time of this update, this translates to Debian Wheezy through Stretch and Ubuntu Trusty through Xenial. Care has been taken to ensure this packages correctly on macOS, and macOS specific issues should still be reported.

Contributing

Please make all pull requests to the develop branch.

Please make sure all pull requests pass the test suite locally.

node-deb's Issues

Package generation is slow

The package generation is very slow. The performance bottleneck is the loop which copies the files at
https://github.com/ehartsuyker/node-deb/blob/develop/node-deb#L451

For every file it spawns a new dirname process which takes a long time if you have 10k js files in your modules.

I don't understand why no cp -Rv is used here....

md5sum will break on OSX if a file has '=' in the path

write_md5sum() {
  # Using $@ and not $1 because of files with whitespace
  declare file="$@"

  # Debian/Ubuntu
  if hash md5sum 2>/dev/null; then
    declare -r sum=$(md5sum "$file" | cut -d ' ' -f1)
  # OSX
  elif hash md5 2>/dev/null; then
    # RIGHT HERE <----------------------
    declare -r sum=$(md5 "$file" | awk -F '=' '{ printf $NF }' | tr -d ' ')
  # Oh dear...
  else
    die 'Unable to find suitable md5 sum program'
  fi

  file=$(echo "$file" | sed -e "s/$(escape "$deb_dir")//" | sed -e 's:^\.::' -e 's:^/*::')
  echo "$sum $file" >> "$deb_dir/DEBIAN/md5sums"
}

Have 'node_deb.start_command' default to 'scripts.start'

Suggestion from a coworker. Need to see if it makes sense.
@jeffhiltz, can you elaborate?

Swap the ordering of checks in `prerm` becuase error message is still displayed if no init is selected

Should probably just copy what was done in postinst.

Failed to install... Need documentation update

$ npm install -g node-deb
$ node-deb
Error: Cannot find module 'jq'

sudo npm install -g jq

recursive calls?
gyp WARN EACCES user "root" does not have permission to access the dev dir "/usr/lib/node_modules/jq/node_modules/contextify/.node-gyp/8.1.2"
gyp WARN EACCES attempting to reinstall using temporary dev dir "/usr/lib/node_modules/jq/node_modules/contextify/.node-gyp"

Need to be run twice to do the job

$ node-deb
Error: Cannot find module 'commander'
$ sudo npm install -g commander
$ node-deb
Error: Cannot find module 'async'
$ sudo npm install -g async
$ node-deb
Error: Cannot find module 'rimraf'
Error: Cannot find module 'winston'
Error: Cannot find module 'colors'

/usr/share/$app/app not removed by dpkg

dpkg: warning: while removing simple, directory '/usr/share/simple/app' not empty so not removed

This is likely because of npm install or other shenanigans. Force kill it on purge/removal.

Entry file is not marked as executable.

I use

"node_deb": { "entrypoints": { "daemon": "server.js " } }

server.js is not marked executable and so install fails.

Consider making a source package instead of directly building a .deb with dpkg-builddeb

Whilst dpkg-builddeb certainly works, you are doing an awful lot of stuff manually and not able to use the multitude of useful packaging utilities by not ultimately building the package with dpkg-buildpackage. Did you reject this approach?

Add option for no init script, upstart, systemd, or both

installing nodejs dependencies

Hello,

there is no npm install of dependencies upon startup, could you add this too ?

`node` not found

On linux it seems you need to use nodejs everywhere, e.g. for the npm start commands, etc, while on the mac node is the program to call. What's the best way to handle this? Not code on mac? Create symlink on mac? Create symlink on linux? When to create it? I guess the best way is to create the symlink on Mac as that is dev only if you need deb-packages.... No real issue, just want your thoughts.

File ownership issues

First reported by @empierre on issue #16. New ticket for organization.

---

also had issues of npm unpak issues, had to add this to correct the issue (clean cache and install as an unprivillegied user:

preinst:
npm cache clean

postint:
chown 'www-data:www-data' '/var/www/'
chown -R 'www-data:www-data' '/usr/share/mydomoathome'

npm_install() {
cd "/usr/share/$1/app"
if [ -e 'node_modules' ]; then
rm -rf 'node_modules'
fi
sudo -u www-data npm install --production
}

suggestion related to package dependencies.

Hi,
while using your module i faced a complication that the package requires legacy nodejs installed in the system, while i prefer to build one from source, whenever i have to install the .deb file built from the module i have to use --ignore-depends flag to bypass dependency failure. Now this is fine when i am manually installing the package, this may not work if for instance i will add a repository to my package manager to provide update automatically, which i am planing to do. so maybe not include forced nodejs as package dependency and ask user to add nodejs if they want in their package.json file. if he feels its necessary.

      `if [ -z "$package_dependencies" ]; then
               package_dependencies=$(jq -r '.node_deb.dependencies' package.json)
              if [[ "$package_dependencies" == 'null' ]]; then
                     package_dependencies="nodejs" #remove nodejs or have some alternate solution.
             else
                     package_dependencies="nodejs, $package_dependencies" #remove nodejs or have some alternate solution.
             fi
       fi`

Maybe i am missing something, What do you think? Thanks You.

Check that entrypoints are exectuable and have a shebang line

postrm script fails to remove group

Figure out why, write tests.

Dockerize test suite

The tests have gotten so unwieldy they are almost useless. vagrant and the guest additions take forever.

Similarly, remove the actual node package and create a fake .deb that "provides" it.

Add md5sums to DEBIAN

debian own repository & package size

I've tried to setup my own debian package repository, but it seems there is an issue in size.

packages available here:
http://www.e-nef.com/domoticz/mdah/

$ du -s :
896 node-mydomoathome-0.0.31.deb

Installed-Size: 1164

error:
Les NOUVEAUX paquets suivants seront installés :
MyDomoAtHome
0 mis à jour, 1 nouvellement installés, 0 à enlever et 1 non mis à jour.
Il est nécessaire de prendre 913 ko dans les archives.
Après cette opération, 1 192 ko d'espace disque supplémentaires seront utilisés.
Réception de : 1 http://www.e-nef.com/domoticz/mdah/ MyDomoAtHome 0.0.31 [913 kB]
915 ko réceptionnés en 0s (1 277 ko/s)
E: Impossible de récupérer http://www.e-nef.com/domoticz/mdah/./node-mydomoathome-0.0.31.deb Taille incohérente

E: Impossible de récupérer certaines archives, peut-être devrez-vous lancer apt-get update ou essayer avec --fix-missing ?

Running `node-deb -h` in directories without a package.json results in errors

root@ops-jenkins1:~# node-deb --help
jq: package.json: No such file or directory
Usage: node-deb [opts] -- file1 file2 ...
...

Solution: move the $(jq ...) commands below the option parsing.

Add defaults for command line args in the node_deb object in package.json

Downloading package dependencies at install time is not ideal

I see that you moved the running of npm install from package time to install time. (Thanks for the warnings - no time was wasted) I understand the reason this was done but it causes some new problems. Both options for npm install are bad:

• package time - if any modules are platform dependent they will be built incorrectly.
• install time - some dependencies may not be available or may download different versions.

I haven't tried this, but I think the solution is to npm install at packaging time and then just do an npm rebuild at install time.

Update README to include System V addition

Yup. Forgot that.

Feature request: Be able to set owner of directories

Hi, great package!

It would be really swell to have a simple way to chown all or some directories to the installing user so that tmp dirs etc can be setup within the package.

Regards,
Tarjei

Binary doesn't run unless run from install directory

Add a chdir to the binary directly. Possibly remove it from the upstart script.

Faster tests

I was lazy and used three VMs. This could have been done with two.

Installing to absolute path

First off, thanks for a very helpful package. How do I install a file to e.g. /etc/myconfigfile.json?

Add log messages to node-deb

Because a coworker "wants to know the program didn't hang when watching Jenkins output."

Move systemd services to /lib/systemd/system

Currently they are in /etc/systemd.

Differ between cli and service invokation

I would like to use the command line feature to configure my app like so

$myapp --update --config --file --or --whatever

It could be there is a different script in the package.json that will be run on cli or so.

The main reason is the use case where the user runs the cli without parameters. Here I would like to display program usage as per usual, but there is no way(?) to differ between service starting and starting from cli.

Cannot Get Real Working Directory

I have built a Node.js CLI packaged with node-deb. It fails to create a tmp folder because of permissions errors because it thinks that the current working directory is /usr/share/$app/app instead of /home/$user. I see why it is doing this but I can't figure out a workaround (aside from running with sudo). Any help would be appreciated.

I am using var tmpDir = process.cwd() + '/tmp' to set the tmp directory location. I've also tried simply var tmpDir = 'tmp' and var tmpDir = __dirname + '/tmpDir' but it always comes out with this directory: /usr/share/$app/app. Not /hone/$user/tmp. It works just fine when installed normally with NPM.

Crash if description is empty

When fixing a bug I added a reproducer project and discovered that if the description is empty in the package.json file like so {"description": ""} node-deb crashes. Not super-critial, and people should really have a description. The error message was pretty good also so it was easy to just add a description. 👍 for good error messages.

systemd.service file is executable

hi,
the systemd service file in /etc/systemd/… that node-deb creates seems to be executable. systemd ignores it but issues a warning like this

Configuration file /etc/systemd/system/mirror-mirror.service is marked executable. Please remove executable permission bits. Proceeding anyway.

the template itself is not executable so this may happen during packaging.

thanks

Error removing package

It's me the issue spammer again. Found another small issue

When sudo apt remove myapp I get this consistently.

Failed to stop myapp.service\x7d.service: Unit myapp.service\x7d.service not loaded.
myapp wasn't even running!

The problem seems to be a spurious closing brace at prerm:12

Deprecate `node_deb_` prefix to template variables

I came at this from Ansible-land where everything has to be prefixed because of the global namespace. This isn't the case here, so deprecate the old variables and start using new ones.

npm rebuild fails when compiling native packages

hi,

first of all thank you very much for working on this. i very much like the fact that i don’t have to get into debian packaging :).

i want to use node-deb with mirror-mirror. the debian-pkg branch contains the necessary changes. packaging itself works fine but npm rebuild --production in postinst fails. it seems like npm is quite picky about the build environment and considering that i compile code from – somewhat – untrusted sources that might be a good thing.
unfortunately this means that my sqlite dependency will throw an exception when npm tries to build it.

this can easily be fixed by using --unsafe-perm for npm install/rebuild but i’m not sure if this would be smart. anyway… i think this needs a solution. maybe it’s a good idea to chown the node_modules directory to the package user than executing rebuild/install and chown-ing it back.

i’ll try to submit a pull request if you favor any solution.

thanks!

Update docs to warn users that `devDependencies` will be included if they exist in `node_modules`

As noted in #34

Make all files pass shellcheck linting with zero errors or warnings

Spurious error printout

I'm getting a spurios error printout whe installing my package using dpkg --install myapp_0.3.15_all.deb

See below

(Reading database ... 178426 files and directories currently installed.)
Preparing to unpack myapp_0.3.15_all.deb ...
Removed symlink /etc/systemd/system/multi-user.target.wants/myapp.service.
Failed to stop myapp.service\x7d.service: Unit myapp.service\x7d.service not loaded.
myapp wasn't even running!
Unpacking myapp (0.3.15) over (0.3.14) ...
Setting up myapp (0.3.15) ...
Directory 'node_modules' exists. Running 'npm rebuild'

> [email protected] install /usr/share/myapp/app/node_modules/websocket
> (node-gyp rebuild 2> builderror.log) || (exit 0)

sh: 1: cannot create builderror.log: Permission denied
[email protected] /usr/share/myapp/app/node_modules/minimist
[email protected] /usr/share/myapp/app/node_modules/websocket
[email protected] /usr/share/myapp/app/node_modules/websocket/node_modules/debug
[email protected] /usr/share/myapp/app/node_modules/websocket/node_modules/debug/node_modules/ms
[email protected] /usr/share/myapp/app/node_modules/websocket/node_modules/nan
[email protected] /usr/share/myapp/app/node_modules/websocket/node_modules/typedarray-to-buffer
[email protected] /usr/share/myapp/app/node_modules/websocket/node_modules/typedarray-to-buffer/node_modules/is-typedarray
[email protected] /usr/share/myapp/app/node_modules/websocket/node_modules/yaeti
[email protected] /usr/share/myapp/app/node_modules/node-deb
Created symlink from /etc/systemd/system/multi-user.target.wants/myapp.service to /etc/systemd/system/myapp.service.

Note the sh: 1: cannot create builderror.log: Permission denied. What is it trying to build? Why can't it write to the log, etc. Everything seems to work fine in spite of this error message. Normally I try to build things so that no errors are emitted when everything works as expected. Please advice.

pure systemd systems

Hello,

Ubuntu 15.x need the preset to activate the service on startup, so I have modified my postinst to match it:

elif hash systemctl 2> /dev/null; then
if [[ "$init_type" == 'auto' || "$init_type" == 'systemd' ]]; then
{
systemctl preset "$service_name.service" &&
systemctl enable "$service_name.service" &&
systemctl start "$service_name.service"

Set `NODE_ENV` environment

As services started from packages built by node-deb typically will run in production mode, the NODE_ENV=production environment should also be set. This is used by e.g. express to suppress stacktraces and other security sensitive information.

It could either be set inside node_deb in conjunction to computing the start_command string or we could juse prepend NODE_ENV=production in templates/executable.

`devDependencies` getting included

For some reason e.g. istanbul code coverage tools gets included even though it's in the devDependencies section of the package.json file. Not supported? Can I help out somehow?

package itself brakes adding files

This line change to working dir so adding files or dirs will not work in node-deb.
https://github.com/heartsucker/node-deb/blob/develop/templates/executable#L6

Add package.json override for architechture, change --arch to --architecture

Move help docs to man-page

Fix or remove references to /var/run

These aren't used anywhere and are actually broken in the systemd template.

use "files" from package.json if CLI args don't specify file paths

This might be too annoying with all the white space problems that exist. Needs investigation.

Use `npm install --production` to install only dependencies

Per default, npm install also installs the devDependencies. If you don't want to have these dependencies, you need to add the --production switch.

The documentation explains this switch:

By default, npm install will install all modules listed as dependencies in package.json.

With the --production flag (or when the NODE_ENV environment variable is set to production), npm will not install modules listed in devDependencies.

For the differences between the dependencies and the dev dependencies see the explanation of package.json:

dependencies
Please do not put test harnesses or transpilers in your dependencies object. See devDependencies, below.

devDependencies
If someone is planning on downloading and using your module in their program, then they probably don't want or need to download and build the external test or documentation framework that you use.
In this case, it's best to map these additional items in a devDependencies object.
These things will be installed when doing npm link or npm install from the root of a package, and can be managed like any other npm configuration param. See npm-config for more on the topic.
For build steps that are not platform-specific, such as compiling CoffeeScript or other languages to JavaScript, use the prepublish script to do this, and make the required package a devDependency.

Preserve root environment variables

The variables in /etc/default/{{app}} didn't load when the upstart script execute, so i add the '-E' option to preserve it.

# start the process
script
  . "/etc/default/uchuva"
  exec sudo -E -u 'uchuva' 'bin/uchuva'
end script

Is there a better way to handle?

& in variable gets replaced with template string

It looks like node-deb has a problem with '&'s in the start command, I wanted to redirect stderr in my start command and 2>&1 came out like this:


2>{{ node_deb_start_command }}1

It seems to include the template string where the '&' was.

TravisCI support

I have been shipping crap software lately. Let the FOSS gods save me from myself as I offer up my humble package to the pedestal of judgement that is TravisCI.