bitwalker/distillery

Simplify deployments in Elixir with OTP releases!

Distillery

NOTE: This project is DEPRECATED. It's functionality was largely integrated into Mix itself as part of the Elixir 1.9 release. While there are some small differences between Distillery and Mix Releases, the vast majority of users should find it quite easy (and very much worthwhile) to migrate. For all others, this repo exists for you to fork and use as you see fit, but it is no longer actively maintained. If there are folks interested in taking over maintenance of the project, please open an issue and we can coordinate on adding you as a maintainer.

• Documentation
• CHANGELOG
• Upgrading to 2.x

About

Every alchemist requires good tools, and one of the greatest tools in the alchemist's disposal is the distillery. The purpose of the distillery is to take something and break it down to its component parts, reassembling it into something better, more powerful. That is exactly what this project does - it takes your Mix project and produces an Erlang/OTP release, a distilled form of your raw application's components; a single package which can be deployed anywhere, independently of an Erlang/Elixir installation. No dependencies, no hassle.

This is a pure-Elixir, dependency-free implementation of release generation for Elixir projects. It is currently a standalone package, but may be integrated into Mix at some point in the future.

Installation

Distillery requires Elixir 1.6 or greater. It works with Erlang 20+.

defp deps do
  [{:distillery, "~> 2.1"}]
end

Just add as a mix dependency and use mix distillery.release.

If you are new to releases or Distillery, please review the documentation, it is extensive and covers just about any question you may have!

Community/Questions/etc.

If you have questions or want to discuss Distillery, releases, or other deployment related topics, a good starting point is the Deployment section of ElixirForum, which can be found here.

I can often be found in IRC on freenode, in the #elixir-lang channel, and there is also an Elixir Slack channel as well, though I don't frequent that myself, there are many people who can answer questions there.

Failing that, feel free to open an issue on the tracker with questions, and I'll do my best to get to it in a timely fashion!

License

MIT. See the LICENSE.md in this repository for more details.

distillery's Issues

Add windows scripts

Need to add Windows support. Should just be a matter of porting the relx batch files, though I'd like to consider porting them to powershell instead.

/cc @OnorioCatenacci

Fail to build an archive with a path in include_erts

When building with a config of:

environment :ci do
  set include_erts: "erts/ci"
  set include_src: true
end

According to the docs this should be supported, unfortunately it crashes when trying to build the tarball:

** (CaseClauseError) no case clause matching: "erts/ci"
    lib/mix/lib/releases/archiver.ex:80: Mix.Releases.Archiver.update_tar/1
    lib/mix/lib/releases/archiver.ex:19: Mix.Releases.Archiver.archive/1
    lib/distillery/tasks/release.ex:108: Mix.Tasks.Release.run/1
    (mix) lib/mix/task.ex:296: Mix.Task.run_task/3
    (mix) lib/mix/cli.ex:58: Mix.CLI.run_task/2

Finish implementing upgrades/downgrades

Appup generation is complete, just need to add in logic for detecting upgrade scenarios and generate appups/relups appropriately.

Archiver failed to make tar

I'm trying to make a prod release but it failed to make the tar:

{{case_clause,
     {'EXIT',
         {function_clause,
             [{filename,join,[[]],[{file,"filename.erl"},{line,395}]},
              {erl_tar,split_filename,4,[{file,"erl_tar.erl"},{line,471}]},
              {erl_tar,create_header,3,[{file,"erl_tar.erl"},{line,400}]},
              {erl_tar,add1,4,[{file,"erl_tar.erl"},{line,323}]},
              {erl_tar,foreach_while_ok,2,[{file,"erl_tar.erl"},{line,1000}]},
              {systools_make,add_to_tar,3,
                  [{file,"systools_make.erl"},{line,1902}]},
              {systools_make,add_appl,7,
                  [{file,"systools_make.erl"},{line,1773}]},
              {systools_make,'-add_applications/5-fun-0-',6,
                  [{file,"systools_make.erl"},{line,1592}]}]}}},
 [{systools_make,'-add_applications/5-fun-0-',6,
      [{file,"systools_make.erl"},{line,1592}]},
  {lists,foldl,3,[{file,"lists.erl"},{line,1263}]},
  {systools_make,add_applications,5,[{file,"systools_make.erl"},{line,1591}]},
  {systools_make,mk_tar,6,[{file,"systools_make.erl"},{line,1585}]},
  {systools_make,mk_tar,5,[{file,"systools_make.erl"},{line,1561}]},
  {systools_make,make_tar,2,[{file,"systools_make.erl"},{line,336}]},
  {'Elixir.Mix.Releases.Archiver',make_tar,1,
      [{file,"lib/mix/lib/releases/archiver.ex"},{line,45}]},
  {'Elixir.Mix.Releases.Archiver',archive,1,
      [{file,"lib/mix/lib/releases/archiver.ex"},{line,17}]}]}
==> Problem generating release tarball:
    {:tar_generation_error, :unknown}

I actually dont need the tar file but didn't find anything in the doc to turn this off.

release to hex

Noticed https://github.com/bitwalker/distillery/releases/tag/0.9.1 is available , but https://hex.pm/packages/distillery does not have 0.9.1

Ubuntu 16.04 Phoenix app failing to start

Okay, I kinda tried to google and try different combos, but still no luck.

Ubuntu 16.04, erlang & elixir are installed there. So, I guess, I can use set include_erts: false.

Versions:

• Erlang/OTP 19 [erts-8.0] [source] [64-bit]
• Elixir 1.3.2

The app is using the latest Phoenix and works if I manually start it, but the release want's something from me.

I'm kinda new with releases, here is the console output:

{"init terminating in do_boot",{load_failed,[error_handler]}}

I can do provide any info needed, just want to make it work finally :)

"/rel/<projectname>/bin/<projectname> stop" does not work

Same problem like in exrm: bitwalker/exrm#385
It works neither on my laptop (current arch linux) nor on my server (Debian wheezy).

Error on mix.exs deps check

Steps to reproduce

mix release

Verbose Logs

** (CaseClauseError) no case clause matching: nil
    lib/mix/lib/releases/models/app.ex:92: Mix.Releases.App.include_dep?/1
    (elixir) lib/enum.ex:807: anonymous fn/4 in Enum.filter_map/3
    (elixir) lib/enum.ex:1623: Enum."-reduce/3-lists^foldl/2-0-"/3
    (elixir) lib/enum.ex:807: Enum.filter_map/3
    lib/mix/lib/releases/models/app.ex:42: Mix.Releases.App.new/2
    lib/mix/lib/releases/utils.ex:198: Mix.Releases.Utils.get_apps/1
    lib/mix/lib/releases/assembler.ex:78: Mix.Releases.Assembler.apply_configuration/2
    lib/mix/lib/releases/assembler.ex:27: Mix.Releases.Assembler.assemble/1
    lib/distillery/tasks/release.ex:106: Mix.Tasks.Release.run/1
    (mix) lib/mix/task.ex:296: Mix.Task.run_task/3
    (mix) lib/mix/cli.ex:58: Mix.CLI.run_task/2
    (elixir) lib/code.ex:363: Code.require_file/2

Description of issue

This error is present since the 0.9.3 version. It might be because of e40fe54

my mix.exs looks like this:

defmodule MyApp.Mixfile do
  use Mix.Project

  def project do
    [app: :my_app,
     version: version_number(),
     elixir: "~> 1.3.0",
     elixirc_paths: elixirc_paths(Mix.env),
     compilers: [:phoenix, :gettext] ++ Mix.compilers,
     build_embedded: Mix.env == :prod || Mix.env == :beta,
     start_permanent: Mix.env == :prod || Mix.env == :beta,
     aliases: aliases(),
     consolidate_protocols: false,
     test_coverage: [tool: ExCoveralls],
     preferred_cli_env: ["coveralls": :test, "coveralls.detail": :test, "coveralls.html": :test],
     deps: deps() ++ dev_test_deps()]
  end

  # Configuration for the OTP application.
  #
  # Type `mix help compile.app` for more information.
  def application do
    [mod: {MyApp, []},
     applications: Enum.map(deps(), fn({package, _version}) -> package end) ++ [:logger, :runtime_tools, :mix]]
  end

  # Specifies which paths to compile per environment.
  defp elixirc_paths(:test), do: ["lib", "web", "test/support"]
  defp elixirc_paths(_),     do: ["lib", "web"]

  # Specifies your project dependencies.
  #
  # Type `mix help deps` for examples and options.
  defp deps do
    [
      {:arc_ecto, "~> 0.4.3"},
      {:arc, "~> 0.5.2"},
      {:bamboo_smtp, "~> 1.2.0"},
      {:bamboo, "~> 0.7"},
      {:calecto, "~> 0.16.0"},
      {:calendar_translations, "~> 0.0.3"},
      {:calendar, "~> 0.16.0"},
      {:comeonin, "~> 2.5"},
      {:cowboy, "~> 1.0"},
      {:credo, "~> 0.4.5"},
      {:earmark, "~> 1.0.0"},
      {:ecto, "~> 2.0.1"},
      {:edeliver, "~> 1.4.0"},
      {:eiconv, git: "https://github.com/zotonic/eiconv.git"},
      {:ex_aws, "~> 0.5.0"},
      {:geo, "~> 1.1"},
      {:gettext, "~> 0.11"},
      {:html_sanitize_ex, "~> 1.0.1"},
      {:httpoison, "~> 0.9.0"},
      {:icalendar, git: "https://github.com/lpil/icalendar.git"},
      {:joken, "~> 1.2.2"},
      {:mailman, "~> 0.2.2"},
      {:oauth2, "0.6.0"},
      {:phoenix_calendar, "~> 0.1.2"},
      {:phoenix_ecto, "~> 3.0.0"},
      {:phoenix_html, ">= 2.6.2"},
      {:phoenix_pubsub, "~> 1.0"},
      {:phoenix_simple_form, ">= 0.0.0"},
      {:phoenix, "~> 1.2.1"},
      {:porcelain, "~> 2.0"},
      {:postgrex, ">= 0.0.0"},
      {:quantum, ">= 1.7.1"},
      {:scrivener_ecto, "~> 1.0"},
      {:scrivener_html, "~> 1.3.1"},
      {:scrivener, "~> 2.0.0"}
      # {:exrm, "~> 1.0"},
    ]
  end

  defp dev_test_deps do
    [
      {:distillery, ">= 0.9.0", warn_missing: false},
      {:ex_machina, "~> 1.0", only: :test},
      {:excoveralls, "~> 0.4", only: :test},
      {:mix_test_watch, "~> 0.2", only: :dev},
      {:phoenix_live_reload, "~> 1.0", only: :dev},
      {:sshex, "2.1.0", only: [:dev, :test]}
    ]
  end

  # See the documentation for `Mix` for more info on aliases.
  defp aliases do
    ["ecto.setup": ["ecto.create", "ecto.load", "ecto.migrate", "run priv/repo/seeds.exs"],
     "ecto.reset": ["ecto.drop", "ecto.setup"],
     "phoenix.server": ["ecto.migrate", "phoenix.server"],
     "s": ["phoenix.server"],
     "t": ["test.watch"],
     "c": ["clean", "deps.clean --all", "deps.get", "compile"],
     "credo": ["credo --strict"],
     "project-version": ["run -e 'IO.puts MyApp.Mixfile.project[:version]'"],
     "test": ["ecto.create --quiet", "ecto.migrate", "test"]]
  end

  defp version_number do
    {output, 0} = System.cmd("git", ["rev-list", "--count", "HEAD"])
    "1.0." <> String.rstrip(output)
  end
end

[minor] Should rel/config.exs live elsewhere?

As an early user of Distillery, I am finding a (minor) inconvenience that the rel dir holds both output and an important file which should be tracked in version control (ie: git).

Should rel/config.exs live elsewhere? Maybe mv rel/config.exs releaseconfig.exs ?

include_erts: false may not really be working?

Currently I am using this config

environment :dev do
  set dev_mode: false
  set include_erts: false
end

to create a release on my Mac that I then copy into a Docker container running Linux.
I was expecting incude_erts: false to simply not include any erts in my release and use whatever is installed on my container.

My release fails to boot because it can't find erts-8.0.2 in the container.
This is correct, as I currently only have 8.0.1 in there.

This prompted me to look into where that comes from.
start_erts.data gets populated with Utils.erts_version which is defined at release time, which means it uses the version in my Mac and not the one in the container.
Adding the full path in include_erts made the release fail because it tried to pickup a path that obviously does not exist on my Mac.

There are probably a bunch of ways to "solve" this:

1. defer the detection to runtime/boot time
2. allow to specify the version in the config file
3. allow to specify a path in the config file, without actually including it

I'd kinda prefer 1) but it could be brittle. 2 and 3 are pretty much the same but mean that I have know and understand all the environments where this will run.

Release crashes in dev mode when Swarm is dependency

Description of issue

The release builds fine with mix release but it fails to start due a missing NIF in the asn1 application. This problem goes away when I change :include_erts to true.

• What are the expected results?
  • I would expect that NIFs in the Erlang runtime system would automatically work in dev without needing to include the runtime.
• What OS, Erlang/Elixir versions are you seeing this issue on?
  • macOS 10.11.6
  • Erlang 18.0.3
  • Elixir 1.3.2

Steps to reproduce

• Build an app with {:swarm, "~> 0.4"} as a dependency.
• run mix release
• start the application with the console script

Verbose Logs

λ rel/my_app/bin/my_app console
Erlang/OTP 18 [erts-7.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

{error_logger,{{2016,8,26},{20,55,46}},"Unable to load asn1 nif library. Failed with error:~n\"~p, ~s\"~n",[load_failed,"Failed to load NIF library: 'dlopen(/Users/scrogson/Projects/my_app/rel/my_app/lib/asn1-4.0.2/priv/lib/asn1rt_nif.so, 2): image not found'"]}
{error_logger,{{2016,8,26},{20,55,46}},crash_report,[[{initial_call,{supervisor,kernel,['Argument__1']}},{pid,<0.1238.0>},{registered_name,[]},{error_info,{exit,{on_load_function_failed,asn1rt_nif},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,352}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}},{ancestors,[kernel_sup,<0.1213.0>]},{messages,[]},{links,[<0.1214.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,376},{stack_size,27},{reductions,117}],[]]}
{error_logger,{{2016,8,26},{20,55,46}},supervisor_report,[{supervisor,{local,kernel_sup}},{errorContext,start_error},{reason,{on_load_function_failed,asn1rt_nif}},{offender,[{pid,undefined},{id,kernel_safe_sup},{mfargs,{supervisor,start_link,[{local,kernel_safe_sup},kernel,safe]}},{restart_type,permanent},{shutdown,infinity},{child_type,supervisor}]}]}
{error_logger,{{2016,8,26},{20,55,47}},crash_report,[[{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,<0.1212.0>},{registered_name,[]},{error_info,{exit,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,asn1rt_nif}}},{kernel,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,134}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}},{ancestors,[<0.1211.0>]},{messages,[{'EXIT',<0.1213.0>,normal}]},{links,[<0.1211.0>,<0.1210.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,376},{stack_size,27},{reductions,117}],[]]}
{error_logger,{{2016,8,26},{20,55,47}},std_info,[{application,kernel},{exited,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,asn1rt_nif}}},{kernel,start,[normal,[]]}}},{type,permanent}]}
{"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,asn1rt_nif}}},{kernel,start,[normal,[]]}}}"}

Crash dump is being written to: erl_crash.dump...done
Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,asn1rt_nif}}},{kernel,start,[normal,[]]}}

make phoenix running according by official site exrm relase section

make phoenix running according by official site exrm relase section

http://www.phoenixframework.org/docs/advanced-deployment

invalid ERTS path, cannot determine version

Steps to reproduce

add include_erts: "/usr/local/Cellar/erlang/19.0.2/lib/erlang/lib" to rel/config.exs
$MIX_ENV=prod mix release --env=prod

Verbose Logs

$ MIX_ENV=prod mix release --env=prod --verbose
==> Loading configuration..
==> Assembling release..
==> Building release omin:0.0.1 using environment prod
==> Discovered applications:
  sasl-3.0
    from: /usr/local/Cellar/erlang/19.0.2/lib/erlang/lib/sasl-3.0
    applications:
      kernel
      stdlib
    includes: none

  iex-1.3.2
    from: /usr/local/Cellar/elixir/1.3.2/bin/../lib/iex
    applications:
      kernel
      stdlib
      elixir
    includes: none

  omin-0.0.1
    from: _build/prod/lib/omin
    applications:
      kernel
      stdlib
      elixir
      phoenix
      phoenix_pubsub
      phoenix_html
      cowboy
      logger
      gettext
      phoenix_ecto
      postgrex
    includes: none

  kernel-5.0
    from: /usr/local/Cellar/erlang/19.0.2/lib/erlang/lib/kernel-5.0
    applications: none
    includes: none

  stdlib-3.0.1
    from: /usr/local/Cellar/erlang/19.0.2/lib/erlang/lib/stdlib-3.0.1
    applications:
      kernel
    includes: none

  elixir-1.3.2
    from: /usr/local/Cellar/elixir/1.3.2/bin/../lib/elixir
    applications:
      kernel
      stdlib
      compiler
    includes: none

  compiler-7.0.1
    from: /usr/local/Cellar/erlang/19.0.2/lib/erlang/lib/compiler-7.0.1
    applications:
      kernel
      stdlib
    includes: none

  phoenix-1.2.1
    from: _build/prod/lib/phoenix
    applications:
      kernel
      stdlib
      elixir
      plug
      poison
      logger
      eex
    includes: none

  plug-1.2.0
    from: _build/prod/lib/plug
    applications:
      kernel
      stdlib
      elixir
      crypto
      logger
      mime
    includes: none

  crypto-3.7
    from: /usr/local/Cellar/erlang/19.0.2/lib/erlang/lib/crypto-3.7
    applications:
      kernel
      stdlib
    includes: none

  logger-1.3.2
    from: /usr/local/Cellar/elixir/1.3.2/bin/../lib/logger
    applications:
      kernel
      stdlib
      elixir
    includes: none

  mime-1.0.1
    from: _build/prod/lib/mime
    applications:
      kernel
      stdlib
      elixir
    includes: none

  poison-2.2.0
    from: _build/prod/lib/poison
    applications:
      kernel
      stdlib
      elixir
    includes: none

  eex-1.3.2
    from: /usr/local/Cellar/elixir/1.3.2/bin/../lib/eex
    applications:
      kernel
      stdlib
      elixir
    includes: none

  phoenix_pubsub-1.0.0
    from: _build/prod/lib/phoenix_pubsub
    applications:
      kernel
      stdlib
      elixir
      logger
      crypto
    includes: none

  phoenix_html-2.6.2
    from: _build/prod/lib/phoenix_html
    applications:
      kernel
      stdlib
      elixir
      logger
      plug
    includes: none

  cowboy-1.0.4
    from: _build/prod/lib/cowboy
    applications:
      kernel
      stdlib
      ranch
      cowlib
      crypto
    includes: none

  ranch-1.2.1
    from: _build/prod/lib/ranch
    applications:
      kernel
      stdlib
    includes: none

  cowlib-1.0.2
    from: _build/prod/lib/cowlib
    applications:
      kernel
      stdlib
      crypto
    includes: none

  gettext-0.11.0
    from: _build/prod/lib/gettext
    applications:
      kernel
      stdlib
      elixir
      logger
    includes: none

  phoenix_ecto-3.0.1
    from: _build/prod/lib/phoenix_ecto
    applications:
      kernel
      stdlib
      elixir
      logger
      ecto
      plug
    includes: none

  ecto-2.0.4
    from: _build/prod/lib/ecto
    applications:
      kernel
      stdlib
      elixir
      logger
      decimal
      poolboy
    includes: none

  decimal-1.1.2
    from: _build/prod/lib/decimal
    applications:
      kernel
      stdlib
      elixir
    includes: none

  poolboy-1.5.1
    from: _build/prod/lib/poolboy
    applications:
      kernel
      stdlib
    includes: none

  postgrex-0.11.2
    from: _build/prod/lib/postgrex
    applications:
      kernel
      stdlib
      elixir
      logger
      db_connection
      decimal
    includes: none

  db_connection-1.0.0-rc.4
    from: _build/prod/lib/db_connection
    applications:
      kernel
      stdlib
      elixir
      logger
      connection
    includes: none

  connection-1.0.4
    from: _build/prod/lib/connection
    applications:
      kernel
      stdlib
      elixir
    includes: none

==> Generated overlay vars:
    release_name=:omin
    release_version="0.0.1"
    is_upgrade=false
    upgrade_from=:latest
    dev_mode=false
    include_erts="/usr/local/Cellar/erlang/19.0.2/lib/"
    include_src=false
    include_system_libs=true
    erl_opts=""
    erts_vsn={:error, "invalid ERTS path, cannot determine version"}
    output_dir="rel/omin"
==> Copying applications to rel/omin
==> Generating nodetool
==> Failed to build release:
    protocol String.Chars not implemented for {:error, "invalid ERTS path, cannot determine version"}

Description of issue

I expected it to successfully build a release.
I'm using OS X 10.11.6, erts-8.0.2, Elixir 1.3.2.
My rel/config.exs file is the default generated by $mix release.init except include_erts: "/usr/local/Cellar/erlang/19.0.2/lib/erlang/lib".

Got an error when opening console in a dev build

When I opened the console after building the dev release($ mix release), I got and error:

$ bin/myapp console
[info] Application myapp exited: Myapp.start(:normal, []) returned an error: shutdown: failed to start child: Myapp.Endpoint
    ** (EXIT) shutdown: failed to start child: Phoenix.CodeReloader.Server
        ** (EXIT) an exception was raised:
            ** (UndefinedFunctionError) function Mix.Project.config/0 is undefined (module Mix.Project is not available)
                Mix.Project.config()
                (phoenix) lib/phoenix/code_reloader/server.ex:29: Phoenix.CodeReloader.Server.init/1
                (stdlib) gen_server.erl:328: :gen_server.init_it/6
                (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3

It's a Phoenix project with code_reloader on in dev env.

erl: not found on app start

Steps to reproduce

run ./bin/my_app console

Verbose Logs

/home/apps/edeliver/my_app/releases/1.0.3602+3602-b84b0be-develop/my_app.sh: 105: /home/apps/edeliver/my_app/releases/1.0.3602+3602-b84b0be-develop/my_app.sh: erl: not found

Description of issue

• the app should start
  (it used to work yesterday)
• Building server: elixir 1.3.2 and Erlang/OTP 18
• on the app server is no erl nor elixir installed. Only nginx and the app are on there

# rel/config.exs
use Mix.Releases.Config,
    default_release: :default,
    default_environment: :dev

environment :dev do
  set dev_mode: true
  set include_erts: false
end

environment :beta do
  set include_erts: true
  set include_src: false
end

environment :prod do
  set include_erts: true
  set include_src: false
end

release :my_app do
  set version: current_version(:my_app)
end

Notice: I used exrm before and switched to distillery. Also I use edeliver for deployment.

Is there anything I didn't think of? I couldn't identify any difference in our app so I figured out that it might be because we switched to distillery.

Starting the same service twice on the same machine

Sorry in advance for this issue, there's a high chance I just didn't get how to deal with distillery properly.

I want to start a release twice on a machine - the only difference being the network port that the service listens on (which is given as an environment variable). Naively, I thought just copying the release in two separate directories (one for the "testing" service and one for production) would be enough since each of them has its own tmp directory. But that wasn't really the case, I could start one of them but the other didn't start. Then I tried having one with dev and one with prod environment in rel/config.exs but that didn't work either.

What's the right way to do this? Maybe it would be worth having a section in "Common Problems" of the documentation for this? I guess I'm not the only one having a testing/production setup and thus running the same project twice on a machine.

Thanks in advance!

Distillery says deps are missing from :applications, but doesn't list them

Steps to reproduce

mix.release --env=prod

Verbose Logs

==> Loading configuration..
==> Assembling release..
==> Building release storm:0.0.1 using environment prod
==> One or more direct or transitive dependencies are missing from
    :applications or :included_applications, they will not be included
    in the release:



    This can cause your application to fail at runtime. If you are sure
    that this is not an issue, you may ignore this warning.

==> Discovered applications:
  sasl-2.7
    from: /home/duijf/.asdf/installs/erlang/18.3/lib/erlang/lib/sasl-2.7
    applications:
      :kernel
      :stdlib
    includes: none

  iex-1.3.2
    from: /home/duijf/.asdf/installs/elixir/1.3.2/bin/../lib/iex
    applications:
      :kernel
      :stdlib
      :elixir
    includes: none

  storm-0.0.1
    from: _build/prod/lib/storm
    applications:
      :kernel
      :stdlib
      :elixir
      :arc
      :arc_ecto
      :cowboy
      :gettext
      :logger
      :phoenix
      :phoenix_ecto
      :phoenix_html
      :phoenix_pubsub
      :postgrex
    includes: none

  kernel-4.2
    from: /home/duijf/.asdf/installs/erlang/18.3/lib/erlang/lib/kernel-4.2
    applications: none
    includes: none

  stdlib-2.8
    from: /home/duijf/.asdf/installs/erlang/18.3/lib/erlang/lib/stdlib-2.8
    applications:
      :kernel
    includes: none

  elixir-1.3.2
    from: /home/duijf/.asdf/installs/elixir/1.3.2/bin/../lib/elixir
    applications:
      :kernel
      :stdlib
      :compiler
    includes: none

  compiler-6.0.3
    from: /home/duijf/.asdf/installs/erlang/18.3/lib/erlang/lib/compiler-6.0.3
    applications:
      :kernel
      :stdlib
    includes: none

  arc-0.5.3
    from: _build/prod/lib/arc
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
    includes: none

  logger-1.3.2
    from: /home/duijf/.asdf/installs/elixir/1.3.2/bin/../lib/logger
    applications:
      :kernel
      :stdlib
      :elixir
    includes: none

  arc_ecto-0.4.3
    from: _build/prod/lib/arc_ecto
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
    includes: none

  cowboy-1.0.4
    from: _build/prod/lib/cowboy
    applications:
      :kernel
      :stdlib
      :ranch
      :cowlib
      :crypto
    includes: none

  ranch-1.2.1
    from: _build/prod/lib/ranch
    applications:
      :kernel
      :stdlib
    includes: none

  cowlib-1.0.2
    from: _build/prod/lib/cowlib
    applications:
      :kernel
      :stdlib
      :crypto
    includes: none

  crypto-3.6.3
    from: /home/duijf/.asdf/installs/erlang/18.3/lib/erlang/lib/crypto-3.6.3
    applications:
      :kernel
      :stdlib
    includes: none

  gettext-0.11.0
    from: _build/prod/lib/gettext
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
    includes: none

  phoenix-1.2.1
    from: _build/prod/lib/phoenix
    applications:
      :kernel
      :stdlib
      :elixir
      :plug
      :poison
      :logger
      :eex
    includes: none

  plug-1.2.0
    from: _build/prod/lib/plug
    applications:
      :kernel
      :stdlib
      :elixir
      :crypto
      :logger
      :mime
    includes: none

  mime-1.0.1
    from: _build/prod/lib/mime
    applications:
      :kernel
      :stdlib
      :elixir
    includes: none

  poison-2.2.0
    from: _build/prod/lib/poison
    applications:
      :kernel
      :stdlib
      :elixir
    includes: none

  eex-1.3.2
    from: /home/duijf/.asdf/installs/elixir/1.3.2/bin/../lib/eex
    applications:
      :kernel
      :stdlib
      :elixir
    includes: none

  phoenix_ecto-3.0.1
    from: _build/prod/lib/phoenix_ecto
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :ecto
      :plug
    includes: none

  ecto-2.0.4
    from: _build/prod/lib/ecto
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :decimal
      :poolboy
    includes: none

  decimal-1.1.2
    from: _build/prod/lib/decimal
    applications:
      :kernel
      :stdlib
      :elixir
    includes: none

  poolboy-1.5.1
    from: _build/prod/lib/poolboy
    applications:
      :kernel
      :stdlib
    includes: none

  phoenix_html-2.6.2
    from: _build/prod/lib/phoenix_html
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :plug
    includes: none

  phoenix_pubsub-1.0.0
    from: _build/prod/lib/phoenix_pubsub
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :crypto
    includes: none

  postgrex-0.11.2
    from: _build/prod/lib/postgrex
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :db_connection
      :decimal
    includes: none

  db_connection-1.0.0-rc.4
    from: _build/prod/lib/db_connection
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
      :connection
    includes: none

  connection-1.0.4
    from: _build/prod/lib/connection
    applications:
      :kernel
      :stdlib
      :elixir
    includes: none

==> Generated overlay vars:
    release_name=:storm
    release_version="0.0.1"
    is_upgrade=false
    upgrade_from=:latest
    dev_mode=false
    include_erts=true
    include_src=false
    include_system_libs=true
    erl_opts=""
    erts_vsn="7.3"
    output_dir="rel/storm"
==> Copying applications to rel/storm
==> Generating nodetool
==> Generating start_erl.data
==> Generating vm.args
==> Generating sys.config from config/config.exs
==> Including ERTS 7.3 from /home/duijf/.asdf/installs/erlang/18.3/lib/erlang/erts-7.3
==> Generating boot script
==> Generating RELEASES
==> Generating start_clean.boot
==> Applying overlays
==> Applying mkdir overlay
    dst: releases/0.0.1/hooks
==> Applying mkdir overlay
    dst: releases/0.0.1/commands
==> Packaging release..
==> Archiving storm-0.0.1
==> Writing tarball to rel/storm/releases/0.0.1/storm.tar.gz
==> Updating tarball
==> Tarball updated!
==> Release successfully built!
    You can run it in one of the following ways:
      Interactive: rel/storm/bin/storm console
      Foreground: rel/storm/bin/storm foreground
      Daemon: rel/storm/bin/storm start

Description of issue

• What are the expected results?

The error message below would either lists which dependencies are missing, or doesn't appear at all.

==> One or more direct or transitive dependencies are missing from
    :applications or :included_applications, they will not be included
    in the release:



    This can cause your application to fail at runtime. If you are sure
    that this is not an issue, you may ignore this warning.

• What OS, Erlang/Elixir versions are you seeing this issue on?

Arch Linux, Erlang 18.3, Elixir 1.3.2.

• If possible, also provide your rel/config.exs, as it is often
  my first troubleshooting question, and you'll save us both time :)

use Mix.Releases.Config,
    default_release: :storm,
    default_environment: :dev

environment :dev do
  set dev_mode: true
  set include_erts: false
end

environment :prod do
  set include_erts: true
  set include_src: false
end

release :storm do
  set version: current_version(:storm)
end

Dependencies are not included if not specified in applications

I have an umbrella scenario with a bit different approach(the app names are here only to make the point)

• MyDistributedCache deploys in multiple machine and it uses pg2 for registering proces group
• MyAwesomeApp deploys in one machine and uses MyDistributedCache

So in my_distributed_cache I create a global process group with name :my_distributed_cache and I used it from :my_awesome_app

in my_awesome_app mix.exs I have included :my_distributed_cache but didn't list it in applications list:

defmodule MyAwesomeApp.Mixfile do
...
  def application do
    [mod: {MyAwesomeApp, []}]
  end
...
  defp deps do
    [{:my_distributed_cache, in_umbrella: true}]
  end
...

I made separate releases for :my_distributed_cache and :my_awesome_app and released them separately and connected both nodes(Node.connect). The problem that I have is even though :my_distributed_cache is in dependency list of :my_awesome_app the beam files of :my_distributed_cache are not included in :my_awesome_app release and therefore I am not able to use MyDistributedCache modules from MyAwesomeApp

Am I missing something here? Or making a mistake in the structure? or it's not supported?

Different config parameters for different environments

Hello.

I tried distillery (with edeliver) and it worked great! 👍

And I've got one question, I couldn't find an answer in the docs.

Say, I have demo env and staging env. For example in demo env an app should send an auth request to auth-demo.mysite.com and in staging it should send an auth request to auth-staging.mysite.com.

Is there a place where I can put these two URLs(somehow special, under different namespaces or whatever) and distillery would automagically figure out which URL should be used when I deploy the same built release to demo or staging environment. Ideally, this magic place should be under version control :)

Thanks!

Make plugins configurable per-release

Currently all plugins are global and there's no way to disable them. An example use case is building a .deb package for the production release, but skipping this for the dev one.

The new configuration features in rel/config.exs are a great place to accommodate enabling plugins explicitly. Not only this goes with elixirs philosophy of explicit > implicit, but this would also allow to remove all the shenanigans with traversing beams to find modules that implement the plugin behaviour, simplifying the code considerably.

Command-line app release?

Any easy way to release command line app like described in bitwalker/exrm#37?

release-init --release-per-app fails in complex umbrella project

Steps to reproduce

A complex umbrella project is failing to init the distillery configuration. Will update ticket as I narrow down on cause... :-)

$ mix release.init --release-per-app
** (UndefinedFunctionError) function nil.project/0 is undefined or private
    nil.project()
    anonymous fn/1 in Mix.Tasks.Release.Init.get_umbrella_bindings/1
    (mix) lib/mix/project.ex:211: Mix.Project.in_project/4
    (elixir) lib/file.ex:1145: File.cd!/2
    (elixir) lib/enum.ex:1184: Enum."-map/2-lists^map/1-0-"/2
    lib/distillery/tasks/init.ex:79: Mix.Tasks.Release.Init.get_umbrella_bindings/1
    lib/distillery/tasks/init.ex:46: Mix.Tasks.Release.Init.run/1
    (mix) lib/mix/task.ex:296: Mix.Task.run_task/3

Description of issue

• What OS, Erlang/Elixir versions are you seeing this issue on?

Elixir 1.3.2, Erlang 18.

• If possible, also provide your rel/config.exs, as it is often
  my first troubleshooting question, and you'll save us both time :)

rel dir is empty.

include_erts: false doesn't solve the cross platform issue

I built the release on a mac without including erlang and the release still failed to run on the production environment. I have other (simple) pheonix application running so it's not an erlang problem.

If what micmus said is true, we should mention that in the guide.

Connection refused/port not open in release

Thanks for a wonderful tool! :)

This may or may not be related to distillery, but I'm not sure where else to ask: when I run MIX_ENV=prod mix phoenix.server everything runs fine; but when I run MIX_ENV=prod mix release it just refuses to open my (hardcoded) port 3000:

$> rel/app/bin/app start
$> rel/app/bin/app ping
pong

$> rel/app/bin/app remote_console
Interactive Elixir (1.3.1) - press Ctrl+C to exit (type h() ENTER for help)
iex([email protected])1> :inet.i
Port  Module   Recv Sent Owner    Local Address   Foreign Address State     Type   
16576 inet_tcp 0    0    <0.47.0> *:36943         *:*             ACCEPTING STREAM 
16784 inet_tcp 4    32   <0.45.0> localhost:48472 localhost:epmd  ????      STREAM 
:ok
iex([email protected])2> 
^C

$> curl localhost:3000
curl: (7) Failed to connect to localhost port 3000: Connection refused

$> lsof -Pan -p `rel/app/bin/app pid` -i
COMMAND   PID      USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
beam    32053 sebastian    7u  IPv4 334710      0t0  TCP *:39078 (LISTEN)
beam    32053 sebastian    8u  IPv4 334712      0t0  TCP 127.0.0.1:59504->127.0.0.1:4369 (ESTABLISHED)
beam    32053 sebastian   10u  IPv4 334721      0t0  TCP 127.0.0.1:38123->127.0.0.1:5432 (ESTABLISHED)
beam    32053 sebastian   11u  IPv4 334725      0t0  TCP 127.0.0.1:50733->127.0.0.1:5432 (ESTABLISHED)
beam    32053 sebastian   13u  IPv4 334729      0t0  TCP 127.0.0.1:45947->127.0.0.1:5432 (ESTABLISHED)
beam    32053 sebastian   14u  IPv4 334733      0t0  TCP 127.0.0.1:47103->127.0.0.1:5432 (ESTABLISHED)
beam    32053 sebastian   15u  IPv4 334737      0t0  TCP 127.0.0.1:35645->127.0.0.1:5432 (ESTABLISHED)
beam    32053 sebastian   16u  IPv4 334741      0t0  TCP 127.0.0.1:49622->127.0.0.1:5432 (ESTABLISHED)
beam    32053 sebastian   17u  IPv4 334749      0t0  TCP 127.0.0.1:54624->127.0.0.1:5432 (ESTABLISHED)
beam    32053 sebastian   18u  IPv4 334755      0t0  TCP 127.0.0.1:49296->127.0.0.1:5432 (ESTABLISHED)
beam    32053 sebastian   19u  IPv4 334759      0t0  TCP 127.0.0.1:50739->127.0.0.1:5432 (ESTABLISHED)
beam    32053 sebastian   20u  IPv4 334764      0t0  TCP 127.0.0.1:37403->127.0.0.1:5432 (ESTABLISHED)

$> curl localhost:39078
curl: (52) Empty reply from server

$> less rel/app/log/erlang.log.1 
=====
===== LOGGING STARTED Fri Aug 12 00:04:15 EDT 2016
=====
Exec: /opt/rcra/erts-8.0/bin/erlexec -boot /opt/rcra/releases/0.0.1/app -boot_var ER
TS_LIB_DIR /opt/rcra/erts-8.0/../lib -env ERL_LIBS /opt/rcra/lib -pa /opt/rcra/lib/a
pp-0.0.1/consolidated -args_file /opt/rcra/releases/0.0.1/vm.args -config /opt/rcra/
releases/0.0.1/sys.config -mode embedded -user Elixir.IEx.CLI -extra --no-halt +iex 
-- console --
Root: /opt/rcra
/opt/rcra
Eshell V8.0  (abort with ^G)
Interactive Elixir (1.3.1) - press Ctrl+C to exit (type h() ENTER for help)
iex([email protected])1> 

$> less rel/app/log/run_erl.log  
run_erl [32675] Fri Aug 12 00:04:15 2016
Args before exec of shell:
run_erl [32675] Fri Aug 12 00:04:15 2016
argv[0] = sh
run_erl [32675] Fri Aug 12 00:04:15 2016
argv[1] = -c
run_erl [32675] Fri Aug 12 00:04:15 2016
argv[2] = exec "/opt/rcra/bin/app" "console" -- 

Here is how I hardcode my port:

config :app, App.Endpoint,
  http: [port: 3000],
  url: [host: "xxxxxxxxxxx.ca", port: 80],
  cache_static_manifest: "priv/static/manifest.json"

Is there something wrong with my configuration? Could this be due to distillery? If yes, could error messaging be improved? I'm at a complete loss where to even start diagnosing this. Thanks a ton in advance for any ideas!

Add documentation about docker / vagrant

From the distillery documentation, you can get a feeling that it is super easy - just add a package, add some configuration and run a console command.

Unfortunately, there is a very important note - your app must be compiled in the same environment as your production server. This step is almost missing in the docs and reading tons of information about VM's (virtual machines) is overwhelming for people, who didn't use VM before.

It would be super awesome to have a chapter or even this issue, where every novice could possibly find quick answers w/o wasting days googling.

---

The task is to setup a basic app (phoenix app), make a release in VM and deploy it to production server. Since we have a chapter, how phoenix app has to be configured in the docs, we only need to setup a VM, make a release there and deploy to production server.

The production server has ubuntu 16.04. The dev server has OS X. The VM options are - vagrant and docker. If I understood correct, vagrant runs os in you prod os while docker uses just your prod os.

So, the first questions are:

1. vagrant or docker, cons and pros
2. dev server has to be the same as production one, what has to be the same (os, cpu, ram, os packages) ?
   2.1) do we need to setup docker / vagrant on production server or just copy the release tarball file?

Elixir 1.3 really required?

I just wanted to make sure that Elixir 1.3 was really required for distillery (awesome talk yesterday @bitwalker!). We wanted to use this in our company, and moving to Elixir 1.3 is definitely on the roadmap, but at this time upgrading elixir would have much more side effects in our project (some erlang projects not compiling on Erlang 19) than just switching from exrm to distillery, and we can't do that in the short term.

Any way to bind ENV variable to config without recompiling project?

Hi guys. Im trying to use ENV variables in release, to make sure that I don’t need to re-deploy everything whenever my config changes (for example, DB password).

I’m using distillery, but it should be much different from exrm), I’m trying following steps:

1. Setup config.exs that looks like this (to try different approaches):

   config :boilerplate, :boilerplate,
     user_test: {:system, "${DB_USER}", "defuser"},
     user_test2: "${DB_USER}",
     user_test3: System.get_env("DB_USER"),
     user_test4: System.get_env("DB_USER") || "default user"
   

2. Compile everything

   MIX_ENV=prod mix release
   

3. This compiles all source and sys.config looks like expected:

   [{sasl,[{errlog_type,error}]},
   {boilerplate,
      [{boilerplate,
           [{user_test,{system,<<"${DB_USER}">>,<<"defuser">>}},
            {user_test2,<<"${DB_USER}">>},
            {user_test3,nil},
            {user_test4,<<"default user">>}]}]}].
   

4. Start app with REPLACE_OS_VARS=true:

   REPLACE_OS_VARS=true DB_USER="someuser2" rel/boilerplate/bin/boilerplate console
   

5. And everything works well, on first start it loads config from environment variables:

   iex > Application.get_env(:boilerplate, :boilerplate)
   [user_test: {:system, "someuser2", "defuser"}, user_test2: "someuser2",
   user_test3: nil, user_test4: "default user»]
   

But it also replaces variables in config and they won’t be able to changed when ENV changes. sys.config after step 5:

[{sasl,[{errlog_type,error}]},
 {boilerplate,
     [{boilerplate,
          [{user_test,{system,<<"someuser2">>,<<"defuser">>}},
           {user_test2,<<"someuser2">>},
           {user_test3,nil},
           {user_test4,<<"default user">>}]}]}].

Futher restart of application with different ENV vars doesn’t change them in settings:

iex > Application.get_env(:boilerplate, :boilerplate)
[user_test: {:system, "someuser2", "defuser"}, user_test2: "someuser2",
 user_test3: nil, user_test4: "default user»]

So. Is there any way to read ENV variables without replacing them in sys.config?

Make hooks into conf.d style dirs [ENH]

It'd be much better to have start_hook and friends be directories, where scripts found following a certain name pattern are sourced/executed in alphabetical order.

A simple name pattern ("must end with .hook, must start with alphanum), helps prevent accidental execution of hidden/temp, conflict files (.rej, .rpmsave), and other floatsam and jetsam.

It'd be a good idea to do it now (pre 1.0), as it's API vs other modules.

Include Consolidated BEAM Files?

We just did a release and noticed the consolidated files were not included in the archive despite being copied to the release. I couldn't find any code that actually handles this and patched it so we can re-release. I am just wonder if we are doing something wrong or is there an oversight?

discord@fc4b92c

Undefined applications

Here are two applications we use that aren't working w/ distillery currently

https://github.com/aaronjensen/distillery_repro

$ mix release
==> Assembling release..
==> Building release distillery_repro:0.1.0 using environment default
==> Including ERTS 7.3 from /Users/aaronjensen/.asdf/installs/erlang/18.3/lib/erlang/erts-7.3
==> Failed to build release:

    Undefined applications: [exq,syslog]

Consider adding runtime_tools to the generated config.exs applications

Runtime tools are important if anyone wants to use most of the awesome tooling erlang gives us for debugging running systems. With the applications configuration in releases I feel like we have a perfect place to define this - it doesn't really make sense for the application itself to depend on the runtime_tools. I think it would be reasonable to include the application by default.

It's relatively large with 296K, but I don't think it matters that much.

a dependency is not included in release

Steps to reproduce

Check out this code https://github.com/slashmili/distillery-sample/tree/ecto-enum

Verbose Logs

mix release --verbose --profile=myapp:prod
==> distillery
Compiling 17 files (.ex)
Generated distillery app
==> myapp
Compiling 1 file (.ex)
Generated myapp app
==> Loading configuration..
==> Assembling release..
==> Building release myapp:0.1.0 using environment prod
==> One or more direct or transitive dependencies are missing from
    :applications or :included_applications, they will not be included
    in the release:

    :ecto_enum

    This can cause your application to fail at runtime. If you are sure
    that this is not an issue, you may ignore this warning.

==> Discovered applications:
  sasl-3.0
    from: /usr/local/Cellar/erlang/19.0.2/lib/erlang/lib/sasl-3.0
    applications:
      :kernel
      :stdlib
    includes: none

  iex-1.3.2
    from: /usr/local/Cellar/elixir/1.3.2/bin/../lib/iex
    applications:
      :kernel
      :stdlib
      :elixir
    includes: none

  myapp-0.1.0
    from: _build/dev/lib/myapp
    applications:
      :kernel
      :stdlib
      :elixir
      :logger
    includes: none

  kernel-5.0
    from: /usr/local/Cellar/erlang/19.0.2/lib/erlang/lib/kernel-5.0
    applications: none
    includes: none

  stdlib-3.0.1
    from: /usr/local/Cellar/erlang/19.0.2/lib/erlang/lib/stdlib-3.0.1
    applications:
      :kernel
    includes: none

  elixir-1.3.2
    from: /usr/local/Cellar/elixir/1.3.2/bin/../lib/elixir
    applications:
      :kernel
      :stdlib
      :compiler
    includes: none

  compiler-7.0.1
    from: /usr/local/Cellar/erlang/19.0.2/lib/erlang/lib/compiler-7.0.1
    applications:
      :kernel
      :stdlib
    includes: none

  logger-1.3.2
    from: /usr/local/Cellar/elixir/1.3.2/bin/../lib/logger
    applications:
      :kernel
      :stdlib
      :elixir
    includes: none

==> Generated overlay vars:
    release_name=:myapp
    release_version="0.1.0"
    is_upgrade=false
    upgrade_from=:latest
    dev_mode=false
    include_erts=true
    include_src=false
    include_system_libs=true
    erl_opts=""
    erts_vsn="8.0.2"
    output_dir="rel/myapp"
==> Copying applications to rel/myapp
==> Generating nodetool
==> Generating start_erl.data
==> Generating vm.args
==> Generating sys.config from config/config.exs
==> Including ERTS 8.0.2 from /usr/local/Cellar/erlang/19.0.2/lib/erlang/erts-8.0.2
==> Generating boot script
==> Generating RELEASES
==> Generating start_clean.boot
==> Applying overlays
==> Applying mkdir overlay
    dst: releases/0.1.0/hooks
==> Applying mkdir overlay
    dst: releases/0.1.0/commands
==> Packaging release..
==> Archiving myapp-0.1.0
==> Writing tarball to rel/myapp/releases/0.1.0/myapp.tar.gz
==> Updating tarball
==> Tarball updated!
==> Release successfully built!
    You can run it in one of the following ways:
      Interactive: rel/myapp/bin/myapp console
      Foreground: rel/myapp/bin/myapp foreground
      Daemon: rel/myapp/bin/myapp start

Description of issue

• What are the expected results?

Since ecto_enum is in my deps list I'm expecting that distillery includes ecto_enum in the OTP release

• What OS, Erlang/Elixir versions are you seeing this issue on?

Elixir 1.3.2, Erlang/OTP 19, Mac 10.12 and Ubuntu 16.04

• If possible, also provide your rel/config.exs, as it is often
  my first troubleshooting question, and you'll save us both time :)

Refer to the repo

• Is there documentation that says one thing, but Distillery does
  another? If so, please link the doc here so it can be updated if
  it's a documentation issue, or so that the fix can be based around
  what's documented.

There is no documents about included_applications like the way that is in exrm
Dependency issues so I'm not sure if distillery is supposed to detect the dependency or I should explicitly mention it

Embedded read-only filesystem support

Review documentation on read-only filesystems here

• Make sure applications can be built to target a RO filesystem
• Make sure applications can be upgraded/downgraded on RO filesystems (if possible)

Could not compile project

Steps to reproduce

Add dependency {:distillery, "~> 0.9"} to mix.exs and try run
mix compile after mix deps.get

Verbose Logs

== Compilation error on file lib/mix/lib/releases/archiver.ex ==
** (CompileError) lib/mix/lib/releases/archiver.ex:121: unhandled operator ->
(stdlib) lists.erl:1353: :lists.mapfoldl/3
(stdlib) lists.erl:1354: :lists.mapfoldl/3

Description of issue

• Windows 10, Elixir 1.2.5

Move release output to _build

Instead of pushing build output into rel/<release_name>, we should leverage _build for these.

• Move rel/<release_name>/* to _build/<env>/rel/<release_name>/*
• Modify release.clean
• Document changes, add notice to README

Foreground mode doesn't work on Ubuntu

When trying to run the generated release on my Ubuntu machine, the release won't start and gives the error "trap: SIGUSR1: bad trap".

I believe this is due to the trap command being give the signals in the form SIGUSR1 / SIGUSR2, which dash (the Ubuntu default shell) doesn't accept. It wants them to be without the SIG prefix: USR1 and USR2.

Apparently, the correct form is to provide the signals without the prefix, at least according to the Open Group's specs (http://pubs.opengroup.org/onlinepubs/007904975/utilities/trap.html): "The condition can be [...] a signal specified using a symbolic name, without the SIG prefix, [...]".

The offending line is in priv/templates/boot.eex, in the case for handling foreground launches:
trap '{}' QUIT TERM KILL HUP SIGUSR1 SIGUSR2

The script starts working when changed into this:
trap '{}' QUIT TERM KILL HUP USR1 USR2

systemd setup?

ubuntu is recommending systemd instead of upstart now so I was wondering how you would set it up. Does this seem right? https://gist.github.com/KozakIvan/fbc992fb1f864de1b907

mix release --env flag problem

To reproduce..

mix phoenix.new sample_app
mix.exs << {:distillery, "~> 0.6"}
mix deps.get
mix release.init

Add staging environment into rel/config.exs

environment :staging do
  set dev_mode: false
  set include_erts: true
end

$ mix release --env=staging
==> Assembling release..
==> Failed to build release:
    :no_environments

If you change default_environment: :default to default_environment: :staging, it will work.

bin/<app_name> dies silently if erl is not found

Steps to reproduce

Remove the directory erl lives in from $PATH. Run /bin/<name>. There will be no output.

Verbose Logs

No output. Exit status of 1.

Description of issue

I expect to see a message explaining why the command failed.

Documentation

Need useful, complete documentation

• Getting started guide
• Glossary of terminology used when discussing this project and releases in general
• Walkthrough of a complete end-to-end deployment/upgrade of a Phoenix application, with modified assets
• Walkthrough of a complete end-to-end deployment/upgrade of an umbrella app
• Document how to do Ecto migrations, or other one time operations when upgrading/downgrading
• Document use of boot script hooks
• Document use of custom commands
• Document known issues
• Clean up and flesh out module/function docs

Pass command line args to start command

See bitwalker/exrm#331

consolidated directory does not exist

Steps to reproduce

mix release

Verbose Logs

  ...
  ...
  mix-1.3.2
    from: /usr/local/Cellar/elixir/1.3.2/bin/../lib/mix
    applications:
      {:arc_ecto, :load}
      {:arc, :load}
      {:bamboo_smtp, :load}
      {:bamboo, :load}
      {:calecto, :load}
      {:calendar_translations, :load}
      {:calendar, :load}
      {:comeonin, :load}
      {:cowboy, :load}
      {:credo, :load}
      {:earmark, :load}
      {:ecto, :load}
      {:edeliver, :load}
      {:eiconv, :load}
      {:ex_aws, :load}
      {:geo, :load}
      {:gettext, :load}
      {:html_sanitize_ex, :load}
      {:httpoison, :load}
      {:icalendar, :load}
      {:joken, :load}
      {:mailman, :load}
      {:oauth2, :load}
      {:phoenix_calendar, :load}
      {:phoenix_ecto, :load}
      {:phoenix_html, :load}
      {:phoenix_pubsub, :load}
      {:phoenix_simple_form, :load}
      {:phoenix, :load}
      {:porcelain, :load}
      {:postgrex, :load}
      {:quantum, :load}
      {:scrivener_ecto, :load}
      {:scrivener_html, :load}
      {:scrivener, :load}
      {:ex_machina, :load}
      {:excoveralls, :load}
      {:mix_test_watch, :load}
      {:phoenix_live_reload, :load}
      {:distillery, :load}
      {:sshex, :load}
      :kernel
      :stdlib
      :elixir
    includes: none

==> Generated overlay vars:
    release_name=:my_app
    release_version="1.0.3541"
    is_upgrade=false
    upgrade_from=:latest
    dev_mode=true
    include_erts=false
    include_src=false
    include_system_libs=true
    erl_opts=""
    erts_vsn=nil
    output_dir="rel/my_app"
==> Copying applications to rel/my_app
==> Failed to build release:
    {:copy_applications, "no match of right hand side value: {:error, :enoent, \"/my_app/_build/dev/consolidated\"}"}

Description of issue

• always fails to build release (also on building server using edeliver)
• the my_app/_build/dev/consolidated directory does not exist but distillery tries to copy it

Fails to run release in foreground

I can successfully build a release, start it and use a Phoenix application on my computer. However, when I try to run the "foreground" script it fails. I'm wanting to run the script in a docker container, but for now everything is on my development machine while I try to just get it working.

$ cd /home/mark/dev/connect/rel/connect/releases/0.0.1/

$ ./connect.sh foreground
Exec: /home/mark/dev/connect/rel/connect/erts-8.0/bin/erlexec -noshell -noinput +Bd -boot /home/mark/dev/connect/rel/connect/releases/0.0.1/connect -mode embedded -config /home/mark/dev/connect/rel/connect/releases/0.0.1/sys.config -boot_var ERTS_LIB_DIR /home/mark/dev/connect/rel/connect/erts-8.0/../lib -args_file /home/mark/dev/connect/rel/connect/releases/0.0.1/vm.args -env ERL_LIBS /home/mark/dev/connect/rel/connect/lib -pa /home/mark/dev/connect/rel/connect/lib/connect-0.0.1/consolidated -extra -- foreground
Root: /home/mark/dev/connect/rel/connect
trap: SIGUSR1: bad trap

As I said, the release is working, I can start it, interact with the application (Phoenix) and stop it. It fails when trying to run in foreground. I can't figure out why. Most of the guides for deploying an elixir release in a docker container recommend running it that way.

Application starts with foreground option but not start option

Steps to reproduce

I am attempting to package and run a "hello world" example using Plug, as described in the supervised handlers section of the Plug documentation.

I kept the rel/config.exs file unchanged from the default produced by mix release.init.

Steps:

1. Run mix release
2. Start the app
3. Send a ping to see if it's running

When running with the foreground option or via iex -S mix, it all works fine. When running the start option, the app doesn't start. So in essence, this doesn't work:

$ ./rel/imby_platform/bin/imby_platform start
$ ./rel/imby_platform/bin/imby_platform ping
Node '[email protected]' not responding to pings.

Whereas this works:

$ ./rel/imby_platform/bin/imby_platform foreground

then from a different terminal:

$ ./rel/imby_platform/bin/imby_platform ping
pong

to add complication to the problem, when I stop the foreground app with ^C, I get the prompt back but the app doesn't stop.

Verbose Logs

$ mix release --verbose
==> Loading configuration..
==> Assembling release..
==> Building release imby_platform:0.1.0 using environment dev
==> Discovered applications:
sasl-3.0
from: /usr/lib/erlang/lib/sasl-3.0
applications:
:kernel
:stdlib
includes: none

iex-1.3.2
from: /usr/local/lib/elixir/bin/../lib/iex
applications:
:kernel
:stdlib
:elixir
includes: none

imby_platform-0.1.0
from: _build/dev/lib/imby_platform
applications:
:kernel
:stdlib
:elixir
:logger
:cowboy
:plug
includes: none

kernel-5.0
from: /usr/lib/erlang/lib/kernel-5.0
applications: none
includes: none

stdlib-3.0.1
from: /usr/lib/erlang/lib/stdlib-3.0.1
applications:
:kernel
includes: none

elixir-1.3.2
from: /usr/local/lib/elixir/bin/../lib/elixir
applications:
:kernel
:stdlib
:compiler
includes: none

compiler-7.0.1
from: /usr/lib/erlang/lib/compiler-7.0.1
applications:
:kernel
:stdlib
includes: none

logger-1.3.2
from: /usr/local/lib/elixir/bin/../lib/logger
applications:
:kernel
:stdlib
:elixir
includes: none

cowboy-1.0.4
from: _build/dev/lib/cowboy
applications:
:kernel
:stdlib
:ranch
:cowlib
:crypto
includes: none

ranch-1.2.1
from: _build/dev/lib/ranch
applications:
:kernel
:stdlib
includes: none

cowlib-1.0.2
from: _build/dev/lib/cowlib
applications:
:kernel
:stdlib
:crypto
includes: none

crypto-3.7
from: /usr/lib/erlang/lib/crypto-3.7
applications:
:kernel
:stdlib
includes: none

plug-1.2.0
from: _build/dev/lib/plug
applications:
:kernel
:stdlib
:elixir
:crypto
:logger
:mime
includes: none

mime-1.0.1
from: _build/dev/lib/mime
applications:
:kernel
:stdlib
:elixir
includes: none

==> Generated overlay vars:
release_name=:imby_platform
release_version="0.1.0"
is_upgrade=false
upgrade_from=:latest
dev_mode=true
include_erts=false
include_src=false
include_system_libs=true
erl_opts=""
erts_vsn="8.0.2"
output_dir="rel/imby_platform"
==> Copying applications to rel/imby_platform
==> Generating nodetool
==> Generating start_erl.data
==> Generating vm.args
==> Generating sys.config from config/config.exs
==> Generating boot script
==> Generating RELEASES
==> Generating start_clean.boot
==> Applying overlays
==> Applying mkdir overlay
dst: releases/0.1.0/hooks
==> Applying mkdir overlay
dst: releases/0.1.0/commands
==> You have set dev_mode to true, skipping archival phase
==> Release successfully built!
You can run it in one of the following ways:
Interactive: rel/imby_platform/bin/imby_platform console
Foreground: rel/imby_platform/bin/imby_platform foreground
Daemon: rel/imby_platform/bin/imby_platform start

Description of issue

See above.

This is on Ubuntu 16.04, Erlang 19.0, ERTS 8.0.2, Elixir 1.3.2

elr/config.exs is unchanged from the default generated by mix release.init:

use Mix.Releases.Config,
    # This sets the default release built by `mix release`
    default_release: :default,
    # This sets the default environment used by `mix release`
    default_environment: :dev

# For a full list of config options for both releases
# and environments, visit https://hexdocs.pm/distillery/configuration.html


# You may define one or more environments in this file,
# an environment's settings will override those of a release
# when building in that environment, this combination of release
# and environment configuration is called a profile

environment :dev do
  set dev_mode: true
  set include_erts: false
end

environment :prod do
  set include_erts: true
  set include_src: false
end

# You may define one or more releases in this file.
# If you have not set a default release, or selected one
# when running `mix release`, the first release in the file
# will be used by default

release :imby_platform do
  set version: current_version(:imby_platform)
end

Setting include_system_libs to false fails the release

Steps to reproduce

Grab a fresh test project, like this one

Open rel/config.exs and change:

environment :prod do
  set include_erts: false
  set include_src: false
end

to

environment :prod do
  set include_erts: false
  set include_src: false
  set include_system_libs: false
end

Execute

MIX_ENV=prod mix do compile, phoenix.digest, release --env=prod --verbose

Verbose Logs

==> Packaging release..
==> Archiving distilleryTest-0.0.2
==> Writing tarball to rel/distilleryTest/releases/0.0.2/distilleryTest.tar.gz
==> Updating tarball
==> Stripping system libs from release tarball
==> Problem generating release tarball:
    Failed to create temporary directory `{'/tmp/.tmp_dir407353268/lib/tmp/.tmp_dir407353268/lib/compiler-7.0', :enoent}`

Description of issue

OS X 10.11.6

conform and umbrella: *.conf file is not copied

Steps to reproduce

When using distillery with conform and an umbrella project, the *.conf file of sub apps is not copied to release folder.

Verbose Logs

$ MIX_ENV=prod mix release --verbose --env=prod
==> Loading configuration..
==> Assembling release..
==> Building release bar:0.1.0 using environment prod
==> Discovered applications:
  foo-0.1.0
    from: _build/prod/lib/foo
    applications:
      :kernel
      :stdlib
      :elixir
      :conform
      :logger
    includes: none

  sasl-3.0
    from: /usr/local/Cellar/erlang/19.0.2/lib/erlang/lib/sasl-3.0
    applications:
      :kernel
      :stdlib
    includes: none

  iex-1.3.2
    from: /usr/local/Cellar/elixir/1.3.2/bin/../lib/iex
    applications:
      :kernel
      :stdlib
      :elixir
    includes: none

  elixir-1.3.2
    from: /usr/local/Cellar/elixir/1.3.2/bin/../lib/elixir
    applications:
      :kernel
      :stdlib
      :compiler
    includes: none

  kernel-5.0
    from: /usr/local/Cellar/erlang/19.0.2/lib/erlang/lib/kernel-5.0
    applications: none
    includes: none

  stdlib-3.0.1
    from: /usr/local/Cellar/erlang/19.0.2/lib/erlang/lib/stdlib-3.0.1
    applications:
      :kernel
    includes: none

  compiler-7.0.1
    from: /usr/local/Cellar/erlang/19.0.2/lib/erlang/lib/compiler-7.0.1
    applications:
      :kernel
      :stdlib
    includes: none

  conform-2.1.1
    from: _build/prod/lib/conform
    applications:
      :kernel
      :stdlib
      :elixir
      :neotoma
    includes: none

  neotoma-1.7.3
    from: _build/prod/lib/neotoma
    applications:
      :kernel
      :stdlib
    includes: none

  logger-1.3.2
    from: /usr/local/Cellar/elixir/1.3.2/bin/../lib/logger
    applications:
      :kernel
      :stdlib
      :elixir
    includes: none

==> conform: loading schema
==> conform: no schema found, skipping
==> Generated overlay vars:
    release_name=:bar
    release_version="0.1.0"
    is_upgrade=false
    upgrade_from=:latest
    dev_mode=false
    include_erts=true
    include_src=false
    include_system_libs=true
    erl_opts=""
    erts_vsn="8.0.2"
    output_dir="rel/bar"
==> Copying applications to rel/bar
==> Generating nodetool
==> Generating start_erl.data
==> Generating vm.args
==> Generating sys.config from config/config.exs
==> Including ERTS 8.0.2 from /usr/local/Cellar/erlang/19.0.2/lib/erlang/erts-8.0.2
==> Generating boot script
==> Generating RELEASES
==> Generating start_clean.boot
==> Applying overlays
==> Applying mkdir overlay
    dst: releases/0.1.0/hooks
==> Applying copy overlay
    src: _build/prod/lib/conform/priv/bin/pre_start.sh
    dst: releases/0.1.0/hooks/pre_start
==> Applying mkdir overlay
    dst: releases/0.1.0/commands
==> Packaging release..
==> Archiving bar-0.1.0
==> Writing tarball to rel/bar/releases/0.1.0/bar.tar.gz
==> Updating tarball
==> Tarball updated!
==> Release successfully built!
    You can run it in one of the following ways:
      Interactive: rel/bar/bin/bar console
      Foreground: rel/bar/bin/bar foreground
      Daemon: rel/bar/bin/bar start

Description of issue

• What are the expected results?
  The *.conf file of the subapp using conform should be copied to the release folder.
• What OS, Erlang/Elixir versions are you seeing this issue on?
  Mac OS X 10.11.6, Erlang/OTP 19 [erts-8.0.2], Elixir (1.3.2)
• If possible, also provide your rel/config.exs, as it is often
  my first troubleshooting question, and you'll save us both time :)
  I created a small project here to reproduce the problem: https://github.com/ahamez/distillery-umbrella-conform

In the foo sub app, there is a config/foo.schema.exs, but logs of distillery seem to say otherwise. Maybe schemas should be in a different place when using distillery?

Circular dependencies among applications

Steps to reproduce

1. Clone https://github.com/chemeris/release_test
2. Run

mix deps.get
mix release.init
mix release

Verbose Logs

$ mix release --verbose
==> Loading configuration..
==> Assembling release..
==> Building release release_test:0.1.0 using environment dev
==> Discovered applications:
  sasl-3.0
    from: /usr/lib/erlang/lib/sasl-3.0
    applications:
      :kernel
      :stdlib
    includes: none

  iex-1.3.1
    from: /usr/local/lib/elixir/bin/../lib/iex
    applications:
      :kernel
      :stdlib
      :elixir
    includes: none

  release_test-0.1.0
    from: _build/dev/lib/release_test
    applications:
      :kernel
      :stdlib
      :elixir
      :lager
      :exometer_core
      :edown
      :exometer_lager
      :meck
      :parse_trans
    includes: none

  kernel-5.0
    from: /usr/lib/erlang/lib/kernel-5.0
    applications: none
    includes: none

  stdlib-3.0
    from: /usr/lib/erlang/lib/stdlib-3.0
    applications:
      :kernel
    includes: none

  elixir-1.3.1
    from: /usr/local/lib/elixir/bin/../lib/elixir
    applications:
      :kernel
      :stdlib
      :compiler
    includes: none

  compiler-7.0
    from: /usr/lib/erlang/lib/compiler-7.0
    applications:
      :kernel
      :stdlib
    includes: none

  lager-3.2.1
    from: _build/dev/lib/lager
    applications:
      :kernel
      :stdlib
      :goldrush
    includes: none

  goldrush-0.1.8
    from: _build/dev/lib/goldrush
    applications:
      :kernel
      :stdlib
      :syntax_tools
      :compiler
    includes: none

  syntax_tools-2.0
    from: /usr/lib/erlang/lib/syntax_tools-2.0
    applications:
      :stdlib
    includes: none

  exometer_core-1.4-39-g9d41a0a
    from: _build/dev/lib/exometer_core
    applications:
      :kernel
      :stdlib
      :hut
      :setup
      :folsom
    includes: none

  hut-1.1.1
    from: _build/dev/lib/hut
    applications:
      :kernel
      :stdlib
    includes: none

  setup-1.6
    from: _build/dev/lib/setup
    applications:
      :kernel
      :stdlib
    includes: none

  folsom-0.8.2
    from: _build/dev/lib/folsom
    applications:
      :kernel
      :stdlib
      :bear
    includes: none

  bear-0.8.2
    from: _build/dev/lib/bear
    applications: none
    includes: none

  edown-0.8.1
    applications:
      :kernel
      :stdlib
      :edoc
    includes: none

  edoc-0.7.19
    from: /usr/lib/erlang/lib/edoc-0.7.19
    applications:
      :compiler
      :kernel
      :stdlib
      :syntax_tools
    includes: none

  exometer_lager-ea15557
    from: _build/dev/lib/exometer_lager
    applications:
      :kernel
      :stdlib
      :lager
      :exometer_core
    includes:
      :hut
  meck-0.8.2
    from: _build/dev/lib/meck
    applications:
      :kernel
      :stdlib
    includes: none

  parse_trans-2.9.2
    from: _build/dev/lib/parse_trans
    applications:
      :kernel
      :stdlib
      :syntax_tools
    includes: none

==> Generated overlay vars:
    release_name=:release_test
    release_version="0.1.0"
    is_upgrade=false
    upgrade_from=:latest
    dev_mode=true
    include_erts=false
    include_src=false
    include_system_libs=true
    erl_opts=""
    erts_vsn="8.0"
    output_dir="rel/release_test"
==> Copying applications to rel/release_test
==> Generating nodetool
==> Generating start_erl.data
==> Generating vm.args
==> Generating sys.config from config/config.exs
==> Generating boot script
==> Failed to build release:

    Circular dependencies among applications: [{exometer_lager,"ea15557"},
                                               {exometer_core,"1.4-39-g9d41a0a"}]

Description of issue

• Expected result: successful release generated
• OS Ubuntu 14.04 64-bit, Erlang/Elixir:

Erlang/OTP 19 [erts-8.0] [source-6dc93c1] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false]
Elixir (1.3.1)

Docker -d stops immediately after container starts in foreground mode

Dockerfile:

# Set the Docker image you want to base your image off.
# I chose this one because it has Elixir preinstalled.
FROM trenpixster/elixir:1.3.2

# Install other stable dependencies that don't change often

# Compile app
RUN mkdir /app
WORKDIR /app

# Install Elixir Deps
ADD mix.* ./
ENV MIX_ENV=prod
RUN mix local.rebar --force
RUN mix local.hex --force
RUN mix deps.get

# Generate release
ADD . .
RUN mix do compile, release --env=default --verbose

# Create folder for logs
RUN mkdir -p ./log

# Clean sources, but save migrations for Ecto.Migrator
RUN if [ -d "priv" ]; then mkdir rel/priv; mv priv/* rel/priv; fi
RUN find . -maxdepth 1 -not -name "rel" -not -name "." -exec rm -rf {} \;

# Change working directory to release dir
WORKDIR rel
RUN rm config.exs

# Allow to read ENV vars for mix configs
ENV REPLACE_OS_VARS=true

# Pre-run hook that allows you to add initialization scripts.
# They should be located in rel/hooks directory, rest of project will be removed on build.
RUN hooks/pre_run.sh

# Runtime config

# Exposes this port from the docker container to the host machine
EXPOSE 4000

# The command to run when this image starts up
#  You can run it in one of the following ways:
#    Interactive: sample/bin/sample console
#    Foreground: sample/bin/sample foreground
#    Daemon: sample/bin/sample start
CMD sample/bin/sample foreground

Sample run:

~/Projects/www/renew/sample(master ✗) docker rm 073aba1db5f9592f9d38133253ef27038b60b2cca9b7f12263cbf0330c027208
073aba1db5f9592f9d38133253ef27038b60b2cca9b7f12263cbf0330c027208
~/Projects/www/renew/sample(master ✗) docker run -p 4000 \
>        --env-file .env \
>        --name sample \
>        -d sample
2a59a0edaea98d47dcdc3fde4ed1f819d622ad84084b5278838f16bb61774d40

~/Projects/www/renew/sample(master ✗) docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

Is there any catches in distillery that doesn't allow run docker container with elixir in detached mode?

Passing arguments to rpc functions

bitwalker, how are function arguments supposed to be passed to rpc.
Using exrm i can do

rel/myapp/bin/myapp rpc Elixir.Application get_all_env 'myapp.'

or

rel/analysis_backend/bin/analysis_backend rpc calendar day_of_the_week "[2016,08,13]."

and it works as expected. But it seems like this is not the correct way with distillery releases..

I couldn't find any tests or examples.

Test suite

Need a solid test suite which can be run in CI, with coverage reporting.

Tests should run/work on Erlang 18.3 and 19.x, using Elixir 1.3+

• Utility tests
• Appup tests
• Integration tests for standard apps
• Integration tests for umbrella apps