onkel-dirtus/logger_file_backend

logger_file_backend's Issues

Fix dialyzer warnings

$ mix dialyzer
==> erlex
Compiling 1 file (.yrl)
src/parser.yrl: Warning: conflicts: 25 shift/reduce, 0 reduce/reduce
Compiling 1 file (.xrl)
Compiling 2 files (.erl)
Compiling 1 file (.ex)
Generated erlex app
==> dialyxir
Compiling 56 files (.ex)
Generated dialyxir app
==> logger_file_backend
Compiling 1 file (.ex)
Generated logger_file_backend app
Finding suitable PLTs
Checking PLT...
[:compiler, :elixir, :kernel, :stdlib]
Looking up modules in dialyxir_erlang-21.2.5_elixir-1.8.1_deps-dev.plt
Looking up modules in dialyxir_erlang-21.2.5_elixir-1.8.1.plt
Finding applications for dialyxir_erlang-21.2.5_elixir-1.8.1.plt
Finding modules for dialyxir_erlang-21.2.5_elixir-1.8.1.plt
Checking 418 modules in dialyxir_erlang-21.2.5_elixir-1.8.1.plt
Finding applications for dialyxir_erlang-21.2.5_elixir-1.8.1_deps-dev.plt
Finding modules for dialyxir_erlang-21.2.5_elixir-1.8.1_deps-dev.plt
Copying dialyxir_erlang-21.2.5_elixir-1.8.1.plt to dialyxir_erlang-21.2.5_elixir-1.8.1_deps-dev.plt
Looking up modules in dialyxir_erlang-21.2.5_elixir-1.8.1_deps-dev.plt
Checking 418 modules in dialyxir_erlang-21.2.5_elixir-1.8.1_deps-dev.plt
Adding 48 modules to dialyxir_erlang-21.2.5_elixir-1.8.1_deps-dev.plt
Starting Dialyzer
[
  check_plt: false,
  init_plt: '/Users/me/workspace/logger_file_backend/_build/dev/dialyxir_erlang-21.2.5_elixir-1.8.1_deps-dev.plt',
  files_rec: ['/Users/me/workspace/logger_file_backend/_build/dev/lib/logger_file_backend/ebin'],
  warnings: [:unknown]
]
Total errors: 4, Skipped: 0
done in 0m1.08s
:0:unknown_function
Function Logger.compare_levels/2 does not exist.
________________________________________________________________________________
:0:unknown_function
Function Logger.Formatter.compile/1 does not exist.
________________________________________________________________________________
:0:unknown_function
Function Logger.Formatter.format/5 does not exist.
________________________________________________________________________________
:0:unknown_type
Unknown type: Logger.level/0.
________________________________________________________________________________
done (warnings were emitted)

Rotate config details when Sharing log file between two or more apps

If you have two or more (phoenix) applications which log to the same file, where would you put the rotate specifications in both the config (not DRY), or put it in either or somehow share the config reference between the apps?

Log routing according to metadata

In other logging settings, I've found value in being able to redirect certain log statements to individual files e.g.: in one application that manages 3 identical serial devices- my logging is configured to create logs <device1>.log, <device2>.log, <device3>.log, each with the serial traffic for just one device, rather than an interleaved log that has UI considerations, DB changes, etc.

Has this idea already been floated? Currently, it looks like a config stanza is needed per file; would it be possible to have a single config stanza create/append to multiple files, and specify a metadata wildcard in the :path key, like such:

config :logger, :serial_devices,
  path: "/path/to/logs/device_$metadata[$device_id].log",
  level: :info

A different, but related capability: being able to deliver messages to a log based on the presence of a metadata key:

config :logger, :ui_logging,
  path: "/path/to/logs/ui.log"
  level: :info,
  required_metadata: [application: :ui]  # or a custom key

There is probably a clever syntax for specifying the filter better than what I have- while I'm suggesting a syntax to be constructive, I'm not wedded to it, and I think there's tons of room for improvement.

Generally, what's the reaction to this idea? Is it something the community would be open to including in this project?

Adding documentation that is provided in Readme to hex

Thanks for this very useful library!

As it currently stands, all the usage related documentation is provided in the README file and none of that is reflected in hexdocs. It would be useful to add this documentation or at-least some part of it to the docs for the library.

Please let me know if this sounds like a good idea. I shall send a PR accordingly.

Removing logger_file_backend does not release file descriptor

I believe this backend is leaking file descriptors after the backend has been removed.

Our application starts a number of temporary GenServers that are started to handle on-demand data processing. I have the following code (actual module and structs renamed) that is called during the GenServer's init/2 callback to add a file backend for the GenServer and remove the backend when the GenServer terminates:

defmodule JobServer.Jobs.Logging do
  @moduledoc """
  Conveniences for configuring logging for job runners.
  """

  alias JobServer.Jobs.Job

  require Logger

  @doc """
  Setup a logger file backend for the current process and given `job_id`.

  The backend will be automatically removed when the current process exits.
  """
  def setup_logger_file_backend(%Job{id: job_id} = job) do
    Logger.metadata(job_id: job_id)

    opts = [
      path: job |> Job.outputs_path() |> Path.join("log"),
      format: "$date $time [$level]$levelpad $message\n",
      level: :info,
      metadata_filter: [application: :job_server, job_id: job_id]
    ]

    {:ok, _pid} = Logger.add_backend({LoggerFileBackend, job_id}, flush: true)
    Logger.configure_backend({LoggerFileBackend, job_id}, opts)

    # Start a background task to monitor the current process and remove
    # the backend when this process exits
    Task.start(__MODULE__, :teardown_logger_file_backend_on_exit, [job_id, self()])
  end

  @doc """
  Monitor the given `pid` and, upon exit, remove the logger file backend
  associated with `job_id`.
  """
  def teardown_logger_file_backend_on_exit(job_id, pid) do
    ref = Process.monitor(pid)

    receive do
      {:DOWN, ^ref, :process, ^pid, _reason} ->
        Logger.remove_backend({LoggerFileBackend, job_id}, flush: true)
    end
  end
end

I can see in Observer that the Erlang process associated with the logger file backend is created and terminated as expected but looking at the operating system file descriptors shows that these log files are still open after the associated backend processes are shutdown:

> ls -l /proc/<os_pid>/fd
...
l-wx------. 1 vagrant vagrant 64 Oct  5 10:41 35 -> /home/vagrant/code/software/jobserver/_tmp/jobs/outputs/e7dea011-bcc5-4f31-bb0d-5e9f16de8616/log
l-wx------. 1 vagrant vagrant 64 Oct  5 10:41 36 -> /home/vagrant/code/software/jobserver/_tmp/jobs/outputs/0af6fe65-4f54-415d-9ff3-c1722feaad08/log
...

I believe this is a fairly serious one IMO as it breaks the assumption that removing a file backend will free all operating system resources claimed when starting said backend. We discovered this issue after our application became unresponsive after exhausting the file descriptors available to the Linux process.

Let me know if there's any additional information that would help with identifying the cause of this issue.

Several warnings with Elixir 1.7.4

Hi

the following warnings are generated by the compiler with the above elixir version

warning: the GenEvent module is deprecated, see its documentation for alternatives
lib/logger_file_backend.ex:5

warning: GenEvent.using/1 is deprecated. Use one of the alternatives described in the documentation for the GenEvent module
lib/logger_file_backend.ex:5

Update hex package to include utf-8 fix

Can you please update the hex package to include fix for #3?

Logger crash

This crashes the Logger:

Logger.debug(" “ ")
(That's a curly quote in there)

20:31:58.104 [error] --> GenEvent handler {LoggerFileBackend, :file_log} installed at Logger
** (exit) an exception was raised:
** (ErlangError) erlang error: :no_translation
(stdlib) :io.put_chars(#PID<0.170.0>, :unicode, [">>>> ", ["20", 58, "31", 58, "58", 46, [48, "94"]], " [", "debug", "] --> ", "“", "\n"])
lib/logger_file_backend.ex:55: LoggerFileBackend.log_event/5
(elixir) lib/gen_event.ex:990: GenEvent.do_handler/3
(elixir) lib/gen_event.ex:831: GenEvent.server_update/5
(elixir) lib/gen_event.ex:816: GenEvent.server_notify/7
(elixir) lib/gen_event.ex:789: GenEvent.server_event/4
(elixir) lib/gen_event.ex:578: GenEvent.handle_msg/5
(stdlib) proc_lib.erl:237: :proc_lib.init_p_do_apply/3

Here's my config.exs:
config :logger, :console,
level: :debug,
format: ">>>> $time [$level] --> $message\n"

  config :logger,
    backends: [{LoggerFileBackend, :file_log}, :console]

  config :logger, :file_log,
    path: "/var/log/riffer4/file_log",
   level: :debug,
  format: ">>>> $time [$level] --> $message\n"

Regards,

Mike

GenEvent module is deprecated

warning: the GenEvent module is deprecated, see its documentation for alternatives
lib/logger_file_backend.ex:5

Logger does not support "truncate" options and truncates output always.

config :logger, :dev_log,
  path: "/tmp/dev.log",
  level: :debug,
  format: "$date\t$time\t$level\t●$message●$metadata\n",
  truncate: :infinity

Still truncates long output with "(truncate)" at end

open file with utf8 mode, otherwise this backend would crash when log message contains utf8 characters

Please adapt it to new Elixir Version(1.5.1) PLS!

GenEvent is depracated in Elixir v1.5.1, pls adapt to the new version, thanks.

0.0.10

I've seen the repo has a fix and the version has been bumped to 0.0.10 however it's not on hex.pm, did you forgot to push it or there is a reason?

File logging in production

When working locally with the configuration below, file logging works. However, when created a release using MIX_ENV=test mix release --env=prod and running the application using _build/dev/rel/hutt/bin/hutt start command file logging does not work

Config.exs
-----------
 config :logger, :hutt,
        format: "$date $time $metadata[$level] $message\n",
        path: "log/hutt.log",
        level: :info`

How to set format on backends?

I am not sure whether this is an issue or a question.

Below is my configuration which works fine:

config :logger, backends: [{LoggerFileBackend, :error_log}]

Now I want to add a format for the log and I am not sure how to do that. I tried a few configuration but none of them work as below. How can I configure the format?

config :logger, backends: [{LoggerFileBackend, :error_log, format: "\n$date $time $metadata $levelpad$message\n"}]

config :logger, backends: [{LoggerFileBackend, :error_log}],
   format: "\n$date $time $metadata $levelpad$message\n"

relative path default absolute location

When specifying a relative log path like in:

config :logger, :debug,
  path: "log/debug.log",
  level: :debug,

Would it be possible to add a config option for something like:

config :logger_file_backend, :relative_path_pwd, "/some/base/directory" # defaults to program global pwd

It would be convenient to have the host of file loggers have a relative location that could then be specified based on the environment at a single location without changing the global pwd. :-)

More detailed README how to use this backend

hi,

I'm having trouble configuring my application to use the LoggerFileBackend and the current README doesn't help much.
Is there a more detailed example somewhere?

Thanks!
Michael

Add fireproofsocks as hex.owner

Is it possible for you to run the mix hex.owner task to either add me as another owner of this Hex package or transfer the package to me? That way I can release the patched versions in hex.pm. Without this, we cannot release updates to this package.

Do you have any plan to add suffix for path?

In this case, the log file is logs/info.log.

config :logger, :foo,
  path: "logs/info.log",
  level: :info,

It's useful that the log file would be logs/info.log.20160105123456, isn't it?

config :logger, :foo,
  path: "logs/info.log.",
  level: :info,

  suffix: some_function_which_returns_datetime()

Failed to start child `Elixir.Logger.Watcher`

I think this is similar to #8 but ..

I'm using distillery, and I can't fix it the same way because.. I put logger_file_backend at the umbrella app level.

Version check for elixir states <= 1.2.0

We're at 1.2.1 now.

I originally forked it with a view to submitting a pull request but then I wasn't entirely sure how you wanted to treat the 1.1.0 series. Wanted to make sure I knew your intent before pulling.

Use application specific configuration keys

I was responding to a question someone had about this package, and I noticed that you're using the :logger application configuration to hold configuration specific to this package.

You should instead be using your application's own configuration space, in this case :logger_file_backend, e.g.:

config :logger_file_backend, :error_log,
  path: "/var/log/my_app/error.log"

does not work

I tried to use it, but I got this error:

=INFO REPORT==== 15-Feb-2017::17:19:34 ===
    application: logger
    exited: {{shutdown,
              {failed_to_start_child,'Elixir.Logger.Watcher',
               {'EXIT',
                {{badmatch,
                  {error,
                   {{{case_clause,
                      {'EXIT',
                       {undef,
                        [{'Elixir.LoggerFileBackend',init,
                          [{'Elixir.LoggerFileBackend',error_log}],
                          []},
                         {gen_event,server_add_handler,4,
                          [{file,"gen_event.erl"},{line,429}]},
                         {gen_event,handle_msg,5,
                          [{file,"gen_event.erl"},{line,274}]},
                         {proc_lib,init_p_do_apply,3,
                          [{file,"proc_lib.erl"},{line,247}]}]}}},
                     [{'Elixir.Logger.Watcher',do_init,3,
                       [{file,"lib/logger/watcher.ex"},{line,78}]},
                      {gen_server,init_it,6,
                       [{file,"gen_server.erl"},{line,328}]},
                      {proc_lib,init_p_do_apply,3,
                       [{file,"proc_lib.erl"},{line,247}]}]},
                    {child,undefined,
                     {'Elixir.Logger.Watcher',
                      {'Elixir.Logger',
                       {'Elixir.LoggerFileBackend',error_log}}},
                     {'Elixir.Logger.Watcher',watcher,
                      ['Elixir.Logger',
                       {'Elixir.LoggerFileBackend',error_log},
                       {'Elixir.LoggerFileBackend',error_log}]},
                     transient,5000,worker,
                     ['Elixir.Logger.Watcher']}}}},
                 [{'Elixir.Logger.Watcher','-start_link/3-fun-0-',2,
                   [{file,"lib/logger/watcher.ex"},{line,16}]},
                  {'Elixir.Enum','-reduce/3-lists^foldl/2-0-',3,
                   [{file,"lib/enum.ex"},{line,1755}]},
                  {'Elixir.Logger.Watcher',start_link,3,
                   [{file,"lib/logger/watcher.ex"},{line,15}]},
                  {supervisor,do_start_child,2,
                   [{file,"supervisor.erl"},{line,365}]},
                  {supervisor,start_children,3,
                   [{file,"supervisor.erl"},{line,348}]},
                  {supervisor,init_children,2,
                   [{file,"supervisor.erl"},{line,314}]},
                  {gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},
                  {proc_lib,init_p_do_apply,3,
                   [{file,"proc_lib.erl"},{line,247}]}]}}}},
             {'Elixir.Logger.App',start,[normal,[]]}}
    type: temporary
** (Mix) Could not start application logger: Logger.App.start(:normal, []) returned an error: shutdown: failed to start child: Logger.Watcher
    ** (EXIT) an exception was raised:
        ** (MatchError) no match of right hand side value: {:error, {{{:case_clause, {:EXIT, {:undef, [{LoggerFileBackend, :init, [{LoggerFileBackend, :error_log}], []}, {:gen_event, :server_add_handler, 4, [file: 'gen_event.erl', line: 429]}, {:gen_event, :handle_msg, 5, [file: 'gen_event.erl', line: 274]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 247]}]}}}, [{Logger.Watcher, :do_init, 3, [file: 'lib/logger/watcher.ex', line: 78]}, {:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 328]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 247]}]}, {:child, :undefined, {Logger.Watcher, {Logger, {LoggerFileBackend, :error_log}}}, {Logger.Watcher, :watcher, [Logger, {LoggerFileBackend, :error_log}, {LoggerFileBackend, :error_log}]}, :transient, 5000, :worker, [Logger.Watcher]}}}
            (logger) lib/logger/watcher.ex:16: anonymous fn/2 in Logger.Watcher.start_link/3
            (elixir) lib/enum.ex:1755: Enum."-reduce/3-lists^foldl/2-0-"/3
            (logger) lib/logger/watcher.ex:15: Logger.Watcher.start_link/3
            (stdlib) supervisor.erl:365: :supervisor.do_start_child/2
            (stdlib) supervisor.erl:348: :supervisor.start_children/3
            (stdlib) supervisor.erl:314: :supervisor.init_children/2
            (stdlib) gen_server.erl:328: :gen_server.init_it/6
            (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3

any idea?

here is my config:

config :logger,
backends: [{LoggerFileBackend, :error_log}]

configuration for the {LoggerFileBackend, :error_log} backend

config :logger, :error_log,
path: "/var/log/myapp/phoenix_error.log",
level: :debug

Rolling file.

I was implement a function that rolling file.
how can I push this?

Suspect LoggerFileBackend crashed Phoenix application

Can you please take a look at this issue and confirm if Logger has anything to do with it? My prod application was crashed yesterday and I have attached the logs with the below given issue. I am using version 0.0.4 of LoggerFileBackend.

phoenixframework/phoenix#1681

Note: I had disabled debug mode in prod. I suspect, somehow, the debug mode is enabled and it tried to write some unicode char into file and that failed. Again I couldn't reproduce or confirm that.

Given below is my prod log configuration from prod.exs file:

backends = [{LoggerFileBackend, :info}, {LoggerFileBackend, :warn}, {LoggerFileBackend, :error}]
# backends = [{LoggerFileBackend, :debug}, {LoggerFileBackend, :warn}, {LoggerFileBackend, :error}, :console]
config :logger, format: "[$date] [$time] [$level] $metadata $message\n", metadata: [:module, :line], backends: backends

config :logger, :info,
  path: "log/prod.log",
  level: :info,
  format: "[$date] [$time] [$level] $metadata $message\n",
  metadata: [:date, :module, :line]

# config :logger, :debug, path: "log/prod.log", level: :debug

config :logger, :warn,
  path: "log/prod.log",
  level: :warn,
  format: "[$date] [$time] [$level] $metadata $message\n",
  metadata: [:date, :module, :line]

config :logger, :error,
  path: "log/prod.log",
  level: :error,
  format: "[$date] [$time] [$level] $metadata $message\n",
  metadata: [:date, :module, :line]

config :logger, :error,
  path: "log/error.log",
  level: :error,
  format: "[$date] [$time] [$level] $metadata $message\n",
  metadata: [:date, :module, :line]

Is this project still supported?

I see a bunch of PRs, issues without merge or reaction, is this project still developing or abandoned?
@onkel-dirtus

failed_to_start_child 'Elixir.Logger.Watcher'

I believe this is related to elixir issue #2649. I am trying to use logger in conjunction with exrm to do an OTP release of my app.

I don't quite understand if the issue should have been resolved. Any ideas?

=INFO REPORT==== 1-Oct-2015::09:17:28 ===
    application: logger
    exited: {{shutdown,
              {failed_to_start_child,'Elixir.Logger.Watcher',
               {'EXIT',
                {{badmatch,
                  {error,
                   {undef,
                    [{'Elixir.LoggerFileBackend',init,
                      [{'Elixir.LoggerFileBackend',file_logs}],
                      []},
                     {'Elixir.GenEvent',do_handler,3,
                      [{file,"lib/gen_event.ex"},{line,990}]},
                     {'Elixir.GenEvent',do_add_handler,5,
                      [{file,"lib/gen_event.ex"},{line,947}]},
                     {'Elixir.GenEvent',handle_msg,5,
                      [{file,"lib/gen_event.ex"},{line,608}]},
                     {proc_lib,init_p_do_apply,3,
                      [{file,"proc_lib.erl"},{line,239}]}]}}},
                 [{'Elixir.Logger.Watcher','-start_link/3-fun-0-',2,
                   [{file,"lib/logger/watcher.ex"},{line,19}]},
                  {'Elixir.Enum','-reduce/3-lists^foldl/2-0-',3,
                   [{file,"lib/enum.ex"},{line,1261}]},
                  {'Elixir.Logger.Watcher',start_link,3,
                   [{file,"lib/logger/watcher.ex"},{line,18}]},
                  {supervisor,do_start_child,2,
                   [{file,"supervisor.erl"},{line,343}]},
                  {supervisor,start_children,3,
                   [{file,"supervisor.erl"},{line,326}]},
                  {supervisor,init_children,2,
                   [{file,"supervisor.erl"},{line,292}]},
                  {gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},
                  {proc_lib,init_p_do_apply,3,
                   [{file,"proc_lib.erl"},{line,239}]}]}}}},
             {'Elixir.Logger.App',start,[normal,[]]}}
    type: permanent

  def application do
    [mod: {MyApp, []},
     applications: applications(Mix.env)]
  end

  defp applications(:test), do: applications(:all) ++ [:blacksmith]
  defp applications(_all),  do: [:phoenix, :phoenix_html, :cowboy, :logger,
                    :phoenix_ecto, :postgrex]

New Release?

Is it possible to get a new release that includes #43 ?

Replace String.strip with String.trim

Could I use the "rotate" in prod environment？

I find the configure of "rotate" in code, but not in README.md, so could the "rotate" be used in prod environment？

colors on log file

I've got this config in my test.exs config

config :logger,
  backends: [{LoggerFileBackend, :test}]

config :logger, :test,
  colors: [enabled: true],
  path: "/tmp/test.log",
  level: :debug,
  format: "$time [$level] [$metadata] - $message\n",
  metadata: [:pid, :mfa, :file, :line]

but I can't see colorful logs in the file. Is this some unsupported feature for file logging?

JSON log format?

Is there a way to format the log entries as JSON?

I have an existing application that writes logs in JSON format (via winston) and with it a bunch of offline post-processors for the logs I'd like to be able to re-use.

The backend level is effectively a minimum level

Hi!

I have a case to log messages exactly of the specified level.
Here https://github.com/onkel-dirtus/logger_file_backend/blob/master/lib/logger_file_backend.ex#L37 the condition allows messages of the specified level plus more serious ones.
Wonder if it's feasible to have "exact" mode and how to implement it in the least invasive way.

TIA

Support elixir 1.6 logger

Elixir logger had an issue on heavy load
So Elixir added some new features in new version like sync_threshold and discard_threshold to improve performance.

Please add these features

Specify log file path as {module, function}

Sometimes I need to log to different files based on some log metadata. Imagine for instance of having a variable number of entities handled by your application, it can be useful to log actions regarding each entity in a separate file.

This can already be done with metadata_filter if you know in advance which are the entities you manage. But, if entities can change at application runtime you can't.

My proposal is to allow specifying the path of the log file as a {module, function} tuple, the function will be called giving log metadata as argument (maybe also log_level?) and it should return the log file path where the log entry should be written.

What do you think about? I can submit a PR if you think is worth.

PS: this is somehow related to the first part of #18 but it allows a more fine customization of the path. I'm perfectly ok also with the solution proposed in #18 if you think is safer.

Warnings when compiling on elixir 1.12.1

When I compile my application I get the following error when compiling the logger_file_backend application.

==> logger_file_backend
Compiling 1 file (.ex)
warning: Logger.compare_levels/2 defined in application :logger is used by the current application but the current application does not depend on :logger. To fix this, you must do one of:

  1. If :logger is part of Erlang/Elixir, you must include it under :extra_applications inside "def application" in your mix.exs

  2. If :logger is a dependency, make sure it is listed under "def deps" in your mix.exs

  3. In case you don't want to add a requirement to :logger, you may optionally skip this warning by adding [xref: [exclude: [Logger]]] to your "def project" in mix.exs

  lib/logger_file_backend.ex:36: LoggerFileBackend.handle_event/2

I think that the library should have :logger added to the extra_applications options in order to fix this.

Add HOW TO install in phoenix framework

Need to add this in the readme

Add {:logger_file_backend, "~> 0.0.10"} to your mix.exs in deps

Run mix deps.get