lifo101/php-daemon

PHP Multiprocessing Daemon. Easily create stable and robust daemons with minimal boilerplate code.

PHP Daemon Library

Synopsis

Create robust and stable PHP multiprocess daemons without the boilerplate code. The core Daemon class handles the main loop and events and can run at any frequency desired (within the limits of PHP). You only have to implement a single method execute to run a daemon process, optionally in the background.

Using Tasks and Workers the daemon can call methods on background processes seamlessly w/o worrying about managing forked children. Plugins allow you to easily create reusable and shareable code for your Daemons. See the Features section below for more information.

Why write a daemon in PHP?

Obviously, writing robust, stable and long-running daemons in PHP is generally not a good idea. It's at least very hard to do, and do well. I personally needed a daemon in PHP because I had an entire website framework built in Symfony that needed a major back-end daemon. I wanted to be able to re-use all my front-end dependencies and entities w/o duplicating resources or configs.

While this library does everything it can to allow you to create a rock solid daemon, care must still be taken in your user-land code to keep things stable.

Requirements

• PHP ^7.4 || ^8.0 (for PHP ^5.4 use the ^1.0 tag and branch)
• A POSIX compatible operating system (Linux, OSX, BSD)
• PHP POSIX and PCNTL Extensions

Documentation

See the Wiki for documentation.

Examples

See the examples directory for examples you can run.

Features

• The Main Loop is maintained by the core Daemon class. All you have to do is implement one method execute that will get called every loop cycle. The loop frequency can be any fractional value in seconds. If set to 0, your execute method will get called as fast as possible (not normally recommended, unless your loop is doing some sort of blocking call, ie: listening on a socket, etc).
• In just a few lines of code you can have parallel processes running in the background.

  • A Task allows you to call any method or callback in a background process. No communication is made between the background process and the parent. Tasks are meant for simple things, for example: Sending an email.

  • A Worker allows you to call any method on an object, or even just a simple callback like a Task. Workers can return a value back to the parent via a simple return statement in your worker method(s). Workers are maintained automatically and can have multiple children running at the same time, which is handled transparently. Even if a worker dies or is killed by the OS the Daemon API will still return a result (or exception) to your code. The return value of a Worker is usually a Promise object. You can use the standard Promise methods like then or otherwise to act on the return value. Or you can register an ON_RETURN callback on the Worker.

    Workers use a Mediator design pattern and use Shared Memory for it's messaging queue and data. Different IPC classes can be created to provide alternate communication methods between the parent and children. I might work on a second IPC class that uses sockets instead of SHM to provide an alternate choice.

• Event Handling. The core Daemon has several events (see: Events) that you can easily interface with by registering a callback. Some events have the means to change the behavior of the daemon.
• Easy Signal Handling via the Event Dispatcher. To catch a signal you simply have to register a ON_SIGNAL callback in your code. Your callback will be passed an SignalEvent with the signal that was caught.
• Simple Plugin architecture allows you to use and create your own plugins that can be injected into the Daemon. Plugins can be lazily loaded.
  • A core plugin FileLock allows you to add a locking mechanism to prevent your daemon from running more than one instance at a time. Simply register the plugin in your daemon and the rest is automatic. A ShmLock is similar but uses Shared Memory to obtain a lock.
• Automatic restarting. The Daemon can automatically restart itself if it's runtime reached a configurable threshold or if a fatal error occurred.
• Built in logging. The Daemon has 3 basic logging methods: log, error, debug. All of these will write to the log file (if configured). If the log file is rotated, overwritten or deleted, the daemon will automatically detect this and will continue to write to the new log file. The DaemonEvent::ON_LOG event allows you to register a callback to change the behavior too. User code can use the LogTrait to easily add native daemon logging to their code.

Credit

The basis for this library was inspired by the PHP-Daemon library from Shane Harter on GitHub. Unfortunately, his library was abandoned (or is on indefinite hiatus), was written for PHP v5.3, had no namespacing, no package management or an auto-loader (ie: Composer).

I choose to create an entirely new library instead of forking and modifying his original library for educational purposes. I also didn't agree with some of his methodologies. I do require some extra dependencies, but Composer makes this a trivial issue.

---

_This library is in a fully working state. I've created very complex daemons that have run for months w/o any memory leaks or crashes. More could be done...

php-daemon's Issues

Using callable in worker initialization

Hello everyone!
Method addWorker() assume object instance, class name, closure or callable. Using object instance described in wiki, but using callable is not. How I can use callable?

what happens if a loop is still executing when the interval comes up?

I think this is what happens:
2018-05-24 00:34:19.3006: 12023 12023: DEBUG: Daemon::wait: Loop took too long. [Interval=10.000000] [Duration=10.245913] [Extra=0.245913]

This comes up in my logs. I'm wondering how the program behaves in this instance? Do I need to make sure that my interval is set long enough that the job can't possibly be running, or will it just gracefully skip the next iteration if the last one is still running?

Thanks,
Dave

Daemon Restart Event Hook

Referring to the wiki pages on both Events and Signal-Handling:
https://github.com/lifo101/php-daemon/wiki/Events
https://github.com/lifo101/php-daemon/wiki/Signal-Handling

Is there some way I can catch a restart event, and prevent it if a condition is met? The restart is being triggered by the daemon's auto restart mechanism setAutoRestartInterval().

<?php
return function () {
    CsvDaemon::getInstance()
        ->setDaemonize(true) # allow main daemon process to fork into the background
        ->setLoopInterval(10) # 10 second loop internal
        ->setAutoRestartInterval(3600) # restart the daemon every 1 hour
        ->setVerbose(true) # verbose logging
        ->setDebug(true) # debug logging enabled
        ->setDebugLevel(1) # 1 = low (not as detailed) ... 5 = high (maximum detail)
        ->setLogFile(DAEMON_LOG_CSV) # where to write the logs
        ->setCommand("sudo /usr/bin/php7.2 /var/daemons/launch.php 'CsvWatcher' '-d' > /dev/null")
        ->run(); # start the daemon
};

Logging Failure after Restart

Whenever a main daemon process restarts, either automatically as the result of a crash/failure in my program, or through setting a restart interval:

->setAutoRestartInterval()

logging wont resume.

require "/var/daemons/config/config.php";


return function () {
    OrderDaemon::getInstance()
        ->setDaemonize(true) # allow main daemon process to fork into the background
        ->setLoopInterval(1) # 1 second loop internal
        ->setAutoRestartInterval(10) # restart the daemon every 10 seconds
        ->setVerbose(true) # verbose logging
        ->setDebug(true) # debug logging enabled
        ->setDebugLevel(1) # high level (not as detailed)
        ->setLogFile(DAEMON_LOG_ORDERS) # where to write the logs
        ->run(); # start the daemon
};


protected function execute()
{
        if ($this->getLoopIterations() % 2 == 0) {
                $this->log("testing");
        }
}

On the first run, logs are written; once restarted, no other log messages are recorded.

I will note that I am using logrotate (ubuntu) to manage log files and that I am using copytruncate so i dont believe its an issue relating to file pointers.

/var/log/daemons/*.log {
        daily
        rotate 14
        compress
        delaycompress
        missingok
        copytruncate
}

   copytruncate
   Truncate the original log file to zero size in place after creating a copy, instead
   of moving the old log file and optionally creating a new one.  It can be used  when
   some  program  cannot  be told to close its logfile and thus might continue writing
   (appending) to the previous log file forever.

Is this a bug, or an issue with something I am doing?

Missing SIGPOLL and SIGPWR signals on some systems

There are some WARNING messages:

PHP Warning: Use of undefined constant SIGPOLL - assumed 'SIGPOLL' (this will throw an Error in a future version of PHP) in …/lifo/php-daemon/src/Lifo/Daemon/Daemon.php on line 1969

PHP Warning: Use of undefined constant SIGPWR - assumed 'SIGPWR' (this will throw an Error in a future version of PHP) in …/vendor/lifo/php-daemon/src/Lifo/Daemon/Daemon.php on line 1969

Systems (where this PHP constants are missing):

• macOS 10.13.2 + PHP 7.2.1
• FreeBSD 11 + PHP 5.6.32
• FreeBSD 11 + PHP 7.1.13
• FreeBSD 10 + PHP 7.2.0

How to check
Execute php -r "echo SIGPOLL;" and php -r "echo SIGPWR;"

Is any way how to exclude some signals that not presented in system?
Maybe overloading of function getSignals helps to resolve? — Unfortunately it is private function now.

Sysv Error: Error fetching message from queue

I have this in my execute() method daemon:

    if ($this->getLoopIterations() % 5 == 0) 
    {
        $this->StartMbcGetWorker();
    }
    if ($this->getLoopIterations() % 8 == 0) 
    {
        $this->StartTrhGetWorker();
    }

But after executing I get this output:

2018-02-12 14:02:47.7789: 17335 17337: SysV Error: Error fetching message from queue: ERR=22 Invalid argument
2018-02-12 14:02:47.7999: 17335 17337: SysV Error: Error fetching message from queue: ERR=22 Invalid argument
2018-02-12 14:02:47.8209: 17335 17337: SysV Error: Error fetching message from queue: ERR=22 Invalid argument
2018-02-12 14:02:47.8418: 17335 17337: SysV Error: Error fetching message from queue: ERR=22 Invalid argument
2018-02-12 14:02:47.8627: 17335 17337: SysV Error: Error fetching message from queue: ERR=22 Invalid argument
2018-02-12 14:02:47.8836: 17335 17337: SysV Error: Error fetching message from queue: ERR=22 Invalid argument
2018-02-12 14:02:47.9045: 17335 17337: SysV Error: Error fetching message from queue: ERR=22 Invalid argument
2018-02-12 14:02:47.9255: 17335 17337: SysV Error: Error fetching message from queue: ERR=22 Invalid argument
2018-02-12 14:02:47.9464: 17335 17337: SysV Error: Error fetching message from queue: ERR=22 Invalid argument
2018-02-12 14:02:47.9673: 17335 17337: SysV Error: Error fetching message from queue: ERR=22 Invalid argument
2018-02-12 14:02:47.9882: 17335 17337: SysV Error: Error fetching message from queue: ERR=22 Invalid argument

... etc ....

This error happens when trying to start a task/worker in the callback of a worker. E.g. when the second worker is supposed to start at 8 seconds this happens when that Task/Worker is supposed to start.

This is from the StartMbcGetWorker method:

$MbcGetWorker = $this->worker('MainMbcGetWorker');
$MbcGetWorker->MainMbcGetWorker()->then(function ($valueArray){
$this->task(new SendToDatabaseTask(), $valueArray, $this->currentMbcCall); // <- Gives SysV error
           $this->Anotherfunction(); // <- Gives Sysv Error
        });

Very confusing, I thought that this was supposed to work? Any idea maybe? Thank you.

PHP 7

Do you plan on updating this to use php 7 anytime in the future?

PHP 8.x

As of PHP 8.x, a warning is thrown:

The magic method Lifo\Daemon\Daemon::__wakeup() must have public visibility

This is related to a change made in 8.0.0 affecting magic methods. Per the PHP doc:

Warning
All magic methods, with the exception of __construct(), __destruct(), and __clone(), must be declared as public, otherwise an E_WARNING is emitted. Prior to PHP 8.0.0, no diagnostic was emitted for the magic methods __sleep(), __wakeup(), __serialize(), __unserialize(), and __set_state().

I grepped the code and found just the 1 occurrence within Daemon.php.

/** @noinspection PhpUnusedPrivateMethodInspection */
private function __wakeup()
{
    // un-serialization is disabled to the public
}

I haven't encountered any issues by making this a public function. However, it's not clear to me how this magic method is being used in the context of this library, and at the same, I only use tasks so I can't confirm the full scope of impact this change would have to workers and related functionality.

Can you advise, are there any concerns with making this a public function?

Premature worker death

I found a log during a daemon worker is running. There was no child worker proc and I don't know where to look and what it means.

2018-09-11 04:22:06.3434: 4192 DEBUG: Mediator [wcm-crawl-worker]: Premature worker death was reaped [PID=9600] [Call=96] [Status=CALLED]

How do I pass variables to workers/tasks?

When I retrieve data on callback from a worker I want to pass that to a separate task i.e. to perform a database query but I don't know how to pass that to the task.

For example:
$this->task("SendToDatabaseTask", $someArray);

Any idea? Thank you.

Edit: I am retarded, I found it.

calling a task leaves a defunct process when complete.

I'm still learning how to use php-daemon so i'm sure this is end user error but it seems every time a task completes its process remains open as a zombie

dsmith 32020 32015 0 16:06 pts/9 00:00:02 [php] <defunct>
I verified from the log that those are the same process ID's from the tasks:

This is how i'm calling the task in my class file:

`
public function execute() {
// no args.....
// $this->task([$this, 'doTask']);
$arg1 = 12;
$arg2 = 21;

  $this->task([$this, 'hitbtcInsert'],0); // arg is which api loop to run... 0 - # of api keys in db.
  $this->task([$this, 'hitbtcInsert'],1); // arg is which api loop to run... 0 - # of api keys in db.
  $this->task([$this, 'hitbtcInsert'],2); // arg is which api loop to run... 0 - # of api keys in db.
  
  

  }

`

and the test script that starts it all:

`
<?php

  require __DIR__ . '/vendor/autoload.php';
  include 'HitbtcInsertDaemon.php';

  declare(ticks = 1); // needed for signal handling

  HitbtcInsertDaemon::getInstance()
    ->setDaemonize(false) 
    ->setLoopInterval(60)
    ->setVerbose(true)
    ->setDebug(true)
    ->setDebugLevel(3)
    ->setLogFile('daemon_hitbtc_insert.log')
    ->run();

`

Where to catch Premature worker death and processed it

Hello, if there is an exception:

2020-02-26 04:02:06.0448: 6526 6526: DEBUG: Mediator [example]: Premature worker death was reaped [PID=6527] [Call=2] [Status=CALLED]

I would like to do a few cleaning operations related to that worker. Can you give me some advice where to catch it and do some operations? Or how?

I would be grateful for a little an example because I am new to Promise pattern.

Here is my my daemon:

<?php
require __DIR__ . '/vendor/autoload.php';

use Lifo\Daemon\Daemon;
use Lifo\Daemon\Mediator\Mediator;

use Lifo\Daemon\LogTrait;
use Lifo\Daemon\Promise;

class MyWorker {
  public function getUrl($endPoint) {
    sleep(1); // fake latency

    // To do something...

    // When finished it return url api
    return $endPoint;
  }
}

class MyDaemon extends Daemon {
  public $api = array('www.api1.com', 'www.api2.com', 'www.api3.com', 'www.api4.com', 'www.api5.com', 'www.api6.com');
  public static $runningApi = array();

  public function getApi() {
    static $loop = 1;
    $counter = sizeof($this->api);
    if ($counter >= $loop) {
      $api = $this->api[$loop-1];
    } else {
      $loop = 1;
      $api = $this->api[$loop-1];
    }
    $loop++;
    return $api;
  }

  public function initialize() {
    $this->addWorker(new MyWorker(), 'example')->setAutoRestart(true)->setMaxProcesses(6);
  }

  public function execute() {
      $api = $this->getApi();
      $key = array_search($api, self::$runningApi);
      if ($key === false) {
        self::$runningApi[] = $api;
        // the worker method will return a Promise
        $this->worker('example')->getUrl($api)->then(function($url) {
          $this->log("URL API received: %s", $url);
          $key = array_search($url, self::$runningApi);
          if ($key !== false) {
            unset(self::$runningApi[$key]);
          }
        });
      }
  }
}

MyDaemon::getInstance()->setVerbose(true)->setDebug(true)->setLogFile('/tmp/MyDaemon.log')->run();

Thank you

Question on performance and multiple daemons

I have a quick question in relation to the performance of your library, CPU and memory consumption predominantly.

If I had an application with say 5 background processing channels, for maximum single process throughput, I would run a daemon per channel. Alternatively, I could run 5 workers with each worker responsible for a single channel, processing one payload at a time. What would be the performance impact of running either scenario using your library, and what would your recommendation be?

I could benchmark this to determine those stats, but given that you have used your library in production to run complex daemons, I was hoping to draw on your experience of it to get a high-level perspective.

SySV Errors after a while.

Hello,

I'm trying to setup a daemon using your package, the daemon pulls n jobs from a queue (beanstalk/sqs/whatever) and for each job it calls to the needed worker, actually the worker does nothing but create a response, fake an execution time and returns the response to the daemon. It all goes fine until I start getting SySV related errors like these after a while:

2018-07-06 14:51:30.9168: 11157   11157: Loop 52
2018-07-06 14:51:31.2001: 11157   11157: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 513: Retrying. Error Code: 0 Success
2018-07-06 14:51:31.4774: 11157   11157: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 513: Retrying. Error Code: 11 Resource temporarily unavailable
2018-07-06 14:51:31.7886: 11157   11157: Mediator [facebook_video]: Error: Method call failed: workIn({arg1}): Unable to put call #513 on queue
2018-07-06 14:51:31.9814: 11157   11157: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 514: Retrying. Error Code: 11 Resource temporarily unavailable
2018-07-06 14:51:32.1996: 11157   11157: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 514: Retrying. Error Code: 11 Resource temporarily unavailable
2018-07-06 14:51:32.5721: 11157   11157: Mediator [facebook_video]: Error: Method call failed: workIn({arg1}): Unable to put call #514 on queue
2018-07-06 14:51:32.7659: 11157   11157: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 518: Retrying. Error Code: 11 Resource temporarily unavailable
2018-07-06 14:51:33.0003: 11157   11157: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 518: Retrying. Error Code: 11 Resource temporarily unavailable
2018-07-06 14:51:33.3191: 11157   11157: Mediator [facebook_video]: Error: Method call failed: workIn({arg1}): Unable to put call #518 on queue
2018-07-06 14:51:33.5107: 11157   11157: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 519: Retrying. Error Code: 11 Resource temporarily unavailable
2018-07-06 14:51:33.0786: 11157   11157: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 519: Retrying. Error Code: 11 Resource temporarily unavailable
2018-07-06 14:51:34.1396: 11157   11157: Mediator [facebook_video]: Error: Method call failed: workIn({arg1}): Unable to put call #519 on queue
2018-07-06 14:51:34.3497: 11157   11157: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 520: Retrying. Error Code: 11 Resource temporarily unavailable
2018-07-06 14:51:34.5784: 11157   11157: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 520: Retrying. Error Code: 11 Resource temporarily unavailable
2018-07-06 14:51:34.8872: 11157   11157: Mediator [facebook_video]: Error: Method call failed: workIn({arg1}): Unable to put call #520 on queue
2018-07-06 14:51:34.0888: 11157   11157: DEBUG: Daemon::wait: Loop took too long. [Interval=2.000000] [Duration=3.971637] [Extra=1.971637]

my main class looks as follows:

class QueueDaemon extends Daemon
{
    /**
     * @var QueueContainer
     */
    private $queues;

    public function execute()
    {
        $this->log("Loop %d", $this->getLoopIterations());
      
        try {
            $jobs = $this->queues->get('youtube_video')->get(10);

            foreach ($jobs as $job) {
                $this->worker($job->getJobType())->workIn($job)->then(function (JobResponse $response) {
                    $this->log('Received response from worker: ' . $response);
                    $job = $response->getJob();

                    if ($response->getStatus() == JobResponse::RESPONSE_STATUS_SUCCESS) {
                        $this->log('Job finished succesfuly. Deleting job with id: ' . $job->getQueueHandler());
                        $this->queues->get('youtube_video')->delete($job);

                    } else {
                        if ($this->shouldRetryJob($job, $this->queues->get('youtube_video'))) {
                            $this->log('Releasing job with id ' . $job->getQueueHandler() . ' back to queue for retry.');
                            $job->incrementRetries();
                            $this->queues->get('youtube_video')->release($job);

                        } else {
                            // Enqueue job in errror/delayed queue
                        }
                    }
                });

            }

        } catch (WrongPayloadAttributeException $e) {
            $this->log($e->getMessage());
        } catch (QueueOperationException $e) {
            $this->log($e->getMessage());
        } catch (\InvalidArgumentException $e) {
            $this->log($e->getMessage());
        }

    }

    protected function initialize()
    {
        parent::initialize(); // TODO: Change the autogenerated stub
        $this->initializeQueueContainer();
    }

    protected function initializeQueueContainer()
    {
        $this->queues = new QueueContainer();
        $queues = ConfigLoader::get('queues');

        foreach ($queues as $queueId => $queueConfig) {
            $queue = QueueFactory::create($queueConfig);
            $queue->setAdapter(
                AdapterFactory::create(
                    $queueConfig[AbstractQueue::QUEUE_CONFIG_TYPE],
                    ConfigLoader::get($queueConfig[AbstractQueue::QUEUE_CONFIG_ADAPTER][AbstractQueue::QUEUE_CONFIG_CONNECTION])
                )
            );

            $this->queues->add($queue,$queueId);
        }
    }

    protected function shouldRetryJob(AbstractJob $job, AbstractQueue $queue) : bool
    {
        return  $job->getRetries() <= $queue->getMaxRetries();
    }


}

about workers, it is set to autorestart and max_processes = 10, my worker code:

class Video extends QueueWorker
{
    use LogTrait;

    const WORKER_NAME = 'youtube_video';

    public function initialize()
    {
        // TODO: Implement initialize() method.
    }

    public function workIn(AbstractJob $job) : JobResponse
    {
        $jobResponse = new JobResponse();
        $jobResponse->setJob($job);
        $jobResponse->setStartedAd(new \DateTime());

        $this->log('Received job on worker with pid ' . posix_getpid() . ' jobId: ' . $job->getQueueHandler() . ' and type: ' . $job->getType() . ' and payload ' . json_encode($job->getPayload()));

        $statuses = [JobResponse::RESPONSE_STATUS_SUCCESS,JobResponse::RESPONSE_STATUS_ERROR];
        $jobResponse->setStatus($statuses[rand(0,1)]);

//        $fakeDelay = rand(0,1000);
//        $this->log('Faking execution delay: ' . $fakeDelay . ' microseconds');
//        usleep($fakeDelay);

        $jobResponse->setFinishedAt(new \DateTime());

        $this->log('Returning job response: ' . json_encode($jobResponse));

        return $jobResponse;

    }

}

and daemon init code:

QueueDaemon::getInstance()
    ->setDaemonize(true)
    ->setLoopInterval(2)
    ->setVerbose(true)
    ->setDebug(true)
    ->setDebugLevel(3)
    ->setLogFile('/tmp/queue_daemon.log')
    ->run();

hope you can help me.

Thanks in advance.
Diego.

Worker and pcntl_exec support

I use pcntl_exec to isolate curl calls because curl is not multi process safe.

I was wondering how I could use php-daemon and workers and be able to handle the return code of the code ran under pcntl_exec()

Use with MQTT Server

Hi,
My PHP Script need to publish to and listen for data from MQTT Server. However, the script can't connect to MQTT server for every request. So I thus a way to keep the connection open forever. Through my search, daemon is a good idea.
I don't know if this library can handle and suitable for my scenario. One more thing, It take around 2 to 5 seconds for the script to wait for data back from MQTT server.

I am new to PHP and don't quite understand this library flow and building block.

Documentation

Do you have any links to good reads for understanding the logic behind your daemon?

Maybe I don't understand the concept of a daemon, but I just don't get how to use things here. For example, you can define the max number of workers to fork via 'setMaxProcesses':

$this->addWorker('myWorker', 'example')
            ->setAutoRestart(true)  # enable auto-restarts of workers, to show them exit and be re-created on-the-fly
            ->setMaxCalls(10)       # how many calls a worker will process before exiting
            ->setMaxRuntime(10)     # how long a worker will run before exiting
            ->setMaxProcesses(3);    # how many workers to fork

But i'm not understanding what that actually does?

Say for example, I have a bucket of 12 apples. I have a worker which eats apples, one at a time. I would assume, by defining that 3 workers should be created on each loop, then 3 apples would be eaten every loop. This is not the behavior I'm seeing. I can fork 30 processes, but only 1 apple ever gets eaten each iteration.

You also note in your documentation that you can call a task from a worker, but I have found no examples and cannot figure out how to make that happen.

I could use some help in understanding all this and I'd very much appreciate any info or links you can provide.

[b]Edit:[/b] Also, is it necessary to use the example.php file? How do I start up this daemon without having to login to the server, execute: php example.php mySampleDaemon then leaving terminal open?

Thank you.

Possible memory leak

Hello everyone,
I try to make some workers and call method on its every second. After few weeks daemon shutdowns because can't put call to IPC.

Output

2018-12-12 11:44:09.9736: 5952     5952: DEBUG: GET MSG=RETURN {"id":5228056,"status":3,"time":1544615048.9774611,"pid":6002}
2018-12-12 11:44:09.9737: 5952     5952: DEBUG: PUT CALL 5228057 UNCALLED
2018-12-12 11:44:09.9738: 5952     5952: DEBUG: PUT MSG=CALL {"id":5228057,"status":0,"time":1544615049.9736831,"pid":5952}
2018-12-12 11:44:09.9739: 5952     6000: DEBUG: GET MSG=CALL {"id":5228057,"status":0,"time":1544615049.9736831,"pid":5952}
2018-12-12 11:44:09.0974: 5952     6000: DEBUG: Mediator [Error 50x]: Call received [Call=5228057] [Method=execute]
2018-12-12 11:44:09.9741: 5952     6000: DEBUG: Mediator [Error 50x]: Received Call 5228057: METHOD=execute()
2018-12-12 11:44:09.9741: 5952     6000: DEBUG: PUT CALL 5228057 RUNNING
2018-12-12 11:44:09.9742: 5952     6000: DEBUG: PUT MSG=RUNNING {"id":5228057,"status":2,"time":1544615049.974082,"pid":6000}
2018-12-12 11:44:09.9744: 5952     6000: DEBUG: PUT CALL 5228057 RETURNED
2018-12-12 11:44:09.9745: 5952     6000: DEBUG: PUT MSG=RETURN {"id":5228057,"status":3,"time":1544615049.974427,"pid":6000}
2018-12-12 11:44:09.9745: 5952     6000: DEBUG: Mediator [Error 50x]: Waiting for call [Calls=1742686/Inf] [Time=1742899s/Inf]
2018-12-12 11:44:12.0952: 5952     5952: DEBUG: PUT CALL 5228058 UNCALLED
2018-12-12 11:44:12.9522: 5952     6001: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 5227143: Retrying. Error Code: -1 Unknown error -1
2018-12-12 11:44:12.9525: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:44:12.9813: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:44:12.9815: 5952     6001: DEBUG: PUT CALL 5227143 RETURNED
2018-12-12 11:44:12.0982: 5952     6001: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:44:12.9871: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:44:12.9872: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:44:12.9872: 5952     5952: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 5228058: Retrying. Error Code: -1 Unknown error -1
2018-12-12 11:44:21.9823: 5952     5952: DEBUG: PUT CALL 5228058 UNCALLED
2018-12-12 11:44:21.9824: 5952     6001: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 5227143: Retrying. Error Code: -1 Unknown error -1
2018-12-12 11:44:21.9833: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:44:22.0489: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:44:22.0492: 5952     6001: DEBUG: PUT CALL 5227143 RETURNED
2018-12-12 11:44:22.0005: 5952     6001: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:44:22.0055: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:44:22.0551: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:44:22.0552: 5952     5952: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 5228058: Retrying. Error Code: -1 Unknown error -1
2018-12-12 11:44:43.0503: 5952     5952: DEBUG: PUT CALL 5228058 UNCALLED
2018-12-12 11:44:43.0504: 5952     6001: Mediator [Slow Web Queries]: Error: Could not send RETURNED ACK for Call 5227143
2018-12-12 11:44:43.0505: 5952     6001: DEBUG: Mediator [Slow Web Queries]: Waiting for call [Calls=1739500/Inf] [Time=1742932s/Inf]
2018-12-12 11:44:43.0507: 5952     6001: DEBUG: GET MSG=CALL {"id":5227176,"status":0,"time":1544614755.880198,"pid":5952}
2018-12-12 11:44:43.0511: 5952     6001: DEBUG: Mediator [Slow Web Queries]: Call received [Call=5227176] [Method=execute]
2018-12-12 11:44:43.0511: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:44:43.0512: 5952     6001: DEBUG: Mediator [Slow Web Queries]: Received Call 5227176: METHOD=execute()
2018-12-12 11:44:43.1966: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:44:43.1967: 5952     6001: DEBUG: PUT CALL 5227176 RUNNING
2018-12-12 11:44:43.1971: 5952     6001: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:44:43.2024: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:44:43.2026: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:44:43.2027: 5952     5952: Mediator [Slow Web Queries]: Error: Method call failed: execute(): Unable to put call #5228058 on queue
2018-12-12 11:44:43.2028: 5952     5952: DEBUG: PUT CALL 5228059 UNCALLED
2018-12-12 11:44:43.2029: 5952     5952: DEBUG: PUT MSG=CALL {"id":5228059,"status":0,"time":1544615083.2027211,"pid":5952}
2018-12-12 11:44:43.0203: 5952     5952: DEBUG: Daemon::wait: Loop took too long. [Interval=1.000000] [Duration=33.283396] [Extra=32.283396]
2018-12-12 11:44:43.2031: 5952     6002: DEBUG: GET MSG=CALL {"id":5228059,"status":0,"time":1544615083.2027211,"pid":5952}
2018-12-12 11:44:43.2032: 5952     6002: DEBUG: Mediator [Bad Backends]: Call received [Call=5228059] [Method=execute]
2018-12-12 11:44:43.2032: 5952     5952: DEBUG: GET MSG=RUNNING {"id":5228057,"status":2,"time":1544615049.974082,"pid":6000}
2018-12-12 11:44:43.2033: 5952     6002: DEBUG: Mediator [Bad Backends]: Received Call 5228059: METHOD=execute()
2018-12-12 11:44:43.2033: 5952     6002: DEBUG: PUT CALL 5228059 RUNNING
2018-12-12 11:44:43.2034: 5952     5952: DEBUG: GET MSG=RETURN {"id":5228057,"status":3,"time":1544615049.974427,"pid":6000}
2018-12-12 11:44:43.2034: 5952     6002: DEBUG: PUT MSG=RUNNING {"id":5228059,"status":2,"time":1544615083.2032771,"pid":6002}
2018-12-12 11:44:43.2036: 5952     5952: DEBUG: GET MSG=RUNNING {"id":5227101,"status":2,"time":1544615049.9470119,"pid":6001}
2018-12-12 11:44:43.2039: 5952     6002: DEBUG: PUT CALL 5228059 RETURNED
2018-12-12 11:44:43.0204: 5952     6002: DEBUG: PUT MSG=RETURN {"id":5228059,"status":3,"time":1544615083.2037981,"pid":6002}
2018-12-12 11:44:43.2041: 5952     6002: DEBUG: Mediator [Bad Backends]: Waiting for call [Calls=1742006/Inf] [Time=1742933s/Inf]
2018-12-12 11:44:43.2043: 5952     5952: DEBUG: GET MSG=RUNNING {"id":5228059,"status":2,"time":1544615083.2032771,"pid":6002}
2018-12-12 11:44:43.2044: 5952     5952: DEBUG: GET MSG=RETURN {"id":5228059,"status":3,"time":1544615083.2037981,"pid":6002}
2018-12-12 11:44:43.2046: 5952     5952: DEBUG: PUT CALL 5228060 UNCALLED
2018-12-12 11:44:43.2047: 5952     5952: DEBUG: PUT MSG=CALL {"id":5228060,"status":0,"time":1544615083.204576,"pid":5952}
2018-12-12 11:44:43.2048: 5952     6000: DEBUG: GET MSG=CALL {"id":5228060,"status":0,"time":1544615083.204576,"pid":5952}
2018-12-12 11:44:43.0205: 5952     6000: DEBUG: Mediator [Error 50x]: Call received [Call=5228060] [Method=execute]
2018-12-12 11:44:43.0205: 5952     6000: DEBUG: Mediator [Error 50x]: Received Call 5228060: METHOD=execute()
2018-12-12 11:44:43.2051: 5952     6000: DEBUG: PUT CALL 5228060 RUNNING
2018-12-12 11:44:43.2051: 5952     6000: DEBUG: PUT MSG=RUNNING {"id":5228060,"status":2,"time":1544615083.2050419,"pid":6000}
2018-12-12 11:44:43.2054: 5952     6000: DEBUG: PUT CALL 5228060 RETURNED
2018-12-12 11:44:43.2055: 5952     6000: DEBUG: PUT MSG=RETURN {"id":5228060,"status":3,"time":1544615083.205384,"pid":6000}
2018-12-12 11:44:43.2055: 5952     6000: DEBUG: Mediator [Error 50x]: Waiting for call [Calls=1742687/Inf] [Time=1742933s/Inf]
2018-12-12 11:44:46.1973: 5952     5952: DEBUG: PUT CALL 5228061 UNCALLED
2018-12-12 11:44:46.1974: 5952     6001: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 5227176: Retrying. Error Code: -1 Unknown error -1
2018-12-12 11:44:46.1976: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:44:46.2231: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:44:46.2232: 5952     6001: DEBUG: PUT CALL 5227176 RUNNING
2018-12-12 11:44:46.2237: 5952     6001: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:44:46.2287: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:44:46.2288: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:44:46.2289: 5952     5952: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 5228061: Retrying. Error Code: -1 Unknown error -1
2018-12-12 11:44:55.0224: 5952     5952: DEBUG: PUT CALL 5228061 UNCALLED
2018-12-12 11:44:55.2241: 5952     6001: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 5227176: Retrying. Error Code: -1 Unknown error -1
2018-12-12 11:44:55.2247: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:44:55.2903: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:44:55.2904: 5952     6001: DEBUG: PUT CALL 5227176 RUNNING
2018-12-12 11:44:55.2909: 5952     6001: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:44:55.0296: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:44:55.2961: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:44:55.2962: 5952     5952: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 5228061: Retrying. Error Code: -1 Unknown error -1
2018-12-12 11:45:16.2912: 5952     5952: DEBUG: PUT CALL 5228061 UNCALLED
2018-12-12 11:45:16.2913: 5952     6001: Mediator [Slow Web Queries]: Error: Could not send RUNNING ACK for Call 5227176
2018-12-12 11:45:16.2919: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:45:16.4375: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:45:16.4376: 5952     6001: DEBUG: PUT CALL 5227176 RETURNED
2018-12-12 11:45:16.4382: 5952     6001: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:45:16.4432: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:45:16.4434: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:45:16.4434: 5952     5952: Mediator [Slow Web Queries]: Error: Method call failed: execute(): Unable to put call #5228061 on queue
2018-12-12 11:45:16.4435: 5952     5952: DEBUG: PUT CALL 5228062 UNCALLED
2018-12-12 11:45:16.4436: 5952     5952: DEBUG: PUT MSG=CALL {"id":5228062,"status":0,"time":1544615116.44348,"pid":5952}
2018-12-12 11:45:16.4437: 5952     5952: DEBUG: Daemon::wait: Loop took too long. [Interval=1.000000] [Duration=33.240578] [Extra=32.240578]
2018-12-12 11:45:16.4438: 5952     6002: DEBUG: GET MSG=CALL {"id":5228062,"status":0,"time":1544615116.44348,"pid":5952}
2018-12-12 11:45:16.4439: 5952     5952: DEBUG: GET MSG=RUNNING {"id":5228060,"status":2,"time":1544615083.2050419,"pid":6000}
2018-12-12 11:45:16.0444: 5952     6002: DEBUG: Mediator [Bad Backends]: Call received [Call=5228062] [Method=execute]
2018-12-12 11:45:16.0444: 5952     5952: DEBUG: GET MSG=RETURN {"id":5228060,"status":3,"time":1544615083.205384,"pid":6000}
2018-12-12 11:45:16.4441: 5952     6002: DEBUG: Mediator [Bad Backends]: Received Call 5228062: METHOD=execute()
2018-12-12 11:45:16.4441: 5952     6002: DEBUG: PUT CALL 5228062 RUNNING
2018-12-12 11:45:16.4442: 5952     5952: DEBUG: GET MSG=RUNNING {"id":5227104,"status":2,"time":1544615049.9478791,"pid":6001}
2018-12-12 11:45:16.4442: 5952     6002: DEBUG: PUT MSG=RUNNING {"id":5228062,"status":2,"time":1544615116.4440949,"pid":6002}
2018-12-12 11:45:16.4446: 5952     6002: DEBUG: PUT CALL 5228062 RETURNED
2018-12-12 11:45:16.4447: 5952     6002: DEBUG: PUT MSG=RETURN {"id":5228062,"status":3,"time":1544615116.4445219,"pid":6002}
2018-12-12 11:45:16.4447: 5952     6002: DEBUG: Mediator [Bad Backends]: Waiting for call [Calls=1742007/Inf] [Time=1742966s/Inf]
2018-12-12 11:45:16.4447: 5952     5952: DEBUG: GET MSG=RUNNING {"id":5228062,"status":2,"time":1544615116.4440949,"pid":6002}
2018-12-12 11:45:16.4449: 5952     5952: DEBUG: GET MSG=RETURN {"id":5228062,"status":3,"time":1544615116.4445219,"pid":6002}
2018-12-12 11:45:16.0445: 5952     5952: DEBUG: PUT CALL 5228063 UNCALLED
2018-12-12 11:45:16.4451: 5952     5952: DEBUG: PUT MSG=CALL {"id":5228063,"status":0,"time":1544615116.444989,"pid":5952}
2018-12-12 11:45:16.4451: 5952     6000: DEBUG: GET MSG=CALL {"id":5228063,"status":0,"time":1544615116.444989,"pid":5952}
2018-12-12 11:45:16.4452: 5952     6000: DEBUG: Mediator [Error 50x]: Call received [Call=5228063] [Method=execute]
2018-12-12 11:45:16.4453: 5952     6000: DEBUG: Mediator [Error 50x]: Received Call 5228063: METHOD=execute()
2018-12-12 11:45:16.4453: 5952     6000: DEBUG: PUT CALL 5228063 RUNNING
2018-12-12 11:45:16.4453: 5952     6000: DEBUG: PUT MSG=RUNNING {"id":5228063,"status":2,"time":1544615116.445271,"pid":6000}
2018-12-12 11:45:16.4456: 5952     6000: DEBUG: PUT CALL 5228063 RETURNED
2018-12-12 11:45:16.4457: 5952     6000: DEBUG: PUT MSG=RETURN {"id":5228063,"status":3,"time":1544615116.4455719,"pid":6000}
2018-12-12 11:45:16.4457: 5952     6000: DEBUG: Mediator [Error 50x]: Waiting for call [Calls=1742688/Inf] [Time=1742966s/Inf]
2018-12-12 11:46:01.4385: 5952     5952: DEBUG: PUT CALL 5228064 UNCALLED
2018-12-12 11:46:01.4386: 5952     6001: Mediator [Slow Web Queries]: Error: Could not send RETURNED ACK for Call 5227176
2018-12-12 11:46:01.4387: 5952     6001: DEBUG: Mediator [Slow Web Queries]: Waiting for call [Calls=1739501/Inf] [Time=1743011s/Inf]
2018-12-12 11:46:01.4389: 5952     6001: DEBUG: GET MSG=CALL {"id":5227743,"status":0,"time":1544614944.9079361,"pid":5952}
2018-12-12 11:46:01.0439: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:01.4394: 5952     6001: DEBUG: Mediator [Slow Web Queries]: Call received [Call=5227743] [Method=execute]
2018-12-12 11:46:01.4395: 5952     6001: DEBUG: Mediator [Slow Web Queries]: Received Call 5227743: METHOD=execute()
2018-12-12 11:46:01.4644: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:01.4646: 5952     6001: DEBUG: PUT CALL 5227743 RUNNING
2018-12-12 11:46:01.4649: 5952     6001: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:01.4702: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:01.4704: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:46:01.4704: 5952     5952: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 5228064: Retrying. Error Code: -1 Unknown error -1
2018-12-12 11:46:04.4651: 5952     5952: DEBUG: PUT CALL 5228064 UNCALLED
2018-12-12 11:46:04.4652: 5952     6001: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 5227743: Retrying. Error Code: -1 Unknown error -1
2018-12-12 11:46:04.4659: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:04.5313: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:04.5314: 5952     6001: DEBUG: PUT CALL 5227743 RUNNING
2018-12-12 11:46:04.5318: 5952     6001: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:04.5369: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:04.5371: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:46:04.5372: 5952     5952: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 5228064: Retrying. Error Code: -1 Unknown error -1
2018-12-12 11:46:13.5321: 5952     5952: DEBUG: PUT CALL 5228064 UNCALLED
2018-12-12 11:46:13.5322: 5952     6001: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 5227743: Retrying. Error Code: -1 Unknown error -1
2018-12-12 11:46:13.5328: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:13.6784: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:13.6787: 5952     6001: DEBUG: PUT CALL 5227743 RUNNING
2018-12-12 11:46:13.6792: 5952     6001: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:13.6843: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:13.6845: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:46:13.6845: 5952     5952: Mediator [Slow Web Queries]: Error: Method call failed: execute(): Unable to put call #5228064 on queue
2018-12-12 11:46:13.6846: 5952     5952: DEBUG: PUT CALL 5228065 UNCALLED
2018-12-12 11:46:13.6847: 5952     5952: DEBUG: PUT MSG=CALL {"id":5228065,"status":0,"time":1544615173.684586,"pid":5952}
2018-12-12 11:46:13.6847: 5952     5952: DEBUG: PUT MSG=CALL {"id":5228065,"status":0,"time":1544615173.684586,"pid":5952}
2018-12-12 11:46:13.6848: 5952     5952: DEBUG: Daemon::wait: Loop took too long. [Interval=1.000000] [Duration=57.241009] [Extra=56.241009]
2018-12-12 11:46:13.6849: 5952     6002: DEBUG: GET MSG=CALL {"id":5228065,"status":0,"time":1544615173.684586,"pid":5952}
2018-12-12 11:46:13.0685: 5952     5952: DEBUG: GET MSG=RUNNING {"id":5228063,"status":2,"time":1544615116.445271,"pid":6000}
2018-12-12 11:46:13.0685: 5952     6002: DEBUG: Mediator [Bad Backends]: Call received [Call=5228065] [Method=execute]
2018-12-12 11:46:13.0685: 5952     6002: DEBUG: Mediator [Bad Backends]: Received Call 5228065: METHOD=execute()
2018-12-12 11:46:13.6851: 5952     6002: DEBUG: PUT CALL 5228065 RUNNING
2018-12-12 11:46:13.6851: 5952     5952: DEBUG: GET MSG=RETURN {"id":5228063,"status":3,"time":1544615116.4455719,"pid":6000}
2018-12-12 11:46:13.6852: 5952     6002: DEBUG: PUT MSG=RUNNING {"id":5228065,"status":2,"time":1544615173.685066,"pid":6002}
2018-12-12 11:46:13.6853: 5952     5952: DEBUG: GET MSG=RUNNING {"id":5227107,"status":2,"time":1544615049.9487391,"pid":6001}
2018-12-12 11:46:13.6854: 5952     6002: DEBUG: PUT CALL 5228065 RETURNED
2018-12-12 11:46:13.6855: 5952     6002: DEBUG: PUT MSG=RETURN {"id":5228065,"status":3,"time":1544615173.6854179,"pid":6002}
2018-12-12 11:46:13.6856: 5952     6002: DEBUG: Mediator [Bad Backends]: Waiting for call [Calls=1742008/Inf] [Time=1743023s/Inf]
2018-12-12 11:46:13.6859: 5952     5952: DEBUG: GET MSG=RUNNING {"id":5228065,"status":2,"time":1544615173.685066,"pid":6002}
2018-12-12 11:46:13.6861: 5952     5952: DEBUG: GET MSG=RETURN {"id":5228065,"status":3,"time":1544615173.6854179,"pid":6002}
2018-12-12 11:46:13.6862: 5952     5952: DEBUG: PUT CALL 5228066 UNCALLED
2018-12-12 11:46:13.6863: 5952     5952: DEBUG: PUT MSG=CALL {"id":5228066,"status":0,"time":1544615173.6862161,"pid":5952}
2018-12-12 11:46:13.6864: 5952     6000: DEBUG: GET MSG=CALL {"id":5228066,"status":0,"time":1544615173.6862161,"pid":5952}
2018-12-12 11:46:13.6865: 5952     6000: DEBUG: Mediator [Error 50x]: Call received [Call=5228066] [Method=execute]
2018-12-12 11:46:13.6866: 5952     6000: DEBUG: Mediator [Error 50x]: Received Call 5228066: METHOD=execute()
2018-12-12 11:46:13.6866: 5952     6000: DEBUG: PUT CALL 5228066 RUNNING
2018-12-12 11:46:13.6867: 5952     6000: DEBUG: PUT MSG=RUNNING {"id":5228066,"status":2,"time":1544615173.6865971,"pid":6000}
2018-12-12 11:46:13.6869: 5952     6000: DEBUG: PUT CALL 5228066 RETURNED
2018-12-12 11:46:13.0687: 5952     6000: DEBUG: PUT MSG=RETURN {"id":5228066,"status":3,"time":1544615173.6869111,"pid":6000}
2018-12-12 11:46:13.0687: 5952     6000: DEBUG: Mediator [Error 50x]: Waiting for call [Calls=1742689/Inf] [Time=1743023s/Inf]
2018-12-12 11:46:34.6796: 5952     5952: DEBUG: PUT CALL 5228067 UNCALLED
2018-12-12 11:46:34.6797: 5952     6001: Mediator [Slow Web Queries]: Error: Could not send RUNNING ACK for Call 5227743
2018-12-12 11:46:34.6801: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:34.7058: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:34.0706: 5952     6001: DEBUG: PUT CALL 5227743 RETURNED
2018-12-12 11:46:34.7065: 5952     6001: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:34.7067: 5952     6001: "Slow Web Queries": IPC "corruption" Error Threshold Reached [11]
2018-12-12 11:46:34.7118: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:34.7121: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:46:34.7122: 5952     5952: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 5228067: Retrying. Error Code: -1 Unknown error -1
2018-12-12 11:46:34.7173: 5952     5952: DEBUG: PUT CALL 5228067 UNCALLED
2018-12-12 11:46:34.7179: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:34.7773: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:34.7833: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:34.7834: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:46:34.7835: 5952     5952: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 5228067: Retrying. Error Code: -1 Unknown error -1
2018-12-12 11:46:34.7835: 5952     5952: DEBUG: PUT CALL 5228067 UNCALLED
2018-12-12 11:46:34.7841: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:34.9294: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:34.9349: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:34.9351: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:46:34.9351: 5952     5952: Mediator [Slow Web Queries]: Error: Method call failed: execute(): Unable to put call #5228067 on queue
2018-12-12 11:46:34.9352: 5952     5952: DEBUG: PUT CALL 5228068 UNCALLED
2018-12-12 11:46:34.9353: 5952     5952: DEBUG: PUT MSG=CALL {"id":5228068,"status":0,"time":1544615194.9351659,"pid":5952}
2018-12-12 11:46:34.9354: 5952     6002: DEBUG: GET MSG=CALL {"id":5228068,"status":0,"time":1544615194.9351659,"pid":5952}
2018-12-12 11:46:34.9355: 5952     6002: DEBUG: Mediator [Bad Backends]: Call received [Call=5228068] [Method=execute]
2018-12-12 11:46:34.9356: 5952     6002: DEBUG: Mediator [Bad Backends]: Received Call 5228068: METHOD=execute()
2018-12-12 11:46:34.9356: 5952     6002: DEBUG: PUT CALL 5228068 RUNNING
2018-12-12 11:46:34.9357: 5952     6002: DEBUG: PUT MSG=RUNNING {"id":5228068,"status":2,"time":1544615194.935575,"pid":6002}
2018-12-12 11:46:34.0936: 5952     6002: DEBUG: PUT CALL 5228068 RETURNED
2018-12-12 11:46:34.0936: 5952     6002: DEBUG: PUT MSG=RETURN {"id":5228068,"status":3,"time":1544615194.935941,"pid":6002}
2018-12-12 11:46:34.9361: 5952     6002: DEBUG: Mediator [Bad Backends]: Waiting for call [Calls=1742009/Inf] [Time=1743044s/Inf]
2018-12-12 11:46:34.9441: 5952     5952: DEBUG: ProcessManager: Reaping child [PID=6001]
2018-12-12 11:46:34.9442: 5952     5952: DEBUG: Daemon::wait: Loop took too long. [Interval=1.000000] [Duration=21.259289] [Extra=20.259289]
2018-12-12 11:46:34.9449: 5952     5952: DEBUG: GET MSG=RUNNING {"id":5228066,"status":2,"time":1544615173.6865971,"pid":6000}
2018-12-12 11:46:34.0945: 5952     5952: DEBUG: GET MSG=RETURN {"id":5228066,"status":3,"time":1544615173.6869111,"pid":6000}
2018-12-12 11:46:34.9452: 5952     5952: DEBUG: GET MSG=RUNNING {"id":5227110,"status":2,"time":1544615049.9495759,"pid":6001}
2018-12-12 11:46:34.9458: 5952     5952: DEBUG: GET MSG=RUNNING {"id":5228068,"status":2,"time":1544615194.935575,"pid":6002}
2018-12-12 11:46:34.0946: 5952     5952: DEBUG: GET MSG=RETURN {"id":5228068,"status":3,"time":1544615194.935941,"pid":6002}
2018-12-12 11:46:34.9461: 5952     5952: DEBUG: PUT CALL 5228069 UNCALLED
2018-12-12 11:46:34.9462: 5952     5952: DEBUG: PUT MSG=CALL {"id":5228069,"status":0,"time":1544615194.946111,"pid":5952}
2018-12-12 11:46:34.9462: 5952     5952: DEBUG: PUT CALL 5228070 UNCALLED
2018-12-12 11:46:34.9463: 5952     6000: DEBUG: GET MSG=CALL {"id":5228069,"status":0,"time":1544615194.946111,"pid":5952}
2018-12-12 11:46:34.9464: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:34.9464: 5952     6000: DEBUG: Mediator [Error 50x]: Call received [Call=5228069] [Method=execute]
2018-12-12 11:46:34.9465: 5952     6000: DEBUG: Mediator [Error 50x]: Received Call 5228069: METHOD=execute()
2018-12-12 11:46:34.9465: 5952     6000: DEBUG: PUT CALL 5228069 RUNNING
2018-12-12 11:46:34.9466: 5952     6000: DEBUG: PUT MSG=RUNNING {"id":5228069,"status":2,"time":1544615194.9464979,"pid":6000}
2018-12-12 11:46:34.9469: 5952     6000: DEBUG: PUT CALL 5228069 RETURNED
2018-12-12 11:46:34.0947: 5952     6000: DEBUG: PUT MSG=RETURN {"id":5228069,"status":3,"time":1544615194.9468861,"pid":6000}
2018-12-12 11:46:34.0947: 5952     6000: DEBUG: Mediator [Error 50x]: Waiting for call [Calls=1742690/Inf] [Time=1743044s/Inf]
2018-12-12 11:46:34.9717: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:34.9773: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:34.9775: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:46:34.9775: 5952     5952: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 5228070: Retrying. Error Code: -1 Unknown error -1
2018-12-12 11:46:34.9775: 5952     5952: DEBUG: PUT CALL 5228070 UNCALLED
2018-12-12 11:46:34.0978: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:35.0435: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:35.0495: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:35.0497: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:46:35.0497: 5952     5952: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 5228070: Retrying. Error Code: -1 Unknown error -1
2018-12-12 11:46:35.0498: 5952     5952: DEBUG: PUT CALL 5228070 UNCALLED
2018-12-12 11:46:35.0505: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:35.0196: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:35.2018: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:35.2019: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:46:35.0202: 5952     5952: Mediator [Slow Web Queries]: Error: Method call failed: execute(): Unable to put call #5228070 on queue
2018-12-12 11:46:35.2021: 5952     5952: DEBUG: PUT CALL 5228071 UNCALLED
2018-12-12 11:46:35.2023: 5952     5952: DEBUG: PUT MSG=CALL {"id":5228071,"status":0,"time":1544615195.2020891,"pid":5952}
2018-12-12 11:46:35.2024: 5952     6002: DEBUG: GET MSG=CALL {"id":5228071,"status":0,"time":1544615195.2020891,"pid":5952}
2018-12-12 11:46:35.2024: 5952     5952: DEBUG: Mediator [Slow Web Queries]: Premature worker death was reaped [PID=6001] [Call=679746] [Status=CALLED]
2018-12-12 11:46:35.2025: 5952     6002: DEBUG: Mediator [Bad Backends]: Call received [Call=5228071] [Method=execute]
2018-12-12 11:46:35.2025: 5952     6002: DEBUG: Mediator [Bad Backends]: Received Call 5228071: METHOD=execute()
2018-12-12 11:46:35.2025: 5952     6002: DEBUG: PUT CALL 5228071 RUNNING
2018-12-12 11:46:35.2026: 5952     6002: DEBUG: PUT MSG=RUNNING {"id":5228071,"status":2,"time":1544615195.202528,"pid":6002}
2018-12-12 11:46:35.2029: 5952     6002: DEBUG: PUT CALL 5228071 RETURNED
2018-12-12 11:46:35.0203: 5952     6002: DEBUG: PUT MSG=RETURN {"id":5228071,"status":3,"time":1544615195.2028961,"pid":6002}
2018-12-12 11:46:35.0203: 5952     6002: DEBUG: Mediator [Bad Backends]: Waiting for call [Calls=1742010/Inf] [Time=1743045s/Inf]
2018-12-12 11:46:35.2132: 5952     5952: DEBUG: Mediator [Slow Web Queries]: Forked [PID=26501]
2018-12-12 11:46:35.2142: 5952     5952: DEBUG: Mediator [Error 50x]: gc
2018-12-12 11:46:35.2143: 5952     5952: DEBUG: Mediator [Slow Web Queries]: gc
2018-12-12 11:46:35.2175: 5952     5952: DEBUG: Mediator [Bad Backends]: gc
2018-12-12 11:46:35.2523: 5952    26501: DEBUG: Mediator [Slow Web Queries]: start
2018-12-12 11:46:35.2526: 5952    26501: DEBUG: Mediator [Slow Web Queries]: Waiting for call [Calls=0/Inf] [Time=0s/Inf]
2018-12-12 11:46:35.2528: 5952    26501: DEBUG: GET MSG=CALL {"id":5228040,"status":0,"time":1544615043.9199469,"pid":5952}
2018-12-12 11:46:35.2535: 5952    26501: DEBUG: Mediator [Slow Web Queries]: Call received [Call=5228040] [Method=execute]
2018-12-12 11:46:35.2536: 5952    26501: DEBUG: Mediator [Slow Web Queries]: Received Call 5228040: METHOD=execute()
2018-12-12 11:46:35.2537: 5952    26501: DEBUG: PUT CALL 5228040 RUNNING
2018-12-12 11:46:35.2539: 5952    26501: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:35.0254: 5952    26501: "Slow Web Queries": IPC "corruption" Error Threshold Reached [16]
2018-12-12 11:46:35.9446: 5952     5952: DEBUG: GET MSG=RUNNING {"id":5228069,"status":2,"time":1544615194.9464979,"pid":6000}
2018-12-12 11:46:35.9448: 5952     5952: DEBUG: GET MSG=RETURN {"id":5228069,"status":3,"time":1544615194.9468861,"pid":6000}
2018-12-12 11:46:35.0945: 5952     5952: DEBUG: GET MSG=RUNNING {"id":5227143,"status":2,"time":1544615049.9508941,"pid":6001}
2018-12-12 11:46:35.9458: 5952     5952: DEBUG: GET MSG=RUNNING {"id":5228071,"status":2,"time":1544615195.202528,"pid":6002}
2018-12-12 11:46:35.9459: 5952     5952: DEBUG: GET MSG=RETURN {"id":5228071,"status":3,"time":1544615195.2028961,"pid":6002}
2018-12-12 11:46:35.9461: 5952     5952: DEBUG: PUT CALL 5228072 UNCALLED
2018-12-12 11:46:35.9461: 5952     5952: DEBUG: PUT MSG=CALL {"id":5228072,"status":0,"time":1544615195.9460471,"pid":5952}
2018-12-12 11:46:35.9462: 5952     5952: DEBUG: PUT CALL 5228073 UNCALLED
2018-12-12 11:46:35.9462: 5952     6000: DEBUG: GET MSG=CALL {"id":5228072,"status":0,"time":1544615195.9460471,"pid":5952}
2018-12-12 11:46:35.9463: 5952     6000: DEBUG: Mediator [Error 50x]: Call received [Call=5228072] [Method=execute]
2018-12-12 11:46:35.9464: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:35.9464: 5952     6000: DEBUG: Mediator [Error 50x]: Received Call 5228072: METHOD=execute()
2018-12-12 11:46:35.9464: 5952     6000: DEBUG: PUT CALL 5228072 RUNNING
2018-12-12 11:46:35.9465: 5952     6000: DEBUG: PUT MSG=RUNNING {"id":5228072,"status":2,"time":1544615195.9463961,"pid":6000}
2018-12-12 11:46:35.9468: 5952     6000: DEBUG: PUT CALL 5228072 RETURNED
2018-12-12 11:46:35.9469: 5952     6000: DEBUG: PUT MSG=RETURN {"id":5228072,"status":3,"time":1544615195.9467771,"pid":6000}
2018-12-12 11:46:35.9469: 5952     6000: DEBUG: Mediator [Error 50x]: Waiting for call [Calls=1742691/Inf] [Time=1743045s/Inf]
2018-12-12 11:46:35.0972: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:35.0978: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:35.9782: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:46:35.9782: 5952     5952: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 5228073: Retrying. Error Code: -1 Unknown error -1
2018-12-12 11:46:35.9783: 5952     5952: DEBUG: PUT CALL 5228073 UNCALLED
2018-12-12 11:46:35.9787: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:36.0443: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:36.0005: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:36.0501: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:46:36.0502: 5952     5952: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 5228073: Retrying. Error Code: -1 Unknown error -1
2018-12-12 11:46:36.0502: 5952     5952: DEBUG: PUT CALL 5228073 UNCALLED
2018-12-12 11:46:36.0507: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:36.1962: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:36.0202: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:36.2021: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:46:36.2022: 5952     5952: Mediator [Slow Web Queries]: Error: Method call failed: execute(): Unable to put call #5228073 on queue
2018-12-12 11:46:36.2023: 5952     5952: DEBUG: PUT CALL 5228074 UNCALLED
2018-12-12 11:46:36.2024: 5952     5952: DEBUG: PUT MSG=CALL {"id":5228074,"status":0,"time":1544615196.202256,"pid":5952}
2018-12-12 11:46:36.2026: 5952     5952: DEBUG: ProcessManager: Reaping child [PID=26501]
2018-12-12 11:46:36.2026: 5952     6002: DEBUG: GET MSG=CALL {"id":5228074,"status":0,"time":1544615196.202256,"pid":5952}
2018-12-12 11:46:36.2027: 5952     6002: DEBUG: Mediator [Bad Backends]: Call received [Call=5228074] [Method=execute]
2018-12-12 11:46:36.2028: 5952     6002: DEBUG: Mediator [Bad Backends]: Received Call 5228074: METHOD=execute()
2018-12-12 11:46:36.2029: 5952     6002: DEBUG: PUT CALL 5228074 RUNNING
2018-12-12 11:46:36.0203: 5952     6002: DEBUG: PUT MSG=RUNNING {"id":5228074,"status":2,"time":1544615196.2028069,"pid":6002}
2018-12-12 11:46:36.2033: 5952     6002: DEBUG: PUT CALL 5228074 RETURNED
2018-12-12 11:46:36.2034: 5952     6002: DEBUG: PUT MSG=RETURN {"id":5228074,"status":3,"time":1544615196.2033,"pid":6002}
2018-12-12 11:46:36.2035: 5952     6002: DEBUG: Mediator [Bad Backends]: Waiting for call [Calls=1742011/Inf] [Time=1743046s/Inf]
2018-12-12 11:46:36.9447: 5952     5952: DEBUG: GET MSG=RUNNING {"id":5228072,"status":2,"time":1544615195.9463961,"pid":6000}
2018-12-12 11:46:36.9449: 5952     5952: DEBUG: GET MSG=RETURN {"id":5228072,"status":3,"time":1544615195.9467771,"pid":6000}
2018-12-12 11:46:36.9451: 5952     5952: DEBUG: GET MSG=RUNNING {"id":5228074,"status":2,"time":1544615196.2028069,"pid":6002}
2018-12-12 11:46:36.9453: 5952     5952: DEBUG: GET MSG=RETURN {"id":5228074,"status":3,"time":1544615196.2033,"pid":6002}
2018-12-12 11:46:36.9454: 5952     5952: DEBUG: PUT CALL 5228075 UNCALLED
2018-12-12 11:46:36.9455: 5952     5952: DEBUG: PUT MSG=CALL {"id":5228075,"status":0,"time":1544615196.945364,"pid":5952}
2018-12-12 11:46:36.9455: 5952     5952: DEBUG: PUT CALL 5228076 UNCALLED
2018-12-12 11:46:36.9456: 5952     6000: DEBUG: GET MSG=CALL {"id":5228075,"status":0,"time":1544615196.945364,"pid":5952}
2018-12-12 11:46:36.9457: 5952     6000: DEBUG: Mediator [Error 50x]: Call received [Call=5228075] [Method=execute]
2018-12-12 11:46:36.9457: 5952     6000: DEBUG: Mediator [Error 50x]: Received Call 5228075: METHOD=execute()
2018-12-12 11:46:36.9457: 5952     6000: DEBUG: PUT CALL 5228075 RUNNING
2018-12-12 11:46:36.9458: 5952     6000: DEBUG: PUT MSG=RUNNING {"id":5228075,"status":2,"time":1544615196.94572,"pid":6000}
2018-12-12 11:46:36.9461: 5952     6000: DEBUG: PUT CALL 5228075 RETURNED
2018-12-12 11:46:36.9461: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:36.9462: 5952     6000: DEBUG: PUT MSG=RETURN {"id":5228075,"status":3,"time":1544615196.94608,"pid":6000}
2018-12-12 11:46:36.9462: 5952     6000: DEBUG: Mediator [Error 50x]: Waiting for call [Calls=1742692/Inf] [Time=1743046s/Inf]
2018-12-12 11:46:36.9715: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:36.9771: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:36.9772: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:46:36.9772: 5952     5952: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 5228076: Retrying. Error Code: -1 Unknown error -1
2018-12-12 11:46:36.9773: 5952     5952: DEBUG: PUT CALL 5228076 UNCALLED
2018-12-12 11:46:36.9777: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:37.0433: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:37.0495: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:37.0496: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:46:37.0497: 5952     5952: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 5228076: Retrying. Error Code: -1 Unknown error -1
2018-12-12 11:46:37.0498: 5952     5952: DEBUG: PUT CALL 5228076 UNCALLED
2018-12-12 11:46:37.0505: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:37.0196: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:37.2018: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:37.2019: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:46:37.0202: 5952     5952: Mediator [Slow Web Queries]: Error: Method call failed: execute(): Unable to put call #5228076 on queue
2018-12-12 11:46:37.2021: 5952     5952: DEBUG: PUT CALL 5228077 UNCALLED
2018-12-12 11:46:37.2022: 5952     5952: DEBUG: PUT MSG=CALL {"id":5228077,"status":0,"time":1544615197.202039,"pid":5952}
2018-12-12 11:46:37.2023: 5952     6002: DEBUG: GET MSG=CALL {"id":5228077,"status":0,"time":1544615197.202039,"pid":5952}
2018-12-12 11:46:37.2024: 5952     6002: DEBUG: Mediator [Bad Backends]: Call received [Call=5228077] [Method=execute]
2018-12-12 11:46:37.2025: 5952     6002: DEBUG: Mediator [Bad Backends]: Received Call 5228077: METHOD=execute()
2018-12-12 11:46:37.2025: 5952     6002: DEBUG: PUT CALL 5228077 RUNNING
2018-12-12 11:46:37.2026: 5952     6002: DEBUG: PUT MSG=RUNNING {"id":5228077,"status":2,"time":1544615197.202482,"pid":6002}
2018-12-12 11:46:37.2029: 5952     6002: DEBUG: PUT CALL 5228077 RETURNED
2018-12-12 11:46:37.0203: 5952     6002: DEBUG: PUT MSG=RETURN {"id":5228077,"status":3,"time":1544615197.2028799,"pid":6002}
2018-12-12 11:46:37.0203: 5952     6002: DEBUG: Mediator [Bad Backends]: Waiting for call [Calls=1742012/Inf] [Time=1743047s/Inf]
2018-12-12 11:46:37.2047: 5952     5952: DEBUG: Mediator [Slow Web Queries]: Forked [PID=26507]
2018-12-12 11:46:37.2372: 5952    26507: DEBUG: Mediator [Slow Web Queries]: start
2018-12-12 11:46:37.2374: 5952    26507: DEBUG: Mediator [Slow Web Queries]: Waiting for call [Calls=0/Inf] [Time=0s/Inf]
2018-12-12 11:46:37.2376: 5952    26507: DEBUG: GET MSG=CALL {"id":5228043,"status":0,"time":1544615044.9210999,"pid":5952}
2018-12-12 11:46:37.2383: 5952    26507: DEBUG: Mediator [Slow Web Queries]: Call received [Call=5228043] [Method=execute]
2018-12-12 11:46:37.2384: 5952    26507: DEBUG: Mediator [Slow Web Queries]: Received Call 5228043: METHOD=execute()
2018-12-12 11:46:37.2385: 5952    26507: DEBUG: PUT CALL 5228043 RUNNING
2018-12-12 11:46:37.2387: 5952    26507: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:37.2388: 5952    26507: "Slow Web Queries": IPC "corruption" Error Threshold Reached [22]
2018-12-12 11:46:37.9449: 5952     5952: DEBUG: GET MSG=RUNNING {"id":5228075,"status":2,"time":1544615196.94572,"pid":6000}
2018-12-12 11:46:37.9451: 5952     5952: DEBUG: GET MSG=RETURN {"id":5228075,"status":3,"time":1544615196.94608,"pid":6000}
2018-12-12 11:46:37.9453: 5952     5952: DEBUG: GET MSG=RUNNING {"id":5228077,"status":2,"time":1544615197.202482,"pid":6002}
2018-12-12 11:46:37.9455: 5952     5952: DEBUG: GET MSG=RETURN {"id":5228077,"status":3,"time":1544615197.2028799,"pid":6002}
2018-12-12 11:46:37.9456: 5952     5952: DEBUG: PUT CALL 5228078 UNCALLED
2018-12-12 11:46:37.9457: 5952     5952: DEBUG: PUT MSG=CALL {"id":5228078,"status":0,"time":1544615197.945581,"pid":5952}
2018-12-12 11:46:37.9457: 5952     5952: DEBUG: PUT CALL 5228079 UNCALLED
2018-12-12 11:46:37.9458: 5952     6000: DEBUG: GET MSG=CALL {"id":5228078,"status":0,"time":1544615197.945581,"pid":5952}
2018-12-12 11:46:37.9459: 5952     6000: DEBUG: Mediator [Error 50x]: Call received [Call=5228078] [Method=execute]
2018-12-12 11:46:37.0946: 5952     6000: DEBUG: Mediator [Error 50x]: Received Call 5228078: METHOD=execute()
2018-12-12 11:46:37.0946: 5952     6000: DEBUG: PUT CALL 5228078 RUNNING
2018-12-12 11:46:37.9461: 5952     6000: DEBUG: PUT MSG=RUNNING {"id":5228078,"status":2,"time":1544615197.9459951,"pid":6000}
2018-12-12 11:46:37.9464: 5952     6000: DEBUG: PUT CALL 5228078 RETURNED
2018-12-12 11:46:37.9464: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:37.9464: 5952     6000: DEBUG: PUT MSG=RETURN {"id":5228078,"status":3,"time":1544615197.9463429,"pid":6000}
2018-12-12 11:46:37.9465: 5952     6000: DEBUG: Mediator [Error 50x]: Waiting for call [Calls=1742693/Inf] [Time=1743047s/Inf]
2018-12-12 11:46:37.9719: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:37.9776: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:37.9777: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:46:37.9778: 5952     5952: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 5228079: Retrying. Error Code: -1 Unknown error -1
2018-12-12 11:46:37.9779: 5952     5952: DEBUG: PUT CALL 5228079 UNCALLED
2018-12-12 11:46:37.9783: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:38.0438: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:38.0498: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:38.0005: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:46:38.0501: 5952     5952: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 5228079: Retrying. Error Code: -1 Unknown error -1
2018-12-12 11:46:38.0501: 5952     5952: DEBUG: PUT CALL 5228079 UNCALLED
2018-12-12 11:46:38.0509: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:38.1962: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:38.2019: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:38.0202: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:46:38.2021: 5952     5952: Mediator [Slow Web Queries]: Error: Method call failed: execute(): Unable to put call #5228079 on queue
2018-12-12 11:46:38.2021: 5952     5952: DEBUG: PUT CALL 5228080 UNCALLED
2018-12-12 11:46:38.2023: 5952     5952: DEBUG: PUT MSG=CALL {"id":5228080,"status":0,"time":1544615198.2021201,"pid":5952}
2018-12-12 11:46:38.2024: 5952     5952: DEBUG: ProcessManager: Reaping child [PID=26507]
2018-12-12 11:46:38.2024: 5952     6002: DEBUG: GET MSG=CALL {"id":5228080,"status":0,"time":1544615198.2021201,"pid":5952}
2018-12-12 11:46:38.2025: 5952     6002: DEBUG: Mediator [Bad Backends]: Call received [Call=5228080] [Method=execute]
2018-12-12 11:46:38.2025: 5952     6002: DEBUG: Mediator [Bad Backends]: Received Call 5228080: METHOD=execute()
2018-12-12 11:46:38.2026: 5952     6002: DEBUG: PUT CALL 5228080 RUNNING
2018-12-12 11:46:38.2027: 5952     6002: DEBUG: PUT MSG=RUNNING {"id":5228080,"status":2,"time":1544615198.202554,"pid":6002}
2018-12-12 11:46:38.2029: 5952     6002: DEBUG: PUT CALL 5228080 RETURNED
2018-12-12 11:46:38.0203: 5952     6002: DEBUG: PUT MSG=RETURN {"id":5228080,"status":3,"time":1544615198.2029231,"pid":6002}
2018-12-12 11:46:38.2031: 5952     6002: DEBUG: Mediator [Bad Backends]: Waiting for call [Calls=1742013/Inf] [Time=1743048s/Inf]
2018-12-12 11:46:38.0945: 5952     5952: DEBUG: GET MSG=RUNNING {"id":5228078,"status":2,"time":1544615197.9459951,"pid":6000}
2018-12-12 11:46:38.9452: 5952     5952: DEBUG: GET MSG=RETURN {"id":5228078,"status":3,"time":1544615197.9463429,"pid":6000}
2018-12-12 11:46:38.9454: 5952     5952: DEBUG: GET MSG=RUNNING {"id":5228080,"status":2,"time":1544615198.202554,"pid":6002}
2018-12-12 11:46:38.9455: 5952     5952: DEBUG: GET MSG=RETURN {"id":5228080,"status":3,"time":1544615198.2029231,"pid":6002}
2018-12-12 11:46:38.9457: 5952     5952: DEBUG: PUT CALL 5228081 UNCALLED
2018-12-12 11:46:38.9457: 5952     5952: DEBUG: PUT MSG=CALL {"id":5228081,"status":0,"time":1544615198.9456439,"pid":5952}
2018-12-12 11:46:38.9458: 5952     5952: DEBUG: PUT CALL 5228082 UNCALLED
2018-12-12 11:46:38.9459: 5952     6000: DEBUG: GET MSG=CALL {"id":5228081,"status":0,"time":1544615198.9456439,"pid":5952}
2018-12-12 11:46:38.0946: 5952     6000: DEBUG: Mediator [Error 50x]: Call received [Call=5228081] [Method=execute]
2018-12-12 11:46:38.0946: 5952     6000: DEBUG: Mediator [Error 50x]: Received Call 5228081: METHOD=execute()
2018-12-12 11:46:38.9461: 5952     6000: DEBUG: PUT CALL 5228081 RUNNING
2018-12-12 11:46:38.9461: 5952     6000: DEBUG: PUT MSG=RUNNING {"id":5228081,"status":2,"time":1544615198.9460509,"pid":6000}
2018-12-12 11:46:38.9464: 5952     6000: DEBUG: PUT CALL 5228081 RETURNED
2018-12-12 11:46:38.9465: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:38.9465: 5952     6000: DEBUG: PUT MSG=RETURN {"id":5228081,"status":3,"time":1544615198.946389,"pid":6000}
2018-12-12 11:46:38.9465: 5952     6000: DEBUG: Mediator [Error 50x]: Waiting for call [Calls=1742694/Inf] [Time=1743048s/Inf]
2018-12-12 11:46:38.9719: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:38.9776: 5952     5952: SysV Error: IPC Test Failed [GUID=0x5302000c]
2018-12-12 11:46:38.9777: 5952     5952: SysV Error: IPC DIAG: Re-Connect failed
2018-12-12 11:46:38.9778: 5952     5952: SysV Error: Lifo\Daemon\IPC\SysV::put failed for call 5228082: Retrying. Error Code: -1 Unknown error -1
2018-12-12 11:46:38.9778: 5952     5952: DEBUG: PUT CALL 5228082 UNCALLED
2018-12-12 11:46:38.9782: 5952     5952: SysV Error: Shared Memory I/O Error [GUID=0x5302000c]
2018-12-12 11:46:38.9783: 5952     5952: "Slow Web Queries": IPC "corruption" Error Threshold Reached [26]
2018-12-12 11:46:38.9783: 5952     5952: Daemon Shutdown
2018-12-12 11:46:43.9914: 5952     5952: DEBUG: ProcessManager: Waiting for 2 processes to exit
2018-12-12 11:46:49.0043: 5952     5952: DEBUG: ProcessManager: Waiting for 2 processes to exit
2018-12-12 11:46:54.0174: 5952     5952: DEBUG: ProcessManager: Waiting for 2 processes to exit
2018-12-12 11:46:59.0322: 5952     5952: DEBUG: ProcessManager: Waiting for 2 processes to exit
2018-12-12 11:47:04.0456: 5952     5952: DEBUG: ProcessManager: Waiting for 2 processes to exit
2018-12-12 11:47:09.0585: 5952     5952: DEBUG: ProcessManager: Waiting for 2 processes to exit
2018-12-12 11:47:14.0717: 5952     5952: DEBUG: ProcessManager: Waiting for 2 processes to exit
2018-12-12 11:47:19.0847: 5952     5952: DEBUG: ProcessManager: Waiting for 2 processes to exit
2018-12-12 11:47:24.0984: 5952     5952: DEBUG: ProcessManager: Waiting for 2 processes to exit
2018-12-12 11:47:29.1111: 5952     5952: DEBUG: ProcessManager: Waiting for 2 processes to exit
2018-12-12 11:47:34.1262: 5952     5952: DEBUG: ProcessManager: Waiting for 2 processes to exit
2018-12-12 11:47:38.9943: 5952     5952: DEBUG: ProcessManager: Reaping child [PID=6002]
2018-12-12 11:47:39.0149: 5952     5952: DEBUG: ProcessManager: Reaping child [PID=6000]

Daemon implementation code:

use App\Config\Config;
use Lifo\Daemon\Mediator\Mediator;


class Daemon extends \Lifo\Daemon\Daemon
{
    /**
     * @var Mediator[]
     */
    private $workers = [];

    /**
     * @inheritdoc
     * @throws \Exception
     */
    protected function initialize()
    {
        cli_set_process_title('php: ' . Config::getApplicationName() . ': Master Process');
        Config::get()->load();

        foreach (Config::get()->getSubmodules() as $submodule) {
            $this->workers[] = $submodule::getWorkerName();
            $this->addWorker($submodule, $submodule::getWorkerName());
        }
    }

    /**
     * @inheritdoc
     * @throws \Exception
     */
    protected function execute()
    {
        foreach ($this->workers as $worker) {
            $this->worker($worker)->execute();
        }
    }
}

Workers execute method:

    /**
     * @inheritdoc
     */
    public function execute() 
    {
        $records = [];

        while ($msg = json_decode(Config::get()->getRedis()->lPop($this->getRedisKey()))) {
            $records[] = $msg;
        }

        if (static::useStats()) {
            $this->processStats($records);
        }

        if (static::useAlerts()) {
            $this->processAlerts($records);
        }
    }

Memory leak in workers

Hi,

I think i found a bug in the worker part. I cannot reproduce this with tasks. Some daemons use tasks instead of workers and they dont have memory leakage.

I setup a new daemon, with a clean worker without any extra code. Debug shows:

2018-06-06 13:02:26.5656: 6303 6303: DEBUG: Daemon startup [Interval=0.1s] [Restart=Never]
2018-06-06 13:02:26.5932: 6303 6303: LEAK process worker started. This controls all outputs.
2018-06-06 13:02:26.6011: 6303 6303: DEBUG: Daemon::run loop started
2018-06-06 13:02:27.5044: 6303 6303: DEBUG: leak memory usage: 2871920
2018-06-06 13:02:28.5062: 6303 6303: DEBUG: leak memory usage: 2873440
2018-06-06 13:02:29.5081: 6303 6303: DEBUG: leak memory usage: 2874960
2018-06-06 13:02:30.0051: 6303 6303: DEBUG: leak memory usage: 2876480
2018-06-06 13:02:31.0512: 6303 6303: DEBUG: leak memory usage: 2878000
2018-06-06 13:02:32.5149: 6303 6303: DEBUG: leak memory usage: 2879520

A daemon that start every cycle a small task shows:
2018-06-06 12:40:10.0787: 3351 3351: DEBUG: in memory usage: 2193552
2018-06-06 12:40:11.0805: 3351 3351: DEBUG: in memory usage: 2193552
2018-06-06 12:40:12.0824: 3351 3351: DEBUG: in memory usage: 2193552
2018-06-06 12:40:13.0847: 3351 3351: DEBUG: in memory usage: 2193552
2018-06-06 12:40:14.0866: 3351 3351: DEBUG: in memory usage: 2193552

As you can see the worker daemon slowly start consuming memory.

leakDaemon.txt
leakWorker.txt
run.txt

How to write unit tests with your daemon?

Hello,

I'm trying to unit test my code using your daemon. Whether I set daemonize to true or false, it blocks execution of everything else, so I can't seem to start it up and then do any more tests.

Do you have any advice or examples on how to test with this? I've got other working examples of phpunit tests, so it's not like I don't know how to set up my unit tests, but I'm stumped on testing with your daemon. I'm guessing it forks and blocks or something.

Here's an example of what's going on.

logfile = __DIR__ . '/../logs/migrate_posts_service.log'; $this->pidfile = __DIR__ . '/../locks/ALO_Client_Daemon_Migrate_Posts.pid'; if (file_exists($this->logfile)) unlink($this->logfile); $this->mp = ALO_CLI_Daemon_Migrate_Posts::getInstance() ->setDaemonize(false) ->setLoopInterval(20) ->setVerbose(true) ->setDebug(true) ->setDebugLevel(3) ->setLogFile($this->logfile) ->run(); echo "Finished setUp()\n"; } And then I have some tests defined later in this test file. If I run this, it hangs when I start the daemon. Even the error log line (error_log("got into Migrate Posts setUp()");) that is called BEFORE starting the daemon doesn't actually print to the error log until after I kill the daemon process. Also I tried echoing instead of error_log, but echo doesn't echo anything. I'm guessing that's also from starting up your daemon, because I can echo in my other unit tests. Your daemon looks promising and I'd really love to get it working, but I need to be able to write testable code. Any advice? Thanks, Dave

Method returns current worker proc count?

I know Daemon sets workers within max proc but I need to check the proc count before a worker instance is created. This causes an issue for me since Promise doesn't return results from sudden death of a forked proc (some worker processes last for more than 6 hours) . I can't seem to find any method returning the current proc count.

Parameters on Task->run causing Fatal Error

PHP Fatal error: Declaration of Daemon\Tasks\Feeds::run($a, $b) must be compatible with Lifo\Daemon\Task\TaskInterface::run() in Test.php on line 5

Run defined as:

    public function run($a, $b) // Two Parameters added.
    {
        $this->log("Task is running in the background! My PID=%d. I will now sleep for 2 seconds", getmypid());
        $this->log("$a, $b");
        sleep(2);
        $this->log("Task is done and will exit now");
    }

Called by:

$this->task('Test', ['AAA', 'BBB']);

Forking Exception: Can't find PDO driver

I am encountering a forking issue -- says PDO driver isn't found.

Error: Forking Exception: could not find driver in file: /var/www/api/daemons/projectX/OrderProcess.php on line: 65
Plain Stack Trace:
#0 /var/www/api/daemons/projectX/OrderProcess.php(65): PDO->__construct('sqlsrv:Server=l...', 'MY_USER', 'MY_PASS')
#1 [internal function]: OrderProcess->run()
#2 /var/www/api/vendor/lifo/php-daemon/src/Lifo/Daemon/Daemon.php(1593): call_user_func_array(Array, Array)
#3 [internal function]: Lifo\Daemon\Daemon->Lifo\Daemon\{closure}()
#4 /var/www/api/vendor/lifo/php-daemon/src/Lifo/Daemon/Plugin/ProcessManager.php(148): call_user_func_array(Object(Closure), Array)
#5 /var/www/api/vendor/lifo/php-daemon/src/Lifo/Daemon/Daemon.php(1612): Lifo\Daemon\Plugin\ProcessManager->fork('task', Object(Closure))
#6 /var/www/api/daemons/projectX/OrderDaemon.php(42): Lifo\Daemon\Daemon->task('OrderProcess')
#7 /var/www/api/vendor/lifo/php-daemon/src/Lifo/Daemon/Daemon.php(1746): OrderDaemon->execute()
#8 /var/www/api/vendor/lifo/php-daemon/src/Lifo/Daemon/Daemon.php(491): Lifo\Daemon\Daemon->loop()
#9 /var/www/api/daemons/projectX/run.php(15): Lifo\Daemon\Daemon->run()
#10 /var/www/api/daemon.php(33): {closure}()
#11 {main}

Line 65, as noted above, is this:

$connection = new PDO($dsn, AZURE_SQL_USER, AZURE_SQL_PASS);

I am using this driver elsewhere on the same server, so it is installed and working.

Thoughts?

Arguments to Tasks

I don't get it. https://github.com/lifo101/php-daemon/wiki/Tasks

Any extra arguments given to Daemon::task($task, [$arg1, $arg2, ...]) will be passed to the Task::run()

How do you consume the argument?

protected function execute()
{
     $this->task('Tester', ['JustTesting']);
}

class Tester extends AbstractTask
{
    use \Lifo\Daemon\LogTrait;

    public function run()
    {
        $this->log('I WANT TO CONSUME THE PASSED ARGUEMENT', getmypid());
    }

}