stil/curl-easy

cURL wrapper for PHP. Supports parallel and non-blocking requests. For high speed crawling, see stil/curl-robot

curl-easy's Issues

Memory Leaks

I'm trying to use this class to make hundreds of thousands of requests over time (8 at a time) in a long running process and I found that there are memory leaks with this system (or the Symphony Event Dispatcher).

Every additional request seems to add 30kb to the memory usage.

Show progress

hi, what correct way to show progress while proccess running (non blocking)
I have tried this

<?php
require"vendor/autoload.php";
file_put_contents( 'progress.txt', '' );
$request = new \cURL\Request('http://ftp.free.org/mirrors/releases.ubuntu-fr.org/11.04/ubuntu-11.04-desktop-i386-fr.iso');
$request->getOptions()
    ->set(CURLOPT_NOPROGRESS,false)
    ->set(CURLOPT_PROGRESSFUNCTION,'progressCallback')
    ->set(CURLOPT_RETURNTRANSFER, true);
$request->addListener('complete', function (\cURL\Event $event) {
     $response = $event->response;
    $data = $response->getContent();
    file_put_contents("test.iso",$data);
    echo"complete";
});
function progressCallback( $download_size, $downloaded_size, $upload_size, $uploaded_size )
{
    static $previousProgress = 0;
    
    if ( $download_size == 0 )
        $progress = 0;
    else
        $progress = round( $downloaded_size * 100 / $download_size );
    
    if ( $progress > $previousProgress)
    {
        $previousProgress = $progress;
        $fp = fopen( 'progress.txt', 'a' );
        fputs( $fp, "$progress\n" );
        fclose( $fp );
    }
}
while ($request->socketPerform()) {
    $array=file("progress.txt");
$count=count($array)-1;
echo $array[$count];
    $request->socketSelect();
    
}

Could you tag it with a version number?

Hi, I would like to use your library, but it does not have a version number - I can only install it via composer if I use * as the version. My employer does not like that (if a backward incompatible update is made to the library, it will automatically be pulled in on the next composer update, even if it has a new major version number). It may well be that you don't intend to make any updates, but as a matter of good practice, a tag with a version number would be appreciated. Thanks!

No error code is displayed

Hi,
I am having trouble getting error code and message. $response->getError(); returns null.

public function curl_req($path, $data = array())
      {
         $request = new \cURL\Request($this->_url . $path);
        $request->getOptions()
          ->set(CURLOPT_TIMEOUT, 5)
          ->set(CURLOPT_RETURNTRANSFER, true)
          ->set(CURLOPT_HEADER,true)
          ->set(CURLOPT_VERBOSE,true)
           ->set(CURLINFO_HEADER_OUT,true)
          ->set(CURLOPT_USERAGENT,'curl/7.54.0')
          ->set(CURLOPT_HTTPHEADER, $this->getHeader());
        if( !empty($data) ) {
           $request->getOptions()
               ->set(CURLOPT_POSTFIELDS, json_encode($data))
               ->set(CURLOPT_SAFE_UPLOAD, true);
        }
        if ($response->hasError()) {
            $error = $response->getError();
            $e = 'Error code: ' . $error->getCode() . 'Message: "' . $error->getMessage() . '"' . "\n";
        }
     
        $this->getLogger()->info('Connect to server: '.$e. ' try: '.$try);
    
    $feed = json_decode($response->getContent(), true);

    return $feed;
}

curl verbose:
< HTTP/1.1 401 Unauthorized
< Server: nginx/1.10.0 (Ubuntu)
< Content-Type: application/json
< Transfer-Encoding: chunked
< Cache-Control: no-cache
< WWW-Authenticate: WSSE realm="Secured API", profile="UsernameToken"
< Date: Tue, 12 Sep 2017 12:09:56 GMT
< Set-Cookie: SERVERID=web1; path=/
<

• Connection #0 to host orocampus.tk left intact
  NULL

How to make a request that run a function asynchronous?

I've camed out with this code: https://pastebin.com/TC288kdq.
But, obviusly, it doesn't work.
How can I do that?

Queuing of CURLOPT_POSTFIELDS

Make it possible to queue requests with arguments passed though CURLOPT_POSTFIELDS. This would allow packages, e.g., XML, to be queued and sent as a HTTP POST.

Set up a queue, add during lifetime of script

I am busy with creating a server that reads info from mqtt and needs to process it (post it to another url). Therefor I found this script.
What I need to do is set up the curl-easy queue, and when I have on MQTT a new message, there should be added a requests.

What I Tried now is on starting of the MQTT handler:

$queue = new \cURL\RequestsQueue;
$queue
    ->getDefaultOptions()
    ->set(CURLOPT_RETURNTRANSFER, true);

Then every time there is a new requests I add it like this
$req = new \cURL\Request('<url>'); $req->getOptions()->set(CURLOPT_POSTFIELDS, $data); $requests[] = $req;

And after every message I call
$queue->attach(array_pop($requests)); $queue->send();

But is that the correct way? Because now I think that the queue is executed multiple times.

In fact the final result should be:

• set up queue
• on new message: add request
• on new message: add request
  ...

and that in the background the requests are done (with a maximum nr of requests at a certain time would be very nice).

Thanks

Reuse same Connection

Is there any way to use the lib as such it does not kill active keep-alive connections, so that it sends multiple requests over the same connection?

using different proxy with each request

Dear All ,
I know that I can set the proxy with

$queue = new \cURL\RequestsQueue;
$queue
    ->getDefaultOptions()
    ->set(CURLOPT_RETURNTRANSFER, true)
    ->set(CURLOPT_PROXY, $proxy)
   ->set(CURLOPT_FRESH_CONNECT,true);

But this queue setting will be used with each request.
I need to use a different proxy with each request, how can I setup such?

multi post fields in one request

Hi,

how can I send in a single request (only one url) post json_encode two or more arrays, for example:

url: my.urlapi.com
post_fields = array (
array (name => Jhon, age => 30),
array (name => Doe, age => 25),
and more 100 arrays...
);

The destination server only allows 1 entry per request. With your library, is it possible to apply more than one entry in the same request?

Thanks!

[Help] Different cookies in parallel

Is there a way to send 2 requests to the same url in parallel, but on one request a cookie is set to "Cookie: user=1" and on the 2nd request it's set to "Cookie: user=2" for example?

Thanks!

'complete' event on individual Requests is not triggered if they're attached to Queue

<?php
use cURL\Request;
use cURL\RequestsQueue;
use cUrl\Event;

$request = new Request('http://foo.bar/');
$request->getOptions()
    ->set(CURLOPT_TIMEOUT, 5)
    ->set(CURLOPT_RETURNTRANSFER, true);
$request->addListener('complete', function (Event $event) {
    $response = $event->response;
    $feed = json_decode($response->getContent(), true);
    echo $feed['entry']['title']['$t'];
});

$queue = new RequestsQueue();
$queue->attach($request);
$queue->send();

You would expect an complete event to be triggered, which isn't true.

$response->getContent() fails when trying to use with loadHTML() function of DOMDocument

Ok, so when I use the normal cURL, if I get a web like a string, later I can use it with DOMDocument, but in this case, the string explote. I don't know why. I read the documentation and I found nothing, maybe it's because is late but I think it should work.

$request = new \cURL\Request(); $request->getOptions() ->set(CURLOPT_TIMEOUT, 5) ->set(CURLOPT_RETURNTRANSFER, true) ->set(CURLOPT_URL, 'https://www.google.com/'); $response = $request->send(); $feed = $response->getContent(); $dom = new DOMDocument(); $dom->formatOutput = true; $dom->loadHTML($feed);

When I load the string $feed, it explotes. I tried to echo the string $feed and it is the website I try to get.

dependency injection in queue

Mr.Stil,

in my crawler i have include a html tidy function : htmLawed.php
in the queue Listener how to injection this htmLawed($contents, $config) function?

include 'htmLawed.php';
$config = array('safe'=>1);
$queue->addListener('complete', function (\cURL\Event $event) use (&$data)
{
    $response = $event->response;
    $dom = new DOMDocument();
    $dom->preserveWhiteSpace = false;
    $dom->loadHTML($response->getContent());
    $xpath = new DomXPath($dom);
    $query = $xpath->query($data['xpath']);
    // tidy the html
    $content = htmLawed($dom->saveXML($query->item(0)), $config);
    // htmLawed() function not in queue
}

Unstable repsonse

My script add 3 request to queue,
but sometime i receive all 3 response, another time i only receive 1 response, another time nothing receive. 👎

`/** @var goi sang ghi chuong trinh $record */
$queue = new RequestsQueue();
$queue->getDefaultOptions()
->set(CURLOPT_TIMEOUT, 5)
->set(CURLOPT_RETURNTRANSFER, true);

    $queue->addListener('complete', function (\cURL\Event $event) {
        $response = $event->response;
        $json = $response->getContent(); // Returns content of response
        $feed = json_decode($json, true);
        echo '<pre>Response';
        print_r($feed);
        echo '</pre>';
    });
    foreach($liveChannelSchedule as $record){
        Yii::error($record->title,'datvt');
        $request = new Request('http://api.xx.vn/record/update?id='.$record->id);
        $queue->attach($request);
        /*$record->uuid = rand(100,99999);
        $record->recorded_time = new Expression("NOW()");
        $record->save(false);*/
        echo 'Record: '.$record->id.' - '.$record->title.PHP_EOL;
    }

    // Execute queue
    while ($queue->socketPerform()) {
        echo '*';
        $queue->socketSelect();
    }`

and response run 1th:
Record: 330 - Phim tài liệu Record: 902 - PHIM TRUYỀN HÌNH: Tình người xứ hoa - Tập 12 Record: 587 - 113 online (2) *************<pre>Response</pre><pre>Response</pre><pre>Response</pre>

run again:

Record: 330 - Phim tài liệu Record: 902 - PHIM TRUYỀN HÌNH: Tình người xứ hoa - Tập 12 Record: 587 - 113 online (2) *************<pre>ResponseArray ( [error] => 0 [uuid] => 8520 [id] => 902 ) </pre>****<pre>Response</pre><pre>Response</pre>

Keep PHP 5.4 Support

It would be great if PHP 5.4 support still could given, we use it in a open source project which runs on severall 5.4 Machines .

$queue->send() returns true even if individual requests fail

Thanks for making a great cURL library!

I think the $queue->send() command seems to return true whether or not individual requests succeed. Is this expected behaviour?

Upgrade symfony event dispatcher

Hi, is this still maintained? I would need a symfony dependencies upgrade...

Allow user set maximum number of processed requests in queue at once

RequestsQueue should provide feature of limiting total amount of requests processed at once.

Performance Non-blocking parallel

hi Mr.Stil,

it's the Non-blocking requests in parallel compare to others methods Non-blocking requests in parallel is more fast and better performance?
https://github.com/stil/curl-easy#non-blocking-requests-in-parallel

getCount on request

Is it possible to add to the Request class a variable where you can get the count of the queue elements which is internally used?

I want to use it to avoid the string comparision on my usage:

        foreach($this->requests as $key => $request) {
            /** @var Request $request */
            try {
                if ($request->socketPerform()) {
                    $request->socketSelect();
                }
            } catch(Exception $e) {
                if ($e->getMessage() == "Cannot perform if there are no requests in queue.") {
                    unset($this->requests[$key]);
                } else {
                    throw $e;
                }
            }
        }

Bug, not always catch error code, as result infinite loop, in non-blocking mode with parallel requests

Example:

$request = new \cURL\Request(....);
// Specially set the variable $logout_url to empty.
$logout_url = '';

$request->getOptions()
    ->set(CURLOPT_URL, $logout_url)
    ->set(CURLOPT_COOKIEJAR, $cookie_jar);

// Execute queue
while ($queue->socketPerform()) {
    $queue->socketSelect();
}

As a result, we obtain an infinite loop running.

Problem in line code: https://github.com/stil/curl-easy/blob/master/src/cURL/RequestsQueue.php#L202
this is bugging code.

The problem described by link http://stackoverflow.com/questions/23612881/detect-curl-timeout-in-php

Solution:

//$runningBefore = $this->runningCount;                                                                           
do {                                                                                                            
      $mrc = curl_multi_exec($this->mh, $this->runningCount);                                                     
      $this->read();                                                                                              
} while ($mrc === CURLM_CALL_MULTI_PERFORM || $this->runningCount);
/*
$runningAfter = $this->runningCount;                                                                            

if ($runningAfter < $runningBefore) {                                                                           
    $this->read();                                                                                            
}
*/

Getting request class not found error

I have tried to include this library using composer but still i am getting class not found error Can any one tell me how can i fix this one

for including library i used :- require_once 'vendor/autoload.php';

how to know when queue is completly done ?

$queue = new \cURL\RequestsQueue;

    $queue->addListener('complete', function (\cURL\Event $event) {
    // do some stuff like add to array 
    });

foreach($buildings as $building){

        $request = new \cURL\Request();
        $request->_myId = $i;
                $postdata="id= $building->id";
        // add options to request
        $opts = new \cURL\Options;
        $opts->set(CURLOPT_URL, 'example,');
        $opts->set(CURLOPT_POSTFIELDS, $postdata);
        $request->setOptions($opts);
        $queue->attach($request);
    }

    $result = $queue->send();   

how can i know when the queue is done with everything ?

Make it easy to find out "which response is which"

It's a very popular problem - how to know which response is which. They may appear in random order, so we need to add a feature allowing to associate responses with their starting parameters.

How to store result and return after completion of all requests ?

I have created a function like this

protected function curl_get_asyn($urls) {
    	$result = [];
    	$queue = new RequestsQueue();
    	$queue->getDefaultOptions()
    	->set(CURLOPT_TIMEOUT, 5)
    	->set(CURLOPT_RETURNTRANSFER, true);
    	// Set function to execute when request is complete.
    	$queue->addListener('complete', function (\cURL\Event $event) {
    		$response = $event->response;
    		$result[] = [
    			'url' => $event->request->_url,
    			'response' => $response->getContent()
    		];
    	});
    	foreach ($urls as $key => $url) {
    		$request = new \cURL\Request($url);
    		$request->_url = $url;
    		// Add request to queue
    		$queue->attach($request);
    	}
    	
    	// Execute queue
    	$timeStart = microtime(true);
    	while ($queue->socketPerform()) {
    		usleep(1000);
    	}	
    	return $result;
    }

return $result is always an empty array because $result array inside listener is different from the one which is returned at the last. How can I achieve this ?

Get array key identifier for queue

Hey all,

I believe it might be related to #14
Anyhow, I will ask :)

Here is the idea.

    $rows = [1 => 'url1', 5 => 'url2', ... ];
    $requests = [];
    foreach ($rows as $id => $url) {
        $requests[$id] = new \cURL\Request($url);
    }

Is there any way to get the key ID?

    $queue->addListener('complete', function (\cURL\Event $event) use (&$requests) {
            $response = $event->response;
            $content = $response->getContent(); // Returns content of response

// TODO: get id here

            if ($next = array_pop($requests)) {
                $event->queue->attach($next);
            }
    });

    $queue->attach(array_pop($requests));
    $queue->attach(array_pop($requests));
    $queue->send();

Thanks in advance.

Codificacion charset error

Hello,
Using curl-easy to download json data from a restfull service, the charset was encoded in another type different of my petition (UTF-8), and this is a huge problem because I need to use the accent marks (like á, é, í, ó, ú, etc.).
Could your library solve this problem?
Reggards!

Event Response

// Set function to be executed when request will be completed
$queue->addListener('complete', function (\cURL\Event $event) {
    $response = $event->response;
    $json = $response->getContent(); // Returns content of response
    $feed = json_decode($json, true);
    echo $feed['entry']['title']['$t'] . "\n";
});

The event object of the EventDispatcher dont have any response property anymore, how to get the response then?

PHP 5.3.10+ curl_multi_select issue

In PHP versions 5.3.10+ curl_multi_select changed its behaviour.
Probably bug: https://bugs.php.net/bug.php?id=63411
It causes problem with cURL\RequestsQueue::send() which is not working now

Temporary workaround: instead of

$queue->send();

You should do:

while ($queue->socketPerform()) {
    $queue->socketSelect();
}

License

Hi;

Can you add Licence?

how to load curl-easy without composer autoload

the composer autoload is use too much memories
request sort by XHProf Incl.MemUse(bytes), as you see the top 10 memories usage list composer use too much.

main()
spl_autoload_call
???_op
Composer\Autoload\ClassLoader::loadClass
Composer\Autoload\includeFile
cURL\RequestsQueue::send
cURL\RequestsQueue::socketPerform
spl_autoload_call@1
ComposerAutoloaderInit3163d95abb499df82cfeb181810bb3d1::getLoader
Composer\Autoload\ClassLoader::loadClass@1
Composer\Autoload\includeFile@1
cURL\RequestsQueue::read

how to load curl-easy without composer autoload?