Is there a way to use WireHttp non-blocking (sending concurrent requests)?

[bernhard]

$http = new WireHttp();
$t = Debug::timer();
$http->get("https://www.google.com");
$http->get("https://www.google.com");
$http->get("https://www.google.com");
db(Debug::timer($t)*1000);

This adds up some hundreds of milliseconds for each request. Is there a way to send those requests concurrently so that it takes roughly the same time for 1 and for multiple requests?

I know that Guzzle supports this, but I'd prefer to use PW core tools 🙂 https://docs.guzzlephp.org/en/stable/quickstart.html#concurrent-requests

I didn't find anything for the terms "concurrent" or "simult" in WireHttp.php so I'm not too optimistic but I'd be happy to be wrong 🙂 

[netcarver]

I know it's not native PW - but you can use parallel curl for this kind of thing.

$urls = [
    'https://www.example.com/',
    'https://www.google.com/',
    'https://www.github.com/'
];

$handles = [];

$multi_handle = curl_multi_init();

foreach ($urls as $url) {
    $handle = curl_init();
    curl_setopt($handle, CURLOPT_URL, $url);
    curl_setopt($handle, CURLOPT_RETURNTRANSFER, true);
    curl_multi_add_handle($multi_handle, $handle);
    $handles[] = $handle;
}

$running = null;
do {
    curl_multi_exec($multi_handle, $running);
} while ($running);

foreach ($handles as $handle) {
    $result = curl_multi_getcontent($handle);
    echo $result;
    curl_multi_remove_handle($multi_handle, $handle);
    curl_close($handle);
}

curl_multi_close($multi_handle);

 

[bernhard]

NICE!

Single url:

6 urls:

Now to you also have a version that limits the number of concurrent requests somehow. 

The idea I have in my mind is to build some basic uptime monitoring service like uptime kuma. If I had 100 domains to monitor and every request takes 1 second I can't monitor all domains every 60s. On the other hand I don't think it would be a good idea to let curl fire 100 concurrent requests, would it?

Also not sure if I'm really going to build this, but it would be fun I guess 🙂 

[netcarver]

I don't see why you couldn't monitor 100 urls every 60s as long as the machine you are using turns over the sockets fast enough (and has high enough limits on how many open handles it can have etc.)  Your server might already have high enough resource limits to allow it, but if it doesn't then ask chatGPT about things like decreasing tcp connection recycling times (wait timeout) and about linux max open files to see how to up the limits.

[bernhard]

1 minute ago, netcarver said:

I don't see why you couldn't monitor 100 urls every 60s as long as the machine you are using turns over the sockets fast enough (and has high enough limits on how many open handles it can have etc.)  Your server might already have high enough resource limits to allow it, but if it doesn't then ask chatGPT about things like decreasing tcp connection recycling times (wait timeout) and about linux max open files to see how to up the limits.

If I did a simple foreach() with WireHttp() and a single request would take 1s then it would take 100s to request all 100 domains. That would mean that after 60s the next cronjob would be started and the timestamp of all requests would be heavily out of range.

Eg a cronjob started at 12:00:00 would get its results at 12:00:00 + 100s = 12:01:40 which is really not what I want 🙂 

[wbmnfktr]

Found this quite some time ago:

https://upptime.js.org/

Might be a better fit or at least an option.

[netcarver]

3 hours ago, bernhard said:

Eg a cronjob started at 12:00:00 would get its results at 12:00:00 + 100s = 12:01:40 which is really not what I want 🙂 

Sure, but just to be clear, I meant with the parallel curl approach :)

[netcarver]

BTW, if you do go for this approach, I'd also consider setting a timeout on each connection - so the whole thing should definitely finish within your 60 second limit.

    curl_setopt($handle, CURLOPT_CONNECTTIMEOUT, 5);

 

[bernhard]

35 minutes ago, netcarver said:

Sure, but just to be clear, I meant with the parallel curl approach 🙂

Ok thx for the clarification 🙂 

18 minutes ago, netcarver said:

BTW, if you do go for this approach, I'd also consider setting a timeout on each connection - so the whole thing should definitely finish within your 60 second limit.

    curl_setopt($handle, CURLOPT_CONNECTTIMEOUT, 5);

Thx! I've just did a quick test on my VPS and it took 11s to fire 140 requests without any caps in place (no timeout, no max concurrency limit). htop showed one cpu core maxed out at 100% during that request. I thought that might come from the do/while so I've added a usleep() of 100ms into the loop but that made no difference (neither in the overall time needed nor in the cpu usage).

[netcarver]

If you are running a CLI-based cronjob for this, then you could take a different approach and try out Framework-X as it's built around react php specifically for this kind of task. Or take a look at react php's http client class if you want to do something more low-level.

[bernhard]

16 minutes ago, netcarver said:

If you are running a CLI-based cronjob for this, then you could take a different approach and try out Framework-X as it's built around react php specifically for this kind of task. Or take a look at react php's http client class if you want to do something more low-level.

What do you think would be the benefit compared to using curl_multi? It definitely looks awesome and I've been following ReactPHP for quite a long time, but it also feels a lot more complex than just adding a cronjob and boostrapping PW 🙂 

[netcarver]

In this case, I'd probably just stick to bootstrapping PW from the cronjob script and then using parallel curl.  I was just throwing out some options in case you wanted to explore using something specifically written for async IO like this.

[bernhard]

Great. Thanks for all your input 🙂 

[bernhard]

Did a little more research on this ... https://susam.net/blog/timing-with-curl.htm

The problem that I'm facing is that curl_multi_exec seems to completely mess up the timings mentioned in that post (and several others). As long as I do single curl requests the timings look reasonable and a page request takes around 200 to 300ms for a fast site and 900 for a slow one.

With curl_multi_exec I get ranges from 4 to 10seconds from curl_getinfo($handle) 😞 

So it looks like it's not so easy to create a little uptime monitor just with php + curl that works with more than a handful of urls.