Htmx, Sse and real time dom swap help.

[bernhard]

Have you tested my module?

[kongondo]

32 minutes ago, bernhard said:

Have you tested my module?

Not the module itself but the code. If you put that in ready.php, i.e., the hook, serve() and sse() everything hangs. For me, this is also a learning opportunity. It could be that sse is better handled in a module, maybe an autoload one. That it might not work with ready.php. This is one of the things I am trying to establish as well. 

[bernhard]

Well please try the module and see if that works first. Then try a custom approach and see if it breaks.

The reason why I created the modules is exactly to avoid hickups when copy/pasting things from this thread or placing things in the wrong spot... So that everybody has a quick test of SSE with a single click (install module) and can proceed from there.

[kongondo]

I am sure your module works just fine but I'll test it anyway ?...doing it right now. 

Having said that, I am not entirely convinced that your module represents a real world scenario since we have to click on a button to init the SSE.

[kongondo]

Just tested. And I take back what I said. Your module also locks system resources. Admin and frontend don't load while the countdown is on. Locking happens on both front and backend pages, pages tree, fields, templates, etc. I have tested with both the 10 countdown and a 50 countdown. I'll see if I can get a video demo.

So, back to my questions (not just directed at @bernhard) but the community.

1. Are we able to to have SSE work without a module? Some people might just want this to work in ready.php without having to install a module. 
2. Are we able to have SSE that is totally event driven? I.e., a page is saved and the frontend gets notified.
3. Are we able to have SSE that doesn't lock system resources?

Maybe we should start a new topic?

[kongondo]

Here's the demo video showing the system lock. Let me know if you cannot watch it. Note, the system will lock even if the count is 10.

 

Edited March 17, 2022 by kongondo

[3fingers]

@kongondo Here I've found an interesting article about SSE and php and one thing mentioned, to avoid session locking (which I think its our culprit here) is to use:

// make session read-only
   session_start();
   session_write_close();

Above everything else in the code used to send data.
I cannot test it at the moment, would you mind to have a check and report it back to us? ?

I've found however possible bad implications doing this, as mentioned here in the forum.
 

[horst]

Hi you all. May I ask a question? AFAIK SSE only works proper over HTTP2 protocol. Is there any working solution known from within the browsers JS to detect if the server supports or delivers in H2 ??

[kongondo]

7 minutes ago, 3fingers said:

I've found an interesting article about SSE and php and one thing mentioned, to avoid session locking (which I think its our culprit here) is to use:

// make session read-only
   session_start();
   session_write_close();

 

Thanks. I have seen that article before (as well as similar posts in SO). It is very helpful. I didn't try the suggested fix since I suspected it might interfere with ProcessWire $session. I wanted to ask in the forums but forgot. Anyone knows if this would happen? 

10 minutes ago, 3fingers said:

I cannot test it at the moment, would you mind to have a check and report it back to us? ?

I'll have a go and maybe also check if $session is affected.

[kongondo]

9 minutes ago, horst said:

Hi you all. May I ask a question? AFAIK SSE only works proper over HTTP2 protocol. Is there any working solution known from within the browsers JS to detect if the server supports or delivers in H2 ??

Not foolproof

https://stackoverflow.com/questions/56301138/how-to-check-whether-server-supports-http-2-in-javascript-on-browser

[horst]

11 minutes ago, kongondo said:

I'll have a go and maybe also check if $session is affected.

 

[bernhard]

Ok now we are talking about a totally different topic... But first things first.

3 hours ago, kongondo said:

Having said that, I am not entirely convinced that your module represents a real world scenario since we have to click on a button to init the SSE.

This is exactly what I'm using SSE for. The user clicks a button, ProcessWire starts doing a long running task (for example trashing lots of pages) and while it does so, the progress is sent to the client so that the client knows what's going on. I'd call that real world ?  The reason why I chose SSE over sockets and over not implementing any client feedback is obvious:

1. Having a long running task without a progress bar is just not an option. What if the user trashed several hundreds of pages and didn't know if that takes 5, 10 maybe 120 seconds?
2. Other techniques are much more complicated to setup (for example web sockets). SSE is built in and does just what I needed: Inform the user of the progress of the task without polling.

3 hours ago, kongondo said:

Just tested. And I take back what I said. Your module also locks system resources. Admin and frontend don't load while the countdown is on. Locking happens on both front and backend pages, pages tree, fields, templates, etc. I have tested with both the 10 countdown and a 50 countdown. I'll see if I can get a video demo.

So, back to my questions (not just directed at @bernhard) but the community.

1. Are we able to to have SSE work without a module? Some people might just want this to work in ready.php without having to install a module. 
2. Are we able to have SSE that is totally event driven? I.e., a page is saved and the frontend gets notified.
3. Are we able to have SSE that doesn't lock system resources?

Maybe we should start a new topic?

Of course while this task is running the system is blocked for all other requests. As far as I understand that's the nature of how PHP works. We have the same limitation everywhere in PW and everything in PHP... For example if you empty the trash with lots of pages and you try to visit the PW backend in another browser tab you'll also be blocked until the first tab has finished it's job - or am I wrong here?

PHP is single threaded so if you wanted to come around this you'd have to do additional things. I've no experience in this territory, but this seems to look like it could be worth a try: https://stackoverflow.com/a/4350418 There's also https://www.php.net/Thread but that is based on a PECL. And so is https://openswoole.com/ which looks like it comes closer to what you are trying to achieve than what is possible with SSE as far as I understand.

1. Of course we are! Why shouldn't we?? The module is just here to have a common foundation to test things out and eliminate confusion about who copied which code snippet to whatever place etc.; You could now test the linked popen() technique and just add that to the module as a PR and everybody could understand and learn.

2+3. We'd need to have a real non-blocking background process for that. But as I said that not a matter of SSE that's a matter of multi threading in PHP.

https://www.youtube.com/watch?v=spDpR2qr-Fs&ab_channel=Dr.RobertDimpsey

So for my use case SSE seem to be a good option. For updating a website's content based on "real time data" polling might still be the better option as it will not block the user.

[kongondo]

10 hours ago, bernhard said:

PHP is single threaded

Yes. I am quite aware of this ?.

10 hours ago, bernhard said:

Of course while this task is running the system is blocked for all other requests.

It is your previous statement below that confused me:

On 3/10/2022 at 1:32 PM, bernhard said:

You might have a process running then in the background.

It seemed to suggest that your code was non-blocking. 

10 hours ago, bernhard said:

The reason why I chose SSE over sockets and over not implementing any client feedback is obvious:

Just curious about this. Was your goal to give the current editor feedback or all editors who use the system? I mean,  did you want the progress to show on different machines simultaneously on any ProcessWire backend page or just the current editor? I have a feeling this won't make sense so will just add. If I am Editor A and I am currently making edits. Across me sits Editor B. Editor B is not making edits. Maybe she is looking at the page tree. If Editor A makes changes and SSE kicks in, Editor B will see the progress on her machine. Was this your use case?

[bernhard]

5 hours ago, kongondo said:

It is your previous statement below that confused me:

On 3/10/2022 at 2:32 PM, bernhard said:

You might have a process running then in the background.

5 hours ago, kongondo said:

It seemed to suggest that your code was non-blocking. 

Ok sorry about that. I try to explain it in other words. What I meant was that if your system is blocked and does not respond to new requests it is likely that you have a process that is still running. This can happen quickly while testing SSE because you can quickly get into an endless loop and the process will run and run and run and block your system until the process is killed because it reaches max_execution_time for example. That's why I suggested to add a hard break while testing so that you exit after 30 iterations and the system will get ready for the next request. Hope that makes sense now?

5 hours ago, kongondo said:

Just curious about this. Was your goal to give the current editor feedback or all editors who use the system? I mean,  did you want the progress to show on different machines simultaneously on any ProcessWire backend page or just the current editor? I have a feeling this won't make sense so will just add. If I am Editor A and I am currently making edits. Across me sits Editor B. Editor B is not making edits. Maybe she is looking at the page tree. If Editor A makes changes and SSE kicks in, Editor B will see the progress on her machine. Was this your use case?

I'm not sure I understand. My usecase was simply trashing multiple pages or updating multiple pages via RockGrid (bulk editing). The user selects multiple rows and then clicks on an action (trash, update, ...). The user is then asked to confirm the action, which triggers the process of updating/trashing pages on the server. Now the only difference to a traditional approach is that this request is kept open using SSE and it keeps waiting for the server to receive the progress (eg trashing page 1 of 10, 2 of 10, 3 of 10... etc). Until this action is finished the system will be blocked for other requests of the same user (I don't understand how that works technically, but I think that other users can still request data from the server in the meantime?!). When the server is done with updating/trashing all pages it will send "DONE" to the client and the connection will be closed which makes the server ready for the next request.

So it's a very special use case but it's also a quite common need! The example you are describing can not be solved using SSE as far as I understand. I think one would need sockets here. But I don't have any knowledge about them so I can't say anything here but I am happy to hear how that would work even if that is not really on topic any more ? 

[kongondo]

On 3/18/2022 at 2:01 PM, bernhard said:

So it's a very special use case but it's also a quite common need! The example you are describing can not be solved using SSE as far as I understand. I think one would need sockets here. But I don't have any knowledge about them so I can't say anything here but I am happy to hear how that would work even if that is not really on topic any more

Thanks for the detailed explanation.

@3fingers, I have come up with an alternative solution to your original question. It does not use SSE but uses htmx and Local Storage. It is detailed and demonstrated in the topic below:

 

[horst]

Hey all.

I have read a bit through all the posts here, got inspired and have changed my personal little local tool for live reloading from using ajax polling to use SSE. If you like, I can share my experiences and thoughts to some of the (somewhere) above mentioned problems.

My setup is manually enabling / disabling SSE via a button.

The SSE-php file does NOT use PWs system.
It is complet solely, means it does not use (and maybe block) a users PW session. 
It uses the basic examples from MDN, as the JS do too. No special buffering, string_padding or the like is needed!
I use h2 (HTTP2) protocol, what also needs https to run. (I'm not sure, but I think h2 is needed for not early disconnecting (and then start polling)).
My polling loop on the server is every second. I send one keep-alive ping around every 15 to 20 seconds.
In the one-second-server-loop I check if something has changed in the GIT tree, in comparison to the previous run. If so, I do an ajax call to PW, where my styles and scripts pre- and post processors are hosted (in a module). (It is similar then included in ProCache, but was coded earlier then that in PC.) They do a in-depth check and do recompile necessary files on demand and return if refresh is needed or not, what, only if true, gets pushed through to the client browser.

I can work in the admin, I can work in the client, also with concurrent fetch/ajax calls while SSE is listening, etc.

My PHP file is:

Spoiler

<?php
/*==============================================================================
    hnDevTools-SSE-LiveReloader.php
    2022-04-04
==============================================================================*/


/**
* This querys the GIT tree for modified files,
* stores their last modified timestamps into array $old
* and returns if there are changes since the last query.
*
* @param mixed $repoPath optional, defaults to root dir
* @return bool if git files have changed since the last query
*/
function gitHasChanged(&$old, $repoPath = null) {
    if(null === $repoPath) $repoPath = __DIR__ . '/';
    $gitRead = shell_exec('cd \ && cd ' . $repoPath . ' && git ls-files -m 2>&1');
    $gitRead = str_replace(["\r\n", "\r"], "\n", trim($gitRead));
    $gitFiles = explode("\n", $gitRead);
    $new = [];
    $changes = false;
    foreach($gitFiles as $f) {
        $ts = filemtime($repoPath . $f);
        if(!isset($old[$f]) || $old[$f] < $ts) $changes = true;
        $new[$f] = $ts;
    }
    $old = $new;
    return $changes;
}


/**
* Querys my module HnSiteMinifyCoder::devRefresh
* and returns it output: no | refreshNeeded
*
* @param string $forWhat [css|js|all]
* @return string  [no|refreshNeeded]
*/
function getAjaxReturn($forWhat = 'css') {
    $context = stream_context_create([
        'http' => [
            'method' => 'GET',
            'header' => "X-Requested-With: XMLHttpRequest\r\n"
        ]
    ]);
    return file_get_contents('https://'. $_SERVER['HTTP_HOST'] .'/ajax/?action=devRefresh&param=' . $forWhat, false, $context);
}





$old = [];

date_default_timezone_set('Europe/Berlin');
header('Cache-Control: no-store');
header('Content-Type: text/event-stream');

while(true) {
    // loop interval is 1 second
    sleep(1);

    // Break the loop if the client aborted the connection (closed the page)
    if(connection_aborted()) exit(0);


    // create / update a counter
    $counter = !isset($counter) || 15 <= $counter ? 0 : $counter;
    $counter++;

    // send a keep-alive ping every 15 seconds
    if(9 === $counter) {
        echo "event: keep-alive\n";
        echo 'data: {"time": "' . date(DATE_ISO8601) . '"}';
        echo "\n\n";
    }

    // check the GIT TREE and only send a message when a reload is necessary
    if(gitHasChanged($old)) {
        // invoke a SCSS and JS check with HnSiteMinifyCoder
        if('refreshNeeded' == getAjaxReturn('css')) {
            // push a reload message to the client(s)
            echo "event: live-reload-css\n";
            echo 'data: {"time": "' . date(DATE_ISO8601) . '"}';
            echo "\n\n";
        }
        if('refreshNeeded' == getAjaxReturn('js')) {
            // push a reload message to the client(s)
            echo "event: live-reload-js\n";
            echo 'data: {"time": "' . date(DATE_ISO8601) . '"}';
            echo "\n\n";
        }
    }

    ob_end_flush();
    flush();
}

 

My JS in browser is:

Spoiler

    // start connection
    evtSource = new EventSource('/hnDevTools-SSE-LiveReloader.php');
    console.log('SSE started: ' + evtSource.readyState);

    // default event handler
    evtSource.onerror = function(err) {
        console.error('EventSource failed: ', err);
    };
    evtSource.onmessage = function(event) {
        console.log(event.data);
    }

    // KEEP-ALIVE PINGS
    evtSource.addEventListener('keep-alive', function(event) {
        console.log('SSE keep-alive: ' + JSON.parse(event.data).time);
    });

    // CSS source file(s) has/have changed
    evtSource.addEventListener('live-reload-css', function(event) {
        // we have to reload all CSS files
        var links = document.getElementsByTagName('link');
        for(var link in links) {
            if(links[link].rel === "stylesheet") {
                links[link].href += "";
            }
        }
        console.log('SSE live-reload-css: ' + JSON.parse(event.data).time);
    });

    // JS source file(s) has/have changed
    evtSource.addEventListener('live-reload-js', function(event) {
        // we have to reload the page, so that all JS files can be loaded by detected conditions and in correct order
        console.log('SSE live-reload-js: ' + JSON.parse(event.data).time);
        evtSource.close();
        location.reload(true); 
        // the current states of my devTool button(s) are stored in cookie, 
        // so after auto reload, it enables itself again!
    });

 

Here is a screen, showing the network tab and console:

Edited April 5, 2022 by horst
fixed typos & added the screen image from extern, as it was not able to get uploaded into the boards editor :-(

[bernhard]

Cool ?  Thx for sharing that @horst this sounds exciting!

18 hours ago, horst said:

The SSE-php file does NOT use PWs system.
It is complet solely, means it does not use (and maybe block) a users PW session. 

How does that work?

[horst]

1 hour ago, bernhard said:

How does that work?

The file source is included in my post.

As I primary need to observe / check the GIT tree, what can be done without PW, I only connect to PW on demand from the PHP file. In my case the most of the time the SSE-PHP file is doing tasks outside PW. Check git tree, send keep-alive pings, etc. I think this is essentiell for not blocking a PW user account. Or, at least, early release of the PW session in the SSE PHP file.

[bernhard]

16 hours ago, horst said:

The file source is included in my post.

As I primary need to observe / check the GIT tree, what can be done without PW, I only connect to PW on demand from the PHP file. In my case the most of the time the SSE-PHP file is doing tasks outside PW. Check git tree, send keep-alive pings, etc. I think this is essentiell for not blocking a PW user account. Or, at least, early release of the PW session in the SSE PHP file.

I saw the code, but I in the way that I understood the whole problem this should not work, that's why I asked how that worked. How do you execute your hnDevTools-SSE-LiveReloader.php ? From the command line?

[horst]

3 hours ago, bernhard said:

How do you execute your hnDevTools-SSE-LiveReloader.php ?

No, it is called as server URL in the JS, with the initializing button click. ?!?

And it runs as a simple PHP file in the apache server?!?

I don't get really where you stuck. ? ?

[horst]

@bernhard Are you talking about apache server timeouts for php script requests?

If so, you must be able to set_time_limit(##) as recuring call, maybe whenever the keep-alive-ping also is sent. (every 15 seconds for me)

Edited April 7, 2022 by horst

[dotnetic]

On 3/15/2022 at 12:23 PM, bernhard said:

So what about the error you previously posted?

The module wasn't working because the home template had a setting of "allow url segments", so the URL hook was not working. Without the setting it works fine.