Htmx, Sse and real time dom swap help.

[3fingers]

Hello all!
A client asked us to implement a landing page where some content would have to swap in real time when needed.
Basically I need to implement a way to select in the backend (via a select dropdown or whatever) which page I want to use as the "replacer", save and then immediately use its html content to swap the old content, in real-time, on the front-end.
I've read about htmx and its sse implementation, and also took a look at this simple example on how to dispach an event via php every second.
The front-end part looks way easier to implement, but the backend one it's over my knowledge at the time of this writing.

I know @netcarver did some test and also @kongondo is playing with those technology a bit lately.

This is the first time I came to such a request and I'm looking for some advice from you guys! :)

[elabx]

I think @kongondo gives here a super simple example and even refers to it as trivial, haven't tried anything yet tho:

 

[netcarver]

On 1/13/2022 at 9:13 PM, 3fingers said:

Basically I need to implement a way to select in the backend (via a select dropdown or whatever) which page I want to use as the "replacer", save and then immediately use its html content to swap the old content, in real-time, on the front-end.

Hi @3fingers

Right, so the demo video I posted did this using a few separate parts. An event generator wrote messages into a Redis pubsub channel, and I had a long-running ReactPHP process subscribed to the channel which would then take those events from Redis and send them via SSE to any connected listeners. A little custom JS in the website's home page would connect via SSE to the server and would be asynchronously invoked when an SSE message came in.  The JS was really simple, it parsed the incoming SSE message to grab an html id and the new content for that part of the DOM and then do the change (along with a little CSS styling to highlight the swap.)

The most difficult part was running the ReactPHP process.

In your case, you could replace the back end event generator with a hook and just publish a change when the PW admin control is changed. Anyone connected would then have their screen updated immediately. PM me if you want to discuss directly.

[3fingers]

If anyone is still interested about this topic I've gracefully solved sse implementation both with alpine.js and vue.js (just for fun).
Let me know if you want some code examples and I'll post them here.

[Andy]

@3fingers

It would be nice if you could share your invention.

[3fingers]

Here is a little breakdown on what I've done to achieve what I needed.

Basically I wanted to have the ability to change page content without browser refresh, using a simple page reference field placed on top of the page tree, allowing the selection of a page (and fields) I want to use to replace the content with.

I've adopted the "kinda new" url hooks, in orded to have and end-point to make my request and have the content I wanted as response:

<?php

namespace ProcessWire;

$wire->addHook('/emit', function ($event) {
    header('Content-Type: text/event-stream');
    header('Cache-Control: no-cache');
  
    $data = [];
  
   	// page_selector is a page reference field
    $contributor = $event->pages->get('/')->page_selector->title;
  
  	// get the current data for the page I have selected
    $data['current'] = $event->pages->getRaw("title=$contributor, field=id|title|text|image|finished, entities=1");
  
  	// other data I wanted to retrieve
    foreach($event->pages->findRaw("template=basic-page") as $key=>$value) {
        $data['contributors'][$key] = $value;
    }
  
    $result = json_encode($data);
  
  	// This is the payload sent to the client, it has to be formatted correctly.
  	// More on this here: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format
  
    echo "event: ping" . PHP_EOL; // The event name could be whatever, "ping" in this case.
    echo "data: " . $result . PHP_EOL;
    echo PHP_EOL;
    ob_end_flush();
    flush();
});

Doing so we now have a url to point our SSE client.

Let's see how it looks like:

	Vue.createApp({
        data() {
            return {
              // This propertied are filled by resolveData() method below.
                fragmentShow: false,
                finished: '',
                id: '',
                title: '',
                text: '',
                image: '',
                imageDescription: '',
                contributors: '',
            }
        },
        methods: {
          // imageUrl(), basePath() and idClass are helper functions.
            imageUrl() {
                return `http://localhost/sse_vue/site/assets/files/${this.id}/${this.image}`;
            },
            basePath() {
                return `http://localhost/sse_vue/site/assets/files/`;
            },
            idClass() {
                return `id-${this.id}`;
            },
          // Retrieve data from the incoming stream
            getData(event) {
                return new Promise((resolve, reject) => {
                    if (event.data) {
                        let result = JSON.parse(event.data);
                      // If the incoming page id is different than the current one, hide the container.
                        if (this.id != result.current.id) {
                            this.fragmentShow = false;
                        }
                      // This allows the <Transition> vue component to complete its animation before resolving the result.
                        setTimeout(() => {
                            resolve(result);
                        }, 300)
                    }
                }).then((result) => {
                    this.resolveData(result);
                });
            },
            resolveData(result) {
              // Once the new values has come show the page again
                this.fragmentShow = true;
              // Set incoming values to vue reactive data() object
                this.finished = result.current.finished,
                    this.id = result.current.id,
                    this.title = result.current.title,
                    this.text = result.current.text,
                    this.image = result.current.image[0].data,
                    this.imageDescription = result.current.image[0].description,
                    this.contributors = result.contributors
            },
        },
        mounted: function() {
            // Init SSE and listen to php page emitter
            let source = new EventSource('/sse_vue/emit/');
            source.addEventListener('ping', (event) => {
              	// Get the incoming data
                this.getData(event);
            });
        }
    }).mount('#app')

On mounted vue lifecycle hook I start listening to the incoming stream and place useful informations inside the reactive data() object properties.

Then I populated the html with those properties:

    <div id="app" class="container">
        <Transition>
            <div :class="idClass()" class="bg-slate-200" v-show="fragmentShow">
                <h1>{{title}}</h1>
                <p v-html="text"></p>
                <img :src="imageUrl()" :alt="imageDescription">
                <div v-for="(contributor, index) in contributors">
                    <p :class="contributor.finished == 1 ? 'finished' : ''">{{contributor.title}}</p>
                    <div v-if="contributor.document">
                        <a :href="basePath() + contributor.id + '/' + contributor.document[0].data">Download
                            document</a>
                    </div>
                </div>
            </div>
        </Transition>
    </div>

Attached the video of the result (please don't look at the styling of it, it sucks).

[Andy]

@3fingers

It's a neat solution. Thank you.

[kongondo]

On 3/8/2022 at 11:15 AM, 3fingers said:

Here is a little breakdown on what I've done to achieve what I needed.

Hi @3fingers,

Thanks for the example. Could you confirm that this is indeed SSE-driven as opposed to polling? For instance, if you console.log like this:

			getData(event) {
				return new Promise((resolve, reject) => {
					console.log(event.data)
					if (event.data) {
						let result = JSON.parse(event.data)
						// If the incoming page id is different than the current one, hide the container.
						// if (this.id != result.current.id) {
						// 	this.fragmentShow = false
						// }
						// This allows the <Transition> vue component to complete its animation before resolving the result.
						setTimeout(() => {
							resolve(result)
						}, 300)
					}
				}).then((result) => {
					this.resolveData(result)
				})
			},

do you see a constant stream (in the console log) or do only see events pushed after you save your ProcessWire page? Thanks.

Edited March 9, 2022 by kongondo

[3fingers]

3 hours ago, kongondo said:

do you see a constant stream (in the console log) or do only see events pushed after you save your ProcessWire page?

Yes, I do see event.data being streamed at regular intervals (five seconds roughly + those 300ms delay driven by setTimeout).

 

 

Of course, for a change to take place and be visible on the browser, a pw page must be saved in order to trigger a change in the data being requested by SSE.

[kongondo]

8 minutes ago, 3fingers said:

Yes, I do see event.data being streamed at regular intervals (five seconds roughly + those 300ms delay driven by setTimeout).

Thanks for confirming. I noticed the same in this w3schools example and also htmx sse. If you run the w3schools example, even just the JavaScript without the server code, you will notice JS running in a continuous loop, much as you describe above. This has left me more confused about SSE. I thought JS would just initiate first contact with the server then sit and wait for a stream. On a change on the server, JS would respond. So far, I don't see much difference between SSE and Ajax polling. I'll need to read up more about it. Any one got any thoughts on this? @netcarver? Thanks.

[3fingers]

The fact is that SSE takes the connection open IF the server is responding something, otherwise it quits and retry a connection after five seconds.
That's why it behaves like polling.
The only way I found to keep the connection opened it's by tweaking server side code, wrapping the query in a while(true) function, considering the various drawbacks and flooding it can cause very quickly.
I might be wrong but, as far as I understand, the only reliable way to keep a connection open, without any hassle, it's to use websockets (a whole different implementation however).

[bernhard]

11 hours ago, kongondo said:

Thanks for confirming. I noticed the same in this w3schools example and also htmx sse. If you run the w3schools example, even just the JavaScript without the server code, you will notice JS running in a continuous loop, much as you describe above. This has left me more confused about SSE. I thought JS would just initiate first contact with the server then sit and wait for a stream. On a change on the server, JS would respond. So far, I don't see much difference between SSE and Ajax polling. I'll need to read up more about it. Any one got any thoughts on this? @netcarver? Thanks.

I've had the same problem - the script seemed to work but sent several requests just like ajax polling:

But I've managed to get it working correctly and the result is a nice stream of data and updating feedback for the user while bulk-editing of data in RockGrid. There's no reload, no ajax-polling:

The key is to echo the correct message. I've built a method for this in my module:

  /**
   * Send SSE message to client
   * @return void
   */
  public function sse($msg) {
    echo "data: $msg\n\n";
    echo str_pad('',8186)."\n";
    flush();
  }

If you do that, you can do something like this:

<?php
header("Cache-Control: no-cache");
header("Content-Type: text/event-stream");

$i = 0;
while(true) {
  $this->sse("value of i = $i");
  if($i>30) return; // manual break after 30s for testing
  while(ob_get_level() > 0) ob_end_flush();
  if(connection_aborted()) break;
  sleep(1);
}

Not sure if the ob_get_level and ob_end_flush are necessary...

[kongondo]

41 minutes ago, bernhard said:

There's no reload, no ajax-polling:

Thanks for sharing. Could you please show us your JavaScript in this case then? For me, it seems the JavaScript is the main issue as it keeps 'polling' as you also noticed. Thanks.

[bernhard]

No, the issue is not the JS, the issue is that the data received is not properly formatted, so the client thinks the request failed and it fires a new request which is the same as if it were ajax polling. If the received data is properly formatted the stream can be read and the connection keeps open. This is my JS:

    const evtSource = new EventSource(url, { withCredentials: true } );
    evtSource.onmessage = function(event) {
      console.log(event);
      if(event.data==='DONE') {
        evtSource.close();
        if(modal) modal.hide();
        if($button.data('reload')) RockGrid2.getGrid($button).reload();
      }
      else if($progress) {
        $progress.fadeIn();
        $progress.html(event.data);
      }
    }

 

[3fingers]

Following @bernhard implementation I've modified my code like this:

    while(true) {
        echo "event: ping" . PHP_EOL;
        echo "data: $result\n\n";
        echo str_pad('',8186)."\n";
        flush();
        while(ob_get_level() > 0) ob_end_flush();
        if(connection_aborted()) break;
        sleep(1);
    }

The difference now is that, on my side, the polling (still present) is acting every 1 second.
Moreover the pw admin (and every other page) doesn't load, the browser loading indicator spins forever....

[bernhard]

2 hours ago, 3fingers said:

Moreover the pw admin (and every other page) doesn't load, the browser loading indicator spins forever....

You might have a process running then in the background... this is why I added the manual break ($i>30) in my example...

This is how a correct SSE implementation looks like:

• one single request stream of +4s
• message appear in "EventStream" tab
• console shows logs (because I'm doing console.log in JS)

[kongondo]

5 hours ago, bernhard said:

No, the issue is not the JS, the issue is that the data received is not properly formatted, so the client thinks the request failed and it fires a new request which is the same as if it were ajax polling.

Great explanation! Thanks.

[kongondo]

8 hours ago, 3fingers said:

The difference now is that, on my side, the polling (still present) is acting every 1 second.
Moreover the pw admin (and every other page) doesn't load, the browser loading indicator spins forever....

Same here. The while loop is locking up everything.

@bernhard, could you please provide more info regarding the endpoint of your url? I.e.

9 hours ago, bernhard said:

const evtSource = new EventSource(url, { withCredentials: true } );

☝️ That url is pointing to some url in ProcessWire. Is it a page URL or a virtual URL that you handle using URL Hooks? Either way could you please show us how you handle requests to that URL in ProcessWire/PHP? I realise you have the thread below as well, but it doesn't show us how you handle 'pings' to your backend either. I am guessing this is where you have your while loop but it would be good to see the whole picture. Thanks.

 

 

[bernhard]

3 hours ago, kongondo said:

Is it a page URL or a virtual URL that you handle using URL Hooks?

It's an URL hook with the code I already posted above:

14 hours ago, bernhard said:

<?php
header("Cache-Control: no-cache");
header("Content-Type: text/event-stream");

$i = 0;
while(true) {
  $this->sse("value of i = $i");
  if($i>30) return; // manual break after 30s for testing
  while(ob_get_level() > 0) ob_end_flush();
  if(connection_aborted()) break;
  sleep(1);
}

Ok I'm missing an $i++ in the example! So it will run forever... Simply add an $i++ in the loop or do while(++$i) ...

[kongondo]

7 hours ago, bernhard said:

Ok I'm missing an $i++ in the example! So it will run forever... Simply add an $i++ in the loop or do while(++$i) ...

Thanks for the info @bernhard. The remaining confusing bit for me is this:

21 hours ago, bernhard said:

if($i>30) return; // manual break after 30s for testing

Your comment suggests that the break was only for testing ?. What are you using in production? Thanks.

I feel like we are still missing something with respect to SSE or it is just the way SSE works. I'd like a situation where, for example, an event would only be pushed to the client after a saveReady fires in ProcessWire. Currently, it looks like SSE is just a glorified long polling, in some respects .

[bernhard]

https://github.com/baumrock/RockSSE-demo

Would be great if many of you could test this on your setups and let me know if it works as expected!

9 hours ago, kongondo said:

Your comment suggests that the break was only for testing ?. What are you using in production? Thanks.

In production I have a long running task like trashing lots of pages. The script will finish when all pages have been trashed.

 

[dotnetic]

I tested the module on Windows 11 with Laragon, Apache 2, PHP 8.0.12 and ProcessWire 3.0.193 and it works ?

[bernhard]

So what about the error you previously posted?

[dotnetic]

It was an installation with redirect hooks in ready.php or something like that. It worked directly in a different project.

[kongondo]

6 hours ago, dotnetic said:

hooks in ready.php

Interesting that you mention this.

@bernhard, I have been testing in ready.php and that makes everything (all pages backend and frontend on the site) hang even with the break  and $i++ present.