Leaderboard

Over the course of 2021, I re-developed the Transferware Collectors Club Database of Patterns and Sources. It is a private, members only database that contains nearly 20,000 patterns of various types of transferware, which in layman's terms is basically art on plates, bowls, cups, etc. (Wikipedia) Many archaeologists and transferware enthusiasts use this database to learn more about the historical nature of their collections or archaeological finds. The system had a long history and there were two versions prior. I believe the first iteration was made with a PHP framework of some sort, then it was re-developed around 2015 in ProcessWire. One special aspect of the previous version was that it utilized Elastic Search. Advanced search capability of the system, both in terms of free-form text entry as well as filters were the crux of the system. In 2020, I was introduced to this project as the previous developer decided to retire. While doing some various maintenance type work, I realized that there was much that could be done to improve the system, from re-imagining the user interface to utilizing all the latest features in the newer versions of ProcessWire, as it had been locked to version 3.0.135 or something at least 3-4 years old. Also, Elastic Search had become deeply outdated and unreliable since the entire site had been moved to a new server. Finally, I saw various aspects of the data model that could be improved as well as reduce several dependencies, while also using new techniques like HTMX. I redeveloped the system over the course of 2021 and the end result has greatly impacted the capabilities of the system as well as its ability to be maintained with little complication years into the future. Search: First, it became apparent that the way in which Elastic Search was being used had little benefit, became unstable and was outdated in the time since it was implemented. I don't like relying on 3rd party solutions, especially for something so critical and I thought about how I could develop a great search experience with ProcessWire alone. To do this, I utilized the WireWordTools module to handle lemmatization and such. Then, I advocated for this feature to be developed into the core, which could allow the results from multiple search queries to be easily stacked one after another and not affect order or pagination. This was a key change allowed for "ranked" search results. Finally, I integrated the BigHugeThesaurus API so that entered search terms can have their synonyms automatically generated. Filters: The previous version of the system had filtering capability, but it was too cumbersome and in my opinion, just didn't feel right. A significant amount of thought went into how to improve it while untangling some issues with the data model (and just understanding what Transferware is in the first place). The end result has has been an extensive set of filters with a special user interface so that users can easily select what they need. Frontend: I take the "classic", server-rendered approach when it comes to web-apps, and during this time, HTMX started becoming a favorite among such developers. By using HTMX, searches can be conducted without having to hit the submit button every time. It gives it that polished ajax-y feel without having to write any JavaScript whatsoever and allowing the use of ProcessWire in the expected way as opposed to it acting as some API end-point. To be able to do this is somewhat of a game-changer and I feel like I'm not "missing out" on these features which in the past were a pain to have to invent as there really wasn't a clean, straight-forward way to do it. HTMX is a gem and it couples very nicely with ProcessWire. I'm also using Alpine.js lightly for toggling things here and there. My goal was to make this project jQuery-free. UIkit 3 is being used as the frontend CSS framework. User Authentication: As mentioned, the database must be accessed after a membership has been purchased. TCC already has a 3rd party membership system, so I developed a handshake that allows a user to log in with the ProcessWire admin form using their credentials from that 3rd party system. If it's the member's first time logging in, it will then automatically create a ProcessWire account for the member so they can bookmark patterns, save searches, etc. The previous version had a different type of handshake that ultimately only logged the users into one master "member" account, so users wouldn't be able to save any information since everyone technically shared the same account. New Server: I put the site on its own dedicated droplet to squeeze out even more performance and be independent of their main marketing site (based on Drupal). We had a couple instances of configuration changes causing some clashes, so this move isolated it from any potential breaking changes that may occur on the main site. A 2-core droplet is all that is needed for now, which is cost effective. Demo Video and Article: While the site can't be visited given it requires an account, I did create an in-depth search tutorial video which shows off quite a bit which I can post here since it's publicly accessible: https://player.vimeo.com/video/666912411?h=30a94132db Feel free to read more about the TCC DB in this interview with Dr. Loren Zeller, who I worked closely with the entire time and who provided invaluable feedback and guidance along the way: https://www.transferwarecollectorsclub.org/sites/default/files/pdf/special-interest-pdf/0225_d_001_analink.pdf More details on my personal website: https://jonathanlahijani.com/projects/transferware-collectors-club/

I don't remember the last time I laughed so hard ?.

I've created a PR for this issue https://github.com/processwire/processwire/pull/219

Here is a little breakdown on what I've done to achieve what I needed. Basically I wanted to have the ability to change page content without browser refresh, using a simple page reference field placed on top of the page tree, allowing the selection of a page (and fields) I want to use to replace the content with. I've adopted the "kinda new" url hooks, in orded to have and end-point to make my request and have the content I wanted as response: <?php namespace ProcessWire; $wire->addHook('/emit', function ($event) { header('Content-Type: text/event-stream'); header('Cache-Control: no-cache'); $data = []; // page_selector is a page reference field $contributor = $event->pages->get('/')->page_selector->title; // get the current data for the page I have selected $data['current'] = $event->pages->getRaw("title=$contributor, field=id|title|text|image|finished, entities=1"); // other data I wanted to retrieve foreach($event->pages->findRaw("template=basic-page") as $key=>$value) { $data['contributors'][$key] = $value; } $result = json_encode($data); // This is the payload sent to the client, it has to be formatted correctly. // More on this here: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format echo "event: ping" . PHP_EOL; // The event name could be whatever, "ping" in this case. echo "data: " . $result . PHP_EOL; echo PHP_EOL; ob_end_flush(); flush(); }); Doing so we now have a url to point our SSE client. Let's see how it looks like: Vue.createApp({ data() { return { // This propertied are filled by resolveData() method below. fragmentShow: false, finished: '', id: '', title: '', text: '', image: '', imageDescription: '', contributors: '', } }, methods: { // imageUrl(), basePath() and idClass are helper functions. imageUrl() { return `http://localhost/sse_vue/site/assets/files/${this.id}/${this.image}`; }, basePath() { return `http://localhost/sse_vue/site/assets/files/`; }, idClass() { return `id-${this.id}`; }, // Retrieve data from the incoming stream getData(event) { return new Promise((resolve, reject) => { if (event.data) { let result = JSON.parse(event.data); // If the incoming page id is different than the current one, hide the container. if (this.id != result.current.id) { this.fragmentShow = false; } // This allows the <Transition> vue component to complete its animation before resolving the result. setTimeout(() => { resolve(result); }, 300) } }).then((result) => { this.resolveData(result); }); }, resolveData(result) { // Once the new values has come show the page again this.fragmentShow = true; // Set incoming values to vue reactive data() object this.finished = result.current.finished, this.id = result.current.id, this.title = result.current.title, this.text = result.current.text, this.image = result.current.image[0].data, this.imageDescription = result.current.image[0].description, this.contributors = result.contributors }, }, mounted: function() { // Init SSE and listen to php page emitter let source = new EventSource('/sse_vue/emit/'); source.addEventListener('ping', (event) => { // Get the incoming data this.getData(event); }); } }).mount('#app') On mounted vue lifecycle hook I start listening to the incoming stream and place useful informations inside the reactive data() object properties. Then I populated the html with those properties: <div id="app" class="container"> <Transition> <div :class="idClass()" class="bg-slate-200" v-show="fragmentShow"> <h1>{{title}}</h1> <p v-html="text"></p> <img :src="imageUrl()" :alt="imageDescription"> <div v-for="(contributor, index) in contributors"> <p :class="contributor.finished == 1 ? 'finished' : ''">{{contributor.title}}</p> <div v-if="contributor.document"> <a :href="basePath() + contributor.id + '/' + contributor.document[0].data">Download document</a> </div> </div> </div> </Transition> </div> Attached the video of the result (please don't look at the styling of it, it sucks). sse.mp4

Hello, I would really appreciate your comments on this blog post I am preparing for our site. =========== Creating a dynamic search with very little code in ProcessWire is easy. This search cannot compete with engines like Elasticsearch, Solr, and others. However, it is suitable for most "showcase" sites. Here is how we did it on the Spiria site using the small library htmx and its companion hyperscript. The goal The recipe Inclusion of libraries htmx and hyperscript (the latter is optional). A field of type textarea integrated to the page models that we want to index. A code for indexing the existing content in the file `ready.php` A search controller which we name here `api.php`. This controller will also be a page with the `api` template. A form placed in the pages requiring the search. Content indexing Before we can program, we need to index the content on which we want to apply our search. In my proof of concept, I have developed two strategies. This is probably overrated, because I am not sure of the gain in speed. Index for a single term search Indexing for a multiple term search To do this, we need to introduce two fields in each model where we want an indexation. The `search_text` field which will contain only one occurrence of each word on a page. The `search_text_long` field which will preserve all sentences without HTML tags. We place a hook in the `ready.php` page in this way: <?php namespace ProcessWire; pages()->addHookAfter("saveReady", function (HookEvent $event) { $p = $event->arguments[0]; switch ($p->template->name) { case "blog_article": $french = languages()->get('fr'); $english = languages()->get('default'); $txt_en = $p->page_content->getLanguageValue($english) . ' ' . $p->blog_summary->getLanguageValue($english); $txt_fr = $p->page_content->getLanguageValue($french) . ' ' . $p->blog_summary->getLanguageValue($french); $title_en = $p->title->getLanguageValue($english); $title_fr = $p->title->getLanguageValue($french); $resultEn = stripText($txt_en, $title_en); $resultFr = stripText($txt_fr, $title_fr); $p->setLanguageValue($english, "search_text", $resultEn[0]); $p->setLanguageValue($english, "search_text_long", $resultEn[1]); $p->setLanguageValue($french, "search_text", $resultFr[0]); $p->setLanguageValue($french, "search_text_long", $resultFr[1]); break; } }); And function stripText($t, $s) { $resultText = []; $t = strip_tags($t); $t .= " " . $s; $t = str_replace("\n", " ", $t); $t = str_replace(",", "", $t); $t = str_replace("“", "", $t); $t = str_replace("”", "", $t); $t = str_replace("'", "", $t); $t = str_replace("?", "", $t); $t = str_replace("!", "", $t); $t = str_replace(":", "", $t); $t = str_replace("«", "", $t); $t = str_replace("»", "", $t); $t = str_replace(",", "", $t); $t = str_replace(".", "", $t); $t = str_replace("l’", "", $t); $t = str_replace("d’", "", $t); $t = str_replace("&nbsp;", "", $t); $t = preg_replace('/\[\[.*\]\]/', '', $t); //$t = preg_replace('/\?|\[\[.*\]\]|“|”|«|»|\.|!|\&nbsp;|l’|d’|s’/','',$t); $arrayText = explode(" ", $t); foreach ($arrayText as $item) { if (strlen(trim($item)) > 3 && !in_array($item, $resultText)) { $resultText[] = $item; } } return [implode(" ", $resultText), $t]; } If you have the ListerPro module, it becomes easy to save all the pages to be indexed in batch and any new page will then be indexed as it is created. The `stripText()` function aims at cleaning up the text as we want. Note that, in my example, I distinguish between French and English. This little algorithm is entirely perfectible! I have commented a shorter way to clean up the text, but at the expense of comprehension. As mentioned before, it is probably unnecessary to create two search fields. The most important thing would be to optimize the text as much as possible, as many small words are useless. The current code restricts to words longer than three characters, which is tricky in a computing context like our site where names like `C#`, `C++`, `PHP` compete with `the`, `for`, `not`, etc. That said, perhaps this optimization is superfluous in the context of a simple content and not and limited in number. So now see the process and the research code. The structure structure.svg This scheme is classical and needs few explanations. The `htmx` library allows a simple Ajax call. The form code01.svg The form has a `get` method that returns to a conventional search page when the user presses the `enter` key. A hidden field with the secret key generated on the fly enhances security. The third field is the `input` of the dynamic search. It has a `htmx` syntax. The first command, `hx-post`, indicates the method of sending the data to the API. Here, it is a `post`. `htmx` allows to handle events on any DOM element. So we could have several calls on different elements of a form. The second line indicates where the API response will be sent when received, which is `div#searchResult` below the form. The `hx-trigger` command describes the context of sending to the API. Here, when the user releases a key, with a delay of 200 ms between each reading of the event. The `hx-indicator` command is optional. It tells the user that something is happening. In the example, the `#indexsearch` image is displayed. This is automatically handled by htmx. We have the possibility to pass other parameters to the search with the `hx-vals` command. The given example is simplified. We send the search language. The last command comes from the `hyperscript` syntax. It indicates that if you click anywhere outside the search field, you will make the contents of `#found` disappear, which will be described below. It is clear from this example that no javascript is called, except the [htmx] and [hyperscript] libraries. It is worth visiting the web site of these two libraries to understand their philosophy and possibilities. The Search API The API resides in a normal ProcessWire page. Although it is published, it remains "hidden" from CMS searches. This page allows requests to be answered and the correct functions to be called. Several requests to the CMS can be gathered in this type of page. <?php namespace ProcessWire; $secretsearch = session()->get('secretToken'); $request = input()->post(); $lang = sanitizer()->text($request["lang"]); if (isset($request['CSRFTokenBlog'])) { if (hash_equals($secretsearch, $request['CSRFTokenBlog'])) { if (!empty($request["search"])) { echo page()->querySite(sanitizer()->text($request["search"]),$lang); } } else { echo __("A problem occurred. We are sorry of the inconvenience."); } } exit; In this case : We extract the secret token of the session, token that will have been created in the search form page. We then process everything that is in the `post` query. Note that this is a simplified example. We compare the token with the one received in the query. If all goes well, we call the SQL query. Our example uses a class residing in `site/classes/ApiPage.php`; it can therefore be called directly by `page()`. Any other strategy is valid. The following code represents the core of the process. <?php namespace ProcessWire; public function querySite($q, $l) { $this->search = ""; $this->lang = $l == 'en' ? 'default' : 'fr'; user()->setLanguage($this->lang); $whatQuery = explode(" ", $q); $this->count = count($whatQuery); if ($this->count > 1) { $this->search = 'template=blog_article,has_parent!=1099,search_text_long~|*= "' . $q . '",sort=-created'; } elseif (strlen($q) > 1) { $this->search = 'template=blog_article,has_parent!=1099,search_text*=' . $q . ',sort=-created'; } if ($this->search !== "") { $this->result = pages()->find($this->search); return $this->formatResult(); } return ""; } protected function formatResult() { $html = '<ul id="found">'; if (count($this->result) > 0) { foreach ($this->result as $result) { $html .= '<li><a href="' . $result->url . '">' . $result->title . '</a></li>'; } } else { $html .= __('Nothing found'); } $html .= '</ul></div>'; return $html; } The `formatResult()` function is simple to understand and it is here that we see the `ul#found` tag appear which, remember, is deleted by the _hyperscript_ line of the form. _="on click from elsewhere remove #found" Actually, I've commented out this line in the production code so far. Instead of clicking anywhere, it would be better to add a background behind the form when doing a search in order to focus the `click` event. But again, all strategies are good! In the current code, it is not necessary to add CSS to display the result. It is placed in an empty `#searchResult` tag, so it is invisible at first. As soon as it is filled by the search result, everything becomes accessible, the CSS being targeted on the `ul#found` list and not on its parent. Conclusion The purpose of this article was to demonstrate htmx and the possibilities that this library allows. There is already an excellent search module for ProcessWire, SearchEngine, which can coexist very well with the code described here.

Hi @3fingers, Thanks for the example. Could you confirm that this is indeed SSE-driven as opposed to polling? For instance, if you console.log like this: getData(event) { return new Promise((resolve, reject) => { console.log(event.data) if (event.data) { let result = JSON.parse(event.data) // If the incoming page id is different than the current one, hide the container. // if (this.id != result.current.id) { // this.fragmentShow = false // } // This allows the <Transition> vue component to complete its animation before resolving the result. setTimeout(() => { resolve(result) }, 300) } }).then((result) => { this.resolveData(result) }) }, do you see a constant stream (in the console log) or do only see events pushed after you save your ProcessWire page? Thanks.

The TCC site actually uses aMember Pro which is what I integrated with (the handshake). I didn't set up aMember Pro originally, but I believe TCC likes the system, although there has been some concern with the recent Russia/Ukraine situation as there are developers from both countries that are involved in the project (and could lead to potential support issues should they arise given the displacement of many people). Creating the handshake was straight-forward and didn't involve too much code. I could give you some tips as to how I did it should you go that route, but in short, when a member logs in via ProcessWire's admin form, a hook it hits a script on the main site (via a get request) that bootstraps aMember's system and runs a check on the credentials entered.

@rick - did you try Robin's version: https://processwire.com/talk/topic/19913-solved-page-reference-field-unpublished-pages-not-visible-by-non-superuser-rolesusers/?do=findComment&comment=172578 That's what I've been using for a long time now.

In most cases when I clone an item (no matter if it's a repeater or a repeater matrix) i'd like to have the "copy" just below the "original" instead at the bottom of the repeater(matrix) field… Especially when there are many items, I have to drag around the cloned item to put it in the right place. I guess there should be just one checkbox in the specific repeater(matrix) field like: "Put cloned items just after the original item instead at the bottom." This way everyone could set up the field as he likes. This would expand ProcessWires flexibility. A different approach could be having 2 cloning icons at every item: one for "clone below", and one for "clone at bottom". This would be even more versatile.

Hello @gerritvanaaken, there are new buttons in Repeater items for cloning before or after an item: https://processwire.com/blog/posts/new-repeater-and-repeater-matrix-features/ Also: Regards, Andreas

It looks like it's just DO that has poor performance on their managed DBs. I presume you've seen this thread - https://www.digitalocean.com/community/questions/managed-db-become-very-very-slow ? As a final test, you could match the DB machine size to the current server and try again, but I have a feeling it might not help much and there's a wider problem. If separating out your DB is the goal, perhaps running it on a separate VM might be more than enough.

@fruid I think the fields is named "seo_title"?? So it would be $page->seo_title = $page->academicrank.' '.$page->givenname.' '.$page->lastname;

Yes, you can use InputfieldTextTags https://processwire.com/blog/posts/pw-3.0.177/ But it would be a lot easier to use HannaCode or a custom Textformatter module which replaces [file=foo-bar.php] with the content of the file foo-bar.php

In the RepeaterMatrix field create two repeater matrix item types: cta1 & cta2. They don't even have to have any fields, i guess. In the template file that should output the repeater items you can output them like this: <?php foreach($page->repeater_matrix_field as $item){ if ($item->type == "cta1"){ echo wireRenderFile("Call-to-Action01.php"); } if ($item->type == "cta2"){ echo wireRenderFile("Call-to-Action02.php"); } } ?> RepeaterMatrix is a pro module, though. Not Core.

Meanwhile i found the way to deal with my approach for translations. Possibly it helps others, so i post it here. Tl;dr: I have a function to enable translation of static template strings. This allows me to reference a _strings.php file as site translation file in the PW Backend. To apply this to the SearchEngine Module, you have to edit some configuration settings before rendering. $searchEngine = $modules->get('SearchEngine'); $config->SearchEngine = [ 'render_args' => [ 'strings' => [ 'form_label' => _t('Suche', 'search'), 'form_input_placeholder' => _t('Ihr Suchbegriff...', 'search'), 'form_submit' => _t('Suche', 'search'), 'results_heading' => _t('Suchergebnisse', 'search'), 'results_summary_one' => _t('Ein Ergebnis für "%s":', 'search'), 'results_summary_many' => _t('%2$d Ergebnisse für "%1$s":', 'search'), 'results_summary_none' => _t('Keine Ergebnisse für "%s".', 'search'), 'errors_heading' => _t('Hm, ihre Suchanfrage konnte nicht ausgeführt werden', 'search'), 'error_query_missing' => _t('Bitte tragen Sie einen Suchbegriff in das Eingabefeld ein', 'search'), 'error_query_too_short' => _t('Bitte verwenden sie mindestens %d Zeichen für den Suchbegriff.', 'search'), ] ] ] <?= $searchEngine->renderForm(); ?> <?= $searchEngine->renderResults(); ?> === Approach to translate: _func.php /** * This function enables the translation of static template strings. * All translations are done in `_strings.php`. More info within the * comments of `_strings.php`. * * @param string $text text to translate * @param string $context Context to allow doubles * @param string $textdomain Point static textdomain * @return function Function for gettext parser */ function _t($text, $context = 'Generic', $textdomain = '/site/templates/_strings.php') { return _x($text, $context, $textdomain); } _strings.php /** * Static translated template strings to use project wide * in any templates. This spares us defining page fields * for every piece of code that doesn't have to be editable * within the admin interface. The function doesn't need to * execute with PHP, so we wrap it in a comment leaving * the code only visible to the language parser. * * Relies on function `_t` (s. `_func.php`) * * Usage: * 1. Define string in markup, e.g. "_t('<string>', '<context>')" * 2. Insert string in `_strings.php`, e.g. "_x('<string>', '<context>')"; * both without the surrounding double quotes. This is just to avoid * this comment is parsed. * 3. Choose the language you like to translate the default language strings * 4. Under (possibly still empty) "Site Translation Files" click on "Translate File" to get * a list of translatable files. Choose the `_strings.php` and click send/save. You'll * see the translatable string phrases. */ /*! _x('Suche', 'search') _x('Ihr Suchbegriff...', 'search') _x('Suchergebnisse', 'search') _x('Ein Ergebnis für "%s":', 'search') _x('%2$d Ergebnisse für "%1$s":', 'search') _x('Keine Ergebnisse für "%s".', 'search') _x('Bitte tragen Sie einen Suchbegriff in das Eingabefeld ein.', 'search') _x('Bitte verwenden sie mindestens %d Zeichen für den Suchbegriff.', 'search') */

If anyone is still interested about this topic I've gracefully solved sse implementation both with alpine.js and vue.js (just for fun). Let me know if you want some code examples and I'll post them here.

I just released a new extension module AppApiPage (waits for approval), which handles the initial steps from my post above completely automatic. You install AppApi and AppApiPage. That makes the /api/page route available and you only have to add the code on top of your template php to add a custom JSON output to your pages. <?php // Check if AppApi is available: if (wire('modules')->isInstalled('AppApi')) { $module = $this->wire('modules')->get('AppApi'); // Check if page was called via AppApi if($module->isApiCall()){ // Output id & name of current page $output = [ 'id' => wire('page')->id, 'name' => wire('page')->name ]; // sendResponse will automatically convert $output to a JSON-string: AppApi::sendResponse(200, $output); } } // Here continue with your HTML-output logic... I hope that this makes it even simpler to add a full-blown JSON api to new and existing pages.

Regarding multi-language fields such as TextLanguage, TextareaLanguage, and PageTitleLanguage: When multi-language fields are shown in Page Edit mode, I often use them next to non-multi-language fields. That makes the page appear a bit disorganized and cluttered because multi-language fields vs non-multi-language fields don't align horizontally. The reason is that multi-language field have "language switcher tabs" that forces the field to "jump down" whereas normal fields stay in the normal position. See screenshot 1. It makes me sea sick! ? Suggested solution is to move the "language switcher tabs" up so that it aligns with the field label. See screenshot 2.

Looks like you’re using @Wanze’s module Pages2Pdf. Unfortunately that module is pretty outdated and not compatible with PHP 8. I recommend switching to @bernhard’s RockPdf: $pdf = $modules->get('RockPdf'); $pdf->settings([ 'mode' => 'utf-8', 'format' => [210, 297], 'margin_top' => 20, 'margin_bottom' => 20, 'margin_left' => 20, 'margin_right' => 20, ]); $pdf->write("<h1>{$page->title}</h1>"); $pdf->write($page->body); $pdf->download($page->name . '.pdf'); It comes with handy output methods:

Just wanted to share the simplest language switcher possible if you only have two languages: <a href="<?= $page->localUrl($languages->findOther()->first()) ?>">DE/EN</a> ??