Search the Community

Dynamic Roles are a powerful access control tool for ProcessWire. They pick up where traditional roles leave off, and allow you to assign permissions at runtime based on any factor present with the user. Once a user receives one or more dynamic roles (at runtime), those dynamic roles then specify what pages the user can view, edit, or add children to. If traditional roles are a sledgehammer, Dynamic Roles are a scalpel, allowing nearly any finely tuned access control scenario. Traditional ProcessWire roles are limited to assignment of view/edit/add access on a per-template basis. Dynamic roles go outside those limitations and enable you to assign that access based on any factors present with a page (i.e. match any field values). Dynamic Roles assign new access, but do not revoke existing access provided by traditional roles. As a result, Dynamic Roles can be used together with traditional roles, and the two work beautifully well together. Though Dynamic Roles can also replace all situations where you would use traditional roles for access control assignments. If using Dynamic Roles to assign page-view access, you would typically want to use traditional roles to revoke view access from at least the "guest" role at the template level. Then use Dynamic Roles to assign view access to those pages in a more granular manner. This module directly affects the results of all page getting/finding operations by applying the access control directly to the database queries before pages are loaded. As a result, it is fast (regardless of scale), pagination friendly, and requires no further intervention by the developer other than configuring the dynamic roles as they see fit. Because it relies upon new features present only in ProcessWire 2.4.6+, it requires the current dev branch. Sponsored by Avoine Concept by Antti Peisa Code by Ryan Cramer PLEASE NOTE: This module is in pre-release state (like the PW dev branch it requires) and is not recommended for production use just yet. Though we do appreciate any testing and/or feedback that you are able to provide. While not required, this module benefits from ProFields Multiplier. If you have ProFields Multiplier installed before installing this module, it will make this module more powerful by making all of your access control selectors have the ability to use OR-group conditions. Depending on your access control needs, this enables you to accomplish more with fewer Dynamic Roles. How to install Make sure you are running ProcessWire 2.4.6 (dev branch) or newer. Download from GitHub (we will add this module to the Modules directory later). Place all files from this module in /site/modules/DynamicRoles/. In your admin, go to Modules > Check for new modules. Click "install" for the Dynamic Roles module (ProcessDynamicRoles). Click to Access > Dynamic Roles for the rest (see example and instructions below). Example and instructions Lets say you ran a Skyscrapers site and wanted a role enabling users with "portmanusa.com" in their email address to have edit access to skyscrapers designed by architect John Portman, with at least 40 floors, and built on-or-after 1970. Yes, this is an incredibly contrived example, but it is an example that also demonstrates the access control potential of this module. 1. In your admin, you would click to Access > Dynamic Roles. 2. Click "Add Dynamic Role". Enter a name for the dynamic role, like: "skyscraper-test-editor" and save. 3. Under "Who is in this dynamic role?" section, click "Add Field" and choose: Email => Contains Text => "portmanusa.com". This will match all users having "portmanusa.com" in their email address. 4. Under "permissions" check the boxes for: page-view and page-edit. 5. For this contrived example, we will assume the user already has view access to all skyscrapers, so we will leave the "What can they view?" section alone. 6. For the "What can they edit?" section: Click "Add Field" and choose: template => Equals => Skyscraper. Click "Add Field" and choose: architect => Equals => John Portman. Click "Add Field" and choose: floors => Greater Than Or Equal => 40. Click "Add Field" and choose: year => Greater Than Or Equal => 1970. 7. Click Save. Now users matching the conditions of your dynamic role will be able to edit the matching pages, but not any others (unless assigned by traditional roles).

Hi @Zeka, Yes, I am using $modules->get("MyModule") in my front-end template. I was just surprised to see MyModule get installed as a result of that code rather than throwing an error, which is what I was expecting. I've achieved my original goal of preventing the install in that situation but I've realised that I still need to use isInstalled() in my front-end templates to avoid runtime errors so, thanks.

First off, just increase raw power. Try bumping up your VPS with more power. Easy solution and this stuff is so cheap nowadays. Upgrade the software if possible I've had instances with 300+ fields on a template with more than 500k pages. Queries started to slow down around 200 fields. I bumped the Mariadb version to the latest, switched to INNOdb, PHP 7.2.x and it was running pretty smoother on a digitalocean medium sized VPS ($40/mo). We also had tons of runtime fields and we cached those so a page save would do the calculations instead of calculating runtime. When we eliminated those the application became really snappy with 200-300 concurrent key users doing all kinds of imports and running reports while the 1000 regular concurrent users didn't notice lots of slowdown. Most of the slowdown nowadays come from the interface itself. We used a lot of jQuery kinda tooling inside ProcessWire admin and since the users don't have the fanciest computers, some browsers are not really capable of rendering the back-end. I guess we needed to switch to a SPA, but I'm not working on the project anymore. Jup, this is the key. Especially while desiging applications this is really important. Try to split up as many as you can. I don't store address information on the order, but on the client, and possible even on an address template which you can re-use. You can also use Page fields to combine all those. To comply to GDPR you can also store the blob of address info in one or two fields on the order itself. So if a client requests to remove his data you can easily remove the accoun and associated addresses. And you still have the address data. Another solution is to use the FieldsetPage to create on instance of all address/personal information. In the background PW creates another page containing the info. You add this field to the page once for invoice and once for the shipping info.

I just had a very weird issue today... I'll try and describe it as best as I can: I have a frontend search form (text search and filters), which used to work fine before I have installed and used the RTMU module. Today the following fatal error appeared: PW basically tripped over these lines: public function getMatchQuery($query, $table, $subfield, $operator, $value) { // we don't allow this field to be queried throw new WireException("Field '{$query->field->name}' is runtime and not queryable"); } Now, the strange thing is: I am carefully constructing my selector (only pages within parent 1041, with template 'project' etc.etc.). The only template that used the RTMU inputfield was NOT the template I was querying. I have used it in another template, along with a page-reference field. Now this page-reference field was configured to show template 'project' pages, and children of page 1041. Now that I have deleted the inpufields / removed them from my other template, everything works fine again. How can it happen that this inputfield is somehow interferring a PW selector / search? For a second, I thought I could also just comment this line: // throw new WireException("Field '{$query->field->name}' is runtime and not queryable"); but then again, I don't know what else I could possibly break somewhere else by accident. @kongondo Do you have any idea? Did you ever run into similar issues? SERVER DETAILS ProcessWire: 3.0.124 PHP: 7.2.14 Webserver: Apache/2.4 MySQL: 10.1.35-MariaDB InputfieldRuntimeMarkup: 0.0.2 TracyDebugger: 4.15.8 PS The module worked fine for me otherwise! I have used it to add a link which opens a frontend-page in a modal: return '<a class="pw-panel pw-panel-reload ui-corner-all ui-button ui-widget ui-corner-all ui-state-default" href="https://dev.webapp.mysite.com/somepage/" data-tab-text="Choose x y z" data-tab-icon="eye" id="proj_chooser" role="menuitem"><i class="fa fa-fw fa-search"></i> search / choose</a>';

Hey @Peter Falkenberg Brown I'm not exactly sure what you are trying to do, but if you want some kind of excel-like data listing (maybe with filter and sorting functionality) you can have a look at RockHitCounter which uses a quick and simple implementation of tabulator.info to list page hit statistics: https://github.com/baumrock/RockHitCounter/blob/main/ProcessRockHitCounter.module.php https://github.com/baumrock/RockHitCounter/blob/main/fields/table.php If you simply want to show aggregated data in the admin page edit then as @kongondo said you can use one of the runtime markup fields. Or a hook: // site/ready.php $wire->addHookAfter("ProcessPageEdit::buildFormContent", function($event) { $page = $event->process->getPage(); if($page->template != 'yourpagetemplate') return; /** @var InputfieldWrapper $form */ $form = $event->return; $f = $this->wire(new InputfieldMarkup()); /** @var InputfieldMarkup $f */ $f->label = 'My Markup Field'; $f->icon = 'exclamation-triangle'; $f->value = 'Dynamic Markup @ ' . date('Y-m-d H:i:s'); // insert dynamic field after the "title" field $form->insertAfter($f, $form->get('title')); });

Typically two choices if you want a custom 'view' in the admin. If custom view per page, i.e. as some sort of field / inputfield: have a look at the growing number of runtime markup fields in the module's directory A dedicated admin page view: You will need to create a custom module. More specifically, regarding data, and lots of it or spreadsheet-like data, have a look at @bernhard's modules and work in the forum. Looks like he has done extensive work in this respect. I am in a bit of a rush so cannot be more concrete. Hope this helps you get started.

Is there a reason why you cannot follow this same pattern in the present project? I can see how this will be painful in the long run. It is also prone to mistakes; posts being added in the wrong sub-sub-category. What comes to mind is a custom module. Could be a process module or a runtime inputfield coupled with a Pages Save Hook. You probably won't even need to save they dynamic field as it is only there to help you drill down to the final parent of the post you are creating. An inputfield may not help with the challenge of going via the tree. It may even confuse authors as they will click on a parent page in the tree to 'Add' a post but the post will then be re-parented. So, maybe a process module is more friendly in this regard. At the top, you would have a dynamic select like field to drill down to sub-sub-category you want. It would then reveal a post add form. On create page, post author is redirected to the edit page of that post. The challenge here is that you will be working away from the usual page edit form. You will need to embed ProcessPageEdit::execute() output in your form. It is not difficult to do. There are posts in the forum that show you how. Just some preliminary thoughts...

In most cases I expect people to use the Select Image inputfield in conjunction with the Dynamic Options fieldtype. And regarding the value of a Dynamic Options field: The Select Images input type is a bit of a special case because it can be a "multiple" or a "single" input type: And beyond that there is a config option for Dynamic Options that can be used if the values that are saved are paths or URLs to Pagefiles or Pageimages, and this could be a good option when you're using Select Images as the inputfield: When that option is enabled you could use the Pageimage sizing methods on the field value. Note that for multiple images the value is an array of Pageimage objects and not a Pageimages object. That's because these modules are intended to be flexible enough to allow selection of images from more than one PW page but a Pageimages object only supports Pageimage objects associated with a single page (see the class constructor). The readme for Select Images says: So you can have the value for each option (thumbnail) be any string you like and output that in a template file or use it in a conditional to affect the page markup in some way. But I think my most common use case will be as an "image reference" field to select URLs to Pageimages and have the formatted value be a Pageimage or array of Pageimage objects. People have suggested different use cases for an image reference field in the GitHub request linked to at the start of the readme: https://github.com/processwire/processwire-requests/issues/207 Personally I have used image references in these scenarios: To allow an editor to select a social media "share" image from among all the uploads to Images fields on a page. In cases where an editor may only select from a collection of existing images and is not allowed to upload new images. Similar to the previous example, but where the editor needs to choose images that have been prepared in advance to fit in spaces with particular aspect ratios. So for a landscape space they can only select landscape images, square space can only select square images, etc. The allowed options for a Dynamic Options field are determined at runtime according to the FieldtypeDynamicOptions::getSelectableOptions hook you are using. These allowed options are used to validate any stored value when it is accessed. So if an image that was referenced is deleted then it will no longer be in the value you get from $page->your_dynamic_options_field

I'm not sure that I fully understand what you need to do, but I have a couple of thoughts. One is that runtime fields and searches won't mix well, and if you need to maintain a field for some kind of sort code, you'd be better off hooking on saveReady to maintain a field. The other is that it can be quicker than you expect to loop through the results of a search (so long as there aren't huge numbers). So a good strategy might be to get as few pages as possible using selectors, and then loop through those pages to produce the final result.

Thanks Rob, got your message and will take a closer look today. Regarding the functions you mentioned: • wakeupValue() converts a file from a basic storage type (typically an array) into it's representation in ProcessWire. Most often this is used to translate the raw value from the DB into the live value for runtime. • sleepValue() does the opposite of wakeupValue() and translates a runtime value back to it's basic type for storage. If you are just dealing with a simple type like an integer or a string that are represented in the database exactly how they are at runtime, then wakeupValue() and sleepValue() don't need to do anything at all (other than return the value they were given). So these functions really only come into play when the field's value has to be stored differently than it would be presented at runtime. This would be the case for anything that is presented as an object at runtime. Since an object can't just be "stored" and still have its components be searchable and sortable in a database, it needs to be translated to/from fields in a DB table. Likewise, if you are sending object field values to/from JSON/XML web services, you probably want that object represented as an array, so that it can be encoded by XML or JSON. • sanitizeValue() is entirely unrelated to the other two functions (though you may see it called in series with them). But it does nothing other than sanitize a value that is assigned to a $page. So lets say that you have a field called 'myfield' and this API call gets executed: $page->myfield = 'my value'; That value gets passed through the sanitizeValue() method for whatever fieldtype is used by 'myfield'. After passing through sanitizeValue() it gets set to the $page. The sanitizeValue() method is just a way to ensure that invalid data doesn't get assigned to the page. So if the fieldtype used by 'myfield' expects it to be a string, then its sanitizeValue() method should ensure that whatever it returns is a string, regardless of what it's given. Or if you want, you can have it throw an exception when it receives invalid data. But here's what a sanitizeValue() method might look like for 'myfield': public function sanitizeValue(Page $page, Field $field, $value) { if(!is_string($value)) $value = (string) $value; return $value; }

@teppo Thanks for the update. I'm migrating a live site to the Wireframe and I need to set altFilename to wireframe for specific templates on the runtime via hook or somehow else only for superuser role. Is there is a way to do that?

If you can build enough discipline in the way you write conditionals, you can avoid this by always putting the constant part first: if ('xx' == $page->template->name) { So if you accidentally miss an equals sign and do an assignment, you get an error from the runtime as you are trying to assign to a constant string literal. I've tried to build this habit over the years and sometimes remember to do this - but I'm not there yet.

Hi, Tks for the module. Wanted to install it to give it a try. I got this error message when the installation started. I have installed the Fieldtype Runtime Only module before. do you have any suggestion ? I am running it on ProcessWire 3.0.164 and php 7.4 Thanks a lot

/** * allowed flag constants (bitmask) for context * @see core/field.php * */ Field::flagAccess // 32 Field::flagAccessAPI // 64 Field::flagAccessEditor // 128 /** * set context via fieldgroup * @see core/fieldgroup.php * */ $fieldGroup->setFieldContextArray($field_title->id, ['flagsAdd' => 192, 'flagsDel' => 32]); $fieldGroup->saveContext(); /** * set context via fields * @see core/fields.php * */ $field_title = wire('fields')->get('title'); $field_title->addFlag(128); // runtime wire('fields')->saveFieldgroupContext($field_title, $fieldGroup); not tested, but should work

Hi all. I've got something that I'm not 100% sure is a problem with AppApi or elsewhere, so I'm hoping someone can just help me cross the AppApi module off my list of the things I'm looking into. I'm currently trying get my API to send an email when a certain POST is made. Locally this works fine, but on our test server I'm getting an unexpected error: { "error": "Internal Server Error", "devmessage": { "message": "set_time_limit() has been disabled for security reasons", "location": "/home/runcloud/webapps/D2U-API/site/assets/cache/FileCompiler/site/modules/WireMailSmtp/WireMailSmtp.module", "line": 750 } } Now a quick look through this forum suggests that this should be generating a PHP warning, that I should just be able to ignore, but for some reason it's stopping execution and generating the API error. I'm wondering whether AppApi is handling this as a hard error and stopping execution, but really it would carry on fine if left, or whether the situation I'm seeing is more serious than the examples in that other forum post? @Sebi or @thomasaull do you have any comments about how a PHP runtime warning would be handled?

Isn't the string you're seeing in the log from your inputfield, which you said to be setting a string currently? Probably the sane way to handling the data would be that the runtime value is always the image obj. The sanitizeValue method will make sure any other valid (user) input, like the db string format, is converted to the runtime obj. sleepValue und wakeupValue will only handle the string to runtime obj conversion for both directions.

Works fine for me here with the dev version, but should also work with the master version (IIRC - sorry, it's been a while since I touched this). Tested in PW 3.0.85. Here's my copy pasted PHP code: // both of these work /*namespace Processwire; return wireRenderFile('runtime-markup-wirerender-test'); */ return ProcessWire\wireRenderFile('runtime-markup-wirerender-test'); The above allows you to have your code inside PHP files. This is the pasted PHP option. It is supported in both front- and back-end. It also allows you to use PHP files, but indirectly. It is not to be confused by the other (currently dev) option of directly rendering PHP files (i.e., just stating name of the PHP file and its path. Are you getting any errors? Tracy? In the template file I echo the field's value like so: echo $page->runtime_markup;

@ryan This is an excellent addition to the core. For a particular use-case I have, I am wondering if something is possible -- either something already possible I've overlooked, or as a feature request. Sometimes it is useful to have the page info when building a selector for a page reference field — so I was wondering whether it could be possible to use placeholder tokens to pass the current page being edited to the ajax request? For example, in the "Text Tags" section of the field setup, we could enter something like this for the "Ajax URL" : /find-fieldname/{page.id}/?q={q} Where "{page.id}" is replaced with the ID of the page being edited at runtime. Similar to other places in the PW admin where we can insert placeholder tokens. Then in the hook itself, we have: $wire->addHook("/find-fieldname/([0-9]{4})/", function($e) { $pageID = $e->arguments(1); // ... the rest of the code }); I'm using page ID as an example, but it could be any valid page or template variable.

er314, I think your intentions are in the right place and I appreciate your determination. You are still asking for an established method with a specific purpose to have a different purpose and an access controlled front-end context, when that is not the purpose or full context of the method. So I hope you can understand why it's not realistic to just change the definition and implementation of an established method. But I get where you are coming from and think you've identified a potential helpful addition in the API, so I go ahead and add a "one" method or change the "findOne" method (in PW3) to behave as a "find just one" method with regard to access control, hidden/unpublished visibility settings, etc. While I don't personally think I will ever use this method, I think that you and Teppo have identified a couple of situations where some might find it handy. Even if one doesn't need it, perhaps just the presence of it in the API will further clarify the intentions of the methods as a whole, for folks that may have thought that the existing find() and get() were the same. I still worry a little bit about people forming an impression that PW's API is doing access control for them, when our entire API is based around providing methods for the developer to control access the way they see fit. The viewable() method is the basis of that. But we've already dipped into that territory with the find() method, and by adding the proposed method, we're not removing any control, just adding more options. So I think it's an okay. What I was stating is that I'm not aware of an instance where anyone has introduced security problems into their site as a result of using the get() method. Though you stated above that you did just that, so now I'm aware of an instance. You are correct that if a developer uses a $pages->get() method to retrieve a page that came from user input, and neglects to validate that the page is one you allow, then that could be a problem. But the same problem could surface anytime a developer neglects to properly sanitize or validate something–anything–that originated from user input. And this is not something you can skip regardless of what method you are using to retrieve a page or group of pages. Simply validating that a page is viewable does not mean it's valid for whatever operation you may be performing upon it. You would need to validate that it uses the expected template, comes from the expected parent, etc. So while we are adding a PW3 method to support your request, be careful not to get the impression that you no longer need to validate a page that originated from user input. Also keep in mind that a $pages->find() or proposed $pages->findOne() method that filters results is based on database-filtering, not runtime filtering. Part of PW's access control model supports runtime hooks to Page::viewable. If your situation includes any runtime access control options and pages you are loading as a result of user input, you shouldn't skip a $page->viewable() call regardless of what method you used to retrieve the page.

You are absolutely right, implementing that forum from scratch is basically implementing stuff in PHP. If there is some interest, I indeed may write an in-depth blog entry, but for now I'll try to make it short (which probably does not work...). Let's start with a quick view on the five templates involved: forum-root well, this is the parent of everything. The template only contains the global welcome text but the associated php file enumerates any contained forum-group and creates the root layout. forum-group just a group around a couple of forums with its own descriptive bodytext forum-forum well, a forum containing the topics. The forum-forum nodes contain an additional "cache" for post and topic counts to provide a massive speedup when rendering forum-root. The counters are flagged with a timestamp, so they are updated on-the-fly only if required. forum-topic this node holds the post-list. In contrast to all other nodes which manage the content only by child-parent-relationship, this one has a page-table on the child node for the only purpose to see the thread also in admin. forum-post basically the post text with automated creation of title and name from current timestamp. Handling topic updates and user notifications was one of the first things implemented due to its expected complexity, but in the end it was rather easy and little efford with the right fields in place. topic-group, -forum and -topic nodes each has a page select field (holding PW user pages) which may be set or cleared by clicking on the grey wall to activate a "forum guard". Whenever there is a change beneath that node, the guard adds the topic to the per-user notification list (placed in the PW user template, which is more complex than usual this implementation). If there are pending notifications an additional button is displayed in the headline next to the user avatar: The user may check active notifications from that button and reading the topic will in turn clear that notification (remove the topic reference in the user's notification-list). Each user may choose to get informed by mail (standard WireMail) for pending notifications (html or text, various intervals, managed by LazyCron). The trick here is, that the user only get informed once per topic until he re-visited the forum to avoid flooding on active topics. Each visit is also recorded per-user to implement the updated-topic functionality. This is basically a topic/forum/group page select with the current timestamp in the PW user, updated topics are flagged with a bright lamp. https://forum.tom-productions.de/?f=test-forum&t=hallo-unglaublicher&p=1 Handling the post content is one of the more interesting and challenging features in this implementation. Mentions, for example, are detected when writing a post (regular @ syntax followed by the user name). This will simply add the post page to the user's notification list so it can be identified during regular processing. But there is more to explain in this screenshot. If someone replies to its own post, both are merged with that "Ergänzung" text beneath the ruler. That additional header is created during runtime from a data-ts field contained in the <hr> tag so it does survive even a later edit. While such an addition pushes the topic's modified timestamp, an edit doesn't. You may also note the phrase "externes Bild" next to an URL. This is a tribute to the european GDPR to not send the user's IP address to external servers without explicit grant. Clicking on that solid shield next to the cite icon will open a popup which has to be acknowledged. You may also notice the colored links in the user's signature, that's another point to avoid clicking on misleading links. If the link text contains the correct target server, its green, otherwise red (changed the signature for that screenshot). The color on the other hand does not qualify the linked content by any means... That's all done during runtime before actually displaying the post content along with replacing the usual emoticon shortcuts (learnt to love preg_replace_callback). Apropos Emoticons: To eliminate copyright issues I'm using regular Unicode emoticons which nicely pass through the PW engine and SQL, so emoticons will look differently depending on platform (Windows7 only shows black and white, for example). Apropos Signature: The signature actually is specified in markdown in an ordinary textarea field so I don't need to fiddle CKEditor into the LoginRegister module. Conversion into HTML is finally done using text formatter Markdown/Parsedown extra before the content has to pass my link and image filter. Well, there's so much more to tell (for example the CKEditor configuration), but that's for a later post... Ah, there is no magic behind the cite feature. It simply encloses the cited post into <blockquote> so the CKEditor opens with that block already filled.

@fedeb Looks like I missed a spot on the updates to the Event class, I have fixed that and removed the references to the old fields (location, notes). 1. In ProcessWire a Page can be in a "output formatted" or unformatted state. You can toggle output formatting on with $page->of(true); or toggle it off with $page->of(false); Or you can get the current state with $formatted = $page->of(); When output formatting is on, values returned from the page are intended for output and thus can contain runtime formatting such as entity encoding. This is good for output, but definitely not something you'd want saved the database, so when you are creating pages or saving values on pages you want to have its output formatting disabled, i.e. $page->of(false); This should be done before getting, setting or modifying values that will be saved. 2. The location and notes aren't supposed to be there. I removed them in favor of making the module simpler by just having "title" and "date", and then people can add any additional columns they want. 3. The set() and get() functions are inherited from the WireData class. (Event extends WireData) 4. InputfieldEvents won't be included unless something asks for it (like the page editor in the admin). If you are just using the API side, then the InputfieldEvents module likely will never be loaded or come into play. It's still nice to have in case you ever want to see or edit your data in the admin.

@ryan for the time being the data (groupID, start, end, sequence) are not supposed to be queryable. Ideally groupID should be, because I would like to display all proteins belonging to a groupID in the group page but I think I will use a workaround for this: I have a file for each group containing this information which I plan to parse when loading the group page. Individual files have at most 1000 lines (proteins). In this way I avoid querying 20+ million entries each time you try to access a particular group page. As you suggested I will load each entry (groupID, start, end, sequence) as a text field and then use php explode method to parse it into an array at runtime. The only doubt is probably on groupID: A single groupID can be referenced by multiple proteins and it does contain additional information displayed in their respective group page (I create the group pages separately). The natural limit is around 20 groups although normally this number is 2 or 3 groups per protein. With this setup is it worth using a Page reference field? What are the other storage possibilities? In the future I think I will end up using ProFields or building a Fieldtype module. For this last approach I think I need to read a bit more about modules since I am new to processwire. This tutorial posted by bernhard is a good start. @Hector Nguyen if you are not constraint by memory then loading the csv into memory all at once is the way to go, right? Thanks for all the useful suggestions. p.s. maybe I am diverging from the original thread. If you prefer I can open a new one.

@fedeb Glad that moving the $parent outside the loop helped there. The reason it helps is because after a $pages->save() is the automatic $pages->uncacheAll(), so the auto-assigned parent from the template is having to be re-loaded on every iteration. By keeping your own copy loaded and assigning it yourself, you are able to avoid that extra overhead in this case. Avoid getting repeaters involved. I wouldn't even experiment with it here. That will at minimum triple the number of pages (assuming every protein page could have a repeater). Repeaters would be just fine if you were working in the thousands-of-pages territory, but in the millions-of-pages territory, it's not going to be worth even attempting. Using a ProFields table field would be the best alternative if you needed it to be queryable data. If you didn't need it to be queryable data (groupID, start, end, sequence), I would leave them as they are, space-separated in a plain textarea field — they can easily be parsed out at runtime so you can access them as as properties of the page. (If that suits your need, let me know and I'll get into how that can be done). When working at large scale, it's also always good to consider custom building a Fieldtype module for the purpose too (that's another topic, but we can get into it too). For your groupID, if the same groupID is referenced by multiple proteins, and there is more information about each "group" (other than just an ID) then I think it would make sense for it to be a Page reference field. What is the max number of groupID+start+end+sequence rows that a protein can have? If there is a natural limit and it's not large, then that would open up some new storage possibilities too. Another optimization you can make in your loop: $page->sort = $i; This prevents it from having to detect and auto-assign a sort value based on the quantity of children the parent page has. For the $page->name, if each page will have a unique "protein-name" then you might also consider using that rather than the ("protein" . $i), as it will be more reflective of the page than a generic index number.

Hi, that's where comes, once more, the beauty of pw... something you can easily do, just create a hidden page, let's say "sliders" (can use an template without file) this page will have children using a template without file too, template to which you'll just need to associate the repeater used to create your sliders elements and this is where the magic comes, Hannacode! create yours, something as easy as [[slider id=123]] or [[slider name="xxx"]] id being. the id od the page containing the slider, name... well guess ? and where you want your user to be abble to use them, just add TextformatterHannaCode to your ckeditor; job done, it works a little like wp shortcodes but; a little difference, it works, it lets you write and use the code you want without generating hundreds of html/js/css lines in the page, it won't be broken by some update and it's fast as hell i let you decide where and how to tell your user what code to use, personally, i use kogondo's https://processwire.com/modules/fieldtype-runtime-markup/ it makes easy to add an easily selectable piece of code inside a readonly input in the backend, live example for a customer who wanted to be able to create carrousels and use them anywhere in any page if french, sorry, but you get the idea and, below, there is... a title and a simple repeater, the user has just to add a p in ckeditor, paste the code and hannacode takes care of what's left to do ? hope it helps have a nice day

@fedeb That's the largest quantity of pages I've heard of anyone creating in ProcessWire, by a pretty large margin. So you are in somewhat uncharted territory. But that's really cool you are doing that. I would be curious how different the graph would be if you split it up into batches so that you aren't creating more than a certain quantity per execution/runtime. For instance, maybe you create 10k in one execution and another 10k in the next, etc., or something like that. Would the same slowdown still occur? If so, I would start to think it might be the database index and increased overhead in maintaining that index as the quantity increases. On the flip side, if restarting the process to create each set in batches solves the slowdown, then I would think it might be memory or resource related. A couple things you can do to potentially (?) improve your page creation time: 1. At the top of your code (before the loop) put: $template = $templates->get('protein'); Then within the loop set: $page->template = $template; 2. I don't see a parent page assignment. How are you doing that? Double check that you aren't asking PW to load the parent page every time in the loop and instead handle it like with the template in #1 above. 3. What kind of fields are on your "protein" template? Depending on their type, there may be potential optimizations. Especially if any are Page references. Can you paste in a line or two from the CSV? 4. If you can assign a $page->name = "protein" . $i; rather than having PW auto-generate a name from the title, that will save some resources too.