Webrocker

Hi, I'm in the process of wiring (heheh) err, tinkering with the structure of a website that should have: One basic structure regarding the main navigation and meta items, but depending on the "country" (which could be interpreted as a subdivision of the construct), which could be adressed via (sub)domains (like de.tld.com for Germany, ch.tld.com for Switzerland, it.tld.com for Italy etc), some pages' contents will be "country"-specific or only be viewable while in that country's (sub)domain, while others will have the same content for all countries involved. The selection of country/language combinations will be available on every page, in meta or footer area. For this structure, multilanguage should be enabled, and DE, EN, FR, IT languages will be used. But, for example on the above mentioned "german" view of the website, only DE and EN translations should be used/selectable, but on a "Swiss" view, there should be DE, FR, and IT translations. I know that I can tackle each of these feature-wishes isolated, but currently I am a little stuck as to how to bring this together in one install. I think I need to make the "country" distinction virtual, have all the subdomains pointing to the same site, and use _one_ page-tree for the basic and common structure, plus an option on every page/content for which "country" this should be display-able. right? then I can use the five translations on every field/content needed and "only" need to filter the unwanted ones out if the pages are viewed in the "wrong" country-(sub)somains… hm (don't ask, I'm in the process of clarifing why an existing translation of a content that is displayed across all "countries" should be hidden in some, but for now the answer to that is pending). Or I could create top-level pages for every "country" needed, but then I need to rebuild the main structure over and over again, this seems not so feasible. What I want to avoid: Having separate page-trees for every country, since this maybe will "pollute" the urls -- where do the pages/contents "live" that are shared across all countries? I am not quite through with thinking of the pro and cons of the different possible ways, which currently to my knowledge are: A) One PW install, one site directory: open question how to make the different "country" pages manageable, how to map the (sub)domains accordingly B) One PW install, different site directories for each "country": since the "look" of the country sites is identical and only some contents and the amount of translations available vary, I think this is overkill. Plus, there's a lot of duplicate content needed (?). ON the other hand this would over the greatest flexibility should the site evolve and the "country"division drift further apart, content-wise. C) .... next gen ProcessWire as announced in the recent blog post? Multiple Domains, one core, shared contents across the above… don't know if this will be production ready soon, and how to handle the new features… black box I think I need some ideas how more seasoned pw-devs would start to tackle this, so any input (links, ideas, rants) is welcomed! Thanks, Tom

Hi, here is one of our recent processwire sites. Since I am really terrible at talking about what and why we did things -- the main reason why our own website is, well, not-so-content-heavy , just have a look at the page and ask stuff if you are interested and I'll try to answer. http://www.krfrm.de The Kulturregion is a group of towns and regions in the RheinMain area around and including the city of Frankfurt, and they orchestrate some common cultural events that are part of a greater topic, like "gardening culture" or "industrial heritage" with guided walks or concerts or talks and so on. The old website didn't allow the editors, who already worked content-wise for the brochures and info leaflets, to add stuff themselves, so the site was not very actual. We were tasked with the responsive webdesign and development, based on ideas and visuals of the graphic designer and collaboratively refined with working prototypes. the budget was super-tight, the time frame from first contact to launch was six months. the site will continously be expanded with new features based on the editors and visitors input. We are very happy with the decision to use processwire, which turned out to be a great tool for the different aspects of the site: FRONTEND: since pw is completly markup agnostic, we could tailor the code using susy-grid and a sass/grunt build process to our needs and the changing art direction (in which we luckily had something to say as well) BACKEND: during the development we had several spec-changes regarding the structur and content. pw's way of sticking together fields in templates could cover quickly every new or changed request EXTERNAL DATA: using pw's api, it was very easy to import all the events' data as then editable pages, categories etc. categorization and relations to projects, searchability over dates - it took some headscratching and good thinking about the structure, but once in place, it was very flexible. and expandable, which is a important thing since this site will grow with each new "project" (the larger theme for events) that will be added. All in all we a very happy with the result, which hopefully will always be work in progress

Hi, here's what happened to one installation yesterday, and since I have no clue what triggered this, I hope some of you may have an idea and are willing to share. While working with a superuser in the backend and modifying a page-template, adding/duplicating fields, suddenly the backend's theme changed and the backend behaved strangely. After logout, the page wasn't viewable - both front- and backend stated that the user had no rights to process the ProcessPageView module. It took two people and four hours of debugging to get to the root of this: the system user template's fieldset had been modified and the only field left in there was 'notifications' -- all other fields were gone, so no user had any roles anymore. Immediately before the problems started a notification was saved to the database, stating 'welcome this is your first notification' (sorry for not being exact here, I'm writing this from memory without access to the actual messages), which is strange since the system notifiactions were active for weeks now and working fine. my question is, how can a system notification / the systemnotification module/ alter the user template, or precisely, the user template's fieldset? This was very hard to debug since at the first, second and third glance everything regarding roles, permissions and users seemed to be correct while looking at the database. the database itself was healthy, no repearable issues, no rogue data or strange overheads. Only the fieldset for the ser template was suddenly missing the usual fields which had been replaced (?) with the notification field. Has anyone else had such an issue? I'll add the processwire version and versions of the modules used as soon as i'm at my working machine, and maybe @inspeCTor has some additional infos since he was the one working with the install when this happened. it may be just a coincidence but since the timestamp of that strange message matches that of the start of the problem and the 'notification' field was the one left in the fieldset, I suspect the system notification module. but that's only guesswork and I'm happy for any idea/explanation for was going on here EDIT: There was no attempt to update or modify the installation when this happened, just 'normal' development work with fields and templates while using a superuser cheers, Tom

Hi, good call, I will check out the autocomplete input for the backend. for the moment I'm quite happy with the backend performance. Currently I'm tackling the problem of how to get the _contents_ of 186000 html post files into the body field of my post pages

Hi Sergio, this is a fairly run of the mill managed server; it is hosted at/by german provider 1&1, who don't excell at their shared hosting offers, but the servers are quite ok Intel® Atom™ C2750 8 Cores x 2,4 GHz (2,6 GHz Turbo Boost) 8 GB DDR3 ECC 1.000 GB (2 x 1.000 SATA) opt: 240 GB ( 2 x 240 GB SSD) Intel® S3500 / Software RAID 1 The issues I encountered were mostly time outs by either the php/apache or the mysql server. calling the import scripts via the php/apache led repeatetly to "gateway timeout" after about 5 minutes. I didn't test the import scripts from the console, though. I maxed the php ini settings, but it seems that the server has some settings that I cannot override; for example the max allocatable memory for php is 152mb, no matter what I may set in php.ini. And the mysql server will time out after about 5 minutes, even with settings as high as 1hr or more in the config. cheers Tom

Hi Pete thx for the link and the hints regarding the auto join… I was wondering about that option while inspecting my template's fields… now this makes more sense to me, great! cheers Tom

Hi, do you want to give those users the right to edit those pages, or are they only visible to logged-in users on the web page? Each can be controlled with the access-roles and permissions for users and the settings on the page's templates. If you sketch what you want to achieve, I'm sure the answer can be found here cheers Tom

Hi, I just wanted to sing a song of praise of how smart actually PW is (designed). From my recent posts here in the forum you may know what I'm currently trying to do in my spare time - I port/migrate a fairly popular handbuilt 16 year old forum over to process wire. Reason is; the forum's users have grown to like the not-like-the forums out there look of the 16 year old originally perl/cgi based thing, which basically is nothing else like a nested list in most views, which then got enhanced whith loads of custom pages, functions, galleries and wutnot. 16yrs worth of procedural php code, created by at least 4 different and differently skilled developers. Bring it on! Whatever the reason, I thought this might be the perfect pet project to get to know PW in and out. And I really have to say: I'm deeply impressed. I have over 15 years experience wiith different CMSses, and I consider myself as fairly skilled in TYPO3 (yes, with that Typoscript riddles), know my way around cutsomizing and theming Wordpress sites since wp2.1 and I messed around with several other popular and not so popular CMSs as a part of my daily work. For like 1.5 years now I use PW as the "motor" of most sites I build. And now, with this "pet project" of mine, I'm really starting to fall in love : Being able to bootstrap the API and creating scripts that I can "fire" via the commandline, directly on the server, connecting to other databases, helped immensly with the task of getting 186000 posts by 5000 users into a processwire install. creating the relations between these posts (which one is the root of a "discussion", what is the "parent" of the current post) was easily done be using "Page" fields and again a script that scraped that info from the old structure and populated the page fields accordingly. I plan to write a detailed how-to once the migration is completed and working, but this could take some time ;-) Now the backend would time out if I wanted to edit one of the post. Caching the templates didn't help, so I thought about what was going on "under the hood" -- and after a little bit of head scratching it occured to me that maybe the 4 "Pages" field each post carried, were to blame, esp. the 2 fields referencing the whole collection of posts: "post_root" and "post_parent" -- each of which could theoretically be one of the other 185999 posts in the database. Since the default option for the visibility of "pages" fields are "select" and "open", PW needs to query the 185999 posts - and that brought down my php script execution limit and/or the allocated memory size of 152MB. Wow. But guess what, and that's where my romance with PW started for real: If you switch the visibility to "closed", or in my case to "closed with ajax on opening" the backend will run fast again. Now the selects will only be populated (with ajax) if I need to change the post_root or post-parent - which should never be necessary in the everyday use of that data. Initially I had imported the 4000 users not with the system's user template, but a custom one. While this worked without a problem, it brought down the lister, which wouldn't respond and time out after like 5 minutes. I still don't know why, but changing those users to the system user template got rid of that problem, and now they are behaving quite well. This is so cool. What started as a late night idea seems now to grow into a real project. With all that importing and moving imported data to pages data, I had like half a million pages in this install, and PW handled this quite nicely. In the meantime I have cleaned up and currently I'm down to about 186000 pages plus 4000 users plus some static pages. This wouldn't have been possible without the option to use the PW API from the outside. Thank you Ryan for this great piece of software. Cheers, Tom

Hi Wanze, thanks for confirming my own nebulous assumptions. at the moment the cleanup script is running in batches of 1000s, and trashes (not delete) the pages via the api. good idea to call the script via the command line, why haven't I thought of that

After identifying the page id with the above, I now try to actually get rid of them. The Problem is; by using the API, the php script timeout will be triggered - that's b/c there are now ~400000 pages involved… So I'm trying to get this done directly with mysql, but I ran into some perf issues as well. SELECT id FROM pages p, field_as_post_legacy_id lid WHERE p.id=lid.pages_id AND p.parent_id=1035 GROUP BY lid.data HAVING COUNT(lid.data) > 1) This will return about 10000 ids which are duplicate pages and can be kicked out of the pages table. if I simply put them in a subquery like this: DELETE FROM pages WHERE id IN ( SELECT id FROM pages p, field_as_post_legacy_id lid WHERE p.id=lid.pages_id AND p.parent_id=1035 GROUP BY lid.data HAVING COUNT(lid.data) > 1 ); it won't work b/c mysql does't allow for the same table in DELETE and SELECT FROM statement. As a workaround I duplicated the pages table, and now this DELETE FROM pages WHERE id IN ( SELECT id FROM pages_backup p, field_as_post_legacy_id lid WHERE p.id=lid.pages_id AND p.parent_id=1035 GROUP BY lid.data HAVING COUNT(lid.data) > 1 ); starts, but eventually this too will time out. I also tried to put this into one query, but the "group by ..having..." seems to throw an error if used in the DELETE statement. :-/ DELETE p FROM pages p, field_as_post_legacy_id lid WHERE p.id=lid.pages_id AND p.parent_id=1035 GROUP BY lid.data HAVING COUNT(lid.data) > 1) As you probably can guess by now, this mysql query crafting thing is not one of my greatest skills, so if anyone has some hints on how to get this to work without grinding the mysqlserver ;-), I would be very thankful cheers Tom

wow, this might come in handy for my forum-migration tests. cool.

Out of curiosity; what is $p->save(array('quiet' => true)); doing differently to $p->save(); ? thx Tom

Hi Wanze & LostKobrakai, thank you for the instant replies!

Hi, I'm still testing my idea to port a 1999 cgi script based forum that has been spaghetti code 'enhanced' over the last 15 yrs over to processwire. (yeah, don't ask ) Currently I have ~ 4000 users and ~190000 "legacepost" pages imported via the api, and I have to say that I'm pretty impressed how easy this went. the legacyposts have fields with values like "rootpostid", "parentpostid", "authorid", which are populated with the values from the old forum. now I created a script that created new "post" pages, and filled "pages" fields like "postauthor", "postroot" and "postparent" with data relating to my processwire install, and of course the post's real data like title, body etc pp. This also went pretty well (I'm amazed how nicely the lister behaves, even with now about 400000 pages under the belt), but during this page creation some thousand duplicate posts have been created, most like because I accidently fired the import twice (I imported the original post in batches). I can identify those by their "legacy_id" - there shouldn't be two "post" pages with the same "legay_id". Is there a "processwire" way to find those, or is this a job for plain mysql? I have some other issues with the design of my "post"page data (by using page fields that query the whole bunch of "post" pages I think I grind php to a halt), but I'll create a separate topic for this and keep this here to the "duplicate" question. cheers, Tom

Hi guys, just a quick follow-up: I now have imported the 4000+ users in PW; using the "normal" user template, spiced up with the custom data fields I need for the user-data stuff from the old forum. This worked, and the lister instance which does put out the user data in the backend is running fine and without problems. So I have no idea what exactly the difference is, since I copied my custom user template off the internal "user" template - but this way now, everything works smoothly. cheers Tom update: (just an aside: i have currently 40,000 posts imported as pages and no hickup in the backend.) So I'll mark this as resolved, since this is not a performance issue of the lister, but something about the combination of a large amount of users, custom-template, and the lister.

Hi franciccio-ITALIANO, ok let me try. depending on what kind of site-profile you chose on setup, the front-end will be rendered with processwire (pw) like this: first, pw "looks" for the _init.php file, wherein you could define some stuff that you may want to use all over your site in different places. this usually then includes the _func.php file, where you would define some custom functions you want to use all over your site (building a custom navigation for example)… then Processwire "looks" for a file that has the same name as the template that is used in the backend (where the input fields are collected). I think this is the point most confusing to people new to processwire (at least I was confused for a while) -- you have the "template" which is responsible for the set of fields where you put in data. And then you can have a "template-file" - that's the php file with the same name as the "template" plus the .php suffix. (there's an option in the "template" to use a file with a different name, but lets ignore this for now). so, processwire now looks for this "template-file", but, and that's what confuses you I think - this template-file alone does not render the output you see in the browser, because usually a "_main.php" file is included afterwards - this is where most of the markup and output will happen. So lets say you have a "title" and a "body" field in your "template". As long as the "_main.php" file uses $page->title and/or $page->body for the output, and as long as your "template-file" does not alter or add something to these variables, you will see your data in the front-end, even if your "template-file" has no code or markup in it. So what are the "template-files" are for? They serve as the entry points for the data you collect via the fields of the template. Since each template can have a variety of different fields and data-types, somewhere before the output to the page these things must be collected and prepared, like a list of small images that should be wrapped in links to the large image versions for example. Have a look at the code inside the files that you find in your "...site/templates/" directory, and maybe with the above explanation things will be a little less confusing, I hope. cheers, Tom

… and yet another update this time I imported "my" 4000 users as normal pages, no roles, no password, just the data. And guess what? The lister is running fine and snappy. Next up is a test how the "users" behave in the lister if I use the systems "user" template, not a custom one, maybe I can narrow down what's causing the timeout.

… for testing purposes I changed the wire('config')->maxPageNum = 99999; (at the very end of the ProcessPageLister.module) to wire('config')->maxPageNum = 1; Now I get a "maximum execution time exceeded" error back. Aha! As I suspected, it was at the "30 seconds" default, so I raised this to 360 seconds. Now after about a minute or so, the "Pages: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away" error returns. Checking with phpinfo I see that the "mysql.connect_timeout" is at 60 seconds. I raised this, too, and now have 600 seconds there. Now it takes considerably longer until the above "server has gone" error will be shown, and afterwards the complete 3897 user entries are there, neatly paged in 25-per-page pages. So it seems that the script execution time and the mysql connection timeout are the bottlenecks. But even if I get this to work with even larger limits - the lister is way too slow to be of use in that scenario, esp. since this will again be fired if I'm going back to the previous view if I had clicked a result in the lister. :-/ I just raised the limits to 2400 seconds in php.ini and then timed the response in the backend - after 1:55:90 (nearly two minutes) the script will be interupted and the "server gone away" error appears. So I searched for "120" entries in the phpinfo output and found: [Core] realpath_cache_ttl 120 ... [Environment] DBENTRY__CPU 120 ... [PHP Variables] _SERVER["DBENTRY__CPU"]120 ... Does anyone know what these are, and if they may be related to my problem? thanks, Tom

Hi thetuningspoon, thanks for answering and even more thanks for confirming that this shouldn't be a problem with PW, but must have something to do with factors that hopefully are in my domain I haven't change the lister's settings, it'll display 25 items per page. I have a hunch that this maybe has something to do with the "users" - these 4000 entries would be users who'd be able to log in somehow, and/or with me borking something while playing around with custom user templates. I'm wondering because the ~4000 "user" pages behave fine and are really quick to browse/page as long as I am in the regular "tree" section. cheers, Tom

Next update: I moved the test install to another host, which usually performs nicely. Here the same thing happens - with 3897 user entries the server will time out and throws 500 error. Adding 500 users slowed the lister's (or the servers?) response by about 12 seconds per set (i started with 500, which took the lister about 24 seconds to show, and added more step by step), the last working amound was around 2300 users with a response time of over 80 seconds - the next batch of imported users made the thing quit. I don't think I've any special fields or data stored at those users, it seems that the pure amount of user/pages brings the lister to its knees. Here's what currently is stored at the user's template (name / fieldtype): pass Password email Email title Title roles Page admin_theme Module language Page as_legacy_id Integer as_legacy_pass Text as_legacy_registered Datetime as_legacy_userlevel Integer as_legacy_numposts Integer as_legacy_color Text interestingly, even though the error states that the "server has gone", the backend is still workable - when the error has been thrown, the "pages/tree" section will work normally. it's just the lister and the access/settings sections that will time out.

Hi, currently I'm testing a rebuild of an 18yr old forum. basically it's a a pet project of mine, to get to know ProcessWire better, but I'm sure it could work, since the core functionality of the forums original script is fairly simple and the added functionality could very well be represented with the cms parts… For testing's sake I have imported the data of about 4000 users via the API, using a custom template file (by using the approach shown here: https://processwire.com/blog/posts/processwire-core-updates-2.5.14/#multiple-templates-or-parents-for-users ) with some additional fields with some meta data from the old forum's user settings. this worked out pretty good, only that I had to split the import in chunks of about 500 entries, b/c otherwise the database connection would time out (or maybe the php script runtime is/was to blame). now I have these users in my PW install, and viewing them via the pages tree is possible. but once I switch to the lister, the complete website will stall, front- and backend, and after some minutes a SQLSTATE error will be thrown: Warning: Error while sending QUERY packet. PID=10464 in processwire/wire/core/Pages.php on line 1992 2015-08-03 12:50:46 Exception: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away (in [...]processwire/wire/core/Pages.php line 1992) #0 PDOStatement->execute() #1 processwire/wire/core/Pages.php(477): Pages->executeQuery(Object(PDOStatement)) #2 processwire/wire/core/PagesType.php(258): Pages->getById(Array, Array) #3 processwire/wire/core/Users.php(29): PagesType->get(41) #4 processwire/wire/core/Session.php(96): Users->get(41) #5 processwire/wire/core/ProcessWire.php(259): Session->__construct() #6 processwire/wire/core/ProcessWire.php(84): ProcessWire->load(Obje 4000 entries shouldn't be a problem, or is it? I know that this is hard to tele-diagnose, so before going deeper into it, I wanted to ask if there are limits performance-wise on what the API can handle? I'm asking because we plan to use the PW backend with listerPro for a project where we expect 10.000 to 20.000 entries which should be search- and filterable in the backend. The above error will be thrown if I try to set a second filter besides the "template" like "forum_user" - in my testcase there's a "legacy_userlevel" field which is an integer and has values through the import of "-100","-1","0", "1", "2", "3". (that's only temporary, I plan to switch to roles based on those old settings). The lister tries to query this field in order to make a select input where I could then select one of those values… this times out. In the "forum_user" template fields I have no settings like "global" or "autojoin" or else… I'm on the ProcessWire 2.6.9 dev version and currently in "advanced" modus. Update: Even if I try to switch to "access" or "roles", the backend will time out. This started to happen with the addition of the many users… without altering the set up, everything worked when there were only the first twenty users. Even though I'm on a managed root server with this testing project, I suspect that the script settings may be a bit… sparse. What ini settings should I look out for to counter this error? cheers, Tom

Good call, LostKobrakai. Using "speaking" values remove the true/false 0/1 string/ornot confusion while checking against the value, too Uhm, yes. That's what my code above is doing in the first example But LostKobrakai's more like it.

Hi Sebastian, if(!$page->article_option == 1) { ... This may only check if the field is present, not what value the field has. maybe if the value is an integer, it'd be better to check with "===", not "==". Depending on how you defined the values in the select opitions setup, for example 1=Hide 2=Feature Article 3=Whatever you can then check for the value switch($page->article_options){ case "1": // do nothing break; case "2": // do something break; case "3": // do something else break; } In your case, where you have the "hide" option for which your code does not need to do something, I would make that the "0" value in the field's setup: 0=Hide 1=Article 2=Whatever So now you can check if the field's value is larger than "0", and only then put out your code $articleIMG = ''; if( $page->article_options > 0 ){ // display image $out = 'code only if image is shown'; switch($page->article_options){ case 1: // do soemthing $out .= 'your code'; break; // do something else $out .= 'your other code'; break; } $articleIMG = $out; } Hope this helps, cheers Tom

Hi, not sure if this is the right place, I have a question regarding the support for "pro" modules like FormBuilder. Last year we purchased a single license, b/c we wanted to try out if this'll fit our needs. A few days ago we upgraded the license to the "big", the agency one. Maybe this was not too smart, since in about ten days our one year support will run out - it seems that with the "upgrade" we didn't prolong the support by another year. Anyways, that's not my actual point, but: Since we now have an "agency" license - how can I, as not being the user who originally purchased the single license, get access to the "pro" support forums? I tried to contact via the "client area" in my profile, but requests there are only directed at the "sales" department, and they didn't respond as for now (I opened the request 2 days ago). I have a problem with the FormBuilder and the date-select, which refuses to work and throws a jqueryUI-JS error even in the admin backend, so getting hints/support on how to debug this would really be… helpful so, sorry for the bother, cheers Tom

Hi, ok, I forgot to mention that I need to use some of the "_init.php" vars in the _header/_footer.php inside ./pages2pdf. I tried to include the _init there, and then in the template as you suggested, but a simple var_dump for one of the "_init" variables returns empty. to give you a bit of background; I use thiis approach: https://processwire.com/talk/topic/10447-using-translatable-strings-across-template-files/#entry99012 for the template translation thing, but somehow the templates inside pages2pdf are ignorant of the "_string.php" file. I have some "translateble" strings in _header/footer.php, but they're ignorant of the translations. cheers Tom UPDATE: Ok, after some more debugging I see that the include_once('../_init.php') works like you presumed, the vars are there. something else must interfer with the translation thing, at least it is not caused by a not-working include. sorry for the bother. UPDATE 2: After even more debugging and testing; the _init needs to be included in all three php files: _header.php, page.php, _footer.php - it seems to be scoped to that file alone where it is included. I somehow was expectng that the incude in the _header.php alone would be enough, but after reflecting how the PDF is looking -- where _header and _footer are repeated on every "page" of the PDF -- it makes sense that this didn't work out. However, I think having three identical includes that are repeatetly called during the creation of the PDF is not really the best way to go about this, even if it now works… ?