gebeer

1 hour ago, bernhard said:

@gebeer I'd love to have better docs, but I don't like github wikis - that's why I built my own docs system on baumrock.com

It reads module docs from markdown files that are stored under the /docs folder within the module. See https://github.com/baumrock/RockFrontend/tree/main/docs for example. Would be great if you could create a PR for RockMigrations that show that info on my website like here: https://www.baumrock.com/en/processwire/modules/rockmigrations/docs/deployments/

I'm busy with client work for the next weeks, so any help is very welcome 🙂 

Great way of managing the docs. I'll make a PR.

On 12/4/2023 at 5:46 PM, bernhard said:

Hi @gebeer no, of course the page class' migrate() method can only be called once the template exists. But that is done automatically by RM if you use the createTemplate() method:

Thank you for the explanations 🙂 Would be awesome to have this in the Wiki. Would you mind me adding it there through a PR?

Splitting up migrations and working with multiple migrations distributed over the code base is something more advanced and not so easy to wrap ones head around. Especially when it comes to order of execution. Can we influence that? Might have overlooked it if it's in the documentation or Wiki...

17 hours ago, Rasso said:

I gave this another try today. Related issue on GitHub: https://github.com/baumrock/RockMigrations/issues/43

I answered you there.

Thank you very much @ryanfor working on this. For larger corporate sites this is a must have. We are using ProDrafts atm and got it kind of working with nested repeaters with some dirty hacks. Approval workflow like mentioned by @Pete would be awesome 🙂

Hi @bernhard,

referring to this post:

Thank you for the detailed explanation there. I was not sure if the custom PageClass files would be called before the associated template exists and therefore the migrations would run initially. For example, I have a site/classes/MemberPage.php with migrations in it, where the migration creates the member template.

Will the migrations be triggered before the member template even exists?

Thanks @bernhardfor the detailed explanations about splitting up the migrations. MAybe you want to add that to the Wiki. That would be awesome.

For that big project where I have that 4000 lines migrations in one file I still need to do that. When I started out with that project I wasn't fully aware on how to properly organize my migrations. Like with any other tool, knowledge grows with usage time 🙂

On 11/27/2023 at 7:00 PM, Rasso said:

I was using the Pro field Repeater Matrix. I tried RockMigrations (as the original Migrations module has been deprecated) with that field and it didn't work (can't remember what exactly went wrong, sorry for not documenting my findings better).

Which version of RockMigrations did you use? I added Repeater Matrix support and this was merged approx. 2 months ago. Here's how to use it: https://github.com/baumrock/RockMigrations#repeatermatrix-field

Repeater Matrix support is also mentioned on the module's page https://processwire.com/modules/rock-migrations/

If you tried the version with Repeater Matrix support and it didn't work for you, please let us know what exactly went wrong so we can help to fix it. In all my tests and using it in a production project, I didn't encounter problems so far.

RockMigrations has been a big time saver and working reliably for me for the past 2 years or so. I haven't tried other solutions mentioned here since they are not actively maintained.

As for having this functionality in the core. Yes, that would be great, indeed. Honestly, I don't understand, why @ryan hasn't implemented it yet. Maybe also because there is a great solution with a well designed API available already as a free module? Still, I think it would benefit the project if this was either in core or available as an optional core module. IMO, migrations are essential, especially when using PW for bigger projects and when working on projects in a team.

As for the discussion here about potential problems with migrations brought up by @MarkE , I was able to work around all of those with RockMigrations since it is also declarative, but not destructive (unless explicitly intended).

On 11/27/2023 at 7:14 PM, d'Hinnisdaël said:

I haven't fully tested RockMigrations yet, but from what I've seen it seems to be doing a few things too many (snippets, "magic pages", "files on demand") and I'd prefer a module that limits itself to migrations, if not just for performance.

The "few things too many" argument is flawed since all the extra functionality is optional and therefore doesn't impact performance. And the addon-features are well designed and can be very useful. Performance with RockMigrations is really good. Only very large migrations take some time. I have a project with about 4000 lines of migrations for templates, fields, pages and roles in my main migration file and then some additional 1000 lines spread throughout modules and these take about 20 seconds. Since you are not doing migrations all too often, this is very tolerable. Migrations on smaller projects are almost not noticable.

On 10/25/2023 at 11:13 PM, Ivan Gretsky said:

thanks to @gebeer's work.

actually the main work for this was done by @bernhardfor his old release RockMigrations V1. I just ported that to V2 and added some improvements/bugfixes :-)

It just so happens, that I needed to do overrides for a FieldTypeFieldsetPage field from the API inside a hook. Took some effort to finally get this working. Just wanted to share this.

/**
 * HOOK for changing label, description, notes for fields inside a FieldtypeFieldsetPage field
 */
$wire->addHookAfter("ProcessPageEdit::buildFormContent", function (HookEvent $event) {
	$page = $event->object->getPage();
	if ($page->template->name !== 'your-template-name') return;
	$form = $event->return;
	// product_description is our target field 
	// which holds fields 'image', 'headline', 'description'
	// product_description is of type FieldtypeFieldsetPage (FieldtypeRepeater under the hood)

	// get InputField
	/** @var InputfieldRepeater $inputField */
	$inputField = $form->get('product_description');
	// get actual Field from InputField
	/** @var RepeaterField $field */
	$field = $inputField->hasField->getContext($page->template);
	// change label, description, notes for 'image' field
	/** @var Field $field */
	$field->fields('image')->label = 'Hero Image';
	$field->fields('image')->description = 'Also shows on overview cards';
	$field->fields('image')->notes = 'Dimensions: 800x600px';
});

@Boostdid you manage to do the overrides in the template settings? On my install this does not work.
Here are my field settings:

And here's how the override settings in a template look like:

Just being curious how that looks on your install :-)

1 hour ago, jploch said:

I will test it a bit more and see if it makes sense for my next project.

RockMigrations makes total sense for all the projects I'm using it on 🙂

17 hours ago, Clarity said:

Edit 1: I think it's field because $this->createField returns Field but not RepeaterMatrixField.

Latest dev branch: https://github.com/baumrock/RockMigrations/blob/da0b08b9cc65ee5573a5041971fe6ec575130b98/RockMigrations.module.php#L4107

In the phpdoc the return value is RepeaterMatrixField|null. But inside the function when the field is created with $this->createField($name, 'FieldtypeRepeaterMatrix', $options); it seems to return the generic class Field.

I also sometimes experienced the problem, that you described. As a workaround, I first created the field and then added the items in a second run inside a conditional which tests for the correct class of the created field from the first run.

But in my latest tests, this did not occur. Unfortunately I can't seem to find out why this is happening sometimes.

@Ivan Gretsky please also take a look at https://github.com/baumrock/RockMigrations/tree/dev#repeatermatrix-field 

I added a convenience method to create the Matrix field and set the types in one go.

On 9/28/2023 at 12:53 AM, Ivan Gretsky said:

I am trying to create RM field and add types in one run (see the code below)  But it seems not possible. Should I put some code in between to make it work?

Unfortunately it is not possible to create a RM field in one go. The way I do it:
 

		// first create the field
		/** @var RepeaterMatrixField $rmfProduct */
		$rmfProduct = $rm->createField('content_product', 'FieldtypeRepeaterMatrix', [
			'label' => 'Product Content Blocks',
			'tags' => 'products',
			'repeaterAddLabel' => 'Add New Block',
		]);
		
		// then populate it
		if (wireInstanceOf($rmfProduct, 'RepeaterMatrixField')) { // TODO quickfix find out why $rmf instanceof Field after create
			$test = $rm->setMatrixItems('content_product', [
				'features' => [
					'label' => 'Features',
					'fields' => [
						'repeater_product_features' => [],
					]
				],
				'gallery' => [
					'label' => 'Image Gallery',
					'fields' => [
						'images' => [],
					]
				],
				...
			]

I will add a hint to the docstring.

Note the issue that I had, that after first creation of the field it returned an instance of Field, not RepeaterMatrixField. So I added some logic to avoid errors. Then needed to run initial migration twice to populate. Haven't found out why that happened, yet.

9 hours ago, Ivan Gretsky said:

Good day, @gebeer!

I've tested setMatrixItems() metod - it seems to work fine so far. There are some bd()'s that cause errors when the field to be added does not exist. But that is minor thing. I am eager to see this PR merged. Ready to do more testing if needed. But just maybe it is better to do it with your code already in the core? We do not want to lose this work!

Hi Ivan, thanks a lot for testing this. I've been working with it on a bigger project for the last 9 months and it seems quite stable. I had some health issues and am getting slowly back on track. Will do the PR on the weekend.

23 hours ago, Gadgetto said:

Which min. required PHP version would be OK for you?
How many of you are still running your servers with PHP < 7.4?

Same for me here, PHP 8 is ok. Also Snipcart v3 only.

Thanks for updating the module!

29 minutes ago, TheMick said:

Does this module work with Repeater / Repeater Matrix in the current version? After the posts of adrian and zoeck in 2018 I did not read anything about this.

And thanks for this awesome module @teppo!

As it happens, I looked into this with the latest version just a couple of days ago on a PW 3.0.208 install with Repeater Matrix and nested Repeaters inside Matrix items. Unfortunately there seems to be no support for Repeater Matrix.

10 hours ago, adrian said:

@gebeer - are you running PHP8? If so, it might make a difference if you update to the latest Tracy - back in March there was an update to the Tracy core (which is PHP 8 only, although I maintain older versions within the package for those on older PHP versions so upgrading won't hurt).

The other thing to make sure is that the "Use Native PHP Session" setting is unchecked.

If those don't help, I am not sure what to do at this point - I can't count the amount of time I have spent workaround issues with SessionHandlerDB and Tracy.

 

Yes, PHP8.1 that is. I updated to latest version. Still same problem. Use Native PHP Session is unchecked. Thanks for all the effort you put into Tracy! We might just resort to not using SessionHandlerDB. The site is in development and about 1 month away from launch. Pre launch we will update PW to latest dev and test again. Maybe that thelps.

4 hours ago, Robin S said:

Are we sure Tracy is causing the error as opposed to only reporting the error? As in, maybe the session lock times out regardless of whether Tracy is installed, but without Tracy installed you never learn about it. The timeout is a configurable setting in SessionHandlerDB so maybe the solution is that if you expect to run tasks that can take longer than the 50 second default you increase that setting, or if the long task is triggered by your own code do a $session->close() immediately before the task.

Or is the sort of exception thrown by SessionHandlerDB when the lock times out supposed to be more silent? There was a Tracy issue similar to that but it was fixed a while back: https://github.com/adrianbj/TracyDebugger/issues/58

I am not sure because I don't see Tracy anywhere in the backtrace. I can only tell that the error doesn't appear when Tracy is disabled. The reported error:

Fatal error: Exception: Unable to obtain lock for session (retry in 30s) (in xxx/wire/modules/Session/SessionHandlerDB/SessionHandlerDB.module line 96) #0 [internal function]: ProcessWire\SessionHandlerDB->read('dbee8f43c207f5c...') #1 xxx/wire/core/Session.php(327): session_start(Array) #2 xxx/wire/core/Wire.php(413): ProcessWire\Session->___init() #3 xxx/wire/core/WireHooks.php(952): ProcessWire\Wire->_callMethod('___init', Array) #4 xxx/wire/core/Wire.php(484): ProcessWire\WireHooks->runHooks(Object(ProcessWire\Session), 'init', Array) #5 xxx/wire/core/Session.php(205): ProcessWire\Wire->__call('init', Array) #6 xxx/wire/core/ProcessWire.php(581): ProcessWire\Session->__construct(Object(ProcessWire\ProcessWire)) #7 xxx/wire/core/ProcessWire.php(315): ProcessWire\ProcessWire->load(Object(ProcessWire\Config)) #8 xxx/index.php(52): ProcessWire\ProcessWire->__construct(Object(ProcessWire\Config)) #9 {main} in xxx/index.php on line 64

Warning: Cannot modify header information - headers already sent by (output started at xxx/index.php:64) in xxx/wire/core/WireHttp.php on line 1705

Warning: Cannot modify header information - headers already sent by (output started at xxx/index.php:64) in xxx/wire/core/WireHttp.php on line 1705
Arrgh… Error: Exception: Unable to obtain lock for session (retry in 30s) (in /wire/modules/Session/SessionHandlerDB/SessionHandlerDB.module line 96) #0 [internal function]: ProcessWire\SessionHandlerDB->read('dbee8f43c207f5c...') #1 /wire/core/Session.php(327): session_start(Array) #2 /wire/core/Wire.php(413): ProcessWire\Session->___init() #3 /wire/core/WireHooks.php(952): ProcessWire\Wire->_callMethod('___init', Array) #4 /wire/core/Wire.php(484): ProcessWire\WireHooks->runHooks(Object(ProcessWire\Session), 'init', Array) #5 /wire/core/Session.php(205): ProcessWire\Wire->__call('init', Array) #6 /wire/core/ProcessWire.php(581): ProcessWire\Session->__construct(Object(ProcessWire\ProcessWire)) #7 /wire/core/ProcessWire.php(315): ProcessWire\ProcessWire->load(Object(ProcessWire\Config)) #8 /index.php(52): ProcessWire\ProcessWire->__construct(Object(ProcessWire\Config)) #9 {main} This error message was shown because: site is in debug mode. ($config->debug = true; => /site/config.php). Error has been logged.

Everytime that error happens, the site gets unresponsive for about a minute or so. After first reload I get another error reported by Tracy:

User Error

Exception: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away (in xxx/wire/core/PageFinder.php line 821)

#0 xxx/wire/core/Wire.php(419): ProcessWire\PageFinder->___find(Object(ProcessWire\Selectors), Array)
#1 xxx/wire/core/WireHooks.php(952): ProcessWire\Wire->_callMethod('___find', Array)
#2 xxx/wire/core/Wire.php(484): ProcessWire\WireHooks->runHooks(Object(ProcessWire\PageFinder), 'find', Array)
#3 xxx/wire/core/PagesLoader.php(424): ProcessWire\Wire->__call('find', Array)
#4 xxx/wire/core/Pages.php(289): ProcessWire\PagesLoader->find('template=admin,...', Array)
#5 xxx/wire/core/Wire.php(419): ProcessWire\Pages->___find('template=admin,...', Array)
#6 xxx/wire/core/WireHooks.php(952): ProcessWire\Wire->_callMethod('___find', Array)
#7 xxx/wire/core/Wire.php(484): ProcessWire\WireHooks->runHooks(Object(ProcessWire\Pages), 'find', Array)
#8 xxx/wire/core/PagesLoader.php(1917): ProcessWire\Wire->__call('find', Array)
#9 xxx/wire/core/Pages.php(244): ProcessWire\PagesLoader->count('template=admin,...', Array)
#10 xxx/wire/core/PagesEditor.php(1877): ProcessWire\Pages->count('template=admin,...')
#11 xxx/wire/core/Pages.php(1912): ProcessWire\PagesEditor->newPageOptions(Array)
#12 xxx/site/modules/RockMigrations/MagicPages.module.php(45): ProcessWire\Pages->newPage(Array)
#13 xxx/wire/core/WireHooks.php(1051): RockMigrations\MagicPages->RockMigrations\{closure}(Object(ProcessWire\HookEvent))
#14 xxx/wir … #20 {main}

And in that lifecycle there's also this warning

PHP Warning: session_write_close(): Failed to write session data using user defined save handler. (session.save_path: /usr/share/cgi-fpm/w01d84f4/tmp/) in Unknown

The whole thing remains mysterious to me. Unfortunately I can't reproduce it on local ddev so I can't step debug it. On staging we don't have xdebug available.

Anyways thanks to everyone for looking into this. As I said above, I'll test again when the site is ready to go live. Fortunately. the error occurs only when using the InputfieldSelector in SearchEngine's module settings. When using it with regular PageFinder, everything is ok.

@netcarverunfortunately I'm now still getting that error with your module still enabled. Heres the session log:

And always shortly after that there the DB goes away. I think those are related.

16 minutes ago, netcarver said:

@gebeer Could you try my spin on the SessionHandlerDB module that I posted here, and let me know if that makes any difference?

I can confirm that with your module I do not get the error.

4 minutes ago, netcarver said:

@gebeer Could you try my spin on the SessionHandlerDB module that I posted here, and let me know if that makes any difference?

I'll give it a shot and get back asap

6 minutes ago, bernhard said:

I'm also seeing this from time to time (but only on local ddev). Do you know WHEN the error occurs? I think in my case it was while debugging via xdebug so I didn't really care.

For me the error occurs every time I try to use the InputfieldSelector field for indexing pages in SearchEngine module after choosing "template":

This is issuing an XHR Request to: https://ziehm.ddev.site/processwire/module/?InputfieldSelector=opval&field=template&type=&name=index_pages_now_selector

Hi,

in reference to https://processwire.com/talk/topic/26070-error-exception-unable-to-obtain-lock-for-session/ I need to bring this up again.

Happens on PW 3.0.208, with TracyDebugger 4.23.33. Only when SessionHandlerDB is installed.

The error

Fatal error: Exception: Unable to obtain lock for session (retry in 30s) (in xxx/wire/modules/Session/SessionHandlerDB/SessionHandlerDB.module line 96) #0 [internal function]: ProcessWire\SessionHandlerDB->read('836ab08de536179...') #1 xxx/wire/core/Session.php(327): session_start(Array) #2 xxx/wire/core/Wire.php(413): ProcessWire\Session->___init() #3 xxx/wire/core/WireHooks.php(952): ProcessWire\Wire->_callMethod('___init', Array) #4 xxx/wire/core/Wire.php(484): ProcessWire\WireHooks->runHooks(Object(ProcessWire\Session), 'init', Array) #5 xxx/wire/core/Session.php(205): ProcessWire\Wire->__call('init', Array) #6 xxx/wire/core/ProcessWire.php(581): ProcessWire\Session->__construct(Object(ProcessWire\ProcessWire)) #7 xxx/wire/core/ProcessWire.php(315): ProcessWire\ProcessWire->load(Object(ProcessWire\Config)) #8 xxx/index.php(52): ProcessWire\ProcessWire->__construct(Object(ProcessWire\Config)) #9 {main} in xxx/index.php on line 64

No error when I disable Tracy.

Sorry if this has been brought up before or even fixed already. I am not aware of that.

This only happens on the staging server, not in the local ddev environment. Both PHP 8.1., MariaDB 10.5.19.

can confirm same behaviour. Disabling TracyDebugger helps.

Fatal error: Exception: Unable to obtain lock for session (retry in 30s) (in xxx/wire/modules/Session/SessionHandlerDB/SessionHandlerDB.module line 96) #0 [internal function]: ProcessWire\SessionHandlerDB->read('836ab08de536179...') #1 xxx/wire/core/Session.php(327): session_start(Array) #2 xxx/wire/core/Wire.php(413): ProcessWire\Session->___init() #3 xxx/wire/core/WireHooks.php(952): ProcessWire\Wire->_callMethod('___init', Array) #4 xxx/wire/core/Wire.php(484): ProcessWire\WireHooks->runHooks(Object(ProcessWire\Session), 'init', Array) #5 xxx/wire/core/Session.php(205): ProcessWire\Wire->__call('init', Array) #6 xxx/wire/core/ProcessWire.php(581): ProcessWire\Session->__construct(Object(ProcessWire\ProcessWire)) #7 xxx/wire/core/ProcessWire.php(315): ProcessWire\ProcessWire->load(Object(ProcessWire\Config)) #8 xxx/index.php(52): ProcessWire\ProcessWire->__construct(Object(ProcessWire\Config)) #9 {main} in xxx/index.php on line 64

1 minute ago, flydev said:

True! 😅 I can't quantify right now how many GPU-hours it will require... we will know it when we will have an idea of the dataset size.

I would go with Weaviate because it's open-source and self-manageable if needed; it's worth noting that both of them support hybrid search (sparse and dense vector concepts), . No idea which one is the most performant, but this details on our level seem not important.

As a note, in the experiments I talked about in my previous post, I am using Qdrant as it have support for storing documents, and the most important part for me, again, you can self-manage it.

Thank you again for the insight. Very much appreciated.

@flydev wow, that is really impressive. I am far below your knowledge level atm. Thanks very much for sharing those links. Will certainly help me to brush up my knowledge.

That idea about the forum chatbot is great. But that would be quite a lot of info to ingest. for large data sets like these, would you use pinecone or weaviate or other solutions?