Big data issue

[rusjoan]

Hi!

This topic is not about a real big data, but nonetheless PW can't hold it

I created a template ("task") with couple of text fields and one repeater field. This repeater has 3 text fields and 2 textarea fields.

There are 2374 pages using this template. When I've decided to restructure my DB I went in a real trouble.

Restructuring is in that I make new template with all of mentioned repeater fields and I remove 2 "massive" textarea fields from repeater. Then task-template pages should become smaller in DB and I'll get ability to do stats analysing faster, and task page still can get 2 textarea fields values from another page if needed.

I moved all of mentioned 2 fields data to new pages then tried to remove these fields from repeater.

But here's what I've got:

Error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 81 bytes)  

Then I launched script to clear values in these fields (set to ""). This was slow and boring but it's done.

And after new attempt I've got the same error.

My suggestion is: script, that removes fields, tries to do $pages->find() for all of pages that use given field. But data size from DB is too big to locate in allowed RAM — textarea uses MEDIUMTEXT innodb type what's about 16 Mib. This case is why I made my script to clear fields values by iterations of 10-50 pages to avoid memory overflow.

How I can remove these fields? The only solution is to temporarily give PW more RAM by performing ini_set( )?

[diogo]

Maybe this helps?

https://processwire.com/talk/topic/2038-distinct-selector-in-processwire/?p=43720#entry43720

[rusjoan]

Maybe this helps?

https://processwire.com/talk/topic/2038-distinct-selector-in-processwire/?p=43720#entry43720

Good solution, thank you.

But my trouble is a bit different...

[WillyC]

Restructuring is in that I make new template with all of mentioned repeater fields and I remove 2 "massive" textarea fields from repeater. 

they.not massive unless you fill. thems up

textarea with.5k text takes up 5 kilobites not 16 megobytes

Then I launched script to clear values in these fields (set to ""). This was slow and boring but it's done.

this do.nothing 

see above

How I can remove these fields? The only solution is to temporarily give PW more RAM by performing ini_set( )?

you see this -> dev commit

mabe u need dev branch ?

[Soma]

I removed a field on template for 40k pages. It took 4 seconds. This was on new dev as willi mention. Before it took 10 + minutes.

[rusjoan]

Woohoo! Thank you guys, now it works!

I simply copied those files of mentioned commit because I have no idea how to work with 2 repos in one git-project while all of your dirs except "wire" should contain only your data but not from PW-repo. Maybe anyone can tell me about the way to keep PW up-to-date?

Then I launched script to clear values in these fields (set to ""). This was slow and boring but it's done.

this do.nothing 

see above

@WillyC there were most of pages with filled fields