Max number of pages before switching to custom tables?

[joer80]

I am planning out a project that will contain a lot of data imported from a csv file, but each row will need its own page.  

I want to be able to use the api if I can and not go an extra custom table route.  My top level is going to be very small, just some of my categories may be large.

Is there a total number of pages that you can hit that makes adding custom tables for the data a better option than importing the content as new pages?

Thanks!

[Martijn Geerts]

Can you tell what a huge amount of pages is?

[joer80]

My current project brought this question up, but I don't think it is going to be too bad, so I guess it's more of a hypothetical question so I know how to structure future projects!  

Maybe a newspaper would be a good example?  To move to a new cms, they would need to import pages by the hundreds of thousands I would imagine?

[diogo]

Welcome to the forum joer!

I found these to threads that may be useful to answer your question:

https://processwire.com/talk/topic/2503-question-about-extreme-scale-hundred-of-thousand-maybe-millions-of-pages/

https://processwire.com/talk/topic/4349-thousands-of-pages-one-site-with-multiple-pages-or-one-install-with-multiple-sites/

[netcarver]

Until recently, there has been a limit to the number of pages with images that PW can handle due to the OS per-directory file limit. However, the current development branch has changes in it that removes this previous limitation and effectively allows you to work to the full capacity of the file system. Previous limit was around 32,000 images on ext3 filesystems (if memory serves) but the dev changes blow that out of the water. Hundreds of thousands of images should now be possible but I have not verified this yet.

With regard to pages, MySQL is able to handle millions of rows in tables, so I think you should be fine as long as the queries don't try to pull a lot of rows (pagination is your friend) with a lot of fields in each row.

Edited to add: Beaten by diogo - again. 

[diogo]

Edited to add: Beaten by diogo - again. 

Your answer is much better. I don't think those threads mention the changes on the dev version

[joer80]

So taking the newspaper example, images would probably be stored in day folders.

domainname.com/content/uploads/pictures/2014/05/image.jpg

So you would never run into a directory limit because that would be a lot of photos in one day.

How does process wire do the default image save path?

(I think wordpress does a day based path based on the page upload date?)

[diogo]

Images are stored in a folder associated with one page.

[joer80]

Good to know!  Should not be any problems there!