How to handle larger amount of static data

[lisandi]

What is the best way to manage larger amounts of constantly necessary static data.

I.e.

Countries and there depending states and the cities in those states with their postal codes which needs to also translatable into different languages.

I.e.

Germany, Deutschland, Allemagne, ประเทศเยอรมัน

North Rhine Westphalia, Nordrhein Westfalen, ...

Cologne, Koeln ,...

...

Or

Mr., Miss., Mrs.,

Or

Other clearly defined lists of static data i.e for Taxes or Currencies, which would be used over and over again in several site

I.e in Search forms, mail forms, user profiles, shops, realestate lists, ....

All this data won't change over years.

One way would be to create pages the processwire way and than search for them an make a search so that a parent shows only the subpages assigned to him.

What about scalability? I.e if you have clearly defined job descriptions - about 30 TSD.

Is there already a module taking care of such kind of static data which probably get much better stored as is in a table direct in the database?

Imho it does not really make sense to create for each item a page.

What is your opinion? How do you handle that static data in your sites?

Thanks Andi

[Philipp]

We always store this in some hidden pages and then select a Page field to include them in templates. This worked so far even with hundres of entries to choose from.

In more detail, I create a template called item only with a title field and then just order them. The Page field then get the right parent as a target.

If you really have 30k entries, you might want to build a module that uses the DB directly I think.

[Pete]

There is also the table profield where you can create useful tables of data in the admin, but for truly huge (thousands and thousands of entries) datasets then a custom module with its own take fan be useful.

The table field is a bit more beginner friendly but might not do everything you want - I think in this case it would though.

I have added countries as pages in the past because they actually do change from time to time. In the UK postal boundaries can change and there are some confusing overlaps too in places so I'm sure there are other countries where these things change. It is never very often though.

I think the beauty of it is that with a small dataset like countries if you have it as pages you might just want the country names to start with but on another project you might want to add currencies and flags, capital cities etc and it's a bit more flexible. If you then cache that data on the frontend (if it was for dropdowns for country or currency) then it's not taking up any more resources than disk space on your server really since even databases are stored on disk. Caching prevents the more expensive database calls but has to be used wisely and where appropriate.

[Soma]

We faced this also at work and while we need it very rare, it's not easy to have all countries for example in all different languages. After lot of searching I found a repo on github from a guy with all the country iso names in many different languages in different formats (php, json, xml, etc)

http://dev.umpirsky.com/list-of-all-countries-in-all-languages-and-all-data-formats/

https://github.com/umpirsky/country-list

Would be possible to create a service for this, but haven't yet come to decide what would be best.

For something more extensive like postal codes etc, a service like http://www.geonames.org/

[Soma]

We also usually import such data as pages. Easy even with thousands entries of data. Depends a lot on the need and quantity. Usually also a xml of csv on server could be considered.

For things like Mr and Mrs we created pages that can be made to radio options using page field. Easy to create an maintain and translate.

There's a currency conversion module by Ryan. http://modules.processwire.com/modules/service-currency-conversion/

It's always possible to create a kind of a static fieldtype with stored data in whatever form (db, xml etc). 

Having pages as metadata has several benefits as you can use API and features already there, and scalability isn't a real issue unless you go into couple millions of data entries, where php mysql is slowly reaching a limit to handle it fast.

[lisandi]

I thought about perhaps reusing the TYPO3 static_info_table (and there are several other tables for other purposes like taxes airports etc - and they get maintained by them quite good.

https://forge.typo3.org/projects/extension-static_info_tables
https://github.com/typo3-ter/static_info_tables
https://github.com/typo3-ter/static_info_tables/blob/master/ext_tables_static%2Badt.sql