How to handle large amounts of data?

[Chris Rosenau]

Problem:

I want to have states and each state I want to have cities. I really don't want to make a page for each city as it is very very time consuming the way the system is designed. There are hundreds of cities for each state. So the module field type select seems like an ok option, but I read that it isn't normalized so data could get corrupt. I also wonder about limited ability to search that data with current built in functions like $page->find.

Can anyone give me advice on how to deal with large amounts of data?

Thanks!

[Macrura]

you could import from a csv into pages

if you're just using them as selects for example you could store all of the data in 1 text field.. and then use php to make the list into options

you could also consider storing in a static file, xml or json, or xml/json in a field

it really depends on what you need to do with this data

[Martijn Geerts]

I would go the pages & API route, it's very quick!

You could make a big array & loop it:

$cities = array(
    'Albany, NY',
    'Albuquerque, NM',
    'Anchorage, AK',
    'Annapolis, MD',
    'Atlanta, GA',
    'and',
    'continue',
    'the',
    'list',
    .....
);

$parent = $pages->get('/cities/'); // assumed that you have a cities parent.

foreach($cities as $city) {
    $p = new Page;
    $p->parent = $parent;
    $p->template = 'city'; // choose a valid template for this
    $p->title = $city;
    $p->save();
}

(written in the browser, but simular concept I have used often)

[adrian]

Or you could try this module:

http://modules.processwire.com/modules/process-page-field-select-creator/

You can easily paste in a CSV list using "Option 2. MULTIPLE FIELDS".

Not sure if it does exactly what you need, but worth a look. Otherwise, Martijn's approach can be customized to whatever your needs are,

[Chris Rosenau]

Thanks everyone, those are awesome answers!