Importing data (dictionary purpose), how to connect fields together ?

[gilles]

Hello !

First post

I am interested in PM wich could be a pertinent tool very close to what I need (because of custom fields and template).

Demo file installed and working. It looks sincerely interesting and attractive.

As of now, I'm confused about many things (mostly how to connect data).

My purpose is to run an online dictionary, quite big. Same kind of work as Skyscrappers demonstration (many data connected, many fields working as filters) I guess.

My online site is designed this way (quick resume) :

table words (55000 items) :

id_word

word

variant

meaning

table quotes (200000 items) :

id_quote

id_word

quote

year_quote

A few other fields and tables (synonyms...) ; but it's enough for my questions.

What would be the classic/usual strategy to get this data run by PW (I will not create the data inside PW : I use microsoft Access daily, then I update the online database once a year, more or less) ?

Do I have to create a field by myself for every information taken from all tables ? I guess yes ?

What about the id fields ? In my current database, they are used for connecting the relevant data together and I need to keep them accessible and usable them to get final url like ..../word/23445 (where 23445 is id of word 23445 of course).

Say I create several fields (word, variant, meaning, quote, year_quote) :

then what to do with my id fields ?

then how does PW connect the data ?

I may be unclear, sorry for that. If needed, please tell me what to correct.

I'm not asking light about how to show informations (templates) --will come next--, but how to import data and how to get working connections (more database related).

Thank you

[ryan]

Welcome to the ProcessWire forums Gilles. I think that PW will work quite well for what you are talking about. While I've not pushed it into the arena of 200k+ pages, I think it should scale nicely.

What would be the classic/usual strategy to get this data run by PW (I will not create the data inside PW : I use microsoft Access daily, then I update the online database once a year, more or less) ?

You'll want to create the fields in ProcessWire, and I would suggest using the same names that you already are in your tables. You will create 2 templates: "word" and "quote". I'm assuming that multiple quotes can reference the same id_word. Unless your words or quotes can be categorized into single categories of some sort, you'll want to bucket them into two parents: /quotes/ and /words/. In your "quote" template, id_word and id_quote will be Page reference fields (select type "Page" when creating a new field). I'd suggest using the Autocomplete Inputfield for them, giving the scale. But using the page reference fields is the manner in which you'll be able to draw relationships between words and quotes.

Since you'll only be updating the online database rarely, you'll want to use the API to handle your import (I'll be happy to follow-up with code examples if you are interested). I would suggest bootstrapping ProcessWire with a command line PHP script (or create a in import page/template). I don't think you'll be able to use the ImportPagesCSV module for this unless you break it down into smaller CSV files. Importing 200k+ records is likely to prevent you from doing any kind of 1-step import due to memory limitations. But when you use the API, you can wire('pages')->uncacheAll(); regularly to clear out pages in memory to make room for more, and this is a good thing to know about whenever doing any kind of large import from the API. But PHP is not MySQL or Access, and you may still find yourself having to limit the import to a few thousand items at a time, depending on how PHP is configured.

Keep us up to date how it goes with this, it sounds like quite an interesting scale and application.

Thanks,

Ryan

[gilles]

Hello Ryan and thank your for your welcome advices.

I'm playing with PW and trying to get it to work according to my needs and (most important) my limited knowledge.

Will play more and, if interested to get involved, come back later for more practical questions.

You show me the me the right direction, fine ; I need to do testing on little data to understand better how it works and how I can use it.

Custom fields and templates are usefull.

(not 200k pages, in my case : main pages are for words --55k-- ; connecting to quotes and more, sort of dictionary).

Gilles.

[apeisa]

Welcome Gilles

(not 200k pages, in my case : main pages are for words --55k-- ; connecting to quotes and more, sort of dictionary).

Almost every item on pw is a "page". You would actually create pages for your quotes also, so that is why Ryan told that 200 000+ pages. You could compare it with Drupal node or single row in database table.

[RyanJ]

Hey Ryan, 

I was wondering if I could take you up on your offer about sharing some code examples on using the api to import. I read this topic here but I am a bit confused where the wire('pages')->uncacheAll(); would go. I currently use the csv import but as you mentioned, I am limited to about 1500 records at a time.

Thanks