Jump to content


Photo

Importing data (dictionary purpose), how to connect fields together ?

database

  • Please log in to reply
3 replies to this topic

#1 gilles

gilles

    Newbie

  • Members
  • Pip
  • 2 posts
  • 0

  • LocationFrance

Posted 31 January 2012 - 01:10 PM

Hello !

First post ;)
I am interested in PM wich could be a pertinent tool very close to what I need (because of custom fields and template).
Demo file installed and working. It looks sincerely interesting and attractive.
As of now, I'm confused about many things (mostly how to connect data).

My purpose is to run an online dictionary, quite big. Same kind of work as Skyscrappers demonstration (many data connected, many fields working as filters) I guess.
My online site is designed this way (quick resume) :

table words (55000 items) :
id_word
word
variant
meaning

table quotes (200000 items) :
id_quote
id_word
quote
year_quote

A few other fields and tables (synonyms...) ; but it's enough for my questions.

What would be the classic/usual strategy to get this data run by PW (I will not create the data inside PW : I use microsoft Access daily, then I update the online database once a year, more or less) ?
Do I have to create a field by myself for every information taken from all tables ? I guess yes ?
What about the id fields ? In my current database, they are used for connecting the relevant data together and I need to keep them accessible and usable them to get final url like ..../word/23445 (where 23445 is id of word 23445 of course).

Say I create several fields (word, variant, meaning, quote, year_quote) :
then what to do with my id fields ?
then how does PW connect the data ?

I may be unclear, sorry for that. If needed, please tell me what to correct.
I'm not asking light about how to show informations (templates) --will come next--, but how to import data and how to get working connections (more database related).

Thank you :)

#2 ryan

ryan

    Reiska

  • Administrators
  • 7,797 posts
  • 6572

  • LocationAtlanta, GA

Posted 01 February 2012 - 10:51 AM

Welcome to the ProcessWire forums Gilles. I think that PW will work quite well for what you are talking about. While I've not pushed it into the arena of 200k+ pages, I think it should scale nicely.

What would be the classic/usual strategy to get this data run by PW (I will not create the data inside PW : I use microsoft Access daily, then I update the online database once a year, more or less) ?


You'll want to create the fields in ProcessWire, and I would suggest using the same names that you already are in your tables. You will create 2 templates: "word" and "quote". I'm assuming that multiple quotes can reference the same id_word. Unless your words or quotes can be categorized into single categories of some sort, you'll want to bucket them into two parents: /quotes/ and /words/. In your "quote" template, id_word and id_quote will be Page reference fields (select type "Page" when creating a new field). I'd suggest using the Autocomplete Inputfield for them, giving the scale. But using the page reference fields is the manner in which you'll be able to draw relationships between words and quotes.

Since you'll only be updating the online database rarely, you'll want to use the API to handle your import (I'll be happy to follow-up with code examples if you are interested). I would suggest bootstrapping ProcessWire with a command line PHP script (or create a in import page/template). I don't think you'll be able to use the ImportPagesCSV module for this unless you break it down into smaller CSV files. Importing 200k+ records is likely to prevent you from doing any kind of 1-step import due to memory limitations. But when you use the API, you can wire('pages')->uncacheAll(); regularly to clear out pages in memory to make room for more, and this is a good thing to know about whenever doing any kind of large import from the API. But PHP is not MySQL or Access, and you may still find yourself having to limit the import to a few thousand items at a time, depending on how PHP is configured.

Keep us up to date how it goes with this, it sounds like quite an interesting scale and application.

Thanks,
Ryan

#3 gilles

gilles

    Newbie

  • Members
  • Pip
  • 2 posts
  • 0

  • LocationFrance

Posted 03 February 2012 - 06:25 PM

Hello Ryan and thank your for your welcome advices.

I'm playing with PW and trying to get it to work according to my needs and (most important) my limited knowledge.
Will play more and, if interested to get involved, come back later for more practical questions.
You show me the me the right direction, fine ; I need to do testing on little data to understand better how it works and how I can use it.
Custom fields and templates are usefull.

(not 200k pages, in my case : main pages are for words --55k-- ; connecting to quotes and more, sort of dictionary).

Gilles.

#4 apeisa

apeisa

    Hero Member

  • Moderators
  • 3,154 posts
  • 1705

  • LocationVihti, Finland

Posted 03 February 2012 - 06:41 PM

Welcome Gilles

(not 200k pages, in my case : main pages are for words --55k-- ; connecting to quotes and more, sort of dictionary).


Almost every item on pw is a "page". You would actually create pages for your quotes also, so that is why Ryan told that 200 000+ pages. You could compare it with Drupal node or single row in database table.





Also tagged with one or more of these keywords: database

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users