To expand on Teppo’s point about very large database extractions, we’ve used a tool called FACT (Fast Extract) from IRI to dump multi-million row tables from RDBs into CSV and other kinds of flat files. On those we run scripts (with another tool from that company called CoSort) to query, reorganize, mask, and do what we need to do with that data without burdening the DB with lookups. Basically we found that flat files are just a more manageable environment for a lot of things, especially bulk data movement and manipulation, a la data warehouse transforms.
