Good questions RecycleRobot. This is really is going to vary from environment to environment, so my recommendations would be more general. Though awhile back Soma reported that some of his colleagues had run a stresstest called a "proxy sniffer load curves report", and the report came back quite good. But I don't know what all that entailed.
I recommend giving it 128M memory (the PHP 5.3 default) or more, in your php.ini. If you need to deal with lots of traffic, it's a good idea for your server to have a PHP opcode cache like APC or eAccelerator.
Beyond the server configuration, other performance factors depend on how you use ProcessWire. If you are doing anything heavyweight with the API towards markup generation, or just need to handle lots of traffic, it's a good idea to make use of PW's various caching options. From the template settings, you can tell it to use a cache from the 'cache' tab. Choose settings that maximize the lifetime of the cache, clearing only what you need to cleared when a page is saved. You'll also want to get familiar with the MarkupCache module, which enables you to cache individual pieces of markup in your templates. This is really handy for caching markup that may be expensive to build on every request, like larger navigation trees... or anything you want.
You'll also want to look at these best practices for long term scalability, as well as the field autojoin option. If you need to work with a lot of pages in memory at once during the same request, the $pages->uncacheAll(); function can clear memory when you need it (at the expense of having to reload any pages you try to access again).
How much memory a Page takes up depends entirely upon how many fields it has and how many are loaded/populated. ProcessWire generally tries to keep Page objects as lightweight as possible, and doesn't load fields until you actually access them. As a result, you could feasibly get thousands of pages in memory if all they had were short 'title' fields. But you might only get a few hundred if those pages have their fields loaded and all in memory at once.
PW does create a new directory for each page at present. That behavior will become on-demand in the near future. But at present, this would mean your maximum quantity of pages (at least, pages that can also store files) could be limited by the file system and any hard limits present there.
ProcessWire is not compatible with PHP 5.1. If you are running it, I'm not sure how because 5.1 lacks some things that PW depends on. I'm guessing 5.1 was a typo, but if not, let us know as I may have missed something. I recommend running PHP 5.3 if possible to maximize performance. Also, I may be wrong about this, but I had thought that there was some overhead with running PHP as a CGI as opposed to running it as a module. If that's still the case (?), you may benefit from running PHP as an Apache module, but I'm out of area of knowledge on this one.