Hi guys, just a quick follow-up:
I now have imported the 4000+ users in PW; using the "normal" user template, spiced up with the custom data fields I need for the user-data stuff from the old forum.
This worked, and the lister instance which does put out the user data in the backend is running fine and without problems.
So I have no idea what exactly the difference is, since I copied my custom user template off the internal "user" template - but this way now, everything works smoothly.
update: (just an aside: i have currently 40,000 posts imported as pages and no hickup in the backend.) So I'll mark this as resolved, since this is not a performance issue of the lister, but something about the combination of a large amount of users, custom-template, and the lister.