Looks like the "Add links to edit previous and next pages" feature results in all sibling pages being loaded - this can be problematic if there are lots of siblings. As soon as I disable that feature it drops down to ~100 pages loaded.