This does suggest to me we should probably change the internal API
so sparse message ranges are handled better; right now all of the
programs access the folder structure members directly and assume that
there will be a msgstat structure in every location in the array.
Sigh. One more thing to add to the list.
If I understand the problem correctly:
a folder with the highest message number of "N" will cause the array to be configured to support N messages, even if there are many fewer (perhaps even one) messages
stat()ing every file in a folder to make sure it is a file (message) instead of a directory (folder) is very expensive (and harms the performance of other programs where this isn't important and is thus a no-go)
I assume we want "close enough" scaling, not perfect. Would not the following work well enough?
Scale the array based upon the number of directory entries in the folder. This will over commit due to subfolders being counted, scratch files, and deleted messages. It seems this would only over commit in interesting cases by 3x (baseline of 1 covers the messages, the 2nd set is scratch files and deleted messages, and 3 is subfolders). Short of malicious actions, you'd end up with, maybe 5x (message, extracted parts of the message, deleted message, folders that look like message numbers). If you want more compactness, you take pains to dump the stuff that isn't a message number (the aforementioned extracted parts and deleted messages).
Or am I missing something about filesystem internals?