Re: ls, mv, rm and directories with over ~13000 files.

[Bob Proulx]

> I'd like to know where the limit in the ability of ls, mv and rm to handle
> more then approximately 13000 files is from. Is it from the shell (i.e.
> bash)? Is it an arbitrary number that can be changed? 

See the (recently posted) FAQ for an answer to your question.

  
http://www.gnu.org/software/fileutils/doc/faq/core-utils-faq.html#Argument%20list%20too%20long

> We do wearable computing research and thus generate a lot of images (e.g.
> 30fps * 60secs * 60min = 108000 files for an hour). Any help would be
> appreciated.

Depending on the filesystem of course, but traditional filesystems
slow down tremendously when the number of files in any single
directory level grow beyond a couple of thousand.  Directories
traditionally have been implemented as linear lists and therefore
requiring linear searches leading to long directory access times when
there are a large number of files in a directory.

Some newer filesystems implement directories as B-trees instead of
linear lists and avoid this.  But if I were you I would avoid the
problem by design in user space and build a hierarchy of names such
that no single directory can grow too large.  A simple example of this
is /usr/share/terminfo (aka on older systems /usr/lib/terminfo) which
do a radix encoding using the first letter.  Not sufficient for your
case but should give an idea of one way to proceed.  Forewarned is
forearmed.

Bob