Re: octave hangs when starting from a directory with too many files

[Mike Miller]

On Fri, Jul 20, 2018 at 11:51:33 -0400, Clinton S wrote:
> I do some work with computational clusters (currently running a mix of
> Ubuntu 14.04 / Octave 4.0.0 and Ubuntu 16.04 / Octave 4.2.1), and I made a
> mistake (with a bash script) that produced around 150000 small files in a
> particular directory "BADDIR" while running over the course of a couple
> days.  If I called 'ls' on BADDIR, it took somewhere between 15 and 60
> minutes to run (the folder is accessed over a 10Gbps network, but there's
> still some latency).  If I tried to start octave with BADDIR as the current
> directory, then octave would hang - at least for 10 minutes; I don't think
> I tried waiting any longer than that when running it interactively.
> However, I could start octave from a different directory and then run files
> from the problematic directory without any delays or problems.  I've since
> removed all the extraneous files and can work normally again.
> 
> Is what I describe expected behavior?  I'm guessing that octave runs some
> operation similar to 'ls' when starting, perhaps to check for startup.m or
> .octaverc?  Can that be done more efficiently?

Yes, you're close. The current directory is always in Octave's load
path, meaning any .m files in the current directory can be called as
scripts or functions. All callable files in the load path are scanned by
Octave and cached internally.

I approximated your setup and got the same behavior. When I inspect
Octave with gdb, sure enough it is in the routine that initializes the
load path, which iterates over a list of the files in the current
directory.

With my setup, it took about 9 minutes for 'ls' to complete, and it took
about the same amount of time for 'octave-cli' to start and show the
prompt.

It may be possible to optimize this better for directories with
thousands of files, none of which are .m files that Octave needs to know
about. For example, in the same directory where 'ls' takes 9 minutes,
the command 'ls *.m' takes only 25 seconds to return an error.

-- 
mike

signature.asc
Description: PGP signature