Re: [Fle3-dev] Import performance

[Tarmo Toikkanen]

> I'm in the process of importing a fle3 system to a newer version. The
> dump is not huge (6 MB), and it has already been at it for some 20
> minutes. The machine I'm doing this on is a p3-933MHz. Now consider
> we have a fle3 system that has a thousand  user, and a million
> messages. Importing this will take days on even the fastest computer
> in the world.
>
> Does anyone have insight on why ImportExport manages to spend such
> ridiculous amounts of time importing stuff?

Importing data to a new Fle3 is done one object at a time. And creating 
each note takes time, as we currently create over a dozen DTML objects 
for each Note object.

I agree that this is much too slow, but I'm not sure there's much we 
can do about it, except perhaps restructure the entire knowledge 
building so that each note doesn't need its own dtml files. This would 
definitely be a change for the better, but it might prove difficult...

I tried this spike once at the beginning of the year, and ran into 
problems... If you replace the dtml files with single copies up in the 
acquisition tree, everything works as long as you have different names 
for all dtml files in the tree (notes, contexts, courses). This means 
that index_html cannot be handled like this. And if we use a python 
method as a mask, we run into weird problems with dtml files calling 
other dtml files. When the chain gets longer than 2 or 3 calls, the 
context object gets dropped in certain cases and things break down.

But apparently we should try to do this again. Of course, if anyone has 
any suggestions...

--
Tarmo Toikkanen
http://www.iki.fi/tarmo/
The Fle3 Team
http://fle3.uiah.fi