Re: [Bug-gnupod] mktunes.pl creates corrupt iTunesDB ?

[Heinrich Langos]

Hi Richard,

Do you have any three iTunesDB's that differ in only one song being added 
from one to the next?

I'd like to have a chain like this:  ok -> ok -> broken

Comparing the first two would show which changes are to be expected.
Comparing the later two would (hopefully) show where it breaks down.

>   <file addtime="3327445787" album="Love, Luther - Disc 1"
> artist="Luther Vandross" artworkcnt="1" bitrate="205" cdnum="0" cds="0"
> comment="" composer="" dbid_1="ca1683ce8c7195e7"
> desc="/d01/mp3/marcel/Luther Vandross/Love, Luther/Luther Vandross - 05
> - Never Too Much (Extended Version).mp3" fdesc="MPEG 1 layer 3 file"
> filesize="8762139" genre="Soul" has_artwork="1" id="11210" mediatype="1"
> path=":iPod_Control:Music:f04:g0__Extended_Version_.mp3" playcount="0"
> songnum="5" songs="0" soundcheck="2661" srate="44100" time="340532"
> title="Never Too Much (Extended Version)" volume="0" year="2007" />
> 
> When this file is present, the iTunesDB is not recognized. When it is
> not in the GNUtunesDB.xml, the iTunesDB works fine up until file numer
> 11531 (id="11558"):

Does this file have anything in its attributes that the files that you
added successfully (until #11531) did not have? artwork maybe? 

>   <file addtime="3327446393" album="Collected" artist="Massive Attack"
> artworkcnt="1" bitrate="156" cdnum="0" cds="0" comment="" composer=""
> dbid_1="261883ce8c7195e7" desc="/d01/mp3/marcel/Massive
> Attack/Collected/Massive Attack - 06 - Protection.mp3" fdesc="MPEG 1
> layer 3 file" filesize="9100998" genre="Electronica" has_artwork="1"
> id="11558" mediatype="1"
> path=":iPod_Control:Music:f47:g0____06___Protection.mp3" playcount="0"
> songnum="6" songs="0" soundcheck="5408" srate="44100" time="465423"
> title="Protection" volume="0" year="2006" />
> 
> The weird thing is, if I put these "broken" files by themself in
> GNUtunesDB.xml it does work (iPod shows 1 song and plays ok).
> 
> I'm kind of lost how to fix this, or to predict which files will cause
> problems. Any ideas anyone?

Could you try to add the SAME file over and over until you get a non 
working state? That would make hunt for an overflow in the internal
structures even easier. (gnupod_addsong "-d" allow duplicates).

Best way to get there is a binary search. Let A be the number of files 
in a working state and B the number of files in a non working state. 

Now take the middle of A and B and test with that number of files. 
If it works, let A be that number, if it doesn't work let B be that number. 
Repeat that until A+1 == B.


> I've saved the GNUtunesDB.xml and iTunesDB during my testing here:
> http://richard.vdberg.org/gnupod/
> The directory names are the number of <file> entries in the
> GNUtunesDB.xml. The ones in the "ok" top dir work on the iPod, the ones
> in "not_ok" do not.
> 
> At first I thought it might be a size limitation of the iTunesDB (around
> 16 million bytes) but this theory does not really hold:
> 

I think there's been a similar issue a while back that ocurred when writing
some huge number of files to an ipod. I don't think it has been resolved
back then, but maybe loking at the archived thread can give you a clue on 
how to proceed with your bug hunt.

> Maybe it doesn't have to do with the iTunesDB size but a limit of the
> internal structures?

Sounds reasonable.

> I used the unaltered mktunes.pl from CVS (to make sure my title patch
> did not cause the issue). I have a 5th generation iPod Video.

Did you try adding your collection with gtkpod? I assume it will be much
faster as it is not written in perl.

Talking about time and performance and your adding of your collection 
taking 23 hours.... that probably depends a great deal on how you add 
your files. The raw copy time (assuming that your device can be written at
20mb/s) would take something like 3 hours and 20 minutes. But each time you 
run gnupod_addsong, you start a new perl process that parses the XML file 
once and writes it once. I guess you add your collection album (directory)
wise? So one gnupod_addsong process for each directory? Or do you add each
file separately? That would be way slower.

To get some numbers on the performance you could do me a favor and add a 
single file to the rather full ipod while profiling the perl process.
This can be done using:

perl -d:DProf /path/to/gnupod_addsong.pl /path/to/file.mp3

Please make sure that you are in a directory that has plenty of space as the
perl profiler will write a "tmon.out" file of several megabytes.
If you then run "dprofpp" this will take the tmon.out file and print a nice
table telling you where your program spent all its time.

After the first run please move the tmon.out file aside and run the same 
perl -d:DProf /path/to/gnupod_addsong.pl /path/to/file.mp3
line once more. (This time gnupod_addsong should abort before writing the
file to your ipod because of the duplicate detection.)

I'd like to see both outputs of "dprofpp" and, if you manage to gzip them
small enough, I'd also like to have those tmon.out files.

I suspect that most of the time will still be spent in GNUpod::XMLhelper::mkfile
and GNUpod::XMLhelper::xescaped eventhough I already did some work on that a
while back.

cheers
-henrik