Re: [lmi] [lmi-commits] master 9c510ad 16/22: Measure elapsed time for MD5 data-file validation

[Greg Chicares]

On 2020-03-30 00:34, Greg Chicares wrote:
> On 2020-03-29 17:58, Vadim Zeitlin wrote:
[...]
>>  In fact, the main reason for writing this reply is that I'm surprised by
>> how long it takes to compute the hash. Using "openssl speed md5", I get
>> ~300MB/s for 64 byte blocks and while I don't know how big the files we're
>> verifying are exactly, I'm pretty sure that they're nowhere close to 30MB
>> in size. So I wonder if it could be useful to profile the code doing this
>> calculation to check if we're not doing something stupid?
> 
> Sounds worthwhile.

Here's evidence that suggests our self-built standalone md5sum program
is as fast as the one in gnu coreutils. Summary:

  clock
   time  program used with "maximal" dataset
  ----- ------------------------------------
  0.194 wine lmi_md5sum.exe [i686]
  0.190 wine lmi_md5sum.exe [x86_64]
  0.200 wine md5sum.exe [i686 binary from the following URL]
    
https://github.com/vadz/lmi/releases/download/new-cygwin-makefiles/md5sum.exe
  0.085 debian native md5sum

so it looks like most of the elapsed time is 'wine' overhead
(which of course wouldn't exist for msw end users).

Details:

These measurements use the "maximal" dataset described in my earlier
email, i.e., a current production distribution including every known
product, all obfuscated MST files, and the no-longer-needed md5sum
binary. As discussed in private email, I've shared [obfuscated] copies
of all these 347 files with a distinctive '.xyz' suffix.

# wine, self-built msw program, i686
/opt/lmi/data[0]$echo $LMI_TRIPLET
i686-w64-mingw32
/opt/lmi/data[0]$time wine ./eraseme_lmi_md5sum.exe *.xyz |wc -l
347
wine ./eraseme_lmi_md5sum.exe *.xyz  0.12s user 0.05s system 86% cpu 0.194 total
wc -l  0.01s user 0.00s system 3% cpu 0.194 total

# wine, self-built msw program, x86_64
/opt/lmi/data[0]$LMI_TRIPLET=x86_64-w64-mingw32
/opt/lmi/data[0]$. /opt/lmi/src/lmi/set_toolchain.sh
/opt/lmi/data[0]$time wine ./eraseme_lmi_x86_64_md5sum.exe *.xyz |wc -l
347
wine ./eraseme_lmi_x86_64_md5sum.exe *.xyz  0.08s user 0.08s system 86% cpu 
0.190 total
wc -l  0.01s user 0.00s system 4% cpu 0.189 total

# wine, historical downloaded msw program, i686
/opt/lmi/data[0]$time wine /cache_for_lmi/downloads/md5sum.exe *.xyz |wc -l
347
wine /cache_for_lmi/downloads/md5sum.exe *.xyz  0.14s user 0.03s system 84% cpu 
0.200 total
wc -l  0.00s user 0.00s system 1% cpu 0.200 total

# debian-testing native, x86_64
/opt/lmi/data[0]$time md5sum *.xyz |wc -l
347
md5sum *.xyz  0.07s user 0.01s system 99% cpu 0.085 total
wc -l  0.00s user 0.00s system 8% cpu 0.085 total