[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[PATCH][simple-md5] Enable hashing of big files
From: |
lou |
Subject: |
[PATCH][simple-md5] Enable hashing of big files |
Date: |
Mon, 16 Dec 2024 01:19:06 +0100 |
Hi,
Here's the promised follow-up to my earlier memory-mapped-files
patch.
Enabling simple-md5 to digest big files by splitting files that
are bigger than what the C implementation can digest into chunks
and recuring over them.
Two notes:
1. This implementation is really only useful for posix
environments. The alternative `mapped-pointer` implementation
for Windows means, as far as I understand, that the Windows
version is limited by physical memory. I don't have a Windows
machine, someone who does should do some testing on whether
the divergence is needed. Nominally `memory-mapped-files`
implements a mmap equivalent for Windows, so it should be
possible to just use the posix code unconditionally.
Putting the `mapped-pointer` call inside the recursion,
mapping the individual chunks, would work around this as
well, but result in an unnecessary amount of mmaps on posix
systems.
2. I hope I got the types right. I got a bit confused because
`unsigned-integer` seems to only accept values up to INT_MAX
not up to UINT_MAX, which wasn't clear to me from the
documentation here:
http://wiki.call-cc.org/man/5/Foreign%20type%20specifiers#integers
I'll take a look at simple-sha1 in a couple of days, I suspect it
would benefit from a similar patch but I haven't checked that out
yet. Hope this is useful to some people.
~~Lou
00-chicken-simple-md5-hash-big-files.patch
Description: Text Data
- [PATCH][simple-md5] Enable hashing of big files,
lou <=