Re: [Sks-devel] sks (fast)build memory/cache problem

[Jeffrey Johnson]

On Jul 1, 2012, at 4:31 PM, Kristian Fiskerstrand <address@hidden> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> On 2012-07-01 22:26, Stephan Beyer wrote:
>> Hi,
>> 
>> On 01.07.2012 04:56, Brian D Heaton wrote:
>>> Having beat my forehead on this one a few weeks ago, I can offer
>>> the following suggestions from JohnC that got me on the right
>>> track:
>>> 
>>> Note:  I used the 5K/file keydump for the initial build.
>> 
>> I tried several parameters like * keys/file (7.5k, 20k, 5k) * bdb
>> version (5.1, 5.3) * -n * -cache * pagesize / ptree_pagesize
>> 
>> Nothing helped.
>> 
>> Perhaps I'll try it one last time with a some 4.x version of bdb.
>> 
> 
> I'd recommend to try 4.6 or 4.7, at least working for me without any
> issue. I'll throw up some development boxes with BDB 5.x and do some
> testing, although I'm quite sure John already is using this.
> 

I think chasing after ancient versions of Berkeley DB
is unnecessary: there aren't any serious issues reported
in the implementation reported (but this is not a claim
that it isn't a serious PITA debugging Berkeley DB).

I have never succeeded with a "build" with any version of Berkeley DB,
most recently tried about a month or so ago because of positive
reports from John. I'd suggest trying the pagesize adjustments
        pagesize:       128 # Use 64k pages for KDB
        ptree_pagesize:  16 # Use  8K pages for PTree
which may well be why my "build" doesn't work and John's does..

I've also seen issues with choosing cache sizes on limited
memory machines (1-2GB). These days I multiply the default
parameters by 100 on 8-16GB machines and "fastbuild" runs well.

On smaller machines (1-2Gb of memory), I've also resorted to
removing some of the dump files, and then bootstrapping
off another key server I run. This can take 12+ hours
and is a serious load if you have to resort to a smaller
number of dump files. Send me private mail if you want
to try to use one of my servers like this … not recommended
but "works".

> Out of curiosity, what is the source of the BDB install? And is it
> configured with pthread support?
> 

Before chasing after a will-o-the-wisp:

        Opening a Berkeley DB dbenv with DB_PRIVATE disables
        all locking: there should not be any pthread_mutex issues
        with all locking disabled.

Find the DBENV->open() call and mask on DB_PRIVATE to the 4th "flags"
parameter, rebuild SKS, load the dumps, then undo the 1-liner hack is
one way to try to get KDB/PTree created (without worrying about pthread_mutexes)

hth

73 de Jeff
> 
> - -- 
> - ----------------------------
> Kristian Fiskerstrand
> http://www.sumptuouscapital.com
> Twitter: @krifisk
> - ----------------------------
> Corruptissima re publica plurimæ leges
> The greater the degeneration of the republic, the more of its laws
> - ----------------------------
> This email was digitally signed using the OpenPGP
> standard. If you want to read more about this
> The book: Sending Emails - The Safe Way: An
> introduction to OpenPGP security is now
> available in both Amazon Kindle and Paperback
> format at
> http://www.amazon.com/dp/B006RSG1S4/
> - ----------------------------
> Public PGP key 0xE3EDFAE3 at http://www.sumptuouscapital.com/pgp/
> 
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.19 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> 
> iQIcBAEBCAAGBQJP8LOMAAoJEBbgz41rC5UIhaYP/i3+Ykck2ez8S9z3zrJoTaPE
> IRt8pzyvMamzWi0Q8aFeTpZzOxhFU+6ooMUfLzugqxkB2z9gUIOMjcLrz94uHKPL
> as7NpDtM8VFeJdEGzTVOKMN0vEixrecj3fbANT3wgVu0dtiUQLN7adNqcBXXPssW
> qRjkq0B181Tn7LLi3vOaU7Ew9Ucpjh3GMaz0TmDIIYfA4LTZiGS3tFtN3DcwyLOr
> olVIcZhxy7x/NbiehbvoVAsRGb6JPLg4h8u/0H3ARUYIZ9BdfD1W6PIWaDTL/Elv
> J5rvrOglu648Y9NN91tNIFbDuVxd7TnGUriTiiEEnACAvOLwGMFLVAVzfX7yAZSn
> PWVSq1F9qSg9oDY3jDN/PpF2t326+F6pV+6e85/guEa0LmIn04scfML5E27w0MO+
> 5yyhowjtpExTRCp3M9gWiKZ+XvyRDdhC7u290M+8097RwLIq5C7cdUyzypK9A3PQ
> uEXFzSXniL/+VjTnqk/BJS+9MUtoiVZJm1uLJW0LlEMI+S6/Q+X6oEzwOhPw7gbM
> ruCrEX180ZxnyIWvy0rdw2JJrjuRr1zPaRiENwnIRTKQmxg8lQf9IXuG1beF9z6x
> yfjX1RxS9tJcTLGueUURUii7Icua/1SdQOv5K5zsbAA313kQLkfqxORQmDd4nwvN
> uWmn70ptbzv74MUpAPdX
> =j3wG
> -----END PGP SIGNATURE-----
> 
> 
> _______________________________________________
> Sks-devel mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/sks-devel