Re: [Sks-devel] sks (fast)build memory/cache problem

[Daniel Austin]

On 07/01/12 21:31, Kristian Fiskerstrand wrote:
> On 2012-07-01 22:26, Stephan Beyer wrote:
>> Hi,
> 
>> On 01.07.2012 04:56, Brian D Heaton wrote:
>>> Having beat my forehead on this one a few weeks ago, I can offer
>>> the following suggestions from JohnC that got me on the right
>>> track:
>>>
>>> Note:  I used the 5K/file keydump for the initial build.
> 
>> I tried several parameters like * keys/file (7.5k, 20k, 5k) * bdb
>> version (5.1, 5.3) * -n * -cache * pagesize / ptree_pagesize
> 
>> Nothing helped.
> 
>> Perhaps I'll try it one last time with a some 4.x version of bdb.
> 
> 
> I'd recommend to try 4.6 or 4.7, at least working for me without any
> issue. I'll throw up some development boxes with BDB 5.x and do some
> testing, although I'm quite sure John already is using this.
> 
> Out of curiosity, what is the source of the BDB install? And is it
> configured with pthread support?

I'm using FreeBSD 9-STABLE (with SKS 1.1.3 from ports) linked against
BDB 5.2.42 (with pthread).

Today I imported keys (5k each) using the following in my sksconf file:


# Tweak DB page sizes
pagesize:       128
ptree_pagesize: 16

and the following commands:

sks build dump/*.pgp -n 4 -cache 128
cp DB_CONFIG KDB/
sks cleandb
sks pbuild -cache 2 -ptree_cache 70
cp DB_CONFIG PTree/

If I increase the cache settings, it causes a segfault.

After the build and pbuild, I put the following in the DB_CONFIG file in
the KDB and PTree directories respectively:

mutex_set_max   262144

I've also re-imported keys on a 2nd FreeBSD server using the same
details as above.

Using a single 3.07GHz Xeon core, here's the timings:

1  mins 39 secs - build command
0  mins 27 secs - cleandb command
18 mins 46 secs - pbuild command

20 mins 52 secs total import time.


Using a single 3.1GHz 2nd gen i5 core, here's the timings:

 3 mins 11 secs - build command
 1 mins  2 secs - cleandb command
20 mins 40 secs - pbuild command

24 mins 53 secs total import time.

The 2nd import has a slower hard disk subsystem than the 1st, so is most
likely the cause of the slightly longer import times.

Hope that helps.


Thanks,

Dan.

signature.asc
Description: OpenPGP digital signature