[Bug-gnubg] Re-evaluation of pruning net

[Jim Segrave]

After Joseph's fix for the choice of nets, I reran my collection of
matches through both the old (pre-pruning) and new versions of gnubg.

I have 297 matches consisting of  2509 games.

First - speed improvement. The pruning net analysis took just under 36
hours, during this period I was also using gnubg to analyse some other
matches, so there were about three games whose analysis took longer
than the old version, almost undoubtedly for this reason. The generall
speedup was 1.74 times with a standard error of 0.37/

297 matches, 2509 games, 79466 moves
3776 forced moves, 570442 positions
The best move was different in 778 cases

Averaging the difference between the two version's evaluations of the
570k positions gives:

            win     win g   win bg  lose g lose bg cubeful equity
avgerage   0.00021 0.00023 0.00002 0.00044 0.00011 0.00061 
std err    0.00076 0.00184 0.00058 0.00454 0.00414 0.00212 

I have, for anyone who is interested, a 1.34Mb file (about 160K
zipped/gzipped/bzip2'ed) with the text export of each position where
the two versions differed in their choice of the best move (it has the
old version's eval of the two moves and the new version's as well).

I also did averages of the differences in the evals of these pairs of
positions:

            win   win g win bg  lose g  lose bg cubeful equity
average 0.00317 0.00569 0.00050 0.00635 0.00061 0.00263 
stderr  0.00711 0.01653 0.00243 0.01470 0.00210 0.00616 

The figures for some of the above are misleading as the pruning
version doesn't always pick moves in the same order when the move
choice doesn't affect the outcome - for example, in one case, where
a loss was inevitable and loss of even one point cost the match, the
old version chose a move which had the minimum gammon risk, the
pruning net chose one with a very high gammon risk. I consider only
the cubeful equity differences to be significant.

As far as the differences in best moves are concerned, here's a short
chart showing the percentages of moves whose equity difference was
below a given value (missing values are percentages where the maximum
difference was the same). There are some outliers where the differnces
are very large indeed - the maximum being 0.083 cubeless equity.

percent      max equity difference
 50.0:       0.00100
 51.0:       0.00100
 54.0:       0.00110
 56.0:       0.00120
 58.0:       0.00130
 59.0:       0.00140
 61.0:       0.00150
 62.0:       0.00160
 63.0:       0.00170
 64.0:       0.00180
 65.0:       0.00190
 66.0:       0.00200
 67.0:       0.00210
 69.0:       0.00220
 70.0:       0.00230
 71.0:       0.00240
 72.0:       0.00250
 73.0:       0.00260
 74.0:       0.00270
 75.0:       0.00290
 76.0:       0.00300
 77.0:       0.00310
 78.0:       0.00330
 79.0:       0.00350
 80.0:       0.00370
 81.0:       0.00390
 82.0:       0.00400
 83.0:       0.00420
 84.0:       0.00450
 85.0:       0.00500
 86.0:       0.00520
 87.0:       0.00550
 88.0:       0.00580
 89.0:       0.00630
 90.0:       0.00680
 91.0:       0.00750
 92.0:       0.00840
 93.0:       0.00960
 94.0:       0.01070
 95.0:       0.01170
 96.0:       0.01250
 97.0:       0.01440
 98.0:       0.01940
 99.0:       0.04330
100,0:       0.08310
-- 
Jim Segrave           address@hidden