Re: [Bug-gnubg] An evalutaion of the pruning nets

[Robert-Jan Veldhuizen]

Hi,

Some more about the pruning net results, I'll forward this from GOL (30 
October):


***********************************************************************
I thought this was interesting:

Jim Segrave on the gnubg mailinglist, about the differences in a large 
sample (although SE are still very high) of matches analysed by both 
versions:

(...) The results:

..............................win.....wing...winbg...loseg...losebg cubeful

Average absolute difference 0.00123 0.00108 0.00009 0.00157 0.00020 0.00343

Std err.....................0.00619 0.00390 0.00051 0.00642 0.00403 0.02100

In 2,375 cases, the choice of best move differed, (0.94% of the time) (...)

[NOTE: Jim later changed this to 2.98% of the time]

I read some conclusions, probably based on this, that 2-ply prune is as 
good as 2-ply no prune, practically speaking. I'm paraphrasing here, 
sorry if I'm wrong about this.

However, looking at these figures I'm not so sure. In an absolute sense 
these differences look very small indeed. However, 2-ply is supposed to 
be playing a world-class (or better!) game. That means even the 
slightest increase in error rates means a clearly lower level of play.

I'm not sure how to quantify and interpret the figures, certainly not 
because the SE's are so high. But isn't it a bit early to draw 
conclusions from here that the pruning has "almost zero" effect on skill?

From some simulations I did myself, mostly letting gnubg play against 
itself, 0-ply vs. 2-ply and 2-ply reduced vs. 2-ply 100%, I think that 
the pruning net choosing a different move almost 1% of the time [NOTE: 
3% even, it seems] is significant, again considering the fact that 2-ply 
is supposed to play world-class or better.
*************************************************************************

It seems like "different move" included all positions where moves have 
equal equity so it doesn't matter. That makes the figures harder to 
interpret; part of the 3% differences is simply irrelevant. But if 
there's still 2% REAL differences, I think that's a significant 
difference, also looking at the equity differences Jim reports here (but 
which may not be of any value considering the SE?).

Greetings,
--
Robert-Jan Veldhuizen