Opening book stats

Discussion about development of draughts in the time of computer and Internet.
Post Reply
Ed Gilbert
Posts: 792
Joined: Sat Apr 28, 2007 14:53
Real name: Ed Gilbert
Location: Morristown, NJ USA
Contact:

Opening book stats

Post by Ed Gilbert » Mon May 25, 2020 19:41

I was curious to know how much the use of an opening book can affect a match result, so I ran some tests. For the first set of tests, I played 2 matches of kingsrow vs scan, one match with books enabled in both engines, and second match with the kingsrow book disabled, but scan's enabled. The kingsrow opening book contains 1.42M positions. Based on the size of the scan book file, and 16 bytes per entry, I get 121K positions, so approximately one twelfth the number of the kingsrow book.

Each match consisted of 12,640 games at 75 moves in 60 seconds time control, GAYP start position. Matches were played on a 16-core Xeon. Each match took about 1 day to complete. I will append the detailed results per core below, along with all the engine settings, but here is the bottom line result:

books on: kingsrow vs scan: +2.3 elo
kr book off: kingsrow vs scan +1.2 elo

So a 1.1 elo drop in kingsrow performance when its opening book is disabled. That's a pretty small difference, and I'm not even sure that 12.6K games is enough to measure that small a difference with any kind of confidence.

I also ran a test to see how many book moves each engine makes on average in a GAYP match. For this test I played another match of around 2000 games, and I used the -t option in tmgr to record the move times in the pdn results. For each engine I made a histogram of the number of book moves made. I assumed that quick moves under 150 msec were book moves. This is not a perfect way to detect book moves, as it also picks up forced capture moves, but these probably average out the same for both engines, so the difference in book depths should still be accurate.

The column headings don't line up very well below, but the table gives the counts of the number of games that each engine had the various number of book moves. For example, scan played 19 games where it dropped out of book after the 3rd move, 348 games where it dropped out of book after the fourth move, etc.

Code: Select all

book moves	scan	kingsrow
0	0	0
1	0	0
2	0	0
3	19	0
4	348	36
5	566	26
6	882	136
7	463	482
8	143	1067
9	157	642
10	32	220
11	0	4
12	0	0
13	3	0
The average number of book moves for scan is 6.0
The average number of book moves for kingsrow is 8.0

-----------------------------------------------------------------

Match stats Kingsrow 1.61 vs. Scan 3.1 (both engines books enabled)

[ 1]: 0.503 score, 790 games, 11 wins, 6 losses, 773 draws, 0 unk
[ 2]: 0.504 score, 790 games, 10 wins, 3 losses, 777 draws, 0 unk
[ 3]: 0.500 score, 790 games, 10 wins, 10 losses, 770 draws, 0 unk
[ 4]: 0.506 score, 790 games, 10 wins, 1 losses, 779 draws, 0 unk
[ 5]: 0.501 score, 790 games, 9 wins, 7 losses, 773 draws, 1 unk
[ 6]: 0.505 score, 790 games, 11 wins, 3 losses, 776 draws, 0 unk
[ 7]: 0.504 score, 790 games, 9 wins, 2 losses, 779 draws, 0 unk
[ 8]: 0.500 score, 790 games, 5 wins, 5 losses, 780 draws, 0 unk
[ 9]: 0.501 score, 790 games, 8 wins, 7 losses, 774 draws, 1 unk
[10]: 0.502 score, 790 games, 9 wins, 6 losses, 775 draws, 0 unk
[11]: 0.504 score, 790 games, 10 wins, 4 losses, 776 draws, 0 unk
[12]: 0.501 score, 790 games, 8 wins, 7 losses, 775 draws, 0 unk
[13]: 0.501 score, 790 games, 9 wins, 8 losses, 773 draws, 0 unk
[14]: 0.508 score, 790 games, 14 wins, 2 losses, 774 draws, 0 unk
[15]: 0.506 score, 790 games, 17 wins, 7 losses, 766 draws, 0 unk
[16]: 0.506 score, 790 games, 17 wins, 7 losses, 766 draws, 0 unk
total 0.503 score, 12640 games, 167 wins, 85 losses, 12386 draws, 2 unk
elo diff 2.3
LOS 1.00

Match stats Kingsrow 1.61 vs. Scan 3.1 (kingsrow book disabled, scan book enabled)

[ 1]: 0.499 score, 790 games, 6 wins, 8 losses, 775 draws, 1 unk
[ 2]: 0.500 score, 790 games, 6 wins, 6 losses, 778 draws, 0 unk
[ 3]: 0.501 score, 790 games, 5 wins, 4 losses, 781 draws, 0 unk
[ 4]: 0.506 score, 790 games, 15 wins, 6 losses, 768 draws, 1 unk
[ 5]: 0.507 score, 790 games, 18 wins, 7 losses, 765 draws, 0 unk
[ 6]: 0.498 score, 790 games, 6 wins, 9 losses, 774 draws, 1 unk
[ 7]: 0.501 score, 790 games, 8 wins, 7 losses, 775 draws, 0 unk
[ 8]: 0.502 score, 790 games, 11 wins, 8 losses, 770 draws, 1 unk
[ 9]: 0.496 score, 790 games, 7 wins, 13 losses, 770 draws, 0 unk
[10]: 0.501 score, 790 games, 10 wins, 8 losses, 771 draws, 1 unk
[11]: 0.500 score, 790 games, 6 wins, 6 losses, 778 draws, 0 unk
[12]: 0.506 score, 790 games, 13 wins, 4 losses, 773 draws, 0 unk
[13]: 0.501 score, 790 games, 7 wins, 5 losses, 778 draws, 0 unk
[14]: 0.504 score, 790 games, 13 wins, 7 losses, 770 draws, 0 unk
[15]: 0.503 score, 790 games, 15 wins, 10 losses, 765 draws, 0 unk
[16]: 0.503 score, 790 games, 11 wins, 6 losses, 773 draws, 0 unk
total 0.502 score, 12640 games, 157 wins, 114 losses, 12364 draws, 5 unk
elo diff 1.2
LOS 1.00


tmgr.ini
---------
[General]
start-pos-file = d:\intl_book\gayp.pdn
games = 790
parallel-matches = 16
time = 60
moves = 75
#inc = 0.5
egdb-wld-path = d:\db_intl\wld_v2

[Engine]
engine-dir = d:\kr_hub
engine-command = d:\kr_hub\kr_hub
draw-count = 5
draw-pieces = 8
draw-score = 0.07
win-score = 20
set-param egdb-wld-pieces = 8
set-param egdb-wld-mem = 5500
set-param egdb-wld-path = d:\db_intl\wld_v2
set-param threads = 1
set-param tt-mem = 256
set-param tt-load = false
set-param book = false
set-param book-quality = best

[Engine]
engine-command = "\projects\scan_31\scan.exe hub"
engine-dir = \projects\scan_31
draw-count = 5
draw-pieces = 7
draw-score = 0.0
win-score = 80
set-param tt-size = 24
set-param bb-size = 6
set-param book = true
set-param book-ply = 4
set-param book-margin = 2
set-param threads = 1

jj
Posts: 180
Joined: Sun Sep 13, 2009 23:33
Real name: Jan-Jaap van Horssen
Location: Zeist, Netherlands

Re: Opening book stats

Post by jj » Mon May 25, 2020 20:05

Ed, thanks for running these tests. As expected the book difference is small for strong engines. To satisfy some people's curiosity, can you run a test with Flits and the anti-Kingsrow book?

Ed Gilbert
Posts: 792
Joined: Sat Apr 28, 2007 14:53
Real name: Ed Gilbert
Location: Morristown, NJ USA
Contact:

Re: Opening book stats

Post by Ed Gilbert » Mon May 25, 2020 23:19

To satisfy some people's curiosity, can you run a test with Flits and the anti-Kingsrow book?
JJ, I have a few other things going on this week so I'm not going to take this on. Anyone with flits can run it though.

Post Reply