Downloadable 8-piece db and open source driver

Discussion about development of draughts in the time of computer and Internet.
Ed Gilbert
Posts: 748
Joined: Sat Apr 28, 2007 14:53
Real name: Ed Gilbert
Location: Morristown, NJ USA
Contact:

Downloadable 8-piece db and open source driver

Post by Ed Gilbert » Tue Aug 09, 2016 21:54

I am making the kingsrow endgame database drivers open source. The 8-piece database files have already been available on a file server for the last couple of years. By making the drivers open source, I hope that some draughts programmers can take advantage of this resource to avoid spending a lot of computing time to duplicate the effort of building them.

The driver source code is on GitHub: https://github.com/eygilbert/egdb_intl

There is detailed documentation in a Readme file. You can view it with markdown formatting at the project's home page on GitHub. I'll paste some of the Description section below.

-- Ed

---------------------

Description

Egdb_intl is a set of C++ source files with functions to access the kingsrow international draughts endgame databases. This package has been designed so that it can be easily integrated into existing C++ draughts programs. The source files are identical to those in the kingsrow draughts program.

The source code can be compiled for Windows using Microsoft Visual Studio 2015, and for Linux. Rein Halbersma ported the code to Linux and improved it in other ways also. Thanks Rein!

The function to lookup the value of a position in the database is thread-safe and can be used in a multi-threaded search engine.

The code can be used to access 2 versions of the win, loss, draw (WLD) db, and the moves-to-conversion (MTC) db. These databases have information for positions with up to 8 pieces and with up to 5 pieces on one side.

From 2010 to 2014 I distributed version 1 of the databases on a portable external hard drive. This db is 407gb of data, which includes a small subset of 9-piece positions, 5 men vs. 4 men (I disabled the 9-piece functionality in this driver because I found through testing that on average it was less effective than not using it). In 2014 I created version 2 by re-compressing the db using better compression techniques, and excluding more positions to make the compression work better. Version 2 is 56gb of data, and is available for download from a file server. See link below.

Joost Buijs
Posts: 317
Joined: Wed May 04, 2016 11:45
Real name: Joost Buijs

Re: Downloadable 8-piece db and open source driver

Post by Joost Buijs » Wed Aug 10, 2016 10:27

Thanks! This is very helpful.

I couldn't resist to try your database code, it compiles without a hitch and I had it working in my program within a few minutes. The only thing I had to do was to convert my board-representation from 'bm,wm,bk,wk' to 'bp,wp,k'.

I'm on the verge of generating a DTW database, comparing my data with yours makes it a lot easier to detect errors.
DTW does not compress that well and it is larger than WLD to begin with so I will probably stick with 7 pieces max.

Joost

Krzysztof Grzelak
Posts: 705
Joined: Thu Jun 20, 2013 17:16
Real name: Krzysztof Grzelak

Re: Downloadable 8-piece db and open source driver

Post by Krzysztof Grzelak » Wed Aug 10, 2016 10:31

Thanks Ed and Rein.

Ed Gilbert
Posts: 748
Joined: Sat Apr 28, 2007 14:53
Real name: Ed Gilbert
Location: Morristown, NJ USA
Contact:

Re: Downloadable 8-piece db and open source driver

Post by Ed Gilbert » Wed Aug 10, 2016 13:18

Thanks for the comments Joost. Nice to hear that you had no trouble using it. Good luck with your DTW project.

-- Ed

Joost Buijs
Posts: 317
Joined: Wed May 04, 2016 11:45
Real name: Joost Buijs

Re: Downloadable 8-piece db and open source driver

Post by Joost Buijs » Fri Aug 12, 2016 20:13

Although I'm on vacation I still have some time for the hobby in the evening hours, I have my computer at home running and can log into it remotely with my laptop.

I put most of your database on SSD and tried how fast I could probe all permutations for (2,2,1,1), this took roughly 261 seconds for all 1,870,120,800 positions, (> 7 mln. pos./sec.) which is not bad at all.
With chess I could never reach this speed, probably because the index function for chess is very complex and therefore consumes more time.

Ed Gilbert
Posts: 748
Joined: Sat Apr 28, 2007 14:53
Real name: Ed Gilbert
Location: Morristown, NJ USA
Contact:

Re: Downloadable 8-piece db and open source driver

Post by Ed Gilbert » Fri Aug 12, 2016 20:44

Joost Buijs wrote: I put most of your database on SSD and tried how fast I could probe all permutations for (2,2,1,1), this took roughly 261 seconds for all 1,870,120,800 positions, (> 7 mln. pos./sec.) which is not bad at all.
With chess I could never reach this speed, probably because the index function for chess is very complex and therefore consumes more time.
The compression encoding uses fixed-width byte codes so the speed is pretty good. I experimented with entropy encoding, and that gives ~25% better compression, but at a big cost in speed. IIRC it was more than 2X longer average lookup times.

The SSD makes a huge difference as I'm sure you know. About a month ago I sent a functional test program to Rein so he could start working on the port to Linux. I told him it takes 5 or 6 minutes to run. The next day he reported that it took him over an hour to run the test on his fairly recent 6-core workstation. I was at the beach at the time with only my laptop, and timed the test at 8 minutes. I had put an SSD on the laptop in the last few months. Well worth the trouble.

-- Ed

Rein Halbersma
Posts: 1625
Joined: Wed Apr 14, 2004 16:04
Contact:

Re: Downloadable 8-piece db and open source driver

Post by Rein Halbersma » Fri Aug 12, 2016 22:27

Ed Gilbert wrote:
Joost Buijs wrote: I put most of your database on SSD and tried how fast I could probe all permutations for (2,2,1,1), this took roughly 261 seconds for all 1,870,120,800 positions, (> 7 mln. pos./sec.) which is not bad at all.
With chess I could never reach this speed, probably because the index function for chess is very complex and therefore consumes more time.
The compression encoding uses fixed-width byte codes so the speed is pretty good. I experimented with entropy encoding, and that gives ~25% better compression, but at a big cost in speed. IIRC it was more than 2X longer average lookup times.

The SSD makes a huge difference as I'm sure you know. About a month ago I sent a functional test program to Rein so he could start working on the port to Linux. I told him it takes 5 or 6 minutes to run. The next day he reported that it took him over an hour to run the test on his fairly recent 6-core workstation. I was at the beach at the time with only my laptop, and timed the test at 8 minutes. I had put an SSD on the laptop in the last few months. Well worth the trouble.

-- Ed
Yes, I'm considering adding an SSD to my pc as well. That, or going to the beach for the other type of silicon :)

Rein Halbersma
Posts: 1625
Joined: Wed Apr 14, 2004 16:04
Contact:

Re: Downloadable 8-piece db and open source driver

Post by Rein Halbersma » Wed Aug 31, 2016 20:53

Joost Buijs wrote:Although I'm on vacation I still have some time for the hobby in the evening hours, I have my computer at home running and can log into it remotely with my laptop.

I put most of your database on SSD and tried how fast I could probe all permutations for (2,2,1,1), this took roughly 261 seconds for all 1,870,120,800 positions, (> 7 mln. pos./sec.) which is not bad at all.
With chess I could never reach this speed, probably because the index function for chess is very complex and therefore consumes more time.
Joost, please see the revamped docs and in particular a complete reference documentation for all functionality.

The interface has been simplified a bit, which should take only few modifications in any of your code. E.g. replace calls like handle->lookup(handle, position, color, cl) with egdb_lookup(handle, position, color, cl) instead. The function pointer calls should still work but have been deprecated (because it relied on the implementation of the EGDB_DRIVER class). Using the global functions egdb_open(), egdb_lookup() and egdb_close() is now the supported interface.

Joost Buijs
Posts: 317
Joined: Wed May 04, 2016 11:45
Real name: Joost Buijs

Re: Downloadable 8-piece db and open source driver

Post by Joost Buijs » Thu Sep 01, 2016 06:51

Rein Halbersma wrote:
Joost Buijs wrote:Although I'm on vacation I still have some time for the hobby in the evening hours, I have my computer at home running and can log into it remotely with my laptop.

I put most of your database on SSD and tried how fast I could probe all permutations for (2,2,1,1), this took roughly 261 seconds for all 1,870,120,800 positions, (> 7 mln. pos./sec.) which is not bad at all.
With chess I could never reach this speed, probably because the index function for chess is very complex and therefore consumes more time.
Joost, please see the revamped docs and in particular a complete reference documentation for all functionality.

The interface has been simplified a bit, which should take only few modifications in any of your code. E.g. replace calls like handle->lookup(handle, position, color, cl) with egdb_lookup(handle, position, color, cl) instead. The function pointer calls should still work but have been deprecated (because it relied on the implementation of the EGDB_DRIVER class). Using the global functions egdb_open(), egdb_lookup() and egdb_close() is now the supported interface.
Rein,

It was already very easy to implement in my code but I will take a look at what has been changed.
I already have a large part of my own EGTB code (depth to win) running, implementing the code of Ed as well gives me the opportunity to compare things.
A DTW EGDB is larger than a WLD one, so it will be 6 or 7 pieces at max, fortunately SSD's get faster, bigger and cheaper each day.
I my main computer sits a 2TB SSD which I mainly use for database storage, Nalimov etc.

Since a couple of days I actually started working on my engine, work on my EGDB has temporary come to a stand still.
The hash-table implementation and evaluation-function are finished, although I still have to tune the latter one which is a different story. Atm. it seems 1 evaluation takes ~100ns which seems fast compared to others.
Right now I'm working on the SMP search, although I can use a lot of code from my chess-engine I want to improve several things and that will take time.

Joost

BertTuyt
Posts: 1303
Joined: Wed Sep 01, 2004 19:42

Re: Downloadable 8-piece db and open source driver

Post by BertTuyt » Sat Sep 17, 2016 12:12

Ed, would you approve if I use your driver and 8P DB as a referee (adjudicate :) )in the Damage GUI.
In quiet endgame positions (so no capture nore capture thread) during (for example) a DXP match, I could add an option that the GUI decides to terminate the game. As you know I dont have commercial intentions with this GUI, and it will still be available to all.

Also as the GUI only needs 1 value, the cache is relatively small, so limited memory impact (altough think I need to deal with the index tables).

Bert

Rein Halbersma
Posts: 1625
Joined: Wed Apr 14, 2004 16:04
Contact:

Re: Downloadable 8-piece db and open source driver

Post by Rein Halbersma » Sat Sep 17, 2016 12:53

BertTuyt wrote:Ed, would you approve if I use your driver and 8P DB as a referee (adjudicate :) )in the Damage GUI.
In quiet endgame positions (so no capture nore capture thread) during (for example) a DXP match, I could add an option that the GUI decides to terminate the game. As you know I dont have commercial intentions with this GUI, and it will still be available to all.

Also as the GUI only needs 1 value, the cache is relatively small, so limited memory impact (altough think I need to deal with the index tables).

Bert
Bert, AFAICS the Boost license gives you this permission, even if you would have commercial intentions. See the doc and license on github. Scan OTOH is GPL so you need to release sources based on it.
Last edited by Rein Halbersma on Sat Sep 17, 2016 13:03, edited 1 time in total.

BertTuyt
Posts: 1303
Joined: Wed Sep 01, 2004 19:42

Re: Downloadable 8-piece db and open source driver

Post by BertTuyt » Sat Sep 17, 2016 12:58

Rein, thanks for sharing.
As you know I have no commercial intentions.
And even if Im legally allowed, I think it is fair to discuss this in the open (also related to the clones discussion).

Bert

Rein Halbersma
Posts: 1625
Joined: Wed Apr 14, 2004 16:04
Contact:

Re: Downloadable 8-piece db and open source driver

Post by Rein Halbersma » Sat Sep 17, 2016 13:03

BertTuyt wrote:Rein, thanks for sharing.
As you know I have no commercial intentions.
And even if Im legally allowed, I think it is fair to discuss this in the open (also related to the clones discussion).

Bert
For competing engines, cloning is a fair topic, but copying code for utilities and gui infrastructure is completely natural.

Ed Gilbert
Posts: 748
Joined: Sat Apr 28, 2007 14:53
Real name: Ed Gilbert
Location: Morristown, NJ USA
Contact:

Re: Downloadable 8-piece db and open source driver

Post by Ed Gilbert » Sat Sep 17, 2016 13:08

Ed, would you approve if I use your driver and 8P DB as a referee (adjudicate :) )in the Damage GUI.
Bert, yes of course, use it for any purpose, commercial or not.

-- Ed

BertTuyt
Posts: 1303
Joined: Wed Sep 01, 2004 19:42

Re: Downloadable 8-piece db and open source driver

Post by BertTuyt » Sat Nov 12, 2016 12:33

Ed, I included the EGDB driver (as-is) with a small wrapper (as my Board configuration is slightly different) in my test program Dwarf.
Also downloaded the 8P DB.
It all worked perfectly.
As one could expect, I tested the Woldouby position, and when the zero score popped up around ply 30, it really felt like a miracle (although we all know these days that it is a draw).

So again thanks for sharing.

Bert

Post Reply