Kingsrow - databases

Discussion about development of draughts in the time of computer and Internet.
Krzysztof Grzelak
Posts: 1050
Joined: Thu Jun 20, 2013 17:16
Real name: Krzysztof Grzelak

Kingsrow - databases

Post by Krzysztof Grzelak » Sun Jan 17, 2021 10:38

I have a question for you, Ed. Can you calculate how much space for base 9 and 10 pieces.

Ed Gilbert
Posts: 829
Joined: Sat Apr 28, 2007 14:53
Real name: Ed Gilbert
Location: Morristown, NJ USA
Contact:

Re: Kingsrow - databases

Post by Ed Gilbert » Mon Jan 18, 2021 01:40

If I assume only 5x4 and 5x5 configurations (no 6x3 or 6x4), and assume that the size for 9 pieces will be the 7-piece size times the ratio of 9-piece positions to 7-piece positions (so assuming the same compression ratio for 9 pieces as was achieved for 7 pieces), then the size for 9 pieces would be 764 GB.

Using similar assumptions applied to 10 pieces relative to 8 pieces, the size for 10 pieces would be 4852 GB.

To use the 9-piece slices, you would of course also need the slices for 2 through 8 pieces, so you would have to add 55 GB to the 764 GB to get the total size of 2 through 9 pieces. And to use the 10-piece slices, you would also need the 2 through 9 pieces, so you would have to add (55 + 764) to the 4852 GB to get the total size of 2 through 10 pieces.

-- Ed

Krzysztof Grzelak
Posts: 1050
Joined: Thu Jun 20, 2013 17:16
Real name: Krzysztof Grzelak

Re: Kingsrow - databases

Post by Krzysztof Grzelak » Mon Jan 18, 2021 08:10

Thanks Ed. And you can describe very precisely how to generate the database 9 pieces. Here a question comes to my mind - after generating the 9 pieces database, you will have to change the program code to handle this database.How do you think Ed how long would it kidnapped.

Ed Gilbert
Posts: 829
Joined: Sat Apr 28, 2007 14:53
Real name: Ed Gilbert
Location: Morristown, NJ USA
Contact:

Re: Kingsrow - databases

Post by Ed Gilbert » Mon Jan 18, 2021 13:28

Krzysztof Grzelak wrote:
Mon Jan 18, 2021 08:10
Thanks Ed. And you can describe very precisely how to generate the database 9 pieces. Here a question comes to my mind - after generating the 9 pieces database, you will have to change the program code to handle this database.How do you think Ed how long would it kidnapped.
I have no plans to build the 9-piece database.

BertTuyt
Posts: 1515
Joined: Wed Sep 01, 2004 19:42

Re: Kingsrow - databases

Post by BertTuyt » Mon Jan 18, 2021 13:35

Ed, although you don't have the intention to build a 9p DB, would you have a gutfeel for the computing time needed on a 16-core machine?
Would resources be sufficient, or do you expect a memory overload?
The SSD most likely is no problem (as these days 4TB SSD are available).
In the end I also do not believe it will add to the playing strength, but for perfect analysis it might bring something.

Bert

Krzysztof Grzelak
Posts: 1050
Joined: Thu Jun 20, 2013 17:16
Real name: Krzysztof Grzelak

Re: Kingsrow - databases

Post by Krzysztof Grzelak » Mon Jan 18, 2021 13:57

I understand that. Can I do it myself with your help? The 764 GB is not much.

Krzysztof Grzelak
Posts: 1050
Joined: Thu Jun 20, 2013 17:16
Real name: Krzysztof Grzelak

Re: Kingsrow - databases

Post by Krzysztof Grzelak » Mon Jan 18, 2021 13:58

BertTuyt wrote:
Mon Jan 18, 2021 13:35
Ed, although you don't have the intention to build a 9p DB, would you have a gutfeel for the computing time needed on a 16-core machine?
Would resources be sufficient, or do you expect a memory overload?
The SSD most likely is no problem (as these days 4TB SSD are available).
In the end I also do not believe it will add to the playing strength, but for perfect analysis it might bring something.

Bert
You need to connect two 4 GB SSDs in Raid. I am thinking to change the processor to an even faster one to draughts core=32 - threads=64 .

Ed Gilbert
Posts: 829
Joined: Sat Apr 28, 2007 14:53
Real name: Ed Gilbert
Location: Morristown, NJ USA
Contact:

Re: Kingsrow - databases

Post by Ed Gilbert » Mon Jan 18, 2021 14:24

I built 2 through 8 pieces in 2008 - 2009. It took 5.6 core-years to build and verify. See http://edgilbert.org/InternationalDraug ... hts_db.htm

The number of positions for 9 pieces is about 11.5 times as many as 2 through 8 pieces.

So if you were to build 9 pieces on 2008 hardware, you could expect it to take 11.5 * 5.6 = 64.4 core-years.

But 2021 hardware is much faster. The CPUs are faster, and now we have SSDs that are much faster than hard drives. In my build the time was dominated by CPU speeds, but SSDs might have sped it up by 25% or so.

Let's say that with 2021 hardware a single core can build dbs at a rate 5x faster than with 2008 hardware. To build 9 pieces would then take (64.4 / 5) = 12.9 core-years. That would be 9 or 10 months on a 16-core machine.

Krzysztof Grzelak
Posts: 1050
Joined: Thu Jun 20, 2013 17:16
Real name: Krzysztof Grzelak

Re: Kingsrow - databases

Post by Krzysztof Grzelak » Mon Jan 18, 2021 14:42

And what version database are you writing about. And here I have a question - is it possible to connect two computers to make the database generation faster.

Rein Halbersma
Posts: 1707
Joined: Wed Apr 14, 2004 16:04
Contact:

Re: Kingsrow - databases

Post by Rein Halbersma » Tue Jan 19, 2021 00:51

IIRC, the 10-pc dbs are in the same ballpark as the chess 7-pc dbs. One key bottleneck would be generating the 5v5 kings sub db, which has 2.6 trillion (10^12) positions. That requires 640+Gb of RAM with 2 bits per position for a single side to move. For 6v4 kings you need at least 1 Tb of RAM. That will set you back at least $6-10K in RAM alone. For chess someone actually had access to two 1.5 Tb machines with 192/384 cores/threads ($90K each) to generate the 18 Tb on disk in 4 months back in 2018. So give it some resources :lol:
Last edited by Rein Halbersma on Tue Jan 19, 2021 12:19, edited 1 time in total.

Krzysztof Grzelak
Posts: 1050
Joined: Thu Jun 20, 2013 17:16
Real name: Krzysztof Grzelak

Re: Kingsrow - databases

Post by Krzysztof Grzelak » Tue Jan 19, 2021 09:05

Rein Halbersma wrote:
Tue Jan 19, 2021 00:51
IIRC, the 10-pc dbs are in the same ballpark as the chess 7-pc dbs. One key bottleneck would be generating the 5v5 kings sub db, which has 2.6 trillion (10^12) positions. That requires 640+Gb of RAM with 2 bits per position for a single side to move. For 6v4 kings you need at least 1 Tb of RAM. That will set you back at least $6-10K in RAM alone. For chess someone actually had access to a 1.5 Tb machine with 384 cores to generate the 18 Tb on disk in 4 months back in 2018. So give it some resources :lol:
Only there is 18TB here. And few chess testers have this base. Ed could make a base 5x5. After all, he has a good computer.

Rein Halbersma
Posts: 1707
Joined: Wed Apr 14, 2004 16:04
Contact:

Re: Kingsrow - databases

Post by Rein Halbersma » Tue Jan 19, 2021 13:07

Krzysztof Grzelak wrote:
Tue Jan 19, 2021 09:05
Rein Halbersma wrote:
Tue Jan 19, 2021 00:51
IIRC, the 10-pc dbs are in the same ballpark as the chess 7-pc dbs. One key bottleneck would be generating the 5v5 kings sub db, which has 2.6 trillion (10^12) positions. That requires 640+Gb of RAM with 2 bits per position for a single side to move. For 6v4 kings you need at least 1 Tb of RAM. That will set you back at least $6-10K in RAM alone. For chess someone actually had access to a 1.5 Tb machine with 384 cores to generate the 18 Tb on disk in 4 months back in 2018. So give it some resources :lol:
Only there is 18TB here. And few chess testers have this base. Ed could make a base 5x5. After all, he has a good computer.
It would take 8 calendar years to compute the 5x5 with 16 cores, but that assumes already ~1Tb of memory which is not nearly what even a high-end workstation has. I think Ed's resources (both cores and RAM) need to be increased 8-fold in order for this to be feasible within 1 calendar year. Assuming Moore's law (doubling every 3 years nowadays) 5x5 will not happen this decade.

Krzysztof Grzelak
Posts: 1050
Joined: Thu Jun 20, 2013 17:16
Real name: Krzysztof Grzelak

Re: Kingsrow - databases

Post by Krzysztof Grzelak » Tue Jan 19, 2021 14:44

Rein Halbersma wrote:
Tue Jan 19, 2021 13:07
It would take 8 calendar years to compute the 5x5 with 16 cores, but that assumes already ~1Tb of memory which is not nearly what even a high-end workstation has. I think Ed's resources (both cores and RAM) need to be increased 8-fold in order for this to be feasible within 1 calendar year. Assuming Moore's law (doubling every 3 years nowadays) 5x5 will not happen this decade.
I wonder over something like that.I just don't know anything about it.Is there any online service - rent a supercomputer to do it, for example, in three months.

Rein Halbersma
Posts: 1707
Joined: Wed Apr 14, 2004 16:04
Contact:

Re: Kingsrow - databases

Post by Rein Halbersma » Tue Jan 19, 2021 15:13

Krzysztof Grzelak wrote:
Tue Jan 19, 2021 14:44
I wonder over something like that.I just don't know anything about it.Is there any online service - rent a supercomputer to do it, for example, in three months.
Of course: https://aws.amazon.com/ec2/pricing/on-demand/

For the peak memory load you want r5b.24xlarge with 96 cores and 768 Gb RAM which costs $7 per hour. That would be enough to run 5v5 all kings and it should be doable to do that within a single day (~$175). But computing all the other 5v5 endgame databases will take ~14.4 months even with that supercomputer. So that's 7 x 14.4 / 12 x 8760 = ~$72K. Let me know if you're interested ;-)

So only for peak computations with very high memory demands (like all kings dbs) you want to rent something for a day. I think with Ed's super-finegrained slicing techniques, he could generate the 9 pieces dbs with his current workstations in a year. But for analysis purposes, balanced 10 piece dbs with equal number of pieces on both sides are much more useful. That would take 8 years with current technology (and a solveable bottleneck for peak memory load).

BertTuyt
Posts: 1515
Joined: Wed Sep 01, 2004 19:42

Re: Kingsrow - databases

Post by BertTuyt » Tue Jan 19, 2021 15:15

Rein, I assume that at least from a memory point of view the 9p would be doable-ish.
My guestimate is that 256 GByte would be sufficient, and at least the Threadripper could address this.
Also with Threadripper mother boards with 8 DIMM slots and 32 GByte Dimm seems not to be impossible.
Last but not least the costs for 256 GByte would be between 1000 - 1200 euro.

Certainly expensive, but not totally out of reach (as this is also my wish next PC system, 32 cores :D)

Bert

Post Reply