Internet engine matches

Discussion about development of draughts in the time of computer and Internet.
Post Reply
jj
Posts: 150
Joined: Sun Sep 13, 2009 23:33
Real name: Jan-Jaap van Horssen
Location: Zeist, Netherlands

Re: Internet engine matches

Post by jj » Sat Oct 12, 2019 14:33

Jelle Wiersma wrote:But switching off pondering, and hence the search threads, alone does not solve it.
There can also be memory conflicts because of the memory used by the two programs for the transposition table and endgame database;
Windows will then start swapping memory blocks from RAM to- and from the disk.
Are these blocks really swapped to/from disk? Even if there is plenty of RAM and everything fits into that? Or is it just swapping to/from RAM cache?
Anyway, I think we can assume that both programs are equally affected by this at the start of their respective searches.
Jelle Wiersma wrote:Besides the search threads, there can also be thread(s) to load the endgame database into memory, especially for 8-piece database.
These threads also need to be postponed when it is not the program's turn to do a move,
otherwise it will interfere with the other program.
Programs such as Scan and Maximus (and Sjende Blyn, I presume) use a 6-piece egdb which is loaded entirely into RAM at program startup. So there will be no threads running that load egdbs. Kingsrow can be set to use only 6 pieces as well. I found there is very little difference in playing strength between Kingsrow 6-piece and 8-piece, maybe Ed can confirm this.
Jelle Wiersma wrote:Everything need to be set/implement such that it prevents interference of the two programs.
That's why I proposed to Krzysztof to use two AMD Ryzen, as in the tournament, seemed to be the most simple.
As it turns out this is not as simple as running on a single computer, for Krzysztof.
Running fair matches on a single computer never seemed a problem before in computer chess (see TCEC) and computer draughts. What do other programmers think about this?

Another matter is time usage. As far as I have observed, programs such as Scan, Kingsrow and Maximus do not use more time than is agreed. Several other programs do use more time than agreed, even showing this on their own clocks, especially when they play against strong(er) opponents and get into difficult positions. It is up to each programmer to implement this correctly, and so far this is not checked automatically.

Jan-Jaap
www.maximusdraughts.org

Krzysztof Grzelak
Posts: 782
Joined: Thu Jun 20, 2013 17:16
Real name: Krzysztof Grzelak

Re: Internet engine matches

Post by Krzysztof Grzelak » Sat Oct 12, 2019 17:02

jj wrote:
Sat Oct 12, 2019 12:36
Krzysztof, does this mean you have it working now?
Yes. The Maximus - Kingsrow test match is in progress.

Krzysztof Grzelak
Posts: 782
Joined: Thu Jun 20, 2013 17:16
Real name: Krzysztof Grzelak

Re: Internet engine matches

Post by Krzysztof Grzelak » Sat Oct 12, 2019 17:14

jj wrote:
Sat Oct 12, 2019 14:33
As it turns out this is not as simple as running on a single computer, for Krzysztof.
Running fair matches on a single computer never seemed a problem before in computer chess (see TCEC) and computer draughts. What do other programmers think about this?
Such a match will always be unfair. Imagine playing Sjende Blyn - Kingsrow. Sjende Blyn uses 16 cores, Kingsrow uses 8 threads. There is already injustice at this point. Unfortunately, this cannot be changed in the program Sjende Blyn. It is going to change in Maximus, Scan, Kingsrow, Dragon. Of course, the game is on one computer.

Krzysztof Grzelak
Posts: 782
Joined: Thu Jun 20, 2013 17:16
Real name: Krzysztof Grzelak

Re: Internet engine matches

Post by Krzysztof Grzelak » Sat Oct 12, 2019 17:44

Test match (not the whole)on two computers Maximus - Kingsrow.
Result - isn't important.
Attachments
dxpgames.pdn
(76.1 KiB) Downloaded 28 times

Ed Gilbert
Posts: 768
Joined: Sat Apr 28, 2007 14:53
Real name: Ed Gilbert
Location: Morristown, NJ USA
Contact:

Re: Internet engine matches

Post by Ed Gilbert » Sat Oct 12, 2019 22:06

As it turns out this is not as simple as running on a single computer, for Krzysztof.
Running fair matches on a single computer never seemed a problem before in computer chess (see TCEC) and computer draughts. What do other programmers think about this?
I've been running matches on a single computer for years. Not only single matches, but many simultaneous matches. It works fine. To require two identical computers is crazy.

The problem appears to be that sjende blyn does not have a user setting to control pondering, or number of search threads. These are very basic controls.
Besides the search threads, there can also be thread(s) to load the endgame database into memory, especially for 8-piece database.
These threads also need to be postponed when it is not the program's turn to do a move,
otherwise it will interfere with the other program.
Kingsrow fills it's egdb cache when the first search is started. This takes about a minute, then it is finished, and after that kingsrow will never use any cpu cycles with pondering off when it is not its turn to move. Simply wait until kingsrow has finished initializing the egdb before starting a match.

Jelle Wiersma
Posts: 47
Joined: Wed Aug 12, 2015 17:45
Real name: Jelle Wiersma

Re: Internet engine matches

Post by Jelle Wiersma » Sat Oct 12, 2019 22:52

May be I should remind you guys that Sjende Blyn was set
to play in a tournament,
against different programs of varying strength,
where each program runs on a seperate computer.
It was not set to play blitz matches, against one program with excellent strength, on one computer.
You asked Krzysztof to play matches with this program, he is allowed to do that if he wants, but as mentioned before the program was not set for matches.

Jelle

Jelle Wiersma
Posts: 47
Joined: Wed Aug 12, 2015 17:45
Real name: Jelle Wiersma

Re: Internet engine matches

Post by Jelle Wiersma » Sun Oct 13, 2019 08:56

Actually this means that it:
Supports to play one game, either manually or via DXP.
Takes risks to win a game against a weaker program / human. The drawback is that it will do the same against a top program,
which increases the chance to loose against a top program.
Utilizes the available processor and memory.

Krzysztof Grzelak
Posts: 782
Joined: Thu Jun 20, 2013 17:16
Real name: Krzysztof Grzelak

Re: Internet engine matches

Post by Krzysztof Grzelak » Mon Oct 21, 2019 16:00

Dziś będę pisał polsku. Tak ostatnio często myślałem o warcabach międzynarodowych i zastanawiałem się na grą silników warcabowych na jednym komputerze i na dwóch komputerach osobno. Większości programistów gra na jednym komputerze. Dlatego postanowiłem zrobić taki krótki test dwóch programów warcabowych : Kingsrow i Scan. Mecz jest rozgrywany na dwóch komputerach osobno. Ustawienie programu Kingsrow 1.60b:

Kingsrow 1.60b x64

Opening Book = Best Moves
HashTable Size = 512 MB
DB cache Size = 30000 MB
The base ends = 8 Pieces
Threads = 8
Time = 1 Min / 75 Moves

Ustawienie program Scan 3.1.

Scan 3.1 x64

Variant = normal
Book = true
Book-ply = 4
Book-margin = 4
Threads = 16
Tt-size = 26
Bb-size = 6
Time = 1 Min / 75 Moves

Tak się zastanawiam czy który z programów przegra. W pierwszy meczu są wszystkie remisy - poniżej plik dxp pierwszego meczu. Już wkrótce kolejne mecze i rezultaty. Wszystkie mecze są rozgrywane na następującym sprzęcie:

Processor - AMD Ryzen Threadripper 1950X
Hard disc - SSD Samsung 860 Pro 1 TB
Memory of frames - 128 GB DDR4 2400
System - Windows 10 64 bit Pro

Kolejny mecz zakończony. Tym razem wygrał program Scan 3-1, resztę gier zakończyło się remisem. Następny mecz zakończony zwycięstwem programu Scan. Scan wygrał 4 gry, pozostałe gry zakończyły się remisem. Dziś kolejny dzień testów programów Kingsrow i Scan. Dzisiejszy mecz zakończył się zwycięstwem programu Scan - wygrał 3 gry, resztę gier zakończyło się remisami.
Attachments
dxpgames.pdn
(151.42 KiB) Downloaded 16 times
dxpgames.pdn
(152.2 KiB) Downloaded 21 times
dxpgames.pdn
(153 KiB) Downloaded 21 times
dxpgames.pdn
(151.27 KiB) Downloaded 20 times

jj
Posts: 150
Joined: Sun Sep 13, 2009 23:33
Real name: Jan-Jaap van Horssen
Location: Zeist, Netherlands

Re: Internet engine matches

Post by jj » Tue Oct 22, 2019 13:21

Krzysztof, what is your point?

Krzysztof Grzelak
Posts: 782
Joined: Thu Jun 20, 2013 17:16
Real name: Krzysztof Grzelak

Re: Internet engine matches

Post by Krzysztof Grzelak » Tue Oct 22, 2019 14:06

jj wrote:
Tue Oct 22, 2019 13:21
Krzysztof, what is your point?
I am doing a two-day test on Kingsrow - Scan on separate computers. I write only in Polish. This time the match ended in a draw. No program has lost or won.
Attachments
dxpgames.pdn
(151.93 KiB) Downloaded 16 times

Ed Gilbert
Posts: 768
Joined: Sat Apr 28, 2007 14:53
Real name: Ed Gilbert
Location: Morristown, NJ USA
Contact:

Re: Internet engine matches

Post by Ed Gilbert » Tue Oct 22, 2019 17:48

I am doing a two-day test on Kingsrow - Scan on separate computers. I write only in Polish. This time the match ended in a draw. No program has lost or won.
This only shows that 158 games is not nearly enough to learn which program is stronger when the programs are not very far apart. Thousands of games are needed. And these are fast 1-minute games. If you play longer games, you get fewer decisive games, so you need even more.

-- Ed

Krzysztof Grzelak
Posts: 782
Joined: Thu Jun 20, 2013 17:16
Real name: Krzysztof Grzelak

Re: Internet engine matches

Post by Krzysztof Grzelak » Tue Oct 22, 2019 19:15

Ed Gilbert wrote:
Tue Oct 22, 2019 17:48
This only shows that 158 games is not nearly enough to learn which program is stronger when the programs are not very far apart. Thousands of games are needed. And these are fast 1-minute games. If you play longer games, you get fewer decisive games, so you need even more.

-- Ed
Of course you are right Ed.

Krzysztof Grzelak
Posts: 782
Joined: Thu Jun 20, 2013 17:16
Real name: Krzysztof Grzelak

Re: Internet engine matches

Post by Krzysztof Grzelak » Tue Oct 29, 2019 20:03

Match SJENDE BLYN - MAXIMUS (2-move ballots)

Sjende Blyn (2019aug31) vs. Maximus 2.01 1 wins, 0 losses, 156 draws, 1 unknowns

The match was played on separate computers with the same parameters:

Processor - AMD Ryzen Threadripper 1950X
Hard disc - SSD Samsung 860 Pro 1 TB
Memory of frames - 128 GB DDR4 2400
System - Windows 10 64 bit Pro

Sjende Blyn (2019aug31)

Cores - 16
Opening book - On
Strength - Maximum
Pondering - On
The base ends - 6
Time - 15 Min / 75 Moves

Maximus 2.01

Opening book - Best moves
Transposition Table - 32 M positions (25 bits)
Threads - 16
Ponder threads - 16
Pondering - On
The base ends - 6
Time - 15 Min / 75 Moves

Attachments
dxp.pdn
(181.1 KiB) Downloaded 19 times

Post Reply