NNUE

Discussion about development of draughts in the time of computer and Internet.
Post Reply
Joost Buijs
Posts: 355
Joined: Wed May 04, 2016 11:45
Real name: Joost Buijs

Re: NNUE

Post by Joost Buijs » Sat May 01, 2021 12:24

Rein Halbersma wrote:
Sat May 01, 2021 11:06
Joost Buijs wrote:
Fri Apr 30, 2021 08:14
Like Alpha Zero you can start with a random network and have the engine play a number of games against itself and update the network on the fly depending upon the outcome of the games. This is called 'reinforcement learning'. The problem with this method is that you have to play a huge number of games before the network reaches an acceptable level of play. I never read the Alpha Zero paper, but I think for chess they used something like 40 million games.
Reinforcement learning should be strictly superior compared to supervised learning, even for NNUE or Scan-pattern draughts programs. Mathematically, supervised learning is just a single iteration in the RL loop, so if you stop after one round, they are equivalent. If you continue the loop and only pick a new network when it improves, you can never get worse.

The question is whether you will gain much from continuous playing and retraining. For Scan-based patterns, I highly doubt this. The eval is almost completely optimized as soon as you have ~100M positions and cannot be made to overfit after that in my experience. For NNUE, there might be more capacity for overfitting that you might then try to reduce by adding more positions and retraining.

The AlphaZero neural networks are a few orders of magnitude more expensive and require much more positions and data to reach the limit of their predictive power. That's why it was so expensive to train and generate all these games (~1700 years in terms of single PC years). IIRC, the training games were with Monte Carlo Tree search with only 1600 nodes per search, that's just a tiny amount of search and a huge amount of CPU/GPU cycles to the eval. The AlphaZero eval also picks up a big part of pattern-based tactics because it's such a large neural network. For Scan-patterns, it's the reverse: a huge amount of search and a tiny amount of eval cycles.
I agree.

It's a pity that it is not worthwhile trying to train larger/deeper networks for 10x10 Draughts because 12 bit Scan-based patterns already seem to cover 99% of the important features. In fact NNUE for Draughts is a step backwards because it is slower, it's just the fun of trying to get it perform at the same level.

For training the huge draw tendency of Draughts is a problem too. Above a certain level engines don't lose anymore, even when you give the opponent a 100 fold speed advantage. From draws the network won't learn anything. It looks like that on current hardware 10x10 Draughts is getting a bit trivial. Maybe it needs other rules like removing compulsory captures, this could make the game more complex.

Joost Buijs
Posts: 355
Joined: Wed May 04, 2016 11:45
Real name: Joost Buijs

Re: NNUE

Post by Joost Buijs » Tue May 04, 2021 19:19

When spending enough time on it a NNUE style network can probably get on par with pattern based evaluation, it solely depends upon the data used for training.

My last attempt was training the network on positions derived from a single 3 move ballot, quiet positions that were taken from the leaves of the quiescence search. After removing all doubles, randomly taking 1 out of 5 positions and labeling them with the score of a 4 ply search based on the evaluation of the previous network, and training the network with this newly generated data, the result clearly got better.

The last 90 move in 1 minute match against KR ended: Kingsrow 1.62 vs Ares v1.2: 2 wins, 0 losses, 156 draws, 0 unknowns, a 4 Elo difference.

This is not so bad considering the slow speed of the network, quantization will make it faster, but I have not the slightest idea what it will do with the quality of the network.
Attachments
dxpgames.pdn
Latest match results
(157.32 KiB) Downloaded 32 times

Sidiki
Posts: 209
Joined: Thu Jan 15, 2015 16:28
Real name: Coulibaly Sidiki

Re: NNUE

Post by Sidiki » Thu May 06, 2021 00:17

Very very good result coming from NNUE, I Just downloaded the pdn of dxp game.

Good work.

Friendly, Sidiki

Joost Buijs
Posts: 355
Joined: Wed May 04, 2016 11:45
Real name: Joost Buijs

Re: NNUE

Post by Joost Buijs » Thu May 06, 2021 08:54

Sidiki wrote:
Thu May 06, 2021 00:17
Very very good result coming from NNUE, I Just downloaded the pdn of dxp game.

Good work.

Friendly, Sidiki
Thanks!

Yesterday I found out that in the last match I forgot to enable Kingsrow's EGTB while in the previous match it was enabled. Maybe this explains the better result.

There is still a lot of work to do, like I already told Bert I was set back by a very subtle bug in the hash-table that costed me a lot of time because I assumed it was a problem of the network and not something in the search.

I'm still busy optimizing the network by using better data and maybe try some other architecture(s), when I'm satisfied with the result I want to convert it from 32 bit floating point to 8 bit integer, this will make it 4 times as fast. Currently it runs at max 1.4 MN/s on a single core and it could go to 5.6 MN/s (possibly even faster because the incremental update is not optimized very well yet). My goal is to reach 8 MN/s.

Sidiki
Posts: 209
Joined: Thu Jan 15, 2015 16:28
Real name: Coulibaly Sidiki

Re: NNUE

Post by Sidiki » Thu May 06, 2021 10:35

Joost Buijs wrote:
Thu May 06, 2021 08:54
Sidiki wrote:
Thu May 06, 2021 00:17
Very very good result coming from NNUE, I Just downloaded the pdn of dxp game.

Good work.

Friendly, Sidiki
Thanks!

Yesterday I found out that in the last match I forgot to enable Kingsrow's EGTB while in the previous match it was enabled. Maybe this explains the better result.

There is still a lot of work to do, like I already told Bert I was set back by a very subtle bug in the hash-table that costed me a lot of time because I assumed it was a problem of the network and not something in the search.

I'm still busy optimizing the network by using better data and maybe try some other architecture(s), when I'm satisfied with the result I want to convert it from 32 bit floating point to 8 bit integer, this will make it 4 times as fast. Currently it runs at max 1.4 MN/s on a single core and it could go to 5.6 MN/s (possibly even faster because the incremental update is not optimized very well yet). My goal is to reach 8 MN/s.
Reach 8 MN/s will give a better level, because if at 1.4,we have these results.
I think that we are slowly discovering power of NNUE.

Thanks again to all programers of this forum for the fun that you give to us, engines users, God bless you all.

Friendly, Sidiki.

Krzysztof Grzelak
Posts: 1050
Joined: Thu Jun 20, 2013 17:16
Real name: Krzysztof Grzelak

Re: NNUE

Post by Krzysztof Grzelak » Thu May 20, 2021 13:59

Excuse me ask Bert, as the GUI Damage.

Post Reply