building failed (Linux)

[tissatussa]

i tried to build chrysaora on Linux but i failed .. i'm new to NIM but i can read code.

i had to install "arraymancer" so i did :

nimble install arraymancer

this installed v0.6.2 (not v0.6.1 as your README states) .. then i gave this command to build chrysaora :

nim c -d:danger src/chrysaora.nim

but the process ends with an error :

/home/roelof/Downloads/tmp5/chrysaora/src/bootstrap.nim(198, 28) template/generic instantiation of optimizerAdam from here
/home/roelof/.nimble/pkgs/arraymancer-0.6.2/arraymancer/nn/optimizers/optimizers.nim(227, 6) Error: 'optimizerAdam' can have side effects

can you help ?
I'm on Xubuntu 20.04

[dylanagreen]

Hello. arraymancer at versions later than 0.6.1 have introduced an issue with certain code used in Chrysaora, namely the generic instantiation problem. I've opened an issue on that repo to address the issue (the exact cause is a little complicated but the gist of it is that it's not my fault) but until it is resolved the only solution is to install arraymancer at 0.6.1. If you're using nimble you can do this with

nimble install arraymancer@0.6.1

This should solve your problem, although let me know if it does not. I had previously coded into chrysaora a workaround for a similar problem so I am leaving this issue open to remind myself to do that for this as well.

I would actually recommend waiting though as I've encountered an engine weakness due to the quiescence search being weaker than expected and reducing the efficacy of the NN evaluation function. I've evaluated a few solutions to this problem and hope that once it is fixed I can push v0.1.0 officially. I hope to have this done by mid-March.

[tissatussa]

thanks. I managed to uninstall arraymancer@0.6.2 and install arraymancer@0.6.1 .. then i compiled Chrysaora and it succeeded .. but after executing i get this error :

$ ./chrysaora
engine.nim(197) initialize_network
Error: unhandled exception: Weights File not found! [IOError]

indeed i have no weights file .. and i prefer not to train one myself .. where can i obtain a (default) weights file ?

[dylanagreen]

Your best bet at the moment if you just want to run chrysaora, skill level be damned, is to compile and run net.nim which will output a random weight file.

nim c --run src/net.nim

Note that this is hardly any better than chrysaora just making random moves.

I would provide you with a better weights file, but I've discovered a bug in my training process and have to try solve that first. When I have a better one I will put it here for you. :)

[tissatussa]

so Chrysaora never had a default NN file ? Can it use formats like NNUE or Seer ? Did you decide to create an own weights format ?

[dylanagreen]

Chrysaora's network was written before Seer even existed. NNUE hadn't even made it's way into the chess world (Stockfish pushed NNUE in July 2020; Seer was published in October 2020; Chrysaora's network was implemented in mid-2019). Chrysaora was, actually, one of the first to use a fully connected layer only NNUE-styled network, but I did it my way since it was programmed in a language for which the machine learning ecosystem is infantile at best, and nonexistent at worst. To rewrite it to use either of those two formats would be tantamount to having to rewrite the entire evaluation function. Not to mention that support for either of these would have to be coded from the ground up anyway, since their implementations are in C or C++ and Chrysaora is in Nim.

All this to say, yes, it uses its own weights format. I did decide to create my own weights format, but only because when I was writing it I had to do it myself. I had no other option. I've trained dozens of weights files for Chrysaora over the years, but have not made them public because I constantly find improvements and bugs. For example, right now the weights files continually come out white-biased, so Chrysaora always plays to a white win even if she is playing as black. You can understand why this would be problematic, I hope.

[tissatussa]

so you're a pioneer in building an NN in Nim .. that's nice .. i guess i have to wait for your NN version but i "Follow" your GitHub channel : do i get a (email) notification whenever you add / change something here ? Anyway, GitHub sends me notifications of changes in issues like this ..

i'm a programmer, but simple .. coding an NN / trainer is beyond me, but i test engines and i find many (recent) NN weight files, mostly NNUE type .. i wonder how the "quality" / "intelligence" of each different NN can be evaluated ? Do some special (benchmark) positions exist ?

[dylanagreen]

I am a bit of a pioneer, although not perhaps the best pioneer so to speak. I have some radical ideas for the network structure for v0.2.0 and onward, but am still heading towards a releasable v0.1.0.

With regards to benchmarks, I don't know if there are many used any more. The Giraffe paper (which defines the Giraffe engine I based Chrysaora's original network structure on) outlines a test suite they used for that engine that used to be much wider used. It is called the Strategic Test Suite. Directly quoting the paper (page 24):

The Strategic Test Suite is a collection of 1500 positions, split into 15 themes of 100 positions each. Each group of 100 positions tests the engine’s understanding on one strategic idea. For example, one theme tests the understanding of control of open files, another tests the understanding of how bishop and knight’s values change relative to each other in different situations, and yet another tests the understanding of centre control. Unlike most other test suites, the STS positions do not have deep tactical lines for the engine to find, making it especially suitable for evaluating positional understanding of an engine.

Each of the positions in the test suite have a list of solution moves with associated scores. The best solutions always have a score of 10, and other solutions have scores between 0 and 10. At the end of a test run, the scores for the engine’s moves for each of the position are added up to a score out of 15000.

Back when that paper was published Stockfish was only up to Version 5, so Stockfish 5 was tested to score a 10505 on that test suite. Would be interesting to see what Stockfish 13 scores.

Although the problem with these kind of tests suites is that they rely upon human intuition as to what is a "good" position and what is "bad", which ignores the effect that engines may discover positions and sequences that a human may classically feel is bad but that may actually be good on a longer term scale. It is an interesting question to ponder, how to test the relative intelligence of a network without making references to a human baseline. It draws an interesting division between intelligence and skill in terms of chess play, since skill can obviously be easily tested by pitting engines against one another. Indeed also from the paper

The test suite likely under-estimates the positional understanding of Giraffe compared to other engines, because most of the themes tested by the test suite are generally well-understood concepts in computer chess that are implemented by many engines, and since the test suite is famous, it is likely that at least some of the engines have been tuned specifically against the test suite.

So there is not perhaps a best choice solution yet. You would have to decide what "intelligence" means in this context as well, which is again hard to do without referencing humans.

As regards to notifications, you may or may not. I'll be pushing half a dozen to a dozen commits sometime over the weekend, you can test to see if you get a notification for that. I think the default if you're following is that you'll only get notified on the things you comment on, whether any PRs you make get accepted, and if there is a new release.