Fishtest displays memory and config file error after using Lenovo Vantage memory cleanup; restart already tried

[kunningKing11]

Fishtest has worked fine for me for a while, until I used Lenovo Vantage's memory cleanup feature (while fishtest was not running). Since then, every time I have tried to open fishtest it shows an error about reserved memory and it says the config file is not written (even if it is).

I don't know if this helps, but here is the system info:
Intel Core Ultra 185H
64 GB RAM
NVIDIA RTX 4070 Mobile Graphics Card (8 GB)

Note: I have tried recreating the worker multiple times and it still doesn't work. I have converted the .config file to a .txt file so it can be uploaded.

fishtest.txt

[Disservin]

I deleted your fishtest.txt file since it included your password... please change it nonetheless

[vdbergh]

Everything seems to be fine with your configfile. I guess something is wrong with how the worker determines the amount of memory. This is the powershell command the worker uses

(Get-CimInstance Win32_OperatingSystem).TotalVisibleMemorySize*1024

Perhaps you can try it directly?

[kunningKing11]

Everything seems to be fine with your configfile. I guess something is wrong with how the worker determines the amount of memory. This is the powershell command the worker uses

(Get-CimInstance Win32_OperatingSystem).TotalVisibleMemorySize*1024

Perhaps you can try it directly?

It works fine when I try it (it outputs 68,305,674,240) but Fishtest still doesn't work.

[vdbergh]

Hmm. That is strange. Can you rename the config file and restart the worker? Then it will create a new config file (it will ask for your username and password again, if you don't provide it on the command line).

[kunningKing11]

Hmm. That is strange. Can you rename the config file and restart the worker? Then it will create a new config file (it will ask for your username and password again, if you don't provide it on the command line).

It still displays the same error after I tried that.

[vdbergh]

Ok. I wanted to exclude that the config file was somehow corrupted.

Unfortunately the only way to make progress seems to be to use an instrumented worker.py script.

[kunningKing11]

Ok. I wanted to exclude that the config file was somehow corrupted.

Unfortunately the only way to make progress seems to be to use an instrumented config file.

What do you mean?

[vdbergh]

Ok. I wanted to exclude that the config file was somehow corrupted.
Unfortunately the only way to make progress seems to be to use an instrumented config file.

What do you mean?

That I post a file worker.py here with some debug prints and that you run it locally.

[vdbergh]

worker.py.zip

If you want to, you can replace worker.py with the above zipped instrumented worker.py and post the output here. It adds three test prints. Hopefully this will clarify at which point the worker goes off the rails.

On my system the output looks as follows:

______ _     _     _            _                        _
|  ___(_)   | |   | |          | |                      | |
| |_   _ ___| |__ | |_ ___  ___| |_  __      _____  _ __| | _____ _ __
|  _| | / __| '_ \| __/ _ \/ __| __| \ \ /\ / / _ \| '__| |/ / _ \ '__|
| |   | \__ \ | | | ||  __/\__ \ |_   \ V  V / (_) | |  |   <  __/ |
\_|   |_|___/_| |_|\__\___||___/\__|   \_/\_/ \___/|_|  |_|\_\___|_|

Worker started in /home/vdbergh/fishtest/worker ... (PID=2388472)
************** Detected amount of memory: 16437702656
Found g++ version 13.3.0
No clang++ or clang++ is not executable
************** Computed max_memory: ('MAX/2', 7838)
************** Computed max_concurrency: 21
...

[kunningKing11]

This is the output:

PS C:\Program Files\Stockfish Workers\worker_3_cores> python worker.py

______ _     _     _            _                        _
|  ___(_)   | |   | |          | |                      | |
| |_   _ ___| |__ | |_ ___  ___| |_  __      _____  _ __| | _____ _ __
|  _| | / __| '_ \| __/ _ \/ __| __| \ \ /\ / / _ \| '__| |/ / _ \ '__|
| |   | \__ \ | | | ||  __/\__ \ |_   \ V  V / (_) | |  |   <  __/ |
\_|   |_|___/_| |_|\__\___||___/\__|   \_/\_/ \___/|_|  |_|\_\___|_|

Worker started in C:\Program Files\Stockfish Workers\worker_3_cores ... (PID=9564)
************** Detected amount of memory: 32
Found g++ version 14.2.0
Found clang++ version 19.1.4
************** Computed max_memory: ('MAX/2', 0)
************** Computed max_concurrency: 0
You need to reserve at least 360 MiB to run the worker!
Error parsing options. Config file not written.

[kunningKing11]

I'm not sure what the detected memory value is supposed to be but it seems far too low (32 bytes compared to 64GB system memory)

[vdbergh]

Yes. So despite the powershell command giving the right answer, the worker still detects the amount of memory incorrectly. We will have to dig a bit deeper.

[vdbergh]

I suspect these lines are the culprit

        with os.popen(cmd) as proc:
            mem_str = str(proc.readlines())
        mem = int(re.search(r"\d+", mem_str).group())

I think the regular expression does not parse the output of the powershell command correctly.

Can you give the full output of the powershell command?

(Get-CimInstance Win32_OperatingSystem).TotalVisibleMemorySize*1024

[kunningKing11]

68305674240

[vdbergh]

68305674240

That's strange. Does it really only print this? Nothing more?

[kunningKing11]

yes

[vdbergh]

Ok. There is still a possibility that somehow the command gives something different when executed by the worker. I will make a new version of worker.py with some extra test prints.

[vdbergh]

Ok here is a new instrumented version

worker.py.zip

On my system it prints

______ _     _     _            _                        _
|  ___(_)   | |   | |          | |                      | |
| |_   _ ___| |__ | |_ ___  ___| |_  __      _____  _ __| | _____ _ __
|  _| | / __| '_ \| __/ _ \/ __| __| \ \ /\ / / _ \| '__| |/ / _ \ '__|
| |   | \__ \ | | | ||  __/\__ \ |_   \ V  V / (_) | |  |   <  __/ |
\_|   |_|___/_| |_|\__\___||___/\__|   \_/\_/ \___/|_|  |_|\_\___|_|

Worker started in /home/vdbergh/fishtest/worker ... (PID=2458386)
************** system_type: linux
************** Memory command returned: ['               total        used        free      shared  buff/cache   available\n', 'Mem:     16437702656 13967527936  1224843264  2201468928  3799961600  2470174720\n', 'Swap:     4294963200  4272640000    22323200\n']
************** Detected amount of memory: 16437702656
Found g++ version 13.3.0
No clang++ or clang++ is not executable
************** Computed max_memory: ('MAX/2', 7838)
************** Computed max_concurrency: 21

[kunningKing11]

this is the output:

from the file output, I thought could have been due to the execution policy but I changed it to RemoteSigned and it still doesn't work

[vdbergh]

So it is a windows configuration issue.

I found this... https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_execution_policies?view=powershell-7.4

I have no idea how Windows knows that the worker has been downloaded from the internet.

I assume you have to set the execution policy to unrestricted (possibly with a limited scope).

Maybe @ppigazzini knows more...