andygrant / openbench Goto Github PK

View Code? Open in Web Editor NEW

134.0 134.0 119.0 65.32 MB

OpenBench is a Distributed SPRT Testing Framework for Chess Engines

License: GNU General Public License v3.0

Python 73.57% HTML 18.05% Shell 0.03% CSS 5.45% JavaScript 2.90%

openbench's People

Contributors

Stargazers

Watchers

Forkers

laldon ratosh killerducky odomobo hub2 malu sayaksam sovaz1997 fabianvdw kyleaforrester shintakezou jtseng20 theo77186 mhouppin justno4b scchess jhonnold gekkehenker connormcmonigle luecx amanjpro ianagbip1oti akanalytics mladuke assassin65170 analog-hors altarchess minuskelvin snowballsh jwatzman skiminki uwuplant disservin marvinkarhan cooffe dav1312 lightningstream rakarchive thomaslemercier crippa1337 sazgr theblackplague rafid-dev kurt1288 lynx-chess snowdrop4 ciekce omerfaruktutkun pedro15 yoshie2000 a1exl1ang matthies jamestiotio dannyj1 jeff-pow willching01 jamadaha aytchell alex2262 sarona-cccf liamt19 haxk20 mrstark7797 knguy22 sroelants ibaibur freemanzlat witek902 jochengehtab codedeliveryservice kjljixx jgilchrist ffloof brito753 zerbinati vincentbab arjunbasandrai lukedaskywalker vast342 stefanmk mlomb beanie496 dewaeq xu-shawn viren6 gahtan-syarif toanth aronpetko yl25946 styxdoto gekomad jnlt3 sscg13 ilayali bftjoe varaton

openbench's Issues

Admin login fails

With the current source, after doing the recommended setup steps for the server in README.md, I am getting this exception after logging in as admin:

Request Method:	GET
http://192.168.0.71:8000/
2.0.6
OperationalError
no such column: OpenBench_profile.engine
/home/jdart/.local/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py in execute, line 303
/usr/bin/python3
3.6.9
['/home/jdart/chess/OpenBench', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/home/jdart/.local/lib/python3.6/site-packages', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages']
Sat, 22 Feb 2020 18:13:42 +0000

Dynamic GAMES_PER_THREAD

Proposal: replace GAMES_PER_THREAD byconcurrency * 32.

This value is a trade-off:

Smaller is better as it gives more responsiveness to server changes (prio/throughput changes, new tests).
But we need to reduce the waste from the stub-effect (ie. unused threads as we reach the end of a chunk).

The result of this trade-off should (all else equal) be proportional to the concurrency value.

For single threaded tests, concurrency = threads, and it means:

Andrew 32 threads: 1024
WorkerPool 16 threads: 512
lucasart 7: 224

For 8 threaded tests, concurrency = int(threads / 8), so we have:

Andrew 32 threads, concurrency 4: 128
WorkerPool 16 threads, concurrency 2: 64
lucasart 7: cannot play 8 threaded tests (concurrency = 0)

Name builds accordingly

It'd be nice if you could add the suffixes _base and _dev for the associated builds so one can understand what's going on:

Started game 6 of 250 (Ethereal 10.86 (POPCNT) vs Ethereal 10.86 (POPCNT))
Finished game 6 (Ethereal 10.86 (POPCNT) vs Ethereal 10.86 (POPCNT)): 1-0 {White wins by adjudication}

Add support for Laser

Requires resolution of many of the other issues posted here

To remove any oddities for Ethereal's repo setup, have the Client search the engine download for a makefile.
Makefile will still have to support CC= and EXE=, parsing for those dynamically may lead to too many issues...

Workers show active test

Mark tests with timeloss or crashes on homepage

Something as simple as throwing a "*" next to any such tests. I believe crashes and timelosses are only tracked in the result model, and not the overall test stats. To save compute we'll need to either add and set a flag on a test model, or accumulate crashes and timelosses

Prefix worker binaries with engine name

It's now hard to tell if any engines are left over and running in background.

CPU, Worker, NPS count on the index page

Error when registering new machine

I now have a user configured and was able to define Minic engine parameters and submit a job. But when I try to use the client I get this

python3 Client/OpenBench.py -U test -P thegoodpassword -S http://127.0.0.1:8000/ -T 1
<Warning> Machine unregistered, will register with Server
Downloading : <!DOCTYPE html><html lang="en"><head><meta content="text/html; charset=utf-8" http-equiv="content-type"/><title>Page not found at //getFiles/</title><meta content="NONE,NOARCHIVE" name="robots"/><style type="text/css">html * { padding:0; margin:0; } body * { padding:10px 20px; } body * * { padding:0; } body { font:small sans-serif; background:#eee; color:#000; } body>div { border-bottom:1px solid #ddd; } h1 { font-weight:normal; margin-bottom:.4em; } h1 span { font-size:60%; color:#666; font-weight:normal; } table { border:none; border-collapse: collapse; width:100%; } td, th { vertical-align:top; padding:2px 3px; } th { width:12em; text-align:right; color:#666; padding-right:.5em; } #info { background:#f6f6f6; } #info ol { margin: 0.5em 4em; } #info ol li { font-family: monospace; } #summary { background: #ffc; } #explanation { background:#eee; border-bottom: 0px none; }</style></head><body><div id="summary"><h1>Page not found <span>(404)</span></h1><table class="meta"><tbody><tr><th>Request Method:</th><td>GET</td></tr><tr><th>Request URL:</th><td>http://127.0.0.1:8000//getFiles/</td></tr></tbody></table></div><div id="info"><p>Using the URLconf defined in <code>OpenSite.urls</code>, Django tried these URL patterns, in this order:</p><ol><li>admin/</li><li>register/</li><li>login/</li><li>logout/</li><li>viewProfile/</li><li>editProfile/</li><li>index/</li><li>index/&lt;int:page&gt;/</li><li>greens/</li><li>greens/&lt;int:page&gt;/</li><li>search/</li><li>viewUser/&lt;str:username&gt;/</li><li>viewUser/&lt;str:username&gt;/&lt;int:page&gt;/</li><li>users/</li><li>machines/</li><li>eventLog/</li><li>eventLog/&lt;int:page&gt;/</li><li>newTest/</li><li>viewTest/&lt;int:id&gt;/</li><li>editTest/&lt;int:id&gt;/</li><li>approveTest/&lt;int:id&gt;/</li><li>restartTest/&lt;int:id&gt;/</li><li>stopTest/&lt;int:id&gt;/</li><li>deleteTest/&lt;int:id&gt;/</li><li>getFiles/</li><li>getWorkload/</li><li>wrongBench/</li><li>submitNPS/</li><li>submitResults/</li><li></li></ol><p>The current path, <code>/getFiles/</code>, didn't match any of these.</p></div><div id="explanation"><p>You're seeing this error because you have <code>DEBUG = True</code> in your Django settings file. Change that to <code>False</code>, and Django will display a standard 404 page.</p></div></body></html>cutechess-linux
Traceback (most recent call last):
  File "Client/OpenBench.py", line 418, in <module>
    getCoreFiles()
  File "Client/OpenBench.py", line 74, in getCoreFiles
    getFile(location + 'cutechess-linux', 'cutechess')
  File "Client/OpenBench.py", line 54, in getFile
    request = requests.get(url=source, stream=True, timeout=HTTP_TIMEOUT)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 640, in send
    adapter = self.get_adapter(url=request.url)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 731, in get_adapter
    raise InvalidSchema("No connection adapters were found for '%s'" % url)
requests.exceptions.InvalidSchema: No connection adapters were found for

Am i doing something wrong ?

Laser engine keeps running when cutechess ended

Happening on your 30-core worker too.

Issue with Django profil

Trying to set up my own instance for Minic I'm running on this issue as soon as I try to login with the superadmin account

Django Version: | 2.0.6
DoesNotExist
Profile matching query does not exist.
/home/vivien/.local/lib/python3.7/site-packages/django/db/models/query.py in get, line 403
/usr/bin/python3
3.7.5

Does that ring a bell ?

Search: Failed and Neutral flags are mixed up, engine names not showing

Filtering by failed gives yellows, while neutral gives red. Should be the other way around.

Add script to auto delete excess data in the database

After some time, all of the Result objects for a given test may be deleted. Leaving only the final results, cutting out the massive amount of data due to the 40+ machines connected. Additionally, the EventLog may also be cleared.

I have been doing this by hand every few months to keep the database small, but each time I forget how to do it.

No workloads for us but a test is pending

I am now able to run and debug the client. I'm facing this

<Warning> Server has no workloads for us

But a test is pending. Any clue ?

recalibrate NPS

Weiss is running at 3.1 on WorkerPool 16 threads, and 2.9 on Andrew 32 threads. So a more reasonable value would be 3m, instead of the current 4m (which inflates tc by 33%).

Might be worth checking other engines while at it. I reckon both Ethereal and Demolito should use 1.4m, as for Laser, I don't know, but 625k looks low.

Buttons for STC and LTC

Reduce User's test count on test deletion.

Is it possible to download the games from openbench?

Like it can be done with fishtest? If yes, how?

Move NPS factor to the server

Firstly it makes sense to have this on the server -- allowing changes without forcing new client versions.
With this, and some makefile changes, the Client may become generalized enough for ALL engines, such that only OpenBench/config.py need be changed for new engines.

Correctly use Foreign Keys to map Tests, Results, Machines, and Events to Profiles

Create and collect pgn of the test games / biased workers

More a feature request than an issue...

It would be great if cutechess would create pgn files of test games and workers would collect, zip and send them to server for download.

There are some really strange results in some tests and I would like to run some evaluation on the pgns to see if its just statistical noise or some real problem of workers.

Example: http://chess.grantnet.us/viewTest/4668/ should be (almost) completely neutral but in this moment:
Andrews worker ist at +1044 -1159 =2725
My worker ist at +772 -647 =1717

Proper HTTP request timeout

Default behavior of requests.get is never, causing worker hanging.

Create tests with tags

I've added this option on my fork, just looks for a tag if the branch is not found.
Changes

invalid bench?

@AndyGrant: your machine stopped this test http://chess.grantnet.us/viewTest/4413/ due to invalid bench. both gcc and clang compile fine, and give me the same bench. workerpool also didn't have problems, and he's using an old version of the client (and gcc). did you change the bench parsing logic again ?

Here are the last lines of Demolito's output:

seal  : 11c28f6d2b272522
time  : 7353ms
nodes : 12681936
nps   : 1724729

Use checksum verification for books

Add a checksum (generated with some python lib.) to the config file with the books.
Send the checksum with the workload.
Have the worker compute the sum and check ... error out if not matching.

Duplicated approve/stop/restart/delete events

Auto append trailing "/" to source path for tests

Qt 4 dependancy ?

When running the OpenBench.py client, it downloads the cutechess (linux) executable, and running it throws this error:

./cutechess: error while loading shared libraries: libQtCore.so.4: cannot open shared object file: No such file or directory

It's the cutechess executable hat fails to run on my machine. But my cutechess-cli (which I compiled from source using Qt5) works fine. Are you using a (very) old version that is based on Qt4 ?

PS: For now, I'll just hack the python client to make it use my own cutechess-cli.

Auto delete machine.txt when invalid

Initially did not do this for testing reasons, to avoid flooding DB when testing.
However, OS name changes when doing kernal updates, and forces an error on the server side.

Better input checking on Test Creation

Issues appear when no TC and no Options are specified.

Add support for RubiChess

Start of support thread for RubiChess

Remove --compiler= and --profile=

Support for Demolito

@lucasart , what time controls and hash settings would you like for Demolito's STC/LTC/SMP testing? Here is the template for Ethereal/Laser

'testmodes' : {
    'stc'     : { 'threads' : 1, 'hash' :   8, 'timecontrol' : '10.0+0.1' },
    'ltc'     : { 'threads' : 1, 'hash' :  64, 'timecontrol' : '60.0+0.6' },
    'smpstc'  : { 'threads' : 8, 'hash' :  64, 'timecontrol' : '5.0+0.05' },
    'smpltc'  : { 'threads' : 8, 'hash' : 256, 'timecontrol' : '20.0+0.2' },
},

Isn't priority redundant with throughput ?

I think priority can be removed, and throughput can be used instead. Example:

default: all tests use normal 1000 throughput.
to put a test in front of all others, instead of setting priority=1, one can simply set throughput to zero for all other tests. At least it makes the intention clear. Of course, this is an anti-social behavior to begin with, but that's a different discussion.
to put a test behind all others, instead of setting priority=-1, one can simply set throughput to zero for the test.

PS: The reason why Fishtest has both priority and throughput is historical. Priority was available first, and throughput was invented later.

Verify machineID when hitting submitNPS and submitResults

machine unregistered

Tried to run the client, but it crashed:

[lucas@megatron Client]$ python3 ./OpenBench.py -U lucasart -P ********* -S http://chess.grantnet.us/ -T 7
<Warning> Machine unregistered, will register with Server
Downloading : <!DOCTYPE html><html lang="en"><head><meta content="text/html; charset=utf-8" http-equiv="content-type"/><title>Page not found at //getFiles/</title><meta content="NONE,NOARCHIVE" name="robots"/><style type="text/css">html * { padding:0; margin:0; } body * { padding:10px 20px; } body * * { padding:0; } body { font:small sans-serif; background:#eee; color:#000; } body>div { border-bottom:1px solid #ddd; } h1 { font-weight:normal; margin-bottom:.4em; } h1 span { font-size:60%; color:#666; font-weight:normal; } table { border:none; border-collapse: collapse; width:100%; } td, th { vertical-align:top; padding:2px 3px; } th { width:12em; text-align:right; color:#666; padding-right:.5em; } #info { background:#f6f6f6; } #info ol { margin: 0.5em 4em; } #info ol li { font-family: monospace; } #summary { background: #ffc; } #explanation { background:#eee; border-bottom: 0px none; }</style></head><body><div id="summary"><h1>Page not found <span>(404)</span></h1><table class="meta"><tbody><tr><th>Request Method:</th><td>GET</td></tr><tr><th>Request URL:</th><td>http://chess.grantnet.us//getFiles/</td></tr></tbody></table></div><div id="info"><p>Using the URLconf defined in <code>OpenSite.urls</code>, Django tried these URL patterns, in this order:</p><ol><li>admin/</li><li>register/</li><li>login/</li><li>logout/</li><li>viewProfile/</li><li>editProfile/</li><li>index/</li><li>index/&lt;int:page&gt;/</li><li>greens/</li><li>greens/&lt;int:page&gt;/</li><li>search/</li><li>viewUser/&lt;str:username&gt;/</li><li>viewUser/&lt;str:username&gt;/&lt;int:page&gt;/</li><li>users/</li><li>machines/</li><li>eventLog/</li><li>eventLog/&lt;int:page&gt;/</li><li>newTest/</li><li>viewTest/&lt;int:id&gt;/</li><li>editTest/&lt;int:id&gt;/</li><li>approveTest/&lt;int:id&gt;/</li><li>restartTest/&lt;int:id&gt;/</li><li>stopTest/&lt;int:id&gt;/</li><li>deleteTest/&lt;int:id&gt;/</li><li>getFiles/</li><li>getWorkload/</li><li>wrongBench/</li><li>submitNPS/</li><li>submitResults/</li><li></li></ol><p>The current path, <code>/getFiles/</code>, didn't match any of these.</p></div><div id="explanation"><p>You're seeing this error because you have <code>DEBUG = True</code> in your Django settings file. Change that to <code>False</code>, and Django will display a standard 404 page.</p></div></body></html>cutechess-linux
Traceback (most recent call last):
  File "./OpenBench.py", line 418, in <module>
    getCoreFiles()
  File "./OpenBench.py", line 74, in getCoreFiles
    getFile(location + 'cutechess-linux', 'cutechess')
  File "./OpenBench.py", line 54, in getFile
    request = requests.get(url=source, stream=True, timeout=HTTP_TIMEOUT)
  File "/usr/lib/python3.8/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3.8/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 640, in send
    adapter = self.get_adapter(url=request.url)
  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 731, in get_adapter
    raise InvalidSchema("No connection adapters were found for '%s'" % url)
requests.exceptions.InvalidSchema: No connection adapters were found for '<!DOCTYPE html><html lang="en"><head><meta content="text/html; charset=utf-8" http-equiv="content-type"/><title>Page not found at //getFiles/</title><meta content="NONE,NOARCHIVE" name="robots"/><style type="text/css">html * { padding:0; margin:0; } body * { padding:10px 20px; } body * * { padding:0; } body { font:small sans-serif; background:#eee; color:#000; } body>div { border-bottom:1px solid #ddd; } h1 { font-weight:normal; margin-bottom:.4em; } h1 span { font-size:60%; color:#666; font-weight:normal; } table { border:none; border-collapse: collapse; width:100%; } td, th { vertical-align:top; padding:2px 3px; } th { width:12em; text-align:right; color:#666; padding-right:.5em; } #info { background:#f6f6f6; } #info ol { margin: 0.5em 4em; } #info ol li { font-family: monospace; } #summary { background: #ffc; } #explanation { background:#eee; border-bottom: 0px none; }</style></head><body><div id="summary"><h1>Page not found <span>(404)</span></h1><table class="meta"><tbody><tr><th>Request Method:</th><td>GET</td></tr><tr><th>Request URL:</th><td>http://chess.grantnet.us//getFiles/</td></tr></tbody></table></div><div id="info"><p>Using the URLconf defined in <code>OpenSite.urls</code>, Django tried these URL patterns, in this order:</p><ol><li>admin/</li><li>register/</li><li>login/</li><li>logout/</li><li>viewProfile/</li><li>editProfile/</li><li>index/</li><li>index/&lt;int:page&gt;/</li><li>greens/</li><li>greens/&lt;int:page&gt;/</li><li>search/</li><li>viewUser/&lt;str:username&gt;/</li><li>viewUser/&lt;str:username&gt;/&lt;int:page&gt;/</li><li>users/</li><li>machines/</li><li>eventLog/</li><li>eventLog/&lt;int:page&gt;/</li><li>newTest/</li><li>viewTest/&lt;int:id&gt;/</li><li>editTest/&lt;int:id&gt;/</li><li>approveTest/&lt;int:id&gt;/</li><li>restartTest/&lt;int:id&gt;/</li><li>stopTest/&lt;int:id&gt;/</li><li>deleteTest/&lt;int:id&gt;/</li><li>getFiles/</li><li>getWorkload/</li><li>wrongBench/</li><li>submitNPS/</li><li>submitResults/</li><li></li></ol><p>The current path, <code>/getFiles/</code>, didn't match any of these.</p></div><div id="explanation"><p>You're seeing this error because you have <code>DEBUG = True</code> in your Django settings file. Change that to <code>False</code>, and Django will display a standard 404 page.</p></div></body></html>cutechess-linux'

Move custom paths to the Server side

Right now the Client can specify a path to compile a given engine. This is useless and requires updates per engine addition. We should store this information on the server and send it with each workload.

This should not impact the old client in any way.

View for Greens, and User Tests, no longer works.

Allow connections via HTTPS

It's 2018 duuude.

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='chess.grantnet.us', port=443):
Max retries exceeded with url: /getFiles/
(Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x0000006703F5FA90>:
Failed to establish a new connection:
[WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond',))

make build command custom (not make, but a user command)
do not depend on a '/src' directory, stay on root directory of the repo, the custom command can cd where it wantsto
possible configuration of own cutechess bin dir

Kill cutechess when worker is terminated.

Working to do list

Parse better to avoid requiring Threads=X (Don't fix space issue)
Better error reporting on bad test creation
Send NPS factor from Server, add to OpenBench/config.py
Write a paging function in OpenBench.utils, add page nums to Index / Event Log
Add information about makefile EXE= to README

Expose partial worker machine identity

So when things went wrong I can know on which machine they happened.