vitogit/pgn-tactics-generator

Generate chess puzzles / tactics from a pgn file

pgn-tactics-generator

About

This is a python application dedicated to creating chess puzzles/tactics from a pgn file. Also it can download your games from lichess.org and use that file.

It's based on the great https://github.com/clarkerubber/Python-Puzzle-Creator by @clarkerubber

Things that I changed:

• Use a local pgn file with games as a source.
• Write results to a file called tactics.pgn
• Default engine depth to 8, so it's faster. Before it was nodes=3500000 this is a depth around 20. So it took several minutes to analyze a game. With depth 8 it takes seconds.
• You can use the depth argument to change the depth if you want more precision.
• chess.pop_count to chess.popcount, because it was failing

This is too complex, give something easy.

There is another option if you don't want to install and manage python scripts I created a more user friendly tactics generator and it's online http://chesstacticsgenerator.vitomd.com It uses a different approach to create tactics, so probably it will generate a different set of tactics.

Installation

This script requires the Requests and Python-Chess libraries to run, as well as a copy of Stockfish Is recommended that you use Python 3 and pip3. But it could work with Python 2.7 and pip (probably you will need to install futures pip install futures )

Please, take a look at development doc for details.

Install requirements

pip3 install -r requirements.txt --user

Setup

MacOS / Linux : sh build-stockfish.sh to obtain the current lichess Stockfish instance.

Launching Application

Downloading games for a specific user

You can download games from a specific user using this command: python3 download_games.py <lichess username>

By default, it will download the last 60 games from blitz, rapid and classical.

Arguments

You can use the max argument to get more games and use the lichess api token with the token argument to make the download faster. https://lichess.org/api#operation/apiGamesUser

It will save the games in the games.pgn file

Example to get 100 games using the token

python3 download_games.py <lichess username> --max 100 --token 123456789

Downloading games from tournaments

You can download games from multiple tournaments using this command:

python3 download_tournaments.py E14kHVwX tdntXNhy

The arguments are the tournaments ids separate by a space

It will save the games in the games.pgn file

Generate tactics

To execute the generator execute this command. By default it will look for the games.pgn file:

python3 main.py

Arguments

• --quiet to reduce the screen output.
• --depth=8 select the Stockfish depth analysis. Default is 8 and will take some seconds to analyze a game, with --depth=18 will take around 6 minutes.
• --games=ruy_lopez.pgn to select a specific pgn file. Default is games.pgn
• --strict=False Use False to generate more tactics but a little more ambiguous. Default is True
• --threads=4 Stockfish argument, number of engine threads, default 4
• --memory=2048 Stockfish argument, memory in MB to use for engine hashtables, default 2048
• --includeBlunder=False If False then generated puzzles won't include initial blunder move, default is True
• --stockfish=./stockfish-x86_64-bmi2 Path to Stockfish binary. Optional. If omitted, the program will try to locate Stockfish in current directory or download it from the net

Example: python3 main.py --quiet --depth=12 --games=ruy_lopez.pgn --strict=True --threads=2 --memory=1024

Tactics output

The resulting file will be a pgn file called tactics.pgn. Each tactic contains the headers from the source game. The result header is the tactic result and not the game result. It can be loaded to a Lichess study or to an app like iChess to practice tactics.

Problems?

Stockfish errors

• If you have problems building Stockfish try downloading Stockfish directly https://stockfishchess.org/download/

Want to see all my chess related projects?

Check My projects for a full detailed list.

pgn-tactics-generator's Issues

Puzzle incomplete?

I use the attached PGN file and the following command:

python3 main.py --games=A.pgn --stockfish=stockfish --depth=12

After a series of outputs I got:

Puzzle incomplete
Generating new puzzle...
k . . . r . . .
. b . . . . . R
. . . . . . . .
. . . . . . . .
. P . . . . . .
. . . . . . P .
P . . . . P . P
. . . . R . K .
Material Value: 2.5
Evaluating Best Move...
Best Move: e8e1
   CP: None
   Mate: 1
Evaluating Legal Moves...
Move: e8e1
   CP: None
   Mate: 0
Move: e8d8
   CP: 1015
   Mate: None
Move: e8c8
   CP: 1038
   Mate: None
... and 20 more moves
Not Going Deeper:
   Ambiguous: False
   Game Over: True
   Has Best Move: True
Puzzle incomplete

A.pgn.zip

New argument to include the blunder move or not

As @karol-brejna-i said here #5 (comment) some sites start with the blunder move like chess.com and others starts directly with the puzzle to solve with the blunder already in the position.

Currently the puzzle generator includes the opponent blunder move.

So my idea is to have a new argument called --includeBlunder with a default value of true but you can set it to false to not include the blunder move and have the position with the blunder already there.

questionable quality

This generator creates puzzles of questionable quality. The collection in https://github.com/xinyangz/chess-tactics-pgn that uses this generator has some issues:
+++++++++++++++++++++++++
Event "0708 49919"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "?"]
[Black "solver"]
[Result "*"][SetUp "1"]
[FEN "2kr4/1p1r1p2/2p1p2p/8/P1P1Rp2/8/1P3PPP/R5K1 w KQkq - 0 1"]

1.Rxf4 Rd1+ 2.Rxd1 Rxd1# *
How can castle flags be KQkq when both kings are not in the initial positions??
How can result be * (both in Tags and in moves) when the game ends with checkmate?
+++++++++++++++++++++++++++++++++++
[Event "Rated Blitz game"]
[Site "https://lichess.org/zY14tLJJ"]
[Date "????.??.??"]
[Round "?"]
[White "RonaldFranco"]
[Black "A_R_A_K_E_L_Y_A_N"]
[Result "1-0"]
[BlackElo "1601"]
[BlackRatingDiff "-13"]
[ECO "C02"]
[FEN "6k1/p5pp/8/P2N4/8/4p3/RP3rPP/2bK4 b - - 3 30"]
[Opening "French Defense: Advance Variation, Nimzowitsch System"]
[SetUp "1"]
[Termination "Normal"]
[TimeControl "360+2"]
[UTCDate "2014.04.23"]
[UTCTime "19:07:56"]
[WhiteElo "1529"]
[WhiteRatingDiff "+13"]

30... Bxb2 31. Nxe3 Bc1 32. Rxf2 Bxe3 1-0
1st Black's move is a blunder and the whole puzzle is confusing. It starts with Black's move and ends with White win.

Use short arguments

A conversation in this pr #20 about arguments

I am wondering if we shouldn't invert the logic (meaning) of the argument: having something like skipBlunder instead of includeBlunder. Then we could have simpler construction, like ./main.py --skipBlunder

I think we can have shorthands for the params, like -sb to skip blunder or just -s, -q for the quiet param, etc

invalid result?

[Event "0728 03099"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "solver"]
[Black "?"]
[Result "*"][SetUp "1"]
[FEN "3r3k/6pp/3Q4/q7/8/4P2P/6P1/5RK1 b KQkq - 0 1"]

1...Rxd6 2.Rf8# *

several issues here:

1. invalid tag [Result "*"][SetUp "1"]
2. invalid result, must be 1-0
3. what kind of puzzle is it? While blunder? Does it mean that the 1st move is always a wrong one?
   In the referred https://github.com/xinyangz/chess-tactics-pgn/ I see these kinds of 'puzzles' along with the right ones, when whoever does the 1st move wins. The guy claims that his puzzles are generated by your program, so how do I distinguish which are which?

Improve Download tournaments file

Update the file https://github.com/vitogit/pgn-tactics-generator/blob/master/download_tourments.py to take tournaments ids as arguments instead of modifying the file to add the tournaments ids.

Something like
python3 download_tourments.py 25MtoToy, E14kHVwX

option to skip the moves from a bin book

I would like an option to skip the moves from a bin book, (or some other pgn file).
I want this because I want to search for tactics in a pgn database from games that could arise after the opining lines of my white repertoire.
I already checked the lines of my white repertoire with an engine many plies deep.
So I want to skip these lines to save time.
So I could create a bin book firs from my white repertoire and tell pgn-tactics-generator to skip these moves.
Or is there some way around, this could be done?

Drop support for Python 2.x

Python 2.x is EOLed.
Let's plan to drop support for it.
Action required:

• clean the code (compatibility patches)
• update docs

Support for Custom Stockfish Binary Path

Hi there! This tool seems to be useful but the problem is it may not work in Apple M1 chips as it is using lichess's fork of Stockfish. Can you add an option to set a custom path for Stockfish-binary in CLI arguments?

License?

What license is developed under? Please, consider adding, for example, Apache 2.0 license.

tactics.pgn is empty

Good morning,

I got following issue when running python3 main.py --quiet --depth=12 --strict=True --threads=2 --memory=1024

There was nothing been generated in the tactics.pgn file and there is no error message when running.

pgn-tactics-generator %python3 main.py --quiet --depth=12 --strict=True --threads=2 --memory=1024
Looking up stockfish-osx-x86_64 ...
Local stockfish-osx-x86_64 is newer than release
pgn-tactics-generator %

The pgn file I used is this one:

[Event "Rated Rapid game"]
[Site "https://lichess.org/u2QGEPmx"]
[Date "2022.04.25"]
[White "Ruslan0209"]
[Black "qH11"]
[Result "0-1"]
[UTCDate "2022.04.25"]
[UTCTime "11:51:07"]
[WhiteElo "2012"]
[BlackElo "1988"]
[WhiteRatingDiff "-6"]
[BlackRatingDiff "+14"]
[Variant "Standard"]
[TimeControl "1200+7"]
[ECO "D10"]
[Opening "Slav Defense"]
[Termination "Normal"]

1. d4 d5 2. c4 c6 3. Nc3 Nf6 4. cxd5 cxd5 5. Nf3 a6 6. Bf4 Nc6 7. e3 Bg4 8. Qb3 Bxf3 9. gxf3 Na5 10. Qa4+ Nc6 11. Bd3 e6 12. h4 Be7 13. h5 h6 14. O-O-O b5 15. Qc2 O-O 16. Rdg1 Kh8 17. Kb1 Qa5 18. Rg2 Rac8 19. Qd1 Nb4 20. a3 Nxd3 21. Qxd3 Bxa3 22. Rhg1 Rg8 23. Bxh6 gxh6 24. Rxg8+ Nxg8 0-1

Could you help me on this ?
Thanks

Unknown class ConfigError in fishnet.py

See: https://github.com/lazydroid/pgn-tactics-generator/blob/master/modules/fishnet/fishnet.py#L90

It looks like we lack a definition for the class ConfigError

Unless there was some "special" logic planned for the class, this simple addition would fix the issue:

class ConfigError(Error):
    """Error in configuration."""

Let me know if it is ok.

strict argument doesn't seem to work as expected

Consider the following pseudocode (in main):

settings = parser.parse_args()
print(f'strict value: {settings.strict}')
print(f'strict type: {type(settings.strict)}')
print(f'strict boolean: {bool(settings.strict)}')

The following invocation:

(.venv) pgn-tactics-generator$ ./main.py --strict=False

would produce unwanted results:

strict value: False
strict type: <class 'str'>
strict boolean: True
Looking up stockfish-x86_64-bmi2 ...

strict variable value is 'False' (string). bool( evaluates to True...

Probably something similar to

pgn-tactics-generator/main.py

Lines 46 to 48 in f81c421

	parser.add_argument("--includeBlunder", metavar="INCLUDE_BLUNDER", default=True,
	type=str2bool, const=True, dest="include_blunder", nargs="?",
	help="If False then generated puzzles won't include initial blunder move")

would fix the issue.

Example games files that are guaranteed to produce output tactics

@vitogit Can you share some PGN files that are guaranteed to produce output tactics (tactics.pgn)?

It would be very helpful for testing/development.

We could even consider adding them to the repo for regression tests.

create tactics only for white or for black

I would like an option to only search for tactics for white or for black. Than for the white tactics black makes a move and the tactic for white has to be given.
The same for black, than white plays a move than the tactic for black has to be given.
Is this possible somehow? If not I would like to request this option.

there are many doubles in the tactics output

There are allot of doubles in the tactics output.
I used this script main.py --quiet --depth=12 --games=games.pgn --strict=True --threads=4 --memory=2048 --includeBlunder=False

And its not easy to find the doubles. If I search for the doubles in Chessbase also not exact doubles get detected. When the tactic starts with a slightly different position.

python-chess breaking the code

python-chess library is updated without backwards compatibility and the code won't work anymore. Seems they replaced chess.uci with chess.engine however the implementation of chess.engine is quite different so your code needs to be updated.

Remove unused imports and code

This is a fork of another library so there are imports and code that is not being use. The idea is to remove all of them.

Any plans to merge https://github.com/lazydroid/pgn-tactics-generator?

Are aware of this fork? Do you plan to merge some changes from the other repo?

I was thinking about using pgn-tactics-generator, do some code cleanup, etc. so I wanted to know just in case...