mklarz / cipher_identifier Goto Github PK
View Code? Open in Web Editor NEWWIP of a script that tries to identify symbol ciphers (for now) available on dcode.fr based on input.
Home Page: https://corax.team
WIP of a script that tries to identify symbol ciphers (for now) available on dcode.fr based on input.
Home Page: https://corax.team
Currently we only base the "weight" on the confidence from Tesseract and each model. Should consider adding a weight calculation process after getting the text output from the OCR that checks if the text contains any words from a wordlist (i.e. the british-english one).
The current codebase is messy, split the methods and constants into separate files.
We need to improve the quality of the input images before we pass them to Tesseract to increase the chance of correctly guessing the cipher.
See https://tesseract-ocr.github.io/tessdoc/ImproveQuality
Also see https://tesseract-ocr.github.io/tessdoc/ImproveQuality#examples
Properly document the methods and add a simple readthedocs page
It's not limited to dcode, but can actually be used for any (symbol, for now) cipher. Find a new better fitting name.
Seems like we don't correctly check the charset of the cipher.
$ python scripts/generate_train_data.py atlantean-language
Generating 10000 training images for cipher: atlantean-language
Cipher image count: 39
Cipher charset: 0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZcst
Generating image #1 [sentence="HMS nutriment shantytown", Traceback (most recent call last):
File "scripts/generate_train_data.py", line 534, in <module>
generate_train_data(cipher, wordlist, limit=limit)
File "scripts/generate_train_data.py", line 442, in generate_train_data
symbols = get_symbols_from_text(symbol_mapping, sentence)
File "scripts/generate_train_data.py", line 401, in get_symbols_from_text
symbols.append(symbol_mapping[character])
KeyError: 'n'
There are other OCR engines that may perform better in our case (OCR on symbols). Test the variants and see if the accuracy of the recognition improve.
Also see https://github.com/OCR4all/OCR4all
We need to somehow make sure the confidence of each model is above a certain expected threshold.
Add unit tests that randomly generates images for the ciphers and uses the models to verify the accuracy of the models.
We need an information file (cipher.json?) that includes the name of the cipher, a description, charset and the list of supported characters.
This would also have to be added when downloading the ciphers from dcode.
In addition, each cipher should have a README.md in their directory with this information, for easier viewing in GitHub.
Looks like the wordlist is running out of words, perhaps randomize the cases if the charset contains upper- and lowercase characters?
$ python scripts/generate_train_data.py babylonian-numbers
Generating 10000 training images for cipher: babylonian-numbers
Cipher image count: 15
Cipher charset: 0123456789ABCDE
Traceback (most recent call last):
File "scripts/generate_train_data.py", line 534, in <module>
generate_train_data(cipher, wordlist, limit=limit)
File "scripts/generate_train_data.py", line 434, in generate_train_data
sentences = generate_sentences(
File "scripts/generate_train_data.py", line 163, in generate_sentences
word = wordlist.pop()
IndexError: pop from an empty deque
Should be easy enough to create a web app that processes input images and outputs the ciphers' confidence and text. Easier than having to install the package locally.
Just to test linting to make sure we have a proper coding style throughout the project.
Might be worth looking into generating variants of the training images, currently they're too perfect. Create variants that are skewed, pixelated, noisy, etc.
It should in theory be possible to handle most (text) ciphers by attempting to decipher them for each available cipher and checking the output against a language wordlist.
This should be fairly easy considering we don't need to do any image processing and can do a weight calculation on how many of the deciphered words exist in the provided wordlist.
See https://github.com/dhondta/python-codext for some ciphers
The different ciphers need to have separate image preprocessing settings as some of the settings could remove essential information. Need to figure out what the optimal settings are and save these to the cipher's information file. See below for a bad image preprocessing for the Braille alphabet:
In the case of Braille this should ideally stop at grayscaling.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.