colindean/optar

OPTical ARchiver - highly compressed 2D barcode for paper or film archiving of digital data

Twibright Optar

Special note: The repository of optar at http://github.com/colindean/optar exists because I (@colindean) had difficulties finding the source and it seemed to be abandoned/dormant. I've made slight improvements over time. Improvements welcomed as pull requests.

This is a program to store data on paper using a 600 dpi black and white laser printer and a 600+ dpi scanner.

Building

You need to install ImageMagick so that the resulting .pgm image can be converted into PostScript with the right dimensions: each pixel must be 3x3 600dpi pixels so that there is no unnecessary jitter.

Make sure you have libpng installed. You'll know that is is installed correctly if you type libpng-config on the commandline, there's a program which prints something. Virtually all desktop Linux distros and every macOS system has this installed already.

Compile with

make

Note that there may be some hardcoded configuration values that you may need to change, for example the page size defaults to A4 instead of US Letter. Read on to learn where to change that.

Installing locally

It's easiest to install your local build with

sudo make install

optar, unoptar, and pgm2ps installed on your system in /usr/local/bin.

To uninstall, run

sudo make uninstall

Encoding (writing)

Run

optar other_guys.ogg other_guys.ogg

which will produce files:

other_guys.ogg_0001.pgm
other_guys.ogg_0002.pgm
other_guys.ogg_0003.pgm
other_guys.ogg_0004.pgm
other_guys.ogg_0005.pgm
other_guys.ogg_0006.pgm

Now convert them into PostScript using the included pgm2ps tool:

pgm2ps *.pgm

Print these using a 600 dpi (or greater resolution) laser printer. Inkjet or dot matrix was never tested and will not probably work at the pre-defined data density. See "Changing the format" below.

Please note that the file will be padded by zeroes and the original length will be lost. Pack your data with tar if you store data that are sensitive to this.

Decoding (reading)

Clean and polish the scanner glass with rubbing alcohol and paper towel. Put yellow pages on the scanner lid to get sharper picture˚. Insert the page so that the text on the bottom is upright. Scan the pages into PNG (not JPEG!) on 600dpi (or 1200dpi, slightly better):

scan_0001.png
scan_0002.png
scan_0003.png
scan_0004.png
scan_0005.png
scan_0006.png

Read the number sequence (format specification) from any of the papers and feed it as 1st argument to the optar, 2nd argument is the filename part before the underscore:

unoptar 0-65-93-24-3-1-2-24 scan > out.ogg

Then play out.ogg with mplayer. You should get first about 41 seconds from the Ogg Vorbis file.

˚ In the scanner I tried (Canoscan), the lid didn't seem to be heavy enough to press the paper down completely - there were blurry spots in the picture. Without yellow pages I got 526 reparable bad bits bad from 3.2 million. With yellow pages the blurry spots were much sharper and I got only 261 reparably bad bits!

Please note the data are padded with zeroes so the original information about file length is lost. If your data format doesn't like this then first pack your data with tar.

A4 <-> US Letter

Change the convert parameters in pgm2ps (see comments). Change XCROSSES and YCROSSES in optar.h (see comments). Recompile. Then you can use US Letter instead of A4.

Changing the format

If your printer is low quality and you are getting irreparable bits, you can try to format the media to lower capacity. Unfortunately, setting by commandline is not implemented yet. Change XCROSSES and YCROSSES in optar.h to lower values which yields bigger pixels and lower capacity per page, but higher reliability. Make sure they are in roughly the same proportion as before, otherwise you get nonsquare pixels and unnecessary waste of channel capacity.

You can also change the decoding parameters in unoptar.c (look for MAGIC CONSTANTS) in attempt to read a difficult recording: unsharp_mask, unsharp_dist, sync_white_cut, white_cut, minmax_filter, pixel_blur, cross_trim.

Future improvement

• manpage could be written for optar and unoptar
• commandline help (-h) could be written for optar and unoptar
• the format could be made configurable. Now it's stored in the optar.h
• the magic constants could be changed by commandline options. Now they are stored in unoptar.c.
• Golay code decoding could be rewritten faster, using a sophisticated algorithm (Kasami algorithm?)
• Easy support for multiple pages per page, so it can be read by a digital camera. Currently it cannot since digital camera blurs at the sides of the picture.

Authorship and Licensing

© GPL 2007 Karel 'Clock' Kulhavy of Twibright Labs.

See COPYING for the text of the GPL license.

e-mail: clock (at) twibright (dot) com

Twibright Optar homepage: http://ronja.twibright.com/optar/

Improvements © 2012-2018 Colin Dean

optar's Issues

Release artifacts should include a PDF version of the source and binary archives

The binary archive is easy: run optar on the the produced archive.

The source archive is a little trickier: download it from Github and do the same as the binary archive.

Why? Proof that it works and an example output for folks to see.

Technical explanation of how it works

I and many others would greatly appreciate it.

My biggest concern is being able to recreate the source code if ever I need to in the future, but I need to understand be able to easily understand what techniques are employed here 😄

Thank you!

I would also like to know how you came up with using Golay codes.

Fail to decode after printing: Error: cannot find upper left corner

I'm printing a file that fits on one single A4 page. The printed version looks good.

However, when decoding after scanning, I hit Error: cannot find upper left corner

Any idea to fix that?

Thanks a lot for the great project! --Martin

$ ./unoptar 0-65-93-24-3-1-2-24 02_sound > foo 
Format: text height=24
Unformatted channel capacity 433.267 kB, formatted raw channel capacity 399.546 kB, net Golay payload capacity 199.773 kB, 133182 Golay symbols, 0 bits unused (incomplete Hamming symbol), border taking 1.50439% of unformatted capacity, border with crosses taking 7.78285% of unformatted capacity, border with crosses and Golay taking 53.8914% of unformatted capacity.
Decoding PNG file 02_sound_0001.png...
Input 595 x 841 pixels, taking 1.00079 megabytes for 2 framebuffers.
Average pixel value 226
Black 40.3372, white 254.53, cutlevel 62 (0x3e), fill cutlevel 147 (0x93)
Black 16.6485, white 251.518, cutlevel 40 (0x28), fill cutlevel 134 (0x86)
Black 12.8823, white 249.818, cutlevel 37 (0x25), fill cutlevel 131 (0x83)
Black 12.3657, white 249.524, cutlevel 36 (0x24), fill cutlevel 131 (0x83)
Black 12.2164, white 249.445, cutlevel 36 (0x24), fill cutlevel 131 (0x83)
Removing dirt from the white border: white border identified, data area identified, erased 64496 pixels of dirt.
Searching for the corners.
Error: cannot find upper left corner
See failure_debug.pgm why.

Comparison of this and OllyDbg's Paperback

See these two repos for reference
https://github.com/timwaters/paperback
https://github.com/Wikinaut/paperback-cli

Modern buildsystem: first attempt at meson

Yes, this is kind of unnecessary and still incomplete. But I wanted to learn Meson and I thought optar would be a nice mini-project to try it on.

Since I did not find a common chat platform to bother you on there, I opened this issue to show you my first attempt: https://github.com/dragetd/optar/tree/draget-make-patch

Any thoughts on this? :D

Perfectly okay if you have no interest, then I will continue my experiments on my fork. :)

Unoptar does not compile

When running make, I get the following:

% make
gcc -c  -O3 -Wall -Wuninitialized -fomit-frame-pointer -funroll-loops -fstrength-reduce -DNODEBUG `libpng-config --I_opts` -o optar.o optar.c
gcc -c  -O3 -Wall -Wuninitialized -fomit-frame-pointer -funroll-loops -fstrength-reduce -DNODEBUG `libpng-config --I_opts` -o common.o common.c
gcc -c  -O3 -Wall -Wuninitialized -fomit-frame-pointer -funroll-loops -fstrength-reduce -DNODEBUG `libpng-config --I_opts` -o golay.o golay.c
gcc -c  -O3 -Wall -Wuninitialized -fomit-frame-pointer -funroll-loops -fstrength-reduce -DNODEBUG `libpng-config --I_opts` -o parity.o parity.c
gcc -lm -o golay golay.o parity.o
./golay > golay_codes.c
gcc -c  -O3 -Wall -Wuninitialized -fomit-frame-pointer -funroll-loops -fstrength-reduce -DNODEBUG `libpng-config --I_opts` -o golay_codes.o golay_codes.c
gcc -lm -o optar optar.o common.o golay_codes.o parity.o
gcc -c -I/usr/local/include/libpng  -O3 -Wall -Wuninitialized -fomit-frame-pointer -funroll-loops -fstrength-reduce -DNODEBUG `libpng-config --I_opts` -o unoptar.o unoptar.c
gcc -lm -o unoptar -L/usr/local/lib -lpng -lz unoptar.o common.o golay_codes.o parity.o
unoptar.o: In function `get_pixel_interp':
unoptar.c:(.text+0x2a): undefined reference to `floor'
unoptar.c:(.text+0x6e): undefined reference to `floor'
unoptar.o: In function `mark':
unoptar.c:(.text+0x1f3): undefined reference to `floor'
unoptar.c:(.text+0x20e): undefined reference to `floor'
unoptar.o: In function `mark_bad_bit':
unoptar.c:(.text+0x301): undefined reference to `floor'
unoptar.c:(.text+0x3e71): undefined reference to `sqrt'
unoptar.o: In function `dump_newary':
unoptar.c:(.text+0x3ebb): undefined reference to `pow'
unoptar.c:(.text+0x3ede): undefined reference to `floor'
unoptar.c:(.text+0x3f09): undefined reference to `pow'
unoptar.c:(.text+0x3f2c): undefined reference to `floor'
unoptar.c:(.text+0x3f57): undefined reference to `pow'
unoptar.c:(.text+0x3f7a): undefined reference to `floor'
unoptar.c:(.text+0x3fa7): undefined reference to `pow'
unoptar.c:(.text+0x3fca): undefined reference to `floor'
unoptar.o: In function `print_badbit':
unoptar.c:(.text+0x443d): undefined reference to `floor'
unoptar.c:(.text+0x445f): undefined reference to `floor'
unoptar.o: In function `read_png':
unoptar.c:(.text+0x4f0a): undefined reference to `png_create_read_struct'
unoptar.c:(.text+0x4f15): undefined reference to `png_create_info_struct'
unoptar.c:(.text+0x4f27): undefined reference to `png_init_io'
unoptar.c:(.text+0x4f32): undefined reference to `png_read_info'
unoptar.c:(.text+0x4f3d): undefined reference to `png_get_image_width'
unoptar.c:(.text+0x4f4e): undefined reference to `png_get_image_height'
unoptar.c:(.text+0x4fac): undefined reference to `png_get_gAMA'
unoptar.c:(.text+0x4fcb): undefined reference to `png_set_gamma'
unoptar.c:(.text+0x4fd6): undefined reference to `png_get_color_type'
unoptar.c:(.text+0x4fe8): undefined reference to `png_get_bit_depth'
unoptar.c:(.text+0x5026): undefined reference to `png_set_interlace_handling'
unoptar.c:(.text+0x5034): undefined reference to `png_read_update_info'
unoptar.c:(.text+0x5257): undefined reference to `png_read_rows'
unoptar.c:(.text+0x529e): undefined reference to `png_read_rows'
unoptar.c:(.text+0x52b9): undefined reference to `png_read_rows'
unoptar.c:(.text+0x52d4): undefined reference to `png_read_rows'
unoptar.c:(.text+0x52ef): undefined reference to `png_read_rows'
unoptar.o:unoptar.c:(.text+0x530a): more undefined references to `png_read_rows' follow
unoptar.o: In function `read_png':
unoptar.c:(.text+0x534c): undefined reference to `png_set_gamma'
unoptar.c:(.text+0x5366): undefined reference to `png_set_expand'
unoptar.c:(.text+0x537f): undefined reference to `png_set_rgb_to_gray'
unoptar.c:(.text+0x5398): undefined reference to `png_set_strip_alpha'
unoptar.c:(.text+0x53bb): undefined reference to `png_set_rgb_to_gray'
unoptar.c:(.text+0x53db): undefined reference to `png_read_rows'
unoptar.c:(.text+0x53ee): undefined reference to `png_read_rows'
unoptar.c:(.text+0x5401): undefined reference to `png_read_rows'
unoptar.c:(.text+0x5414): undefined reference to `png_read_rows'
unoptar.c:(.text+0x5427): undefined reference to `png_read_rows'
unoptar.o:unoptar.c:(.text+0x543a): more undefined references to `png_read_rows' follow
unoptar.o: In function `read_png':
unoptar.c:(.text+0x5478): undefined reference to `png_read_end'
unoptar.c:(.text+0x54a4): undefined reference to `png_set_expand'
unoptar.c:(.text+0x54b4): undefined reference to `png_set_strip_16'
unoptar.o: In function `main':
unoptar.c:(.text.startup+0x1113): undefined reference to `sqrtf'
unoptar.c:(.text.startup+0x113b): undefined reference to `sqrtf'
unoptar.c:(.text.startup+0x119b): undefined reference to `floor'
unoptar.c:(.text.startup+0x11fd): undefined reference to `floor'
unoptar.c:(.text.startup+0x23c7): undefined reference to `sqrt'
unoptar.c:(.text.startup+0x2432): undefined reference to `sqrt'
unoptar.c:(.text.startup+0x24b3): undefined reference to `asin'
unoptar.c:(.text.startup+0x256a): undefined reference to `asin'
unoptar.c:(.text.startup+0x2615): undefined reference to `asin'
unoptar.c:(.text.startup+0x26b5): undefined reference to `asin'
unoptar.c:(.text.startup+0x274d): undefined reference to `remainder'
unoptar.c:(.text.startup+0x4738): undefined reference to `floor'
unoptar.c:(.text.startup+0x4752): undefined reference to `floor'
unoptar.c:(.text.startup+0x4c90): undefined reference to `floor'
unoptar.c:(.text.startup+0x4d2b): undefined reference to `sqrt'
unoptar.c:(.text.startup+0x4d55): undefined reference to `floorf'
unoptar.c:(.text.startup+0x55ac): undefined reference to `sqrtf'
unoptar.c:(.text.startup+0x55c3): undefined reference to `sqrtf'
unoptar.c:(.text.startup+0x6346): undefined reference to `floor'
unoptar.c:(.text.startup+0x6bc7): undefined reference to `floor'
unoptar.c:(.text.startup+0x6be9): undefined reference to `floor'
unoptar.c:(.text.startup+0x6c19): undefined reference to `floor'
unoptar.c:(.text.startup+0x6f3c): undefined reference to `floor'
unoptar.o:unoptar.c:(.text.startup+0x6f68): more undefined references to `floor' follow
unoptar.o: In function `main':
unoptar.c:(.text.startup+0x83d1): undefined reference to `asin'
unoptar.c:(.text.startup+0x842f): undefined reference to `asin'
unoptar.c:(.text.startup+0x8488): undefined reference to `asin'
unoptar.c:(.text.startup+0x8500): undefined reference to `asin'
collect2: error: ld returned 1 exit status
make: *** [unoptar] Error 1

I'm running Ubuntu 12.10, 64-bit.

I found a hint over at http://chaource.livejournal.com/87817.html but had no luck with any permutation of "-lm -lpng".

debian package

There has been a Debian Request for Packaging which got closed as stale. Adding a pointer to it in case anybody wants to unarchive and reopen it:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=633689

I'd definitely be interested to do it myself, but it's not high prio.

Python, JS and Go version of the software

Objective C is being deprecated over Swift, so if it is possible, please have these versions ready.

• Python (a good scripting language)
• JS (for web-based implementation)
• Go (A Java/C#/Swift/Kotlin replacement)

Add GitHub release

Since the original tool is no longer maintained, I also thought about archiving the sources on GitHub and fixing minor things. But I saw that you already did that. :)

How about spinning a v1.0.1 (or 1.0.0-r1 - as the fixes you added were only build-fixes and not actual updates to the software) and adding a GitHub release - I'd package this for Gentoo then. :-)

Online version

An online version of Optar could be awesome! I know it can be a lot of work but I am sure it could become a great inspiration for archive projects, art, and research. I don't have the skills to do such a fork, at least not the dev part. Maybe we can talk about that?
Awesome idea by the way. I am part of a collective named http://webarchivists.org/ and we love your project.