opentimestamps/opentimestamps-client

OpenTimestamps client

OpenTimestamps Client

Command-line tool to create and validate timestamp proofs with the OpenTimestamps protocol, using the Bitcoin blockchain as a timestamp notary. Additionally this package provides timestamping of PGP signed Git commits, and verification of timestamps for both Git commits as a whole, and individual files within a Git repository.

Requirements

• Python3

While OpenTimestamps can create timestamps without a local Bitcoin node, to verify timestamps you need a local Bitcoin Core node (a pruned node is fine).

Installation

Either via PyPi:

$ pip3 install opentimestamps-client

or from source:

$ python3 setup.py install

On Debian (Stretch) you can install the necessary system dependencies with:

sudo apt-get install python3 python3-dev python3-pip python3-setuptools python3-wheel

Usage

Creating a timestamp:

$ ots stamp README.md
Submitting to remote calendar https://a.pool.opentimestamps.org
Submitting to remote calendar https://b.pool.opentimestamps.org
Submitting to remote calendar https://a.pool.eternitywall.com

You'll see that README.md.ots has been created with the aid of three remote calendars. We can't verify it immediately however:

$ ots verify README.md.ots
Assuming target filename is 'README.md'
Calendar https://alice.btc.calendar.opentimestamps.org: Pending confirmation in Bitcoin blockchain
Calendar https://bob.btc.calendar.opentimestamps.org: Pending confirmation in Bitcoin blockchain
Calendar https://finney.calendar.eternitywall.com: Pending confirmation in Bitcoin blockchain

It takes a few hours for the timestamp to get confirmed by the Bitcoin blockchain; we're not doing one transaction per timestamp.

However, the client does come with a number of example timestamps which you can try verifying immediately. Here's a complete timestamp that can be verified locally:

$ ots verify examples/hello-world.txt.ots
Assuming target filename is 'examples/hello-world.txt'
Success! Bitcoin block 358391 attests existence as of 2015-05-28 CEST

You can specify JSON-RPC credentials (USER and PASS) for a local bitcoin node like so:

$ ots --bitcoin-node http://USER:[email protected]:8332/ verify examples/hello-world.txt.ots
Assuming target filename is 'examples/hello-world.txt'
Success! Bitcoin block 358391 attests existence as of 2015-05-28 CEST

Incomplete timestamps are ones that require the assistance of a remote calendar to verify; the calendar provides the path to the Bitcoin block header:

$ ots verify examples/incomplete.txt.ots
Assuming target filename is 'examples/incomplete.txt'
Got 1 new attestation(s) from https://alice.btc.calendar.opentimestamps.org
Success! Bitcoin block 428648 attests existence as of 2016-09-07 CEST

The client maintains a cache of timestamps it obtains from remote calendars, so if you verify the same file again it'll use the cache:

$ ots verify examples/incomplete.txt.ots
Assuming target filename is 'examples/incomplete.txt'
Got 1 attestation(s) from cache
Success! Bitcoin block 428648 attests existence as of 2016-09-07 CEST

You can also upgrade an incomplete timestamp, which adds the path to the Bitcoin blockchain to the timestamp itself:

$ ots upgrade examples/incomplete.txt.ots
Got 1 attestation(s) from cache
Success! Timestamp is complete

Finally, you can get information on a timestamp, including the actual commitment operations and attestations in it:

$ ots info examples/two-calendars.txt.ots
File sha256 hash: efaa174f68e59705757460f4f7d204bd2b535cfd194d9d945418732129404ddb
Timestamp:
append 839037eef449dec6dac322ca97347c45
sha256
 -> append 6b4023b6edd3a0eeeb09e5d718723b9e
    sha256
    prepend 57d46515
    append eadd66b1688d5574
    verify PendingAttestation('https://alice.btc.calendar.opentimestamps.org')
 -> append a3ad701ef9f10535a84968b5a99d8580
    sha256
    prepend 57d46516
    append 647b90ea1b270a97
    verify PendingAttestation('https://bob.btc.calendar.opentimestamps.org')

Timestamping and Verifying PGP Signed Git Commits

See doc/git-integration.md

Privacy Security

Timestamping inherently records potentially revealing metadata: the current time. If you create multiple timestamps in close succession it's quite likely that an adversary will be able to link those timestamps as related simply on the basis of when they were created; if you make use of the timestamp multiple files in one command functionality (./ots stamp <file1> <file2> ... <fileN>) most of the commitment operations in the timestamps themselves will be identical, providing an adversary very strong evidence that the files were timestamped by the same person. Finally, the REST API used to communicate with remote calendars doesn't currently attempt to provide any privacy, although it could be modified to do so in the future (e.g. with prefix filters).

File contents are protected with nonces: a remote calendar learns nothing about the contents of anything you timestamp as it only ever receives an opaque and meaningless digest. Equally, if multiple files are timestamped at once, each file is protected by an individual nonce; the timestamp for one file reveals nothing about the contents of another file timestamped at the same time.

Compatibility Expectations

OpenTimestamps is production software that has been in use for many years. The timestamp format itself is stable and future OpenTimestamp's clients will always be able to verify OTS timestamps created in the past, provided that the relevant calendar data is available. The only case where this guarantee could fail is if cryptographic hashing functions themselves suffer a catestrophic failure; a failure of Bitcoin itself, eg due to 51% attack, is not sufficient to make Bitcoin timestamps from the past unverifiable, as the Bitcoin chain is widely witnessed.

It's likely that the REST protocol used to communicate with remote calendars will change, including in backwards incompatible ways. If this happens you'll just need to upgrade your client software; existing timestamps will be unaffected.

Calendar Mirroring

Calendars can be backed up with the otsd-backup.py tool from the OpenTimestamps Server package.

Development

Use the setuptools development mode:

python3 setup.py develop --user

Known Issues

• Need unit tests for the client.

• Git tree re-hashing support fails on certain filenames with invalid unicode encodings; this appears to be due to bugs in the underlying GitPython library. As a work-around, you may find the convmv tool useful to find and rename these files.

• Git annex support only works with the SHA256 and SHA256E backends.

• Errors in the Bitcoin RPC communication aren't handled in a user-friendly way.

• Not all Python platforms check SSL certificates correctly. This means that on some platforms, it would be possible for a MITM attacker to intercept HTTPS connections to remote calendars. That said, it shouldn't be possible for such an attacker to do anything worse than give us a timestamp that fails validation, an easily fixed problem.

• ots-git-gpg-wrapper doesn't yet check for you if the timestamp on the git commit makes sense.

• bitcoin package can cause issues, with ots confusing it with the required python-bitcoinlib package. A symptom of this issue is the message AttributeError: module 'bitcoin' has no attribute 'SelectParams' or JSONDecodeError("Expecting value", s, err.value) from None. To remedy this issue, one must do the following:

# uninstall the packages through pip
pip3 uninstall bitcoin python-bitcoinlib

# remove the bitcoin directory manually from your dist-packages folder
rm -rf /usr/local/lib/python3.5/dist-packages/bitcoin

# reinstall the required package
pip3 install python-bitcoinlib

opentimestamps-client's Issues

Checking for completeness / doublespend protection

Hi Peter,

I'm interested in using OpenTimestamps for signed git transactions, but I have an additional requirement: The user checking the timestamp needs to know that they have the most recent commit, and that there isn't a competing git tree also signed by the same developer.

This is a feature request.

Ryan

RFC: Use git config to specify additional options for ots-git-wrapper.sh

Hi,
what do you think about adding a custom git config (like gpg.ots-args) to specify additional options for ots-git-wrapper.sh (like --socks5-proxy)?

For example:
git config --global gpg.ots-args "--socks5-proxy 127.0.0.1:9050"

and then, with a slightly patch of ots-git-wrapper.sh:

diff --git a/ots-git-gpg-wrapper.sh b/ots-git-gpg-wrapper.sh
index 73d61bb5..97d42aaf 100755
--- a/ots-git-gpg-wrapper.sh
+++ b/ots-git-gpg-wrapper.sh
@@ -5,4 +5,4 @@
 # Required because git's gpg.program option doesn't allow you to set command
 # line options; see the doc/git-integration.md

-"`dirname "$0"`"/ots-git-gpg-wrapper --gpg-program "`which gpg`" -- "$@"
+"`dirname "$0"`"/ots-git-gpg-wrapper --gpg-program "`which gpg`" `git config gpg.ots-args` -- "$@"

Tweetable timestamps

When the proof is short enough, it's possible to link the proof in a URL shortener and post on twitter, including the entire matured timestamp. It would be nice if the client would give me a link in the following format:

https://opentimestamps.org/info.html?ots={serialized_timestamp}

This can get URL shortened and the URL shortener stores the proof and gives you a short URL to pass around in public. Fits nicely into a tweet, too.

Also, I believe the serialized timestamp should only include one attestation, not all of the available attestations from the OTS file.

When a timestamp was made from a merkle tree of many other elements, maybe the client should complain to the user and say that a URL shortener will not be likely to accept the input. These services often have a length limit, it's not like they gladly accept gif/webm data: URIs.

Feature: Allow filenames in file, or nul-terminated a la xargs, or allow sha256sum file format to provide sums directly

The argument list can only be so long on some platforms; and in any event, a limit on open filehandles currently limits the OTS client from e.g. stamping a large directory full of files. If the hashing work is already done in another file (for example, in a sha256sum format file) then feeding that file, or at least a list of files, into the command-line would be a great way to help mass-stamp.

Thanks for the awesome tool!

(P.S. git file extraction works for trees of very small files, but it isn't great for extremely large trees of extremely large files and anyway the overhead is significant for no appreciable gain.)

MITM proxies aren't sanely detected

Resulting error message:

opentimestamps.core.serialize.DeserializationError: Unknown operation tag 0x3c

Python version 3.4.2 (which doesn't check SSL certs IIRC)

Timestamping public PGP keys

I want to timestamp my public PGP key. OpenTimestamps lets me do that. This will allow me to prove to future acquaintances that I did indeed create my public PGP key at a particular time in the past. This is great.

But here's the thing. I also want to prove to these future acquaintances that I created no other public PGP keys around a similar time - or at least that I didn't create more than hundreds (or thousands) of other public PGP keys around a similar time without expending millions (or billions) of dollars in computing power.

Is this possible? I wonder if a public PGP key can be attached with information proving that a certain computing power was performed to create it.

I realize this is not directly related to OpenTimestamps, but I'm hoping you guys have ideas on this.

TLDR, I want to show to future folks that a particular and no other PGP was mine in the past. I'm hoping I can do better than using trusted third parties for this.

ValueError: 'timeout' must be a non-negative number

ots: Submitting to remote calendar https://a.pool.opentimestamps.org
ots: Submitting to remote calendar https://b.pool.opentimestamps.org
ots: Submitting to remote calendar https://a.pool.eternitywall.com
Traceback (most recent call last):
  File "/home/kanzure/local/opentimestamps/opentimestamps-client/ots-git-gpg-wrapper", line 133, in <module>
    otsclient.cmds.create_timestamp(final_timestamp, args.calendar_urls, args)
  File "/home/kanzure/local/opentimestamps/opentimestamps-client/otsclient/cmds.py", line 112, in create_timestamp
    result = q.get(block=True, timeout=remaining)
  File "/usr/lib/python3.5/queue.py", line 166, in get
    raise ValueError("'timeout' must be a non-negative number")
ValueError: 'timeout' must be a non-negative number

Setup.py does not create entry points

After grabbing the latest code and verifying all the dependencies I noticed that pip did not create/reference the entry points ots and ots-git-gpg-wrapper as it state on the tin:

 # To provide executable scripts, use entry points in preference to the
 # "scripts" keyword. Entry points provide cross-platform support and allow
 # pip to create the appropriate form of executable for the target platform.
    entry_points={
        'console_scripts': [
            'ots = otsclient.ots:main',
            'ots-git-gpg-wrapper = otsclient.git_gpg_wrapper:main',
        ],

Ran setup on Ubuntu 16.04 LTS. Also tried running setup with sudo, because permissions, but same result. Just to be clear - setup runs fine, the entry points just aren't created. Should pip be creating them or is the above code just a reference to the entry points that should be there (but are missing)?

Easily worked around by copying ots entry point from the previous version.

Failed to create timestamp: need at least 2 attestations but received 0 within timeout

Not sure what's happening when I try to timestamp the Qubes OS canaries. Here is my output:

Submitting to remote calendar https://a.pool.opentimestamps.org
Submitting to remote calendar https://b.pool.opentimestamps.org
Submitting to remote calendar https://a.pool.eternitywall.com
Failed to create timestamp: need at least 2 attestations but received 0 within timeout

Are the servers down or something?

git-gpg-wrapper is a too generic name

As the "creator" of unofficial (AUR) Arch Linux package of opentimestamps-client-git I had to rename the wrapper, since git-gpg-wrapper filename doesn't let the user know it's an opentimestamps "tool".

In my case I renamed it as ots-git-gpg-wrapper.sh.
I think something similar should be done also in the git repository in order to have the same filename on any distribution

Strange calendar server 'https://ots.btc..catallaxy.com'

I'm seeing this (malformed?) calendar server url when verifying a stamp:
verify PendingAttestation('https://ots.btc..catallaxy.com')

ots recognizes it's malformed at least:
Ignoring attestation from calendar https://ots.btc..catallaxy.com: Calendar not in whitelist

MacOS 10.13.6
ots installed via pip3 and working fine for everything else.

Software code signing guidelines and considerations

Hey, I'm working on FOSS project, we are in process of integrating public CI service and public package repositories, would be happy to use opentimestamps for code signing, we evaluated traditional code signing certificates, but perfer to stick with decentralized solutions as much as possible for now. would appreciate if we can get any feedback and thoughts about using opentimestamps for software code signing rather than traditional methods.

Any thoughts and feedback on this would be appreciated.

ValueError: 'timeout' must be a non-negative number

ots: Submitting to remote calendar https://a.pool.opentimestamps.org
ots: Submitting to remote calendar https://b.pool.opentimestamps.org
ots: Submitting to remote calendar https://a.pool.eternitywall.com
Traceback (most recent call last):
  File "/home/kanzure/local/opentimestamps/opentimestamps-client/ots-git-gpg-wrapper", line 133, in <module>
    otsclient.cmds.create_timestamp(final_timestamp, args.calendar_urls, args)
  File "/home/kanzure/local/opentimestamps/opentimestamps-client/otsclient/cmds.py", line 112, in create_timestamp
    result = q.get(block=True, timeout=remaining)
  File "/usr/lib/python3.5/queue.py", line 166, in get
    raise ValueError("'timeout' must be a non-negative number")
ValueError: 'timeout' must be a non-negative number
error: gpg failed to sign the data
fatal: failed to write commit object

Are there plans to integrate with this spec: https://chainpoint.org/ ?

I've used OTS with success and now am working on some solutions on ethereum. It seems the chainpoint spec is flexible to accommodate both. Would you be interested in a converter?

error: Could not backup timestamp: 'errorfile.bak' already exists

Was attempting to do bulk (globbed) upgrade of a bunch of mixed upgraded/non-upgraded timestamp files. Some of the already-upgraded files seem to work just fine, but sometimes I run into this error about a backup already existing.

If backup files must really not be tampered with (although for upgrades, it should be a strict improvement over the previous backup file), then I recommend a CLI utility that looks at a directory and finds out which timestamps have not been upgraded, and which ones do not yet have backups, and only upgrade the ones that do not yet have backups.

This can be achieved manually by some crazy shell scripting, but really this should be handled by a good timestamping client.

Stamp verification without bitcoin full node

Is it possible to verify a timestamp without running bitcoin node? Even pruned one is not too easy to install as it requires downloading of the entire blockchain IIRC. Is it possible to use an SPV node for verification?

Installation in Debian Stretch

I had some problems installing in a Debian Stretch distribution where I never did any python3 development before. I had to search around a bit to solve this with:

sudo apt-get install python3-dev python3-setuptools python3-wheel 

You might want to add this to the README or somewhere else, as I don't think it's obvious to everyone. Also I hope longer term you do plan to make this a proper pip3-installable package?

Apart from this, great work, I am going to timestamp my archive now...

OTS verification with web browser

Since Sunday I can only verify OTS with my full node. Unfortunately, the block numbers are no longer displayed via the web browser.

Daniel
OTS Check.zip

socks-proxy should have a second option so local bitcoin nodes can be used

The socks-proxy is being used to connect to a bitcoin-node also; this means that if a bitcoin node is running on another machine and outgoing connections are desired to operate over a Tor socks proxy, that the tor socks proxy will be used to try to connect to the bitcoin node too.

It would be cool to have a -tor or .. like -onion or something for non local connections and then bitcoin-node can work for a local bitcoind on the same network. Assuming remote rpc works. :) Still working on getting that functional.

How to find the attestation transaction id from an ots info file?

I get the following output from running ots info on one of my .ots files:

File sha256 hash: 3e8931c7a0ff54b4c77ec845dd17a385d8705c285c88521d1428b39ab06421e7
Timestamp:
append b1cbf92d328dee6f4c7ecd19ab94969e
sha256
prepend ed788ebf68555688fd41e529d1eae955fbb0b7e8e3d63ed59c9f3eed29b309e1
sha256
 -> append 8be1d43da44c76954746861b77a63ff4
    sha256
    prepend 58b9e977
    append 8bda26c3e2fce829
    verify PendingAttestation('https://alice.btc.calendar.opentimestamps.org')
    sha256
    append a5b22aa26333881605966bb2fe2fe17f0a088064806c3c2e4ec15bf440a6c313
    sha256
    append 367bb32757561efee9331a8e0900f9c25f2e0949fb7584961791a9b51d480c76
    sha256
    prepend e69566cbb104e5090e15feff190ff6167677cd8d126c561272e2a924527ecdb9
    sha256
    append 168afd2a028d45f901a496e6f60477c4d4242b33ad1d16ab153e3a56c678822f
    sha256
    prepend d79c3fb7dccd0b28250f47e4291d76339c2c99cc0807f6e33d0863fd35e176bf
    sha256
    append 3796a0de7d3138e47e340663d4483cc52fa49ebe5f8ce62ce5debde91d2da81a
    sha256
    prepend 0100000001eded3975b617be12b4d9d26f600dbff09e122ce69222b6e8a038a43f14c77da2000000004847304402200d18dd24aba0b18cdb43ded74e5fe44f2e7491195a3d726218f35a797d5c164902207c8bf3ecb9c5cc8c2553cb6c6fc5d99716334f2dca14676190fe45985d24fee001fdffffff0282733100000000002321037b28af5bfc4ed21e78e541fc12a11bbcc4bdbf7ca59bef70aade644f97bb2d04ac0000000000000000226a20
    append f6f30600
    sha256
    sha256
    append def74ac2a943a41eb16169d8b30fcd8a06725ccc18cf006edd9d940e36fd9631
    sha256
    sha256
    append 4952ed708ffdde79d0a86e63711c7b6bcc5dc1b18e9034fdfc87afa64c570b19
    sha256
    sha256
    prepend 8d25dd5023fdfba3de713ff4df9e88fbf0e6792e190a87287f9b76915aec31bb
    sha256
    sha256
    append a5110f3b1f7a704a7481544e4740e1c7d79f2fddf71c4a9d0e8faff969b5a7ed
    sha256
    sha256
    append 5d9af3aa699fcae46a2a94638d0e9b4324fed74c613109333dddf18b2ee41c21
    sha256
    sha256
    append 0a603cd8d29516dc5477068d6b51defce1ce77e7189d6bb9b341b5cb8eeca536
    sha256
    sha256
    append 40e4872db15121979ed490225c0f963ded20722e35d2c1e8062c7bfd9a9c4a3f
    sha256
    sha256
    prepend b5074d6ab193566a1cc4d8bfcde8602a0b6b3720da69c10e0e994ec58a91d016
    sha256
    sha256
    prepend 133ddf23c6eca380669b91fd025417b89524a555f378e984777b5dec1006a78c
    sha256
    sha256
    append dddc977af3f5dce8721d26e1331f80a76ef472f9bff3b4e735a4364dddae443e
    sha256
    sha256
    prepend 03cf82f6463395be306282122933ba3b774a0f6c278529ca4b9c8df7f005d3b8
    sha256
    sha256
    append 0d26ec24c36996d60adeab2e80b374185a897b00fbc0e582084ed43c9cc91cbf
    sha256
    sha256
    verify BitcoinBlockHeaderAttestation(455671)
 -> append c37914ced3ae02f1c43e8ef19ec11c1c
    sha256
    prepend 58b9e977
    append bfea8361e1410f2f
    verify PendingAttestation('https://bob.btc.calendar.opentimestamps.org')

I see the attestation transaction was included in block 455671. But how can I tell which transaction roots the OpenTransactions merkle tree? Pointing me to more information on how to read an ots info file would also be helpful.

Remove local node dependency for client

While having this dependency on the server is completely expected and justifiable, requiring it for simple [commit] stamping/verification is not and adds a lot of setup friction.

Translating block height to it's merkle root can be done with one request to some public APIs, ex:

https://blockchain.info/block-height/<height>?format=json.

Understandably, 3rd party API calls might not be desired in the core code, but how about:

• making it available on the opt-in basis (ex. --btc-plugin blockchain flag),
• making it an external, installable module:

echo 'OpentimestampPlugin[plugin-name]>=[version]' >> requirements.txt && pip3 install -r requirements.txt

Example implementation: #11

[feature-request] Sphinx documentation / manpages

As the title says - some basic documentation in sphinx format would be great for packaging into distributions which require man pages.

I report this issue mainly so others can find it and notify me if they want to work on it. I'll presumably work on it in order to get ots into debian.

I would use sphinx-argparse for generating documentation from the command line parameters.

Logic of is_timestamp_complete

According to the is_timestamp_complete function you need one BitcoinBlockHeaderAttestation to consider the timestamp complete. However it could happen than another calendar server (in PendingAttestation) has a BitcoinBlockHeaderAttestation which has an eventually lower than the first height returned.
This information will never be asked to the calendar if I have a timestamp upgraded from the other calendar.
What about having an option in upgrade to consider a timestamp complete if all PendingAttestation are served?

ots in examples aren't stand alone executable

Considering the output

$ ./ots info examples/hello-world.txt.ots 
File sha256 hash: 03ba204e50d126e4674c005e04d82e84c21366780af1f43bd54a37816b6ab340
Timestamp:
ripemd160
...

Since the first operation of the path is ripemd160 while I got the file hash in sha256, could I verify the ots without the file hello-world.txt?

Two calendars writing separate commitments in the same block

If two calendars wrote two different different commitment transaction in the same block, the timestamp will never upgrade the second PendingAttestation because the BitcoinBlockHeaderAttestation is the same (same height, same merkle root)
This line will not return new_attestations even if the path to reach the merkle root is different.
This behaviour is reproducible with:
./ots upgrade example/merkle2.txt.ots
The PendingAttestation from the bob calendar will not upgrade even if the calendar correctly return the timestamp.

Even if no relevant information is added by the second calendar a client may wrongly consider the calendar unreliable so I think this should be fixed.
What do you think @petertodd ?

Git integration - ots-git-gpg-wrapper.sh instructions

it's not well documented in the git integration document that ots-git-gpg-wrapper.sh should be installed from https://raw.githubusercontent.com/opentimestamps/opentimestamps-client/master/ots-git-gpg-wrapper.sh

I think it would be easier to follow if the url is explicitly mentioned in the instructions.
Something like this would be helpful and save some time:

cd ~
wget https://raw.githubusercontent.com/opentimestamps/opentimestamps-client/master/ots-git-gpg-wrapper.sh
git config --global gpg.program ~/ots-git-gpg-wrapper.sh

Reading from stdin is broken due to argparse bug

@haraldschilly opened opentimestamps/python-opentimestamps#31 in the wrong repo, so opening an issue here to track it.

Your service is sh*t fot bitcoin world - UTXO

Hi,

You service is sh*t for Bitcoin community :-E
You use UTXO for proof and a database of UTXO permanently grows from your proof transactions!
You could use OP_RETURN as it do https://proofofexistence.com/

It's right way!

ots-git-gpg-wrapper: error: unrecognized arguments: --rehash-trees

Looks like ots-git-gpg-wrapper has lost a feature?

ots-git-gpg-wrapper: error: unrecognized arguments: --rehash-trees

AttributeError: 'bool' object has no attribute 'use_btc_wallet'

  File "/home/kanzure/local/opentimestamps/opentimestamps-client/otsclient/cmds.py", line 55, in create_timestamp
    setup_bitcoin = args.setup_bitcoin if args.use_btc_wallet else False
AttributeError: 'bool' object has no attribute 'use_btc_wallet'

Trivial, but there it is.

Failure to timestamp a git commit

Reported by @laanwj on commit bitcoin/bitcoin@a36834f

Possibly because the commit message was so long?

Client is failing when some callendar is unavailable ...

Client is failing when some callendar is unavailable ...
Submitting to remote calendar https://a.pool.opentimestamps.org <<<< not reponding
Submitting to remote calendar https://b.pool.opentimestamps.org << working fine
Submitting to remote calendar https://a.pool.eternitywall.com <<<< working fine
Submitting to remote calendar https://ots.btc.catallaxy.com <<< not reponding

See this:
#########################################################
/opts/opentimestamps-client/examples$ ots -v stamp lalal
Submitting to remote calendar https://a.pool.opentimestamps.org
Traceback (most recent call last):
File "/usr/lib/python3.5/urllib/request.py", line 1254, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "/usr/lib/python3.5/http/client.py", line 1106, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python3.5/http/client.py", line 1151, in _send_request
self.endheaders(body)
File "/usr/lib/python3.5/http/client.py", line 1102, in endheaders
self._send_output(message_body)
File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
self.send(msg)
File "/usr/lib/python3.5/http/client.py", line 877, in send
self.connect()
File "/usr/lib/python3.5/http/client.py", line 1252, in connect
super().connect()
File "/usr/lib/python3.5/http/client.py", line 849, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/usr/lib/python3.5/socket.py", line 711, in create_connection
raise err
File "/usr/lib/python3.5/socket.py", line 702, in create_connection
sock.connect(sa)
OSError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/rodrigo/opts/opentimestamps-client/ots", line 34, in
args.cmd_func(args)
File "/home/rodrigo/opts/opentimestamps-client/otsclient/cmds.py", line 128, in stamp_command
create_timestamp(merkle_tip, args.calendar_urls, args.setup_bitcoin if args.use_btc_wallet else False)
File "/home/rodrigo/opts/opentimestamps-client/otsclient/cmds.py", line 98, in create_timestamp
calendar_timestamp = remote.submit(timestamp.msg)
File "/home/rodrigo/opts/opentimestamps-client/opentimestamps/calendar.py", line 36, in submit
with urllib.request.urlopen(req) as resp:
File "/usr/lib/python3.5/urllib/request.py", line 163, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.5/urllib/request.py", line 466, in open
response = self._open(req, data)
File "/usr/lib/python3.5/urllib/request.py", line 484, in _open
'_open', req)
File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
result = func(*args)
File "/usr/lib/python3.5/urllib/request.py", line 1297, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/usr/lib/python3.5/urllib/request.py", line 1256, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 101] Network is unreachable>
rodrigo@rodrigo-ayi-notebook:~/opts/opentimestamps-client/examples$
#########################################################

But if I try with -c option it works fine
#########################################################
~/opts/opentimestamps-client/examples$ ots -v stamp -c https://a.pool.eternitywall.com lalal
Submitting to remote calendar https://a.pool.eternitywall.com

~/opts/opentimestamps-client/examples$ ots info lalal.ots
File sha256 hash: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Timestamp:
append 958a3527b7cbf948ab06ed0e76a20961
sha256
append 43d745f28f3ca3f54ab54f1195fcdfb7
sha256
append 060be7b6d2e2f0fe496cc16570385352
sha256
prepend 5b885c97
append 8bf6fcc9b329913d
verify PendingAttestation('https://finney.calendar.eternitywall.com')

~/opts/opentimestamps-client/examples$
#########################################################

Is there some response validation on calendar server availability to prevent this error????

Shrink ots receipts

Once a timestamp is complete, a user may find useful to drop the redundant data included in the ots receipt, storing only the linear proof from the file hash to the best attestation.

This feature has a draft implementation in java-opentimestamps and it would be nice if also Python clients have access to it. However, if shrinking becomes a default procedure by clients, then new problems and/or risks may occur.

In case of a complete receipt with 4 branches (one for each calendar): 2 branches end in 2 BitcoinBlockHeaderAttestations while the other 2 end in 2 PendingAttestations. The shrunk receipt would consist in only the linear branch to the best BitcoinBlockHeaderAttestation (minimum height, in case of parity minimum length of the timestamp, in case of parity first one read). All the other three branches (along with the PendingAttestation in the remaining branch) are dropped.

It may occur that one of the two calendars whose PendingAttestation was not resolved actually has a better proof for the user. By dropping that branch, the user won't never be able to obtain that proof.

Furthermore, it may happen that one calendar lied and the BitcoinBlockHeaderAttestation doesn't actually match the merkleroot published in the corresponding block header. If that attestation is the best attestation, then the shrunk receipt would be completely useless.

Considering those issues, I was wondering which should be the correct way to set up this feature:

• as a new command for the OpenTimestamps client, or
• as an independent Python script (stored somewhere else)

In the latter case, it would be something like:

python3 ots-shrink.py my-file.ots my-file.ots.shrunk [--notary {btc,ltc,eth}]

To avoid the problems mentioned above, the user should then check that the shrunk receipt my-file.ots.shrunk is actually valid before overwriting my-file.ots. A proof may contain attestations from different notaries, a shrunk receipt would have only the best attestation from the selected notary (default is btc).

Summing up, does it make sense to implement this feature as a new command for the ots client, or it is too risky?
Moreover, are there other risks that I haven't pointed out?

ots upgrade of multiple timestamps halted by single incomplete timestamp

The Manubot uses OpenTimestamps during continuous deployment to timestamp scholarly manuscripts. The .ots files are saved in directories and can be upgraded at a later date.

To upgrade, I ran the following command:

ots upgrade  v/*/*.ots

The support for upgrading multiple files at once was very convenient. However, the command exits as soon as a single failure occurs. In my case, one of the timestamps had yet to be confirmed, resulting in:

Calendar https://bob.btc.calendar.opentimestamps.org: Pending confirmation in Bitcoin blockchain
Failed! Timestamp not complete

This triggered the program to exit as per:

opentimestamps-client/otsclient/cmds.py

Lines 379 to 381 in 2bffedb

	else:
	logging.warning("Failed! Timestamp not complete")
	sys.exit(1)

As a result, many timestamps were not upgraded that could have been upgraded. Would it make sense to not exit on failure and to proceed with additional inputs?

Change git wrapper to use .gitconfig

Basically, rather than the current wrapper.sh uglyness, command line arguments should be replaced with a .gitconfig section for [ots]. Similar to how git-annex works.

Send git commit message to stdout after ots git wrapper failure

When the calendar servers are offline, signing with git commit -S leads to losing the commit message. Send the git commit message to stdout so that the author doesn't have to write a new message when retrying.

ots verify should verify timestamp in parallel with hashing the file

Slightly faster, but more importantly, gives feedback sooner if the timestamp will fail due to it not yet being confirmed.

[feature request] Timestamp an arbitrary hash

I see this is an option for the js client: https://github.com/opentimestamps/javascript-opentimestamps#stamp

Is this possible in the python client?

gpg wrapper fails in git worktree

When using the gpg wrapper in a git worktree (out-of-tree checkout), the following error happens when signing:

Traceback (most recent call last):
  File "/.../.local/bin/ots-git-gpg-wrapper", line 11, in <module>
    sys.exit(main())
  File "/.../.local/lib/python3.5/site-packages/otsclient/git_gpg_wrapper.py", line 115, in main
    repo = git.Repo()
  File "/.../.local/lib/python3.5/site-packages/git/repo/base.py", line 161, in __init__
    raise InvalidGitRepositoryError(epath)
git.exc.InvalidGitRepositoryError: /.../projects/bitcoin/bitcoin/.git/worktrees/bitcoin-0.16

I think this is a problem with the upstream git python module.

(help offer) Why is python-opentimestamps source copied in this project?

Is there a reason for having the sources of python-opentimestamps project are copied verbatim in this project under python-opentimestamps/ folder?

• Is it a WIP for moving those sources out of this project, into a separate one, to be pip installed?
• Is the separate project going to be dropped, and have all sources in this project?
• Why both projects do not have a setup.py script, and support proper executable command?
• Can I help in one of the above tasks?

libssl missing some constants

From this machine:

[xxx@localhost ~]$ uname -a
Linux localhost.localdomain 2.6.18-194.17.1.0.1.el5 #1 SMP Wed Sep 29 15:40:03 EDT 2010 i686 i686 i386 GNU/Linux
[xxx@localhost ~]$ cat /etc/issue
Enterprise Linux Enterprise Linux Server release 5.5 (Carthage)

It appears libssl is missing some constant needed by python-bitcoinlib.

Traceback (most recent call last):
  File "./ots", line 16, in <module>
    import otsclient.args
  File "/home/xxx/opentimestamps-client-opentimestamps-client-v0.2.2/otsclient/args.py", line 22, in <module>
    import otsclient.cmds
  File "/home/xxx/opentimestamps-client-opentimestamps-client-v0.2.2/otsclient/cmds.py", line 22, in <module>
    import bitcoin.rpc
  File "/home/xxx/opentimestamps-client-opentimestamps-client-v0.2.2/bitcoin/rpc.py", line 48, in <module>
    from bitcoin.wallet import CBitcoinAddress, CBitcoinSecret
  File "/home/xxx/opentimestamps-client-opentimestamps-client-v0.2.2/bitcoin/wallet.py", line 29, in <module>
    import bitcoin.core.key
  File "/home/xxx/opentimestamps-client-opentimestamps-client-v0.2.2/bitcoin/core/key.py", line 102, in <module>
    _ssl.EC_GROUP_get_curve_GFp.restype = ctypes.c_int
  File "/usr/local/lib/python3.5/ctypes/__init__.py", line 360, in __getattr__
    func = self.__getitem__(name)
  File "/usr/local/lib/python3.5/ctypes/__init__.py", line 365, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /lib/libssl.so.6: undefined symbol: EC_GROUP_get_curve_GFp

Since AFAIK shouldn't be a problem for opentimestamps, I brutally commented out all the problematic line in key.py and ./ots is working.

I am publishing here for reference.

Issuing the command sudo yum install openssl-devel which sould be platform equivalent to sudo apt-get install libssl-dev does not help.

"File does not match original!" error when verifying provided example timestamps

Running ots verify examples/hello-world.txt.ots or any other timestamps in the examples directory results in "File does not match original!" error.

Running on Windows 10, using Python 3.6.5.

OTS version: v0.6.1dev (built from latest commit: 1b32826)

I presume this might have something to do with windows.

Neeed to give users more feedback while timestamp tx is confirming

E.g. it'd be cool if you got a display of how many more blocks needed to be mined for the tx to fully confirm.

Display commit message after calendar submission failure

In the git wrapper, it would be mighty polite to display the git commit message before failing to complete the git commit operation. Otherwise the author loses the commit message text.

Also, same for getting the passphrase wrong n-many times.

'bitcoin' has no attribute 'SelectParams'

Completing the example with README.md

$ ~ > ./ots verify README.md.ots     
Assuming target filename is 'README.md'
Got 2 attestation(s) from cache
Traceback (most recent call last):
  File "./ots", line 34, in <module>
    args.cmd_func(args)
  File "/home/mike/Software/opentimestamps-client/otsclient/cmds.py", line 410, in verify_command
    if not verify_timestamp(detached_timestamp.timestamp, args):
  File "/home/mike/Software/opentimestamps-client/otsclient/cmds.py", line 338, in verify_timestamp
    proxy = args.setup_bitcoin()
  File "/home/mike/Software/opentimestamps-client/otsclient/args.py", line 99, in setup_bitcoin
    bitcoin.SelectParams('mainnet')
AttributeError: module 'bitcoin' has no attribute 'SelectParams'

I have installed all the dependencies. Am I missing something?

upgrade w/ multiple timestamps fails too early

$ ./ots upgrade requirements.txt.ots release-notes.md.ots TODO.md.ots README.md.ots LICENSE.ots
Success! Timestamp complete
Calendar https://alice.btc.calendar.opentimestamps.org: Pending confirmation in Bitcoin blockchain
Failed! Timestamp not complete

Quite at release-notes.md.ots when the other timestamps could have been completed.

Secondly, there's no feedback as to which timestamp files are complete/not complete.

Windows support

I have run into 2 problems running on Windows (10).

1. The opentimestamps symlink folder did not work. Deleting and recreating it will fix it...(run as admin) MKLINK /D opentimestamps python-opentimestamps\opentimestamps

2. The SSL certificate is not verifying, possibly due to a known problem when client and server are running different versions of SSL/TLS (https://docs.python.org/2/library/ssl.html#socket-creation). I have temporarily set 'unverifiable=True' in the calendar requests to avoid this.

info subcommand fails with bad arg `verbosity`

When running ./ots info foo.ots I get this exception:

File sha256 hash: 2b1d2e7d12b239beec8d02379e32ad974383c357c0b96ba301ef89af142a6de0
Timestamp:
Traceback (most recent call last):
  File "./ots", line 36, in <module>
    args.cmd_func(args)
  File "D:\Work\ots-client.git\otsclient\cmds.py", line 501, in info_command
    print(detached_timestamp.timestamp.str_tree(verbosity=args.verbosity))
TypeError: str_tree() got an unexpected keyword argument 'verbosity'```

What does the timestamp stamp?

What part of the commit does the timestamp stamp when the git wrapper is used? Does the timestamp ignore the signature or does it stamp the signature as well as the rest of the commit?

If the timestamp does not stamp the signature, then it may be possible for an attacker to timestamp a malicious git commit and sign it later when a victim's key is compromised. Then using the timestamp to verify that the commit was signed before the key was compromised is useless.

Stamp command should prompt if file exists

I think it would be nice if stamp command prompt the user if file exists.

Encountered behavior:

1. First run
➜  opentimstamps git:(master) ✗ ots stamp wee
Submitting to remote calendar https://a.pool.opentimestamps.org
Submitting to remote calendar https://b.pool.opentimestamps.org
Submitting to remote calendar https://a.pool.eternitywall.com

2. Second run
➜  opentimstamps git:(master) ✗ ots stamp wee
Submitting to remote calendar https://a.pool.opentimestamps.org
Submitting to remote calendar https://b.pool.opentimestamps.org
Submitting to remote calendar https://a.pool.eternitywall.com
Failed to create timestamp 'wee.ots': [Errno 17] File exists: 'wee.ots'

Suggested behavior:

1. Maybe show an error before submitting at all.
2. Or, prompt the user, file already exists and this operation will overwrite.