opentimestamps/opentimestamps-server

OpenTimestamps Calendar Server

OpenTimestamps Calendar Server

This package provides the otsd daemon, a calendar server which provides aggregation, Bitcoin timestamping, and remote calendar services for OpenTimestamps clients. You do not need to run a server to use the OpenTimestamps protocol - public servers exist that are free to use. That said, running a server locally can be useful for developers of OpenTimestamps protocol clients, particularly with a local Bitcoin node running in regtest mode.

Installation

You'll need a local Bitcoin node (version 24.0 is known to work) with a wallet with some funds in it; a pruned node is fine. While otsd is running the wallet should not be used for other purposes, as currently the Bitcoin timestamping functionality assumes that it has exclusive use of the wallet.

Install the requirements:

pip3 install -r requirements.txt

Create the calendar:

mkdir -p ~/.otsd/calendar/
echo "http://127.0.0.1:14788" > ~/.otsd/calendar/uri
echo "bitcoin donation address" > ~/.otsd/calendar/donation_addr
dd if=/dev/random of=~/.otsd/calendar/hmac-key bs=32 count=1

The URI determines what is put into the URI field of pending attestations returned by this calendar server. For a server used for testing, the above is fine; for production usage the URI should be set to a stable URL that OpenTimestamps clients will be able to access indefinitely.

The donation address needs to be a valid Bitcoin address for the type of network (mainnet, testnet, regtest) you're running otsd on. It's displayed on the calendar info page.

The HMAC key should be kept secret. It's meant to allow for last-ditch calendar recovery from untrusted sources, although only part of the functionality is implemented. See the source code for more details!

To actually run the server, run the otsd program. Proper daemonization isn't implemented yet, so otsd runs in the foreground. To run in testnet or regtest, use the --btc-testnet or --btc-regtest options. The OpenTimestamps protocol does not distinguish between mainnet, testnet, and regtest, so make sure you don't mix them up!

To use your calendar server, tell your OpenTimestamps client to connect to it:

ots stamp -c http://127.0.0.1:14788 -m 1 FILE

OpenTimestamps clients have a whitelist of calendars they'll connect to automatically; you'll need to manually add your new server to that whitelist to use it when upgrading or verifying:

ots -l http://127.0.0.1:14788 upgrade FILE.ots

If your server is running on testnet or regtest, make sure to tell your client what chain to use when verifying. For example, regtest:

ots --btc-regtest -l http://127.0.0.1:14788 upgrade FILE.ots

Tip: with regtest you can mine blocks on demand to make your timestamp confirm with the generate RPC command. For example, to mine ten blocks instantly:

bitcoin-cli -generate 10

By default otsd binds to localhost; otsd is not designed to be exposed directly to the public and requires a reverse proxy for production usage. An example configuration for nginx is provided under contrib/nginx.

Unit tests

python3 -m unittest discover -v

opentimestamps-server's Issues

Unconfirmed txs clearing

Is possible that the folllowing line:

opentimestamps-server/otsserver/stamper.py

Line 272 in b0bdfe6

self.unconfirmed_txs.clear()

Isn't considering the case where someone else is doing the transaction for the calendar but the digest is older than the most recent?
In that case we should probably remove only all the older than the current unconfirmed_tx (because the external tx is not conflicting with the calendar one)

Missing Hardware Requirements

Can you add Hardware Requirements of OpenTimestamps Server?
I need also some details about High Availability of Server.

JSONRPCError while running bitcoin-core 0.21.1 version in regtest mode

Hi,

I use Bitcoin-0.21.1. Stamping happens correctly.
But during upgrade and verification I receive error.

__do_bitcoin() failed: JSONRPCError({'code': -32601, 'message': 'Method not found'})

daemonization

"To actually run the server, run the otsd program. Proper daemonization isn't implemented yet, so otsd runs in the foreground. "

Is it worth noting using screen, solves issues with terminal disconnect and background tho not a daemon?

screen -S opentimestampserver
./otsd

• Detach: Ctrl+a d
• Find: screen ls
• Reattach: screen -r

Sent timestamp tx, then stuck at Can't timestamp; no spendable outputs

Server Running at https://legaltimestamp.com/

ok it worked but i forgot to unlock wallet first

2021-02-08 12:49:50,259 Opening journal for appending...
2021-02-08 12:49:50,259 Journal has 1 entries
2021-02-08 12:49:50,263 Starting aggregator loop
2021-02-08 12:49:50,263 Starting stamper loop
2021-02-08 12:49:50,271 New block 00000000000000000009371acf1b0f4a79b854962d2e1ddef818fd4a513ae251 at height 669693
2021-02-08 12:49:50,780 Sent timestamp tx b3303a1a76baa2513863f0bee4bec74d7223a2daa43c0f5d762ff0ca3bf24e5c; 1 total commitments
2021-02-08 12:53:05,058 New block 00000000000000000002664dd7f9dd0126191d7d8de9d32d4ac4a3b2fe2401e0 at height 669694
2021-02-08 12:53:05,569 __do_bitcoin() failed: JSONRPCError({'code': -13, 'message': 'Error: Please enter the wallet passphrase with walletpassphrase first.'})
Traceback (most recent call last):
File "/home/ec2-user/opentimestamps-server/otsserver/stamper.py", line 490, in __loop
self.__do_bitcoin()
File "/home/ec2-user/opentimestamps-server/otsserver/stamper.py", line 425, in __do_bitcoin
r = proxy.signrawtransaction(unsigned_tx)
File "/usr/local/lib/python3.7/site-packages/bitcoin/rpc.py", line 708, in signrawtransaction
r = self._call('signrawtransaction', hextx, *args)
File "/usr/local/lib/python3.7/site-packages/bitcoin/rpc.py", line 238, in _call
raise JSONRPCError(response['error'])
bitcoin.rpc.JSONRPCError: {'code': -13, 'message': 'Error: Please enter the wallet passphrase with walletpassphrase first.'}

now no matter what i do it

2021-02-08 14:08:26,683 Opening journal for appending...
2021-02-08 14:08:26,683 Journal has 1 entries
2021-02-08 14:08:26,687 Starting aggregator loop
2021-02-08 14:08:26,690 Starting stamper loop
2021-02-08 14:08:26,699 New block 00000000000000000009369a02cd2ca038d8a3fb94398b6648fda8ddc3e4e24c at height 669705
2021-02-08 14:08:27,150 Couldn't find a confirmed output, trying unconfirmed
2021-02-08 14:08:27,190 Can't timestamp; no spendable outputs

restart bitcoin node and otsd

[ec2-user@ip-172-31-61-7 ~]$ bitcoin-cli getwalletinfo
{
"walletname": "",
"walletversion": 169900,
"balance": 0.00030304,
"unconfirmed_balance": 0.00000000,
"immature_balance": 0.00000000,
"txcount": 2,
"keypoololdest": 1612119205,
"keypoolsize": 1000,
"keypoolsize_hd_internal": 1000,
"unlocked_until": 0,
"paytxfee": 0.00000000,
"hdseedid": "c86cfb653b5db71d1c566329736e744af91cfd15",
"hdmasterkeyid": "c86cfb653b5db71d1c566329736e744af91cfd15",
"private_keys_enabled": true
}

unlock wallet for a year

[ec2-user@ip-172-31-61-7 ~]$ bitcoin-cli walletpassphrase [password] 31556952
[ec2-user@ip-172-31-61-7 ~]$ bitcoin-cli getwalletinfo
{
"walletname": "",
"walletversion": 169900,
"balance": 0.00030304,
"unconfirmed_balance": 0.00000000,
"immature_balance": 0.00000000,
"txcount": 2,
"keypoololdest": 1612119205,
"keypoolsize": 1000,
"keypoolsize_hd_internal": 1000,
"unlocked_until": 1644350490,
"paytxfee": 0.00000000,
"hdseedid": "c86cfb653b5db71d1c566329736e744af91cfd15",
"hdmasterkeyid": "c86cfb653b5db71d1c566329736e744af91cfd15",
"private_keys_enabled": true
}
2021-02-08 14:13:37,975 Opening journal for appending...
2021-02-08 14:13:37,976 Journal has 1 entries
2021-02-08 14:13:37,980 Starting aggregator loop
2021-02-08 14:13:37,980 Starting stamper loop
2021-02-08 14:13:37,987 New block 00000000000000000009cf1b219efe88e23014808b2714fd2e185eb240397948 at height 669707
2021-02-08 14:13:38,279 Couldn't find a confirmed output, trying unconfirmed
2021-02-08 14:13:38,300 Can't timestamp; no spendable outputs

any ideas how to get back on track? do i need more btc ?

more efficient transactions

i found a way of committing a merkle tree to a bitcoin transaction that costs less fees and is compatible with the current opentimestamps client
instead of an op_return output, the merkle tree is embedded in a p2wsh script

the op return output costs 8 bytes value, script length (22), OP_RETURN (6a), data length (20), 32 bytes data = 43 vbytes total
when spending, the last witness element of 33/4 vbytes (p2wpkh pubkey) is replaced with a script <pubkey> checksigverify <commitment> ((34 + 1 + 33)/4 vbytes) and the p2wsh output costs 12 vbytes more compared to p2wpkh.
timestamp transactions produced this way are 22.25 vbyte smaller, and will cost 23 sats less at 1 sat/vB (i like sats)

example (i timestamped a message with this type of transaction on testnet):
timestamped text:
testnet timestamps example 1
put this is a file named 'example'

ots file as hex: (echo <hex> | xxd -r -p > example.ots to convert to ots file)

004f70656e54696d657374616d7073000050726f6f6600bf89e2e884e8929401081d31be05e85083599f737b57093d6e30ec571a8473a5d1415ba3135ce56a03c0f1242102fd4e04cdb53aaba4a42f8c5ef2351512167a5d47022cccb66250312b519ccda6ad2008f13a0100000001f121a3cbddd4ca29758e9017d8c53638b24a51100b8da5ba03c13425af2bb6960000000000ffffffff018d26000000000000220020f004000000000808f020af10816130558cac98ea58e94b767f7ea9bbef818455c63db3ee5365ef2394190808f02011a84a66c3bf689f6298daafe39833b9046f6c5686f2121dcd86fb16a336f3e80808f0208e9b34418af67c0f8f1648616dc3535048973d40414a911b7972eab8106a84c10808f120b43540f35f0165b55c58a88915d8bd5d17a0938626ede9871bd1aee6a485ce4f0808f1200bd207f905c3b51c3d4c1feb83cbbbcf1d47b8f68a182dd4f6631467b6d0cadb0808000588960d73d7190104e0eb9201

verify:

$ ots --btc-testnet verify example.ots 
Assuming target filename is 'example'
Success! Bitcoin block 2405856 attests existence as of 2022-11-11 CET

txid (testnet): e880fa56a829a3890393a2213fe43f20a1a2ea51d4d8c4758b9bf65a4cfa3216 (forgot to use low r so only 22 vbyte smaller)

btw, merkle trees can also be committed using a taproot tweak like pubkey tweak, leaving no traces on the blockchain, but the opentimestamps client does not support this yet

You service is sh*t for Bitcoin community :-E

Hi,

You service is sh*t for Bitcoin community :-E
You use UTXO for proof and a database of UTXO permanently grows from your proof transactions!
You could use OP_RETURN as it do https://proofofexistence.com/

It's right way!

Standalone aggregator questions

• Should standalone aggregators submit to a single calendar server or could the be configured to post to several of them?
• How should the standalone aggregator respond to a network error or calendar responding with an error? Log the error, rebuild tree with new digests and retry submission?

Finney crash

Finney which was running cb25ede9 wasn't timestamping after this happened in the log:

File "/home/ots/opentimestamps-server/otsserver/stamper.py", line 464, in __loop
                      self.__do_bitcoin()
                    File "/home/ots/opentimestamps-server/otsserver/stamper.py", line 305, in __do_bitcoin
                      block = proxy.getblock(block_hash)
                    File "/home/ots/.local/lib/python3.5/site-packages/bitcoin/rpc.py", line 442, in getblock
                      r = self._call('getblock', block_hash, False)
                    File "/home/ots/.local/lib/python3.5/site-packages/bitcoin/rpc.py", line 217, in _call
                      self.__conn.request('POST', self.__url.path, postdata, headers)
                    File "/usr/lib/python3.5/http/client.py", line 1106, in request
                      self._send_request(method, url, body, headers)
                    File "/usr/lib/python3.5/http/client.py", line 1151, in _send_request
                      self.endheaders(body)
                    File "/usr/lib/python3.5/http/client.py", line 1102, in endheaders
                      self._send_output(message_body)
                    File "/usr/lib/python3.5/http/client.py", line 936, in _send_output
                      self.send(message_body)
                    File "/usr/lib/python3.5/http/client.py", line 908, in send
                      self.sock.sendall(data)
                  BrokenPipeError: [Errno 32] Broken pipe

I am not sure why this wasn't a problem before, I am restarting with this fix RCasatta@fcc251a
but I am not convinced.

calendar uri & hmac-key files not documented

First run requires two undocumented files in the calendar directory: uri & hmac-key.

I suppose for uri something like this:

http://myhost:port/path?query

But a working example is very useful.

I suppose that hmac-key should be generated with something like this:

import hmac
import hashlib
import base64
dig = hmac.new(b'secret', msg=bytes_string, digestmod=hashlib.sha256).digest()
base64.b64encode(dig).decode() 

Where 'secret' is a password and bytes_string is the string saved inside the uri file above.

os.fdatasync on Mac OS

There is an issue running otsd on Mac OS because the os.fdatasync function is not supported on the platform.

This is just for reference, it's not very useful to have the server running on Mac OS

here is the stack trace:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.5.2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python3/3.5.2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/casatta/workspaces/github/RCasatta/opentimestamps-server/otsserver/calendar.py", line 192, in __loop
    self.calendar.submit(digests_commitment)
  File "/Users/casatta/workspaces/github/RCasatta/opentimestamps-server/otsserver/calendar.py", line 98, in submit
    self.journal.submit(commitment.msg)
  File "/Users/casatta/workspaces/github/RCasatta/opentimestamps-server/otsserver/calendar.py", line 74, in submit
    os.fdatasync(self.append_fd.fileno())
AttributeError: module 'os' has no attribute 'fdatasync'

Create Code of Conduct

Maybe a Code of Conduct might be nice. What do you all think?

Can't timestamp; no spendable outputs

Hi. I mounted a bitcoin node and it's syncronized.
I have 0.0005 BTC in it.
When I run Opentimestamps server, I throws an error.

Opening journal for appending...
Journal has 1 entries
Starting aggregator loop
Starting stamper loop
New block 00000000000000000012c625df8bd746933f71aac61c10cba5f4ea4219ea3264 at height 551502
Couldn't find a confirmed output, trying unconfirmed
Can't timestamp; no spendable outputs

Can anybody help me?

Memory issues while performing LevelDB batch write

In my server memory usage spikes here

opentimestamps-server/otsserver/calendar.py

Line 205 in ac67218

self.db.Write(batch, sync=True)

I was wondering if we should write batch incrementally, for example after this log call

opentimestamps-server/otsserver/calendar.py

Line 200 in ac67218

logging.debug("Added %d timestamps to LevelDB; %f stamps/second" %

Public Accessibility - urlopen error [Errno 111] Connection refused

Everything is going great, now that the server is working, i was trying to remote stamping to work, but found issues with just using the domain name over loopback, not quite sure what I am missing.

I have ports 80 / 443 / 14788 open

Server: https://legaltimestamp.com

Loopback
[ec2-user@ip-172-31-61-7 ~]$ touch FILE2
[ec2-user@ip-172-31-61-7 ~]$ ots -vv stamp -c http://127.0.0.1:14788 -m 1 FILE2
Doing 1-of-1 request, timeout is 5 second
Submitting to remote calendar http://127.0.0.1:14788
0.96 seconds elapsed
[ec2-user@ip-172-31-61-7 ~]$

Hostname Same Machine
[ec2-user@ip-172-31-61-7 ~]$ ots -vv stamp -c http://legaltimestamp.com:14788 -m 1 FILE2
Doing 1-of-1 request, timeout is 5 second
Submitting to remote calendar http://legaltimestamp.com:14788
<urlopen error [Errno 111] Connection refused>
Failed to create timestamp: need at least 1 attestation but received 0 within timeout
[ec2-user@ip-172-31-61-7 ~]$

Hostname Diff Machine
[root@ip-172-31-11-91 ~]# ots -vv stamp -c http://legaltimestamp.com:14788 -m 1 FILE2
Doing 1-of-1 request, timeout is 5 second
Submitting to remote calendar http://legaltimestamp.com:14788
<urlopen error [Errno 111] Connection refused>
Failed to create timestamp: need at least 1 attestation but received 0 within timeout

.otsd/calendar/uri
[ec2-user@ip-172-31-61-7 ~]$ cat ~/.otsd/calendar/uri
http://legaltimestamp.com:14788
[ec2-user@ip-172-31-61-7 ~]$

/etc/nginx/conf.d/ots.conf
[ec2-user@ip-172-31-61-7 ~]$ cat /etc/nginx/conf.d/ots.conf
proxy_cache_path /tmp/nginx-cache levels=1:2 keys_zone=otsd:10m inactive=3600m max_size=1000m;

server { listen 443 default_server; listen [::]:443 default_server; server_name opentimestampserver.ddns.net;

root /dev/null;

location calendar { autoindex on; alias var/lib/otsd/calendar/exported; }

location / { proxy_cache otsd; add_header X-Proxy-Cache $upstream_cache_status always;

add_header 'Access-Control-Allow-Origin' '*'; add_header 'Access-Control-Allow-Methods' 'POST GET';

proxy_pass http://127.0.0.1:14788; proxy_set_header Host $host; }

ssl on; ssl_certificate /etc/letsencrypt/live/opentimestampserver.ddns.net/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/opentimestampserver.ddns.net/privkey.pem;

ssl_session_timeout 5m;

ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';

ssl_prefer_server_ciphers on; ssl_dhparam /etc/ssl/private/dhparams.pem;
}
[ec2-user@ip-172-31-61-7 ~]$

How to start a mirror server

Hi there!

Just wondering if it is possible to have a local server with mirrored data por upgrading the OTS's. I've seen the real-time requirement in the TODO file in the future, but I do not really have real-time requirements, i.e., it would fine for me to upgrade only files that are a couple of days old, and I would be more than willing to run a mirror on my internal network so that I do not send too much traffic to your servers.

Is there a way to do this? We are currently sending too many upgrade requests so we are having too many rejections. ¿Is there a better way to do the upgrades that to call the calendar server with every its?

BTW, thank you guys for a great library :-)

Cheers,

Manuel

Suggestion: OpenTimeStamping should be also working via mail (receive a mail, timestamp/sign it, and reply to sender)

See http://www.itconsult.co.uk/stamper.htm for an implementation started back in 1995 which uses PGP.

Still pending attestation

Hello there.

I saw the pending attestation issue was closed but I've found I have the same problem.

Using the browser, when I verify an OTS file with the stamped file, I obtain the green flag of successfully verified file but if I click on the small "i" in the right upper corner, where you can see the details of the operation, there's still a Pending attestation: server https://alice.btc.calendar.opentimestamps.org

Can you check what's going on?

Using Bitcoin Core 0.19.0

Is supporting newer Bitcoin Core versions planned? Is it possible to get it to work with an already running instance of Bitcoin Core 0.19.0?

[unittest] AttributeError: CTransaction.GetHash() has been removed; use GetTxid() instead

python 3.6.2
centos-release-7-4.1708.el7.centos.x86_64

======================================================================
ERROR: test (opentimestamps.tests.test_bitcoin.Test_make_timestamp_from_block)

Traceback (most recent call last):
File "/data/otsd/opentimestamps-server/opentimestamps/tests/test_bitcoin.py", line 26, in test
root_stamp = make_timestamp_from_block(digest, block, 0)
File "/data/otsd/opentimestamps-server/opentimestamps/bitcoin.py", line 77, in make_timestamp_from_block
assert commitment_tx.GetHash() == txid_stamp.msg
File "/data/otsd/lib64/python3.6/site-packages/bitcoin/core/init.py", line 363, in GetHash
raise AttributeError("CTransaction.GetHash() has been removed; use GetTxid() instead")
AttributeError: CTransaction.GetHash() has been removed; use GetTxid() instead

---

Ran 69 tests in 1.324s

FAILED (errors=1)

High cumulated cpu usage

I recently noticed high cumulated cpu usage from the calendar server.
From the log I saw my server took more than 2 minutes to process a block. (I admit it's a shitty cpu but it's too much anyway)
After investigation I found the bottleneck is the make_timestamp_from_block method which is called for every past unconfirmed tx. Recently network conditions caused my server to have about 300 prior transactions and this caused the long processing.
I wrote a simple test case that confirmed my suspect and created an optimized version of the function in this commit.
The optimization rely on serializing the entire block out of the cycle and searching the digest inside this data, immediately exiting from the actual function if not found (avoiding serializing every tx every time). This is about 50 times faster in the worst case scenario.
I am currently testing this code in the finney server before making the PR.
@petertodd what do you think?

Bitcoin node: Remote end closed connection without response

I am running client and server, this seemed way more server related.

expected
[root@ip-172-31-11-91 ~]# ots verify WORKING_FILE.txt.ots
Assuming target filename is 'WORKING_FILE.txt'
Ignoring attestation from calendar http://legaltimestamp.com:14788: Calendar not in whitelist
expected
[root@ip-172-31-11-91 ~]# ots -l http://legaltimestamp.com:14788 upgrade WORKING_FILE.txt.ots
Got 1 attestation(s) from http://legaltimestamp.com:14788
Success! Timestamp complete
unexpected but makes sense
[root@ip-172-31-11-91 ~]# ots verify WORKING_FILE.txt.ots
Assuming target filename is 'WORKING_FILE.txt'
Could not connect to Bitcoin node: Cookie file unusable ([Errno 2] No such file or directory: '/root/.bitcoin/.cookie') and rpcpassword not specified in the configuration file: '/root/.bitcoin/bitcoin.conf'
unexpected and unsure
[root@ip-172-31-11-91 ~]# ots --bitcoin-node http://legaltimestamp.com:8333 verify WORKING_FILE.txt.ots
Assuming target filename is 'WORKING_FILE.txt'
Could not connect to local Bitcoin node: Remote end closed connection without response

I have confirmed the node is open via https://bitnodes.io/ and tracking connections at https://io.adafruit.com/insidenothing/dashboards/blockchain

My guess is there are RPC configurations (different port?) that need to be made to bitcoin core so "--bitcoin-node" will work, any suggestions on where to focus?

Attestation pending for over two days

Here's the attestation I submitted, still pending in all the three calendars:

Attestation

Opentimestamp server prevented wallet to see transaction

Hello,
I'm running a public full node using bitcoin core v0.17.1 with the options deprecatedrpc=signrawtransaction deprecatedrpc=accounts addresstype=bech32 and otsd with only the --rpc-port option specified (I know that I can use the public servers but I've explained in another issue why I'll run my own server).
About two days ago I've transferred some funds from a coinbase account to the wallet on my server, and even if I was sure that the transaction was confirmed my node was showing the wallet as empty. You can read the details here: bitcoin-dot-org/Bitcoin.org#3299
Now (about two days after the transaction) I was trying to understand why the wallet wasn't up to date and I tried to run bitcoin-cli getblockchaininfo, stop opentimestamps server and run it again to see if I could determine if it was having some influence on bitcoin core. Sadly I don't know how to interpret getblockchaininfo's output so I wasn't able to see if there was an important difference, nor I have the output before and after stopping otsd (I've called getblockchaininfo multiple times and now I'm unable to tell which calls where while otsd was running and which while it wasn't); but after starting it again I saw that now my wallet has the right balance:

$ bitcoin-cli getwalletinfo
{
  "walletname": "",
  "walletversion": 169900,
  "balance": 0.00228164,
  "unconfirmed_balance": 0.00000000,
  "immature_balance": 0.00000000,
  "txcount": 1,
[other fields omitted]
}

So seems that somehow otsd prevented the wallet from updating the balance. Is this a symptom of a configuration error on my side, a known problem or a new bug?

Connection with multichain

Hi everyone
I want to use opentimestamp with private blockchain.
For the private blockchain i use Multichain .
someone can help with integrate opentimestamp and multichain

thanks

Timestamp returned content for each GET request

Provide a base64-encoded immature timestamp at the bottom of each index page (/ route) returned by GET requests.

Missing intermediate certificate on https://*.opentimestamps.org

Running ./ots stamp README.md produces:

Submitting to remote calendar https://a.pool.opentimestamps.org
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1254, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1106, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1151, in _send_request
    self.endheaders(body)
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1102, in endheaders
    self._send_output(message_body)
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 934, in _send_output
    self.send(msg)
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 877, in send
    self.connect()
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1260, in connect
    server_hostname=server_hostname)
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 377, in wrap_socket
    _context=self)
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 752, in __init__
    self.do_handshake()
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 988, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 633, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./ots", line 34, in <module>
    args.cmd_func(args)
  File "/Users/mee/dev/repos/opentimestamps-client/otsclient/cmds.py", line 141, in stamp_command
    create_timestamp(merkle_tip, args.calendar_urls, args.setup_bitcoin if args.use_btc_wallet else False)
  File "/Users/mee/dev/repos/opentimestamps-client/otsclient/cmds.py", line 98, in create_timestamp
    calendar_timestamp = remote.submit(timestamp.msg)
  File "/Users/mee/dev/repos/opentimestamps-client/opentimestamps/calendar.py", line 36, in submit
    with urllib.request.urlopen(req) as resp:
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 163, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 466, in open
    response = self._open(req, data)
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 484, in _open
    '_open', req)
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1297, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 1256, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)>

Accessing https://a.pool.opentimestamps.org via either Chrome, Opera or wget produce similar errors (Firefox works fine).

SSLLabs and whatsmychaincert provide some further info and suggestions.

(Not sure if that bug belongs here)

Extract the HTML templates into a folder

I think it might be useful to remove the hard-coded HTML and create a "templates" folder.
If you agree I can make this operation for participating to hacktoberfest! :P

Tests error on Mac OS X

running python3 -m unittest discover -v on Mac OS X gives the following error

======================================================================
ERROR: test_blobs (opentimestamps.tests.core.test_git.Test_GitTreeTimestamper)
Git blob hashing
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/xxx/github/opentimestamps/opentimestamps-server/opentimestamps/tests/core/test_git.py", line 58, in test_blobs
    stamper = self.make_stamper("30f6c357d578e0921dc6fffd67e2af1ce1ca0ff2")
  File "/Users/xxx/github/opentimestamps/opentimestamps-server/opentimestamps/tests/core/test_git.py", line 37, in make_stamper
    db = dbm.open(self.db_dir.name + '/db', 'c')
  File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/dbm/__init__.py", line 94, in open
    return mod.open(file, flag, mode)
_gdbm.error: [Errno 35] Resource temporarily unavailable

----------------------------------------------------------------------

looks a known bug of dbm on the platform

Timestamps don't start in "pending" state

The timestamp state (unknown/pending/complete) seems very time-sensitive: a POST /digest that is immediately followed corresponding by GET /timestamp/$hash yields a 404 Not Found.

However, when inserting a one-second delay (the aggregation interval?), the response is instead 404 Calendar localhost: Pending confirmation in Bitcoin blockchain.

This can be pretty confusing when writing a client. I suggest the server should either respond with the "Pending confirmation" from the start, or with a different "Pending" message.

Script to reproduce:

# in opentimestamps-client
# running opentimestamps-server on localhost

rm -f release-notes.md.ots && \
  ./ots s -m 1 -c http://localhost:14788 release-notes.md && \
  sleep 0 && \
  ./ots u -c http://localhost:14788 release-notes.md.ots

Changing sleep 0 to sleep 1 yields the expected response.

Minor enhancement for WEB-UI: more info on last transactions

Using gettransaction RPC call we could get a lot more info on the latest transactions which would make it easier to observer server statistics and behavior from the web-ui.

This is low-priority and could be done in the future as part of a re-design of the web-ui.

Suggestions:

1. Blockhash
2. Transaction fee

Not sure if there's an easy way to get the blockheight directly from core RPC though

Question about using a pruned node

Hello everyone.
I will setup a calendar server, but since I'm new to blockchain technology and I know it's internals only up to a certain point I'm wondering: if I use opentimestamp server with a pruned node that keeps, let's say, only the last 550 blocks, would it be able to confirm a document whose hash was registered in a block before the last 550?
Thanks for your help.

Does opentimestamps send currency to "random" Bitcoin addresses?

I'm having trouble groking opentimestamps. Does it send Bitcoin to an address that is a hash of some data? If so, doesn't that essentially destroy the currency, in that it will never be retrievable? Thanks for your input, and please excuse any naivety.

Exposing last tip for external tx

In the context of allowing external actor to make the timestamp transaction the server must expose his last tip.
Simple proposed implementation could be something like this:
https://github.com/RCasatta/opentimestamps-server/compare/perf-fix-with-order...RCasatta:get-tip?expand=1

Calendar server is down

Hi Team,

One of the calendar servers https://bob.btc.calendar.opentimestamps.org/ is down it seems.

Also, we are getting "Handshake failure" due to the recent SSL certificate(with EC signature algorithm) changes in the Alice server.

Thanks,
Anand

API to get recent probable permanent block hash

In order to provide an early bound on a piece of content, a known block hash can be included in timestamped data. I see the home page lists best block but didn't find an API to get that from OTS. Taking into account reorgs, it would also be useful to get a block hash 6 or 12 blocks from best.

Running a bitcoin node is an alternate way to get this info but I'm not sure if a lightweight client in java that downloads headers only would be much of an improvement over trusting ots servers to report these block hashes.

The other question is, would this APIs be accepted here if a quality PR were submitted?

Question - Frequency of transactions

Just curious, how many minutes/how many requests does it take for an opentimestamps server to start a bitcoin transaction? Is it hourly or based upon the number of waiting proofs?

Use Sphinx for documentation

http://sphinx-doc.org/ext/autodoc.html#module-sphinx.ext.autodoc

Potential issue with implementation: More open to hash collisions when it doesn’t have to be

I found this project because I’ve been pondering similar questions for a few years, and publicly storing file hashes is viable for specific kinds of files (it’s suboptimal for photos/videos since you really need to store perceptual hashes, but I digress), but there is one major (resolvable) obstacle:

Given enough time, it is likely that any specific hash method will be “cracked” and allow someone to create a different file that matches the same hash.

One technique is to record the file-size as well since it’s a trivial change that makes an intentional collision much harder. Git does this to avoid accidental collisions at scale.

Another is to store/compare multiple different hashes for the same file. The more algorithms used, the more confident the verification can be.

I’d suggest at minimum adding the file sizes to the hash as it’s a tiny amount of easily-obtained data.

Public Server Availability and Reliability

Hi,

I want to know if the public server is reliable for production usage. Like if we have 100,000 entries generated every day and use the public server to generate proofs for each of them, is it proper? From my understanding, the public server is already aggregating data, so wondering do we still need to aggregating our daily data in batch and timestamp it once every day.
Thanks

problem own calendar server

We are trying to start calendar server in regtest mode, but there is an error:

./otsd --btc-regtest --btc-min-confirmations 1 --btc-min-tx-interval 60
Opening journal for appending...
Journal has 2 entries
Starting aggregator loop
Starting stamper loop
New block 358b81119a13ba553a3f178601e466e2eab5255cd5b542b4e611ecc28c45a420 at height 109
Sent timestamp tx 2da1295f2399153b17c81bcd15f313b64f16eb112dd7386b0ad6f105448a20c8; 2 total commitments

127.0.0.1 - - [12/Jul/2017 01:12:10] "GET / HTTP/1.0" 200 -
Aggregated 1 digests under commitment 1cb33f719e10fb673e78cd80055b002470da8d683cb95bdcc6afacc3b981736b
127.0.0.1 - - [12/Jul/2017 01:12:33] "POST /digest HTTP/1.0" 200 -
127.0.0.1 - - [12/Jul/2017 01:12:47] "GET /timestamp/596577811cb33f719e10fb673e78cd80055b002470da8d683cb95bdcc6afacc3b981736ba6a18d8036965fd0 HTTP/1.0" 404 -
New block 163b023c6b1f2d9f1be525d5a2c4af731b8fad5cd94fdefd3fe4f72f1dfd62cb at height 110
New block 26917fba243ba1e01bc2916b431657c0ce5d2de64e7e3d404b76843d4fd50c5c at height 111
New block 52c962c801c28979a3512310ae27489d358f269b4d4f0576b3f7510b02f9a001 at height 112
Sent timestamp tx 76fb610c7992a55a9b7ff0caa5f9fdfc0994b82bcf6b4775ec017060d571c312, replacing 2da1295f2399153b17c81bcd15f313b64f16eb112dd7386b0ad6f105448a20c8; 3 total commitments
New block 08ecdf6a822216352601aef1ef948434379bf5214fb28d68cd9fcf44eedf5642 at height 113
__do_bitcoin() failed: AssertionError()

Docker version of opentimestamps-server

Hi everyone, is there a official docker version of the opentimestamps-server?
i find this diginex/opentimestamps-server on docker-hub...

signrawtransaction is deprecated and will be fully removed in v0.18

I must init bitcoin with -> bitcoind --daemon -deprecatedrpc=signrawtransaction

bitcoin.rpc.JSONRPCError: {'code': -32, 'message': 'signrawtransaction is deprecated and will be fully removed in v0.18. To use signrawtransaction in v0.17, restart bitcoind with -deprecatedrpc=signrawtransaction.\nProjects should transition to using signrawtransactionwithkey and signrawtransactionwithwallet before upgrading to v0.18'}

getaccountaddress is deprecated and will be fully removed in v0.18

From 0.17.x the rpc module fails calling getaccountaddress method and the web ui result blank to the user.

[Help] Question about specifying otsd port

Hello,
The readme states that for public servers I should put inside ~/.otsd/calendar/uri an URL that can be consistently reached by clients and a port. My question is: since for public servers it's suggested to run otsd behind a proxy (I will use nginx) should I put in the above file the port on which the proxy is listening and specify otsd's port in some other way or should I put otsd's port no matter on which port the proxy will be listening?
Thanks for the clarification and have a nice weekend.

BTC node and OTS server on different machines

Hi, I am trying to create a custom calendar server. I currently have the bitcoin node in the same machine. And it works fine. Is it possible to run the Bitcoin node and the calendar server in different machines? How can we configure the calendar server to connect to the bitcoin node running in the other machine? Thanks

Failed to sign transaction

Bicoin version 0.17.01
Bitcoin command to init: bitcoind --daemon -deprecatedrpc=signrawtransaction

This is the error:

bitcoin.rpc.InWarmupError: {'code': -28, 'message': 'Loading P2P addresses...'}
New block 0000000000000000003031a88b78124fabeb0b33a89830184143f3a6cbe51e67 at height 552626
Failed to sign transaction! r = {'tx': CTransaction((CTxIn(COutPoint(lx('25a9847931a1507bed514367be8bfb69647df8b8b8085292ab8a14ac64e48c8f'), 0), CScript([x('0014ffd033b71790484c44715d0aa9786b977e104c58')]), 0xfffffffd),), (CTxOut(0.00127166COIN, CScript([OP_HASH160, x('c7c1ca6a27cd07b1087972b4dd5b00c1ca961d14'), OP_EQUAL])), CTxOut(0.0COIN, CScript([OP_RETURN, x('2174915a6bc55f92e23cb17306f1a06129ba1176313ae07add43f4958a861e15')]))), 552626, 1, CTxWitness()), 'errors': [{'vout': 0, 'txid': '25a9847931a1507bed514367be8bfb69647df8b8b8085292ab8a14ac64e48c8f', 'error': 'Witness program hash mismatch', 'sequence': 4294967293, 'scriptSig': '160014ffd033b71790484c44715d0aa9786b977e104c58', 'witness': []}], 'complete': False}

https://opentimestamps.org/ not showing pretty breakdown of OTS any more.

Sorry - know this is not really the right place to raise this but could not see anywhere else.

Issue is just when you drad/drop an OTS then file on the website then click on the (i) link the next page with the pretty breakdown of the proof does not load.

Thanks