mit-dci/opencx

An open-source cryptocurrency exchange toolkit for implementing experimental exchange features

opencx

Cryptocurrency exchanges are some of the largest businesses in the cryptocurrency space, and their reserves are often viewed as "honeypots" of cryptocurrencies. Because of this, cryptocurrency exchanges have been a hotbed of crime in the form of hacks, front-running, wash trading, fake orderbooks, and much more. In order for cryptocurrency to be successful, we need safe, trustworthy ways to exchange cryptocurrencies, without fear that coins will be stolen, or trades executed unfairly. Additionally, the vast majority of exchange software is closed-source, and exchanges have historically not implemented technological upgrades that would substantially decrease risk for users.

OpenCX hopes to solve this problem by making it trivially easy to run a secure, scalable cryptocurrency exchange which implements many of these features, including:

• Layer two compatibility
• Non-custodial exchange
• Anti front-running
• Public orderbook auditability
• More to come...

Additionally, all of the components of OpenCX are designed to be swappable, secure, and scalable. The goal is to fit those requirements and implement features similar to that of modern cryptocurrency exchanges, while keeping high quality software.

DO NOT use in a production environment, this project is a work in progress!

Pull requests and issues encouraged!

Demo

Contributing

Please see the contributing file to get started with contributing!

Setup

Requirements

• Go 1.12+
• A MySQL Database (not needed for client)
• GMP (GNU Multiple Precision Arithmetic Library)

Installing

Installing GMP

Debian

sudo apt-get install libgmp3-dev

macOS

brew install gmp

Clone repo and install dependencies

git clone [email protected]/mit-dci/opencx.git
cd opencx
go get -v ./...

Running opencx server / exchange

You will need to run MariaDB or any other MySQL database in-order to run the server. You can configure authentication details for your database at ~/.opencx/db/sqldb.conf

Start your database (MariaDB in this case)

Linux

sudo systemctl start mariadb

macOS

mysql.server start

Now build and run opencx

go build ./cmd/opencxd/...
./opencxd

Running opencx CLI client

go build ./cmd/ocx/...
./ocx

You can now issue any of the commands in the cxrpc README.md file.

Configuration

There are configuration options (both command line and .conf) for the client and the server, and by default home folders for these files will be created at ~/.opencx/opencxd/ and ~/.opencx/ocx/ respectively. You can decide whether or not to use the NOISE protocol for authentication, which hostnames and ports to use for connecting to certain clients, which coins you would like to support, and whether or not to support lightning.

If you'd like to add your own coins, just add a coinparam struct like in lit.

opencx's Issues

Create set of tests for new DB API

Is your feature request related to a problem? Please describe.
Now that the database is refactored into something more logical, it would be useful to create a set of tests for the following APIs:

• SettlementEngine
• AuctionEngine
• LimitEngine
• AuctionOrderbook
• LimitOrderbook
• PuzzleStore
• DepositStore

Describe the solution you'd like
Each of the APIs are fairly descriptive, so creating tests for the API can serve as a test for all future datastore implementations. Creating inputs / outputs for each call is probably enough, given many of them are designed to have inputs as well as outputs.

Describe alternatives you've considered
The database implementations could also be tested individually, but this would probably be the same amount of work, except more useful.

Additional context
N/A

Web API

Currently we only have RPC, it may be useful to include a cxweb package that is a web API

GPU timelock puzzle solver

Is your feature request related to a problem? Please describe.
Currently if I want to run an exchange that is supposed to be front-running resistant, I have to solve a bunch of timelock puzzles in parallel. CPUs are okay at this if they have lots of cores, but since we're doing a lot of the same kind of operation, GPUs might be way better.

Describe the solution you'd like
First it would be useful to figure out the latency difference between GMP on the CPU, and whatever is out there for GPUs. If that's way too high (2-3x or so) then this isn't really viable. Also figuring out whether or not GPUs would even help in this scenario is important.
If a GPU would help in this scenario and isn't too much worse latency-wise then a package should be written to take advantage of the parallelism on modern graphics cards.

Wesolowski and Pietrzak proof generation on graphics cards might also be useful (the logn space proof strategy), if it makes sense.

Describe alternatives you've considered
Another solution to this problem would be figuring out an easy way to distribute the work across multiple machines.

Additional context
I know nothing about how graphics cards work or what they're good at, I have just heard they're good at doing things in parallel, and that's sort of what we need.

Fix matching algorithm

I think the matching algorithm only runs for orders that are equal in price, disregarding whether or not the current best sell order is a lower price than the current best buy order. This is a simple fix, like 6 lines of code + database debugging, which gets tiring.

Handle errors in cmd package init files

In the initialization files such as opencxdinit.go, ocxinit,go, and fredinit.go, there are some calls to os and file methods that return errors, but the errors are ignored. They should be handled at some point.

These files are fairly messy, so refactoring these files and going through any other code which creates files / directories would be very helpful.

Dockerfile for opencxd

Is your feature request related to a problem? Please describe.
Currently opencxd requires building from source, and takes more than a few steps to get up and running. Docker is a great solution to this problem, and helps standardize the build process.

Describe the solution you'd like
Write a dockerfile and relevant documentation so we can use docker for quickly running opencxd.

Describe alternatives you've considered
N/A

Additional context
N/A

Configurable chain support

The decision to include traditional exchange features that would require deposit addresses, withdrawing to address etc. is something that could be configurable now that lightning fund i/o is now a thing.

Interactive first-time setup

Is your feature request related to a problem? Please describe.
cmd/opencxd, while highly configurable, would benefit user-experience wise if there were some form of interactive setup during the first run.
The configuration options can be explained and prompted if there is no configuration file.

Describe the solution you'd like
Since cmd/opencxd already has a configuration file, it would make sense to add a prompt that generated one if one doesn't exist. Here's an idea of what it should look like:

Welcome to OpenCX! Here are some questions to help you get set up.
Would you like to create a cryptocurrency or assets exchange? 
    crypto, or 
    assets
opencx-setup# crypto
Which of the following coins would you like to support? 
    bitcoin, 
    litecoin, 
    vertcoin
opencx-setup# bitcoin, litecoin, vertcoin
Would you like to support lightning or other layer-2 solutions? 
    yes, or 
    no
opencx-setup# yes
... etc etc

Describe alternatives you've considered
A GUI setup would also work, but this gets the same functionality without having to incorporate any new frontend libraries.

Additional context
N/A

Refactor DB code

Right now the code for database stuff is really ugly - it does sadly need refactoring. There are two things that can be done, one is have some persistent data storage solution that doesn't require management like mariadb, the other is cleaning up the current SQL implementation so it has more stuff in, say, a config file, rather than globals.

If different things are supposed to be pluggable as well, like auction vs normal exchange, we should have a nice way of making sure the database is set up for that. For example, if we wanted to use the same SQL database for either an auction orderbook, or peer storage, or both.

Wesolowski and Pietrzak VDF Proof for RSW timelocks

Is your feature request related to a problem? Please describe.
Currently we have timelock puzzles in the crypto package, but we can very easily create a VDF interface, and implement the Wesolowski/Pietrzak proofs for rsw, and then we'll have a VDF implementation.

Describe the solution you'd like
An implementation of https://eprint.iacr.org/2018/623, https://eprint.iacr.org/2018/627 for what we've built in rsw and an abstract VDF interface in the crypto package.

The hope is to get this as fast as possible verification wise, so using the Rjected/gmp fork of ncw/gmp would be a good idea.

Describe alternatives you've considered
There are also VDF proofs for iterated hashing, but those involve snarks and iterated-hashing as timelock puzzles aren't really that useful, since there's no trapdoor. It's better to add to what we've already written.

Transaction Replay

Is your feature request related to a problem? Please describe.
It would be useful for simulations if transactions could be inferred from orderbook state changes from other exchanges, so the transactions could be replayed later. This would be useful for benchmarking especially, since it's realistic data with realistic timestamps.

Describe the solution you'd like
Potentially work out benchclient and create a new daemon (like opencxd but much more limited) since these orders would have to conform to our format and pass validation, so pubkeys, etc would most likely have to be filled in. This really just tests the matching engine so maybe a new daemon is all that's necessary, just remove validation.

Describe alternatives you've considered
This is sort of similar to backtesting but is more for benchmarking and simulations. Backtesting isn't quite what we want, since we might want to determine realistic overhead or hardware required for various operations and configurations.

Additional context
Collecting, timestamping, and replaying this data is generally helpful for testing.

Proofs of solvency

Implementing a proof of solvency / Provisions module would be a good addition

Is this because lit needs to be running?

did not see clear doc on what daemon should be running?

2022/11/21 10:36:33.548299 Error setting up wallets:
Error when starting wallet:
Unsupported coin daemon or coin daemon not running

Pro-rata matching and auction support

Currently the matching algorithm is embedded in the structs implementing the OpencxStore interface. The matching algorithm should be abstracted and pro-rata should be implemented. Pro-rata should help to implement batch auctions.

Better encoding/decoding

Need to figure out a better way for encoding/decoding. Right now we only really need to decode one thing from bytes, AuctionOrder, but it's good practice to have a standard way to encode and decode various things we might want to send over the wire.

Update Redis DB implementation

Is your feature request related to a problem? Please describe.
Redis is extremely fast, and was originally chosen as the database implementation (cxdb/cxdbredis). However, it was much easier to do transactions in MySQL, and cxdb/cxdbsql was created. #11 should help to more thoroughly define the interaction between the server and datastore, so once there is a more general, easy to use, and modular interface for defining datastore implementations, the redis implementation should be much easier to update.

Describe the solution you'd like
Once #11 is finished, the cxdb/cxdbredis package should be essentially rewritten to conform to the interfaces that we will have defined. Writing a DB implementation for OpenCX that is also ACID would be very helpful.

Describe alternatives you've considered
There are other databases that we could implement but there should really be a fast database option alongside the SQL database, and redis is pretty good. Memcached is another option but we don't want to be maintaining too many database implementations unless it's absolutely necessary. Another alternative could be Cassandra, which would be very cool, but again it's hard to maintain multiple database implementations unless we've decided on a final interface, which is unlikely to happen anytime soon.

Additional context
N/A

Add successful transcript creation & validation benchmarks

Title, need to add transcript creation and validation performance benchmarks.

Make matching testable

Writing tests for the matching engine and orderbook operations would be a good thing to do at some point

Channel use optimization

Optimize channel use when doing deposits/withdraws to maximize the use of the total channel capacities.

Out of memory error with GMP timelock puzzles

Describe the bug
When running the fred daemon, solving many timelock puzzles in one long auction makes GMP crash.

To Reproduce
Steps to reproduce the behavior:

1. In ~/.opencx/fred/fred.conf, set auctiontime=30000000000
2. go build ./cmd/fred/...
3. go build ./cmd/ocx/...
4. ./fred -vvv in one terminal
5. ./ocx placeauctionorder buy regtest/litereg 10000 10
6. Repeat the previous step 4 times

Expected behavior
I expected the puzzles to be solved without running out of memory

dan@dan-pc ~/D/P/opencx> ./fred -vvv
passphrase: 

2019/05/15 13:50:57.953518 [DEBUG] Asset 0: regtest
2019/05/15 13:50:57.953562 [DEBUG] Asset 1: vtcreg
2019/05/15 13:50:57.953570 [DEBUG] Asset 2: litereg
2019/05/15 13:50:57.953598 [DEBUG] Pair 0: regtest_vtcreg
2019/05/15 13:50:57.953605 [DEBUG] Pair 1: regtest_litereg
2019/05/15 13:50:57.953614 [DEBUG] Pair 2: vtcreg_litereg
2019/05/15 13:50:58.101533 [INFO] Starting an auction with auction time 30000000000
2019/05/15 13:50:58.101678 [INFO] Starting Auction Clock!
2019/05/15 13:50:58.101711 [INFO] Auction clock tick!
2019/05/15 13:50:58.101735 [INFO] Waiting for tick
2019/05/15 13:50:58.101776 [INFO] Notifying signals
2019/05/15 13:50:58.101895 [INFO]  === will start to listen on noise-rpc ===
2019/05/15 13:50:58.101927 [INFO] Registering RPC API over Noise protocol ...
2019/05/15 13:50:58.102075 [INFO] Starting RPC Server over noise protocol
2019/05/15 13:50:58.102514 [INFO] Running RPC-Noise server on [::]:12345
2019/05/15 13:51:06.954237 [INFO] Received timelocked order!
2019/05/15 13:51:06.954503 [INFO] Got a new puzzle for auction f32e6f018a64bcd1108f9affd472b2afbec8b9e2707965a8e7da76eccbb3ec29
2019/05/15 13:51:08.076423 [INFO] Received timelocked order!
2019/05/15 13:51:08.076590 [INFO] Got a new puzzle for auction f32e6f018a64bcd1108f9affd472b2afbec8b9e2707965a8e7da76eccbb3ec29
2019/05/15 13:51:09.312454 [INFO] Received timelocked order!
2019/05/15 13:51:09.312779 [INFO] Got a new puzzle for auction f32e6f018a64bcd1108f9affd472b2afbec8b9e2707965a8e7da76eccbb3ec29
2019/05/15 13:51:10.556470 [INFO] Received timelocked order!
2019/05/15 13:51:10.556654 [INFO] Got a new puzzle for auction f32e6f018a64bcd1108f9affd472b2afbec8b9e2707965a8e7da76eccbb3ec29
GNU MP: Cannot reallocate memory (old_size=8 new_size=3750000040)
SIGABRT: abort
PC=0x7fab88d5982f m=12 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7fab88d5982f
stack: frame={sp:0x7fab6d739b00, fp:0x0} stack=[0x7fab6cf3a288,0x7fab6d739e88)
00007fab6d739a00:  0000000000000000  0000000000000000 
00007fab6d739a10:  0000000000000000  0000000000000000 
00007fab6d739a20:  0000000000647178 <github.com/ncw/gmp.(*Int).doinit.func1+56>  000000380019dfc8 
00007fab6d739a30:  0000000000000000  0000000000000000 
00007fab6d739a40:  0000000000000000  00007fab6d739b00 
00007fab6d739a50:  0000000000451c43 <runtime.tracebackdefers+83>  0000000000000000 
00007fab6d739a60:  0000000000000000  0000000000000000 
00007fab6d739a70:  0000000000e869a0  0000000000000000 
00007fab6d739a80:  0000000000e869a0  0000000000bd6a58 
00007fab6d739a90:  0000000000bd6a58  0000000000000000 
00007fab6d739aa0:  0000000000000000  0000000000000000 
00007fab6d739ab0:  0000000000000000  0000000000000000 
00007fab6d739ac0:  0000000000000000  0000000000000000 
00007fab6d739ad0:  0000000000bd6a58  0000000000e869a0 
00007fab6d739ae0:  000000000045e7b1 <runtime.goexit+1>  000000000045e7b1 <runtime.goexit+1> 
00007fab6d739af0:  0000000000000000  000000c00005efd0 
00007fab6d739b00: <0000000000000000  000000c00005efd0 
00007fab6d739b10:  000000c00005efd8  0000000000000000 
00007fab6d739b20:  0000000000000000  00007fab6d739ce0 
00007fab6d739b30:  0000000000449cc6 <runtime.copystack+630>  00007fab50000020 
00007fab6d739b40:  0000000000001000  00007fab50000020 
00007fab6d739b50:  0000000000001000  00007fab88ee1aa0 
00007fab6d739b60:  00000000df868000  00000000df8475b0 
00007fab6d739b70:  0000000001de7270  000000000001ed90 
00007fab6d739b80:  fffffffe7fffffff  ffffffffffffffff 
00007fab6d739b90:  ffffffffffffffff  ffffffffffffffff 
00007fab6d739ba0:  ffffffffffffffff  ffffffffffffffff 
00007fab6d739bb0:  ffffffffffffffff  ffffffffffffffff 
00007fab6d739bc0:  ffffffffffffffff  ffffffffffffffff 
00007fab6d739bd0:  ffffffffffffffff  ffffffffffffffff 
00007fab6d739be0:  ffffffffffffffff  ffffffffffffffff 
00007fab6d739bf0:  ffffffffffffffff  ffffffffffffffff 
runtime: unknown pc 0x7fab88d5982f
stack: frame={sp:0x7fab6d739b00, fp:0x0} stack=[0x7fab6cf3a288,0x7fab6d739e88)
00007fab6d739a00:  0000000000000000  0000000000000000 
00007fab6d739a10:  0000000000000000  0000000000000000 
00007fab6d739a20:  0000000000647178 <github.com/ncw/gmp.(*Int).doinit.func1+56>  000000380019dfc8 
00007fab6d739a30:  0000000000000000  0000000000000000 
00007fab6d739a40:  0000000000000000  00007fab6d739b00 
00007fab6d739a50:  0000000000451c43 <runtime.tracebackdefers+83>  0000000000000000 
00007fab6d739a60:  0000000000000000  0000000000000000 
00007fab6d739a70:  0000000000e869a0  0000000000000000 
00007fab6d739a80:  0000000000e869a0  0000000000bd6a58 
00007fab6d739a90:  0000000000bd6a58  0000000000000000 
00007fab6d739aa0:  0000000000000000  0000000000000000 
00007fab6d739ab0:  0000000000000000  0000000000000000 
00007fab6d739ac0:  0000000000000000  0000000000000000 
00007fab6d739ad0:  0000000000bd6a58  0000000000e869a0 
00007fab6d739ae0:  000000000045e7b1 <runtime.goexit+1>  000000000045e7b1 <runtime.goexit+1> 
00007fab6d739af0:  0000000000000000  000000c00005efd0 
00007fab6d739b00: <0000000000000000  000000c00005efd0 
00007fab6d739b10:  000000c00005efd8  0000000000000000 
00007fab6d739b20:  0000000000000000  00007fab6d739ce0 
00007fab6d739b30:  0000000000449cc6 <runtime.copystack+630>  00007fab50000020 
00007fab6d739b40:  0000000000001000  00007fab50000020 
00007fab6d739b50:  0000000000001000  00007fab88ee1aa0 
00007fab6d739b60:  00000000df868000  00000000df8475b0 
00007fab6d739b70:  0000000001de7270  000000000001ed90 
00007fab6d739b80:  fffffffe7fffffff  ffffffffffffffff 
00007fab6d739b90:  ffffffffffffffff  ffffffffffffffff 
00007fab6d739ba0:  ffffffffffffffff  ffffffffffffffff 
00007fab6d739bb0:  ffffffffffffffff  ffffffffffffffff 
00007fab6d739bc0:  ffffffffffffffff  ffffffffffffffff 
00007fab6d739bd0:  ffffffffffffffff  ffffffffffffffff 
00007fab6d739be0:  ffffffffffffffff  ffffffffffffffff 
00007fab6d739bf0:  ffffffffffffffff  ffffffffffffffff 

goroutine 27 [syscall]:
runtime.cgocall(0x869d80, 0xc00005ec98, 0xc0000a5bc0)
        /usr/lib/go/src/runtime/cgocall.go:128 +0x5b fp=0xc00005ec68 sp=0xc00005ec30 pc=0x4098db
github.com/ncw/gmp._Cfunc_mpz_pow_ui(0xc0000a5b80, 0xc0000a5bc0, 0x6fc23ac00)
        _cgo_gotypes.go:1095 +0x45 fp=0xc00005ec98 sp=0xc00005ec68 pc=0x642615
github.com/ncw/gmp.(*Int).Exp.func1(0xc0000a5b80, 0xc0000a5bc0, 0xc0000a5a00)
        /home/dan/go/pkg/mod/github.com/ncw/[email protected]/int.go:713 +0x175 fp=0xc00005ece8 sp=0xc00005ec98 pc=0x64acb5
github.com/ncw/gmp.(*Int).Exp(0xc0000a5b80, 0xc0000a5bc0, 0xc0000a5a00, 0x0, 0xc0000a5a80)
        /home/dan/go/pkg/mod/github.com/ncw/[email protected]/int.go:713 +0x9a fp=0xc00005ed18 sp=0xc00005ece8 pc=0x64575a
github.com/mit-dci/opencx/crypto/rsw.(*PuzzleRSW).SolveGMPCkXOR(0xc0000a5480, 0x0, 0x0, 0x0, 0x203000, 0x203000)
        /home/dan/Documents/Projects/opencx/crypto/rsw/rswtimelock.go:220 +0x493 fp=0xc00005edc8 sp=0xc00005ed18 pc=0x650ce3
github.com/mit-dci/opencx/crypto/rsw.(*PuzzleRSW).Solve(0xc0000a5480, 0x0, 0x0, 0xc0000ffec0, 0x411f79, 0xc0000a58e0)
        /home/dan/Documents/Projects/opencx/crypto/rsw/rswtimelock.go:210 +0x2b fp=0xc00005ee08 sp=0xc00005edc8 pc=0x65080b
github.com/mit-dci/opencx/crypto/timelockencoders.SolvePuzzleRC5(0xc0001d2000, 0xb1, 0xb1, 0xb462e0, 0xc0000a5480, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/dan/Documents/Projects/opencx/crypto/timelockencoders/blockciphers.go:112 +0x52 fp=0xc00005ef00 sp=0xc00005ee08 pc=0x6c4a22
github.com/mit-dci/opencx/match.SolveRC5AuctionOrderAsync(0xc0001d0000, 0xc000078720)
        /home/dan/Documents/Projects/opencx/match/auctionorder.go:30 +0x94 fp=0xc00005efd0 sp=0xc00005ef00 pc=0x6c5ae4
runtime.goexit()
        /usr/lib/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00005efd8 sp=0xc00005efd0 pc=0x45e7b1
created by github.com/mit-dci/opencx/cxauctionserver.(*OpencxAuctionServer).PlacePuzzledOrder
        /home/dan/Documents/Projects/opencx/cxauctionserver/orders.go:27 +0x21f

goroutine 1 [chan receive]:
main.main()
        /home/dan/Documents/Projects/opencx/cmd/fred/fred.go:188 +0x5a9

goroutine 19 [syscall]:
os/signal.signal_recv(0x0)
        /usr/lib/go/src/runtime/sigqueue.go:139 +0x9c
os/signal.loop()
        /usr/lib/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
        /usr/lib/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 70 [syscall]:
github.com/ncw/gmp._Cfunc_mpz_pow_ui(0xc0004a7c40, 0xc0004a7c60, 0x6fc23ac00)
        _cgo_gotypes.go:1095 +0x45
github.com/ncw/gmp.(*Int).Exp.func1(0xc0004a7c40, 0xc0004a7c60, 0xc0004a7b80)
        /home/dan/go/pkg/mod/github.com/ncw/[email protected]/int.go:713 +0x175
github.com/ncw/gmp.(*Int).Exp(0xc0004a7c40, 0xc0004a7c60, 0xc0004a7b80, 0x0, 0xc0004a7bc0)
        /home/dan/go/pkg/mod/github.com/ncw/[email protected]/int.go:713 +0x9a
github.com/mit-dci/opencx/crypto/rsw.(*PuzzleRSW).SolveGMPCkXOR(0xc0004a7a40, 0x0, 0x0, 0x0, 0x203000, 0x203000)
        /home/dan/Documents/Projects/opencx/crypto/rsw/rswtimelock.go:220 +0x493
github.com/mit-dci/opencx/crypto/rsw.(*PuzzleRSW).Solve(0xc0004a7a40, 0xc000464f6c, 0x2, 0xc000464ec0, 0x411f79, 0xc0004a7ae0)
        /home/dan/Documents/Projects/opencx/crypto/rsw/rswtimelock.go:210 +0x2b
github.com/mit-dci/opencx/crypto/timelockencoders.SolvePuzzleRC5(0xc0004b00c0, 0xb1, 0xb1, 0xb462e0, 0xc0004a7a40, 0xc000464f68, 0x2, 0x0, 0x1, 0x0)
        /home/dan/Documents/Projects/opencx/crypto/timelockencoders/blockciphers.go:112 +0x52
github.com/mit-dci/opencx/match.SolveRC5AuctionOrderAsync(0xc0000b6370, 0xc000078720)
        /home/dan/Documents/Projects/opencx/match/auctionorder.go:30 +0x94
created by github.com/mit-dci/opencx/cxauctionserver.(*OpencxAuctionServer).PlacePuzzledOrder
        /home/dan/Documents/Projects/opencx/cxauctionserver/orders.go:27 +0x21f

goroutine 10 [select]:
database/sql.(*DB).connectionOpener(0xc0004bc240, 0xb4ade0, 0xc00048c500)
        /usr/lib/go/src/database/sql/sql.go:1000 +0xe8
created by database/sql.OpenDB
        /usr/lib/go/src/database/sql/sql.go:670 +0x15e

goroutine 11 [select]:
database/sql.(*DB).connectionResetter(0xc0004bc240, 0xb4ade0, 0xc00048c500)
        /usr/lib/go/src/database/sql/sql.go:1013 +0xfb
created by database/sql.OpenDB
        /usr/lib/go/src/database/sql/sql.go:671 +0x194

goroutine 12 [select]:
github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc000078600, 0xc0004bc300, 0xc0004d4300)
        /home/dan/go/pkg/mod/github.com/go-sql-driver/[email protected]/connection_go18.go:178 +0xbf
created by github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher
        /home/dan/go/pkg/mod/github.com/go-sql-driver/[email protected]/connection_go18.go:175 +0xbe

goroutine 14 [chan receive]:
github.com/mit-dci/opencx/cxauctionserver.(*OpencxAuctionServer).AuctionOrderHandler(0xc0004b22d0, 0xc000078720)
        /home/dan/Documents/Projects/opencx/cxauctionserver/handlers.go:11 +0x42
created by github.com/mit-dci/opencx/cxauctionserver.InitServer
        /home/dan/Documents/Projects/opencx/cxauctionserver/auctionserver.go:41 +0x2a4

goroutine 15 [chan receive]:
github.com/mit-dci/opencx/cxauctionserver.(*OpencxAuctionServer).AuctionClock(0xc0004b22d0)
        /home/dan/Documents/Projects/opencx/cxauctionserver/clock.go:33 +0xb5
created by github.com/mit-dci/opencx/cxauctionserver.InitServer
        /home/dan/Documents/Projects/opencx/cxauctionserver/auctionserver.go:44 +0x2c6

goroutine 16 [chan receive]:
main.main.func1(0xc0003e01a0)
        /home/dan/Documents/Projects/opencx/cmd/fred/fred.go:163 +0x167
created by main.main
        /home/dan/Documents/Projects/opencx/cmd/fred/fred.go:156 +0x476

goroutine 50 [chan receive]:
github.com/mit-dci/opencx/cxauctionrpc.OffButtonCloseListener(0xc0003e01a0, 0xb49ae0, 0xc000493440)
        /home/dan/Documents/Projects/opencx/cxauctionrpc/listener.go:91 +0x3c
github.com/mit-dci/opencx/cxauctionrpc.NoiseListenAsync(0xc0000ee0e0, 0xc000493260, 0xc0003e01a0, 0x9746ab, 0x9, 0xc0004b3039)
        /home/dan/Documents/Projects/opencx/cxauctionrpc/listener.go:47 +0x2bb
created by main.main
        /home/dan/Documents/Projects/opencx/cmd/fred/fred.go:186 +0x592

goroutine 51 [IO wait]:
internal/poll.runtime_pollWait(0x7fab84292310, 0x72, 0x0)
        /usr/lib/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc00046a498, 0x72, 0x0, 0x0, 0x9728cf)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc00046a480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/lib/go/src/internal/poll/fd_unix.go:384 +0x1ba
net.(*netFD).accept(0xc00046a480, 0xc0004d4360, 0x7fab88cdfd98, 0x0)
        /usr/lib/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc0000100c8, 0xc000119f28, 0xc000119f30, 0x30)
        /usr/lib/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc0000100c8, 0xc000119f68, 0x2, 0x0, 0x1)
        /usr/lib/go/src/net/tcpsock.go:260 +0x48
github.com/mit-dci/opencx/cxnoise.(*Listener).listen(0xc000493440)
        /home/dan/Documents/Projects/opencx/cxnoise/listener.go:82 +0xc9
created by github.com/mit-dci/opencx/cxnoise.NewListener
        /home/dan/Documents/Projects/opencx/cxnoise/listener.go:64 +0x200

goroutine 52 [select]:
github.com/mit-dci/opencx/cxnoise.(*Listener).Accept(0xc000493440, 0xaba6c0, 0xc0004b2370, 0x7fab6efc2088, 0xc00048cb80)
        /home/dan/Documents/Projects/opencx/cxnoise/listener.go:207 +0xbb
net/rpc.(*Server).Accept(0xc0004b2370, 0xb49ae0, 0xc000493440)
        /usr/lib/go/src/net/rpc/server.go:632 +0x69
created by github.com/mit-dci/opencx/cxauctionrpc.NoiseListenAsync
        /home/dan/Documents/Projects/opencx/cxauctionrpc/listener.go:46 +0x294

goroutine 46 [syscall]:
github.com/ncw/gmp._Cfunc_mpz_powm(0xc0004a78a0, 0xc0004a77c0, 0xc0004a78c0, 0xc0004a7840)
        _cgo_gotypes.go:1110 +0x45
github.com/ncw/gmp.(*Int).Exp.func2(0xc0004a78a0, 0xc0004a77c0, 0xc0004a78c0, 0xc0004a7840)
        /home/dan/go/pkg/mod/github.com/ncw/[email protected]/int.go:716 +0x285
github.com/ncw/gmp.(*Int).Exp(0xc0004a78a0, 0xc0004a77c0, 0xc0004a78c0, 0xc0004a7840, 0xc0004a78c0)
        /home/dan/go/pkg/mod/github.com/ncw/[email protected]/int.go:716 +0xe3
github.com/mit-dci/opencx/crypto/rsw.(*PuzzleRSW).SolveGMPCkXOR(0xc0004a7380, 0x7fab88cdfd98, 0x0, 0x8, 0x203000, 0x203000)
        /home/dan/Documents/Projects/opencx/crypto/rsw/rswtimelock.go:220 +0x4c2
github.com/mit-dci/opencx/crypto/rsw.(*PuzzleRSW).Solve(0xc0004a7380, 0x10, 0x917e60, 0xc00004d6c0, 0x411f79, 0xc0004a7760)
        /home/dan/Documents/Projects/opencx/crypto/rsw/rswtimelock.go:210 +0x2b
github.com/mit-dci/opencx/crypto/timelockencoders.SolvePuzzleRC5(0xc00015c000, 0xb1, 0xb1, 0xb462e0, 0xc0004a7380, 0x412808, 0x30, 0x93cd60, 0xc0000bc301, 0xc00009cd20)
        /home/dan/Documents/Projects/opencx/crypto/timelockencoders/blockciphers.go:112 +0x52
github.com/mit-dci/opencx/match.SolveRC5AuctionOrderAsync(0xc0000b62d0, 0xc000078720)
        /home/dan/Documents/Projects/opencx/match/auctionorder.go:30 +0x94
created by github.com/mit-dci/opencx/cxauctionserver.(*OpencxAuctionServer).PlacePuzzledOrder
        /home/dan/Documents/Projects/opencx/cxauctionserver/orders.go:27 +0x21f

goroutine 48 [syscall]:
github.com/ncw/gmp._Cfunc_mpz_powm(0xc0004a65c0, 0xc0004a64e0, 0xc0004a65e0, 0xc0004a6560)
        _cgo_gotypes.go:1110 +0x45
github.com/ncw/gmp.(*Int).Exp.func2(0xc0004a65c0, 0xc0004a64e0, 0xc0004a65e0, 0xc0004a6560)
        /home/dan/go/pkg/mod/github.com/ncw/[email protected]/int.go:716 +0x285
github.com/ncw/gmp.(*Int).Exp(0xc0004a65c0, 0xc0004a64e0, 0xc0004a65e0, 0xc0004a6560, 0xc0004a65e0)
        /home/dan/go/pkg/mod/github.com/ncw/[email protected]/int.go:716 +0xe3
github.com/mit-dci/opencx/crypto/rsw.(*PuzzleRSW).SolveGMPCkXOR(0xc00000ece0, 0x0, 0x0, 0x0, 0x203000, 0x203000)
        /home/dan/Documents/Projects/opencx/crypto/rsw/rswtimelock.go:220 +0x4c2
github.com/mit-dci/opencx/crypto/rsw.(*PuzzleRSW).Solve(0xc00000ece0, 0x0, 0x0, 0xc0001036c0, 0x411f79, 0xc0004a6480)
        /home/dan/Documents/Projects/opencx/crypto/rsw/rswtimelock.go:210 +0x2b
github.com/mit-dci/opencx/crypto/timelockencoders.SolvePuzzleRC5(0xc00015c0c0, 0xb1, 0xb1, 0xb462e0, 0xc00000ece0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/dan/Documents/Projects/opencx/crypto/timelockencoders/blockciphers.go:112 +0x52
github.com/mit-dci/opencx/match.SolveRC5AuctionOrderAsync(0xc0004b2140, 0xc000078720)
        /home/dan/Documents/Projects/opencx/match/auctionorder.go:30 +0x94
created by github.com/mit-dci/opencx/cxauctionserver.(*OpencxAuctionServer).PlacePuzzledOrder
        /home/dan/Documents/Projects/opencx/cxauctionserver/orders.go:27 +0x21f

rax    0x0
rbx    0x6
rcx    0x7fab88d5982f
rdx    0x0
rdi    0x2
rsi    0x7fab6d739b00
rbp    0x8
rsp    0x7fab6d739b00
r8     0x0
r9     0x7fab6d739b00
r10    0x8
r11    0x246
r12    0x0
r13    0x1
r14    0x7fab50000c30
r15    0x200
rip    0x7fab88d5982f
rflags 0x246
cs     0x33
fs     0x0
gs     0x0

Desktop (please complete the following information):

• OS: Arch Linux
• Version 5.0.13-arch1-1-ARCH

Additional context
This might be a bug in ncw/gmp.

Make Assets conform to SLIP-0044

Is your feature request related to a problem? Please describe.
Currently the Asset struct assigns arbitrary, non-standard bytes to be used for asset identification by the server. It would be useful to include a standard that can be used for both BIP-0044 (as the HDCoinType) and the exchange (as an asset identifier). SLIP-0044 solves this problem, so it would make sense to implement this as the definition of the Asset struct.

Describe the solution you'd like
Implement SLIP-0044 as the definition of the Asset struct and make sure all of the cointypes that come from lit repo use it, or are compatible with the rest of opencx.

Describe alternatives you've considered
There do not seem to be many alternatives other than using the coin types in lit coinparams.

Additional context
N/A

gmp should only be used for solving puzzles

Describe the bug
ocx does not build if gmp (libgmp-dev on Ubuntu) is not installed.

To Reproduce
Steps to reproduce the behavior:

1. Uninstall gmp if you have it installed
2. go build ./cmd/ocx/...
3. See error

Expected behavior
cmd/ocx should not need gmp installed in order to build, only cmd/fred. Creating timelock puzzles is quick since there is a trapdoor, so gmp is not needed.

Desktop (please complete the following information):

• OS: Ubuntu

Additional context
This should be solved by separating the creating and solving dependencies for auction orders.

Figure out better key management

Right now the exchange has two required parts, and one optional, external part. The two required parts are:

The required parts

The OpenCX Client, currently implemented in benchclient and utilized as a shell client in ocx, sends RPC commands to the Opencx Server. The client has its own keys, and signs the commands it sends for authentication. All registration is done with this key, and this is how the server keeps track of deposit addresses (which are used to link users with deposit transactions). ocx stores its private key in ~/.ocx/

The OpenCX Server, which is currently not very abstracted. The server has its own key which it uses as a master key for pretty much everything. All of the key management here is done by wallits, so there's basically none.

The optional part

A lightning node - Currently I use lit for everything. When the lightning node and OpenCX Server successfully complete channel creation, the OpenCX Server registers the public key used for the channel. The indicator for this on the server side is a sigproof, since the lightning node creates a channel with the server, not the other way around. When the funding transaction is confirmed, the public key is credited with the initial push amount. This works so far, but has some issues, most caused by key management being more internal than external for both of these.

How does the server authenticate?

Currently the server authenticates by having the client sign data for each command, and send the signature in the RPC request. The pubkey is extracted from the signature, verified, and the command is executed if the signature is valid. This pubkey is used similarly to a "user" field for things like placing an order.

The issue

The first issue is that while the lightning node has its own public key that is registered on the exchange with the establishment of a channel, the OpenCX Client isn't aware of the key management going on within the lightning node. Since commands need signatures, and the pubkeys associated with the signatures are what the exchange uses to associate commands with accounts, if a client wants to do anything with the balance obtained by a lightning deposit, then it needs to be able to sign RPC requests that authenticate for the same public key that the deposit was made with.

Because there's no notion of key management at all to begin with, this is a roadblock for authentication using only public key stuff. So lightning withdrawal, and withdrawal in general, is not easy to implement right now for balances originating from lightning channels. However, if lightning withdrawal were to be implemented any other way you would probably have to specify a public key, and specify a node for withdrawal, or keep them together. Because there are public keys created at essentially every fund, it gets somewhat difficult to keep track of things compared to the whole "one deposit address per user" model.

The solution(s)

Currently I can only think of ways to solve this problem that involve modifying lit or another lightning node alongside the OpenCX Client. The ocx client itself could be a daemon, with a modified lightning node running, and all key management done within that. There might also be a more robust key management solution.

DB configuration

Right now the database client has a bunch of stuff hard coded in as globals. I'd like to turn those globals into defaults but provide configuration options as input for when the db client is created / started.

Add a precision to Assets based on coin

Is your feature request related to a problem? Please describe.
In many places, we use uint64's to represent values of various coins, which for Bitcoin is generally fine, but for other assets which have higher capacities, this may not be the case.
Also, doing floating point arithmetic with coin values should conform to the specific coin's precision.

Describe the solution you'd like
Figure out how to easily create new Assets (maybe do #15 before this one) with a specified precision for arithmetic, then create said arithmetic methods.

Describe alternatives you've considered
N/A

Additional context
N/A

Hi,I found a SQL error in 'opencx/cxdb/cxdbsql/depositstore.go' line 38.

When I run 'go build ./cmd/opencxd/...' and executed './opencxd',it threw an error
.
I think its because this code is incorrect,the two 'pubkey' isn't match.
'depositAddrStoreSchema = "pubkey VARBINARY(66), address VARCHAR(34), CONSTRAINT unique_pubkeys UNIQUE (pubkeys, address)"'

"Pinky swear" settlement

Is your feature request related to a problem? Please describe.
Traditional securities exchanges (like NASDAQ) have what I call "pinky swear" settlement, where the broker placing the order promises they can provide the funds for the trade that is being placed. This allows exchanges not to take custody of assets, and instead just make sure that they maintain some sort of reputation per user. Allowing users that regularly cause settlement failures means honest traders won't want to use the exchange.
Settlement failures are recorded by the SEC here.
We should build out this type of settlement to compare the architectures of traditional securities exchanges and cryptocurrency exchanges, just as we compare custodial and non-custodial settlement architectures (database / internal accounting vs channels, atomic swaps).

Describe the solution you'd like
"Pinky swear" settlement is, at most, just making sure users placing orders are on a whitelist.
The SettlementEngine's CheckValid method will extremely easy, just verifying that the *match.SettlementExecution Pubkey is on a whitelist.
A proper API for the engine (for use by the server / exchange only) will have to be made if you want to implement changes to the whitelist.
A minimal pinky swear settlement engine written in literally five minutes:

import (
    "sync"
)
type PinkySwearEngine struct {
    whitelist     map[[33]byte]bool
    whitelistMtx *sync.Mutex
    coin         *coinparam.Params
}
// CreateSettlementEngine creates a settlement engine for a specific coin
func CreateSettlementEngine(coin *coinparam.Params, whitelist [][33]byte) (engine match.SettlementEngine, err error) {
    pe := &PinkySwearEngine{
        coin: coin,
        whitelist: make(map[[33]byte]bool),
        whitelistMtx: new(sync.Mutex),
    }
    pe.whitelistMtx.Lock()
    for _, pubkey := range whitelist {
        pe.whitelist[pubkey] = true
    }
    pe.whitelistMtx.Unlock()
    return
}
// ApplySettlementExecution applies the settlementExecution, this assumes that the settlement execution is
// valid
func (pe *PinkySwearEngine) ApplySettlementExecution(setExec *match.SettlementExecution) (setRes *match.SettlementResult, err error) {
    // this highlights potentially how not generic the SettlementResult struct is, what to put in the NewBal field???
    setRes = &match.SettlementResult{
        // this is where we have sort of undefined on what we do
        NewBal: uint64(0),
        SuccessfulExec: setExec,
    }
    return
}
// CheckValid returns true if the settlement execution would be valid
func (pe *PinkySwearEngine) CheckValid(setExec *match.SettlementExecution) (valid bool, err error) {
    pe.whitelistMtx.Lock()
    if valid, ok = pe.whitelist[setExec.Pubkey]; !ok {
        // just being really explicit here, all this does is check if you're in the whitelist and your value
        // is true. There's no reason for it to be false.
        valid = false
        pe.whitelistMtx.Unlock()
        return
    }
    pe.whitelistMtx.Unlock()
    return
}

Describe alternatives you've considered
N/A

Additional context
N/A

Refactor lit & chain interaction

Right now the lit node is fairly baked in to the lightning functionality, whereas I'd like to define an interface on what operations we really need the lit node to provide. This would require creating an interface and figuring out what's a good way to treat the lit node.

The same goes for the chainhook, that's even more baked in and an even simpler I/O device.

Add networking middleware

Currently if we wanted to add new networking features, like HTTP(S), or custom networking, we would have to make a separate package and basically copy-paste a lot of code. We should make it easy on both the client and server side to use a different networking or command protocol. It should also be easier to potentially implement a custom protocol.

Need a better way to represent a trading pair

Is your feature request related to a problem? Please describe.
Currently, we can represent the same trading pair two different ways (BTC/LTC or LTC/BTC). In cxdbsql we just keep one table which has one of the assets first, and the other second. So there will only be regtest_litereg and not litereg_regtest. Instead, there should be one way of representing an asset pair. #15 should be done before this.

Describe the solution you'd like
#15 will give a nice ordering of assets, so maybe the lower asset ID can always go first. Trading pairs can have two non-equal Asset ID's, pairs will be unique if they are ordered.

Describe alternatives you've considered
The greater asset ID could go first, but it really doesn't matter.

Additional context
N/A