raiden-network/raidex

raidex's Issues

Implement the api in the webapp to place orders

see /docs/api.md

Cleanup

Some thoughts on cleanup. Didn't give it too much thinking though. Please discuss.

naming:

orders vs. offers:

• orders are received by the local api
• offers are sent to or received by the broadcast service

structure & naming:

• raidex_node/

  • api (the local user facing restful api used by UIs or bots)
  • api_client (lib to talk to the raidex_node/api, mainly used in tests)
  • service (registers markets i..e token_pairs and their commitment_service and message_broker endpoints)
  • order_manager (spawns and manages orders_tasks triggered by api requests)
  • order_task (spawns and manages exchange_tasks)
  • exchange_task (does the hard work: spawns and manages offers & commitments, executes the token swap
  • offer_book (reconstructs the book of all broadcasted offers)

• comittment_service/ (subfolder)

  • server
  • client

• message_broker/ (subfolder)

  • server
  • client

• raiden_mock/ (mocks the raiden API, we need this in order to test Raidex, consider to add this in Raiden)

test:

• move fixtures from conftest to respective or topic specific test modules
• where are the pubsub (message broker) tests? and or the client lib

transport/network:

• extend pubsub to be a message broker and allow for direct (non broadcasted) messages (i.e. filtered in the message_broker/.
• remove the network/ folder
• rename pubsub.py to message_broker (consider sub module)

other stuff:

• add docstrings to all classes
• mv: ui/ raidex_api_mock
• rm: outdated cmFlow.txt
• rm: #1
• fix: #5 to reflect the use of DirectMessages
• rm: smart_contracts
• rm: app.py and raidex.py
• write tests for each line of code which is used

Fix the feature to show different time intervals webapp

The bar width should stay the same.
Why are there different distances between the bars?

Implement usage of CommitmentServiceAdvertisements

A CS should spawn it's CommitmentServiceAdvertisements to the broadcast while it is online.
This serves for discovery (later on), but most importantly:

A node doesn't need to know the current commitment-fee-rate beforehand and can observe and set it automatically. (He should know it for balance checks of a Commitment-Refund)

Trade history

Implement way to track the trade history and latest price and so on.
Good place is properly the Offerbook

Improve making orders

Improve the limit order like other strategy how to split up the offers.
React on offers not happening.
Support checking status of order, cancel the order.

Change to `decimal` module for internal arithmetic to handle float/rounding errors

Pythons decimalmodule provides decimal fixed-point and floating-point arithmetic,

Decimal “is based on a floating-point model which was designed with people in mind, and necessarily has a paramount guiding principle – computers must provide an arithmetic that works in the same way as the arithmetic that people learn at school.” From the python doc

where the user has good controls over rounding and precision-behaviour.
In addition to that, the module offers "Signalling" for computations, indicating e.g. that a computation produced an Inexact or rounded result.

With this module it's also very easy to set quantizing behaviour (e.g. for grouping atomic offers into bigger chunks for UI-representation) on a per user basis to fit the users needs.
Look here for a first implementation of this using the decimal module.

The goal is to have all relevant numbers as decimal.Decimal objects inside the application and
only encode them to strings for the user facing API.

Preprocess the Offerbook and the Trade to group offers and trades

The Offers and Trades needs to be grouped.
For example by roughly the same time, same price, so that it is not so much noise.

ConnectionError with many offers

When submitting a lot of offers, I get the following error after some time (I assume the time it tries to resubmit for the first time):

ConnectionError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /api/topics/c08b5542d177ac6686946920409741463a15dddb (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1839c64c50>: Failed to establish a new connection: [Errno 11] Resource temporarily unavailable',))
<LimitOrderTask at 0x7f185a5ef050> failed with ConnectionError

To reproduce, add the following lines to __main__.py:

    from raidex.raidex_node.offer_book import OfferType
    import random
    for _ in range(1000):
        node.limit_order(OfferType.BUY, int(1e18), random.randrange(1, 10000))

I assume that there are too many requests that take too long to process.

Known Problems

• DDOS (of the centralized services like Commitment Service and Message Broker).
• Proof of Burn (we need another trusted party that confirms, when a commitment service burned tokens)
• Stating the exchange happened is the rational choice for both parties, even if one was betrayed

Implement commitment clientside

Will only be a dummy implementation for the first iteration

Offers seem to disappear

When adding more than a few offers, they seem to randomly disappear and come back again.

The easiest way to reproduce is to add the following lines in __main__.py, start everything and refresh the UI at the right points in time:

from raidex.raidex_node.offer_book import OfferType
import random
for _ in range(20):
    node.limit_order(OfferType.BUY, int(1e18), random.randrange(1, 10000))

I assume the reason is that the time between offer timeout and resubmission is too long. If during this dead time the UI requests the set of offers it gets an incomplete list.

Setup a Message Broker (pubsub service) for publishing offers

In iteration 1 this will not support any sort of user authentication.

Color the LimitOrder-table's rows according to type

To save some space the explicit column of the Order-type should be excluded.
Instead, all text in the other columns should be coloured green/red according to their type.
This was started using PrimeNG's rowStyleClass, but this is not yet working as expected.

Commitment Service (own channel)

The commitment service CS helps two parties to commit to a token swap agreement prior to its execution. It helps to bring down the "free option" from minutes to a few seconds. It's implemented as a service in the public network. In order to interact with the commitment manager one needs to open a "channel" which allows to do off-chain commitments (value transfers).

It works as follows:

1. The Maker creates an Offer: O
2. The Maker hashes its offer H(O)
3. The Maker sends a signed "Maker Commitment" message MC to CS, saying it is committed up to certain timestamp T to the offer O. Part of it is a certain amount of ether as a deposit D.
4. CS confirms a successful commitment by signing a message MCS = Signed([CM, committed_amount], CS). And a message signing the new channel credit (i.e. the updated number of tokens left to do commitments after deducting the now locked tokens for the recent commitment).
5. Maker broadcast the signed offer O and the commitment MCS
6. A Taker sends a commitment TC to CS (see #16)
7. CS confirms as in 4) but signing TC (instead of MC), i.e. TCS = Signed(TC, CS)
   7a) if the offer timed out or was already taken, CS responds by a channel credit message refunding the commitment
8. Taker contacts Maker proofing commitment by showing TCS. Maker and Taker eventually swap the tokens.
   9a) Taker and Maker notify of successful execution of the swap
   9b) Only one or none of the parties reports execution of the swap
9. CS sends a channel credit message refunding the commitment (minus some fee) to each of the parties
10. CS sends the a signed message indicating that the deal completed to the broadcast channel (this is used by the clients to learn which of the orders was executed, so they can have last prices).

Notes:

1. O = [bid_token, bid_amount, ask_token, ask_amount, offer_id]
2. Commitment Message: CM = Signed([H(O), timeout, remaining_deposit, channel_nonce], maker) . the remaining_deposit implicitly informs the CS about the committed value.
3. Channel Credit Message:CC = Signed([amount, channel_nonce], CS)
4. Commitment Message: CM = Signed([H(MCS), timeout, deposit_amount, channel_nonce], maker) where timeout and deposit_amount must equal those in MC.

add slack integration

Trading Client: API

The API is a means to interact with the exchange in a high-level fashion:
It should work similarly than a normal exchange where you just place orders (Limit Order, Market Order) and a time the order is valid. The client handles the requests to the actual trusted parties and tries to construct multiple asset swaps in a way that the order is executed in a satisfactory fashion.

Add something to the ui to see currently placed orders

Commitment strategies

Maker strategy:

Maker sends a Commitment message (timeout) and follows it up with a raiden transfer to the CS.
If transfer arrives before the timeout, the CS sends a CommitmentProof

❌ Taker strategy 1:

Taker sends a Commitment message (offer_id, offer_timeout) to the CS. Now the CS waits for a raiden transfer of the first taker. Once it arrives, the CS sends a CommitmentProof.

Problem:

When a taker posts an Commitment to the CS but refuses to send the raiden transfer, he stalls the linked offer for other takers without any costs, what creates a free option.

This renders the whole exchange unusable, as it introduces a free option in the same order as an exchange without a CommitmentService

❓ Taker strategy 2:

Every taker has to first transfer the commitment deposit to the CS and then sends the commitment message.
The CS accepts the first taker that manages to follow up the commitment deposit with a commitment message as the engaged taker.

Problem:

This introduces a lot of transaction traffic on the CS and in the raiden network.
A lot of takers are competing for the cheapest offer.

❓ Taker strategy 3:

Taker sends only a commitment deposit (offer-id) to the CS. The CS approves the first commitment deposit with a commitment proof. Other commitment deposits are put on a backlog queue and will be refunded as soon as feasible for the CS.

Problem:

Potential takers are unsure about their state of engagement/commitment until they are refunded or receive the CommitmentProof.
This introduces a lot of transaction traffic on the CS and in the raiden network.
A lot of takers are competing for the cheapest offer. (But at least no additional Commitment message per taker-try).

• The CS should broadcast a message to indicate the offer isn't available anymore.

This should allow failed takers to move on with their trading strategy and exclude the Offer from their Orderbook and will stop further traffic on the CS for this Offer.

Also the offer_id has potential collisions, since it is no enforced logic and chosen by the maker.

Remove broken code

We should go through the repo and check all code if it is doing something and working
If code works, there should be a test for it and it should pass
if code doesnt work, if it is useful as pseudo code it should be marked as such and not break anything
or it should be removed.

If no one is willing to do this, we should remove all code and readd the parts later, that we need.

Trading Client|MessageBroker: Message Formats

There are these kinds of messages:

• Offer
• CommitmentAnnouncement
• CommitmentProof
• ProvenOffer
• Commitment Service Advertisements
• Swap Execution Notification
• Swap Confirmation

The messages module will provide objects that allow for easy (de-)serialization.

_!!! Below is partially outdated. Please read the docstrings in https://github.com/ezdac/rex/blob/master/rex/messages.py_

Offers

Status: okay
An offer is published by a market maker and pushed to one ore more broadcast services.
Data

offer = rlp([ask_token, ask_amount, bid_token, bid_amount, offer_id, timeout])
timeout = <UTC milliseconds since epoch>
offer_sig = sign(sha3(offer), maker)
commitment = rlp([sha3(offer), amount])
commitment_sig = raiden signature of the commitment transfer by the committer
commitment_proof = sign(commitment_sig, cs)
cs = <commitment service ethereum address>

Broadcast

{
  "msg": "offer",
  "version": 1, 
  "data": "rlp([offer, offer_sig, commitment, commitment_sig, commitment_proof, cs])"
}

Commitment Service Advertisements

Status: revised
These can be send by the Commitment Service (CS) to broadcast services in order to announce its service to users.
Data:

address = <ethereum/raiden address>
commitment_asset = <asset_address of the accepted commitment currency>
fee_rate = <uint32 (fraction of uint32.max_int)>

Broadcast:

{
  "msg": "commitment_service",
  "data": rlp([address, commitment_asset, fee_rate]),
  "version": 1
}

Fee calculations

Users of the service have to expect to have a fee of

fee = int(uint32.max_int / fee_rate * commitment + 0.5)

deducted from each commitment.

An optional (unverifiable) statistics field informing about the past performance could be discussed.

Commitment Announcement/Commitment Proof

Status: revised
A Commitment is a commitment message sent to the CS, followed by a hashlocked (direct) raiden transfer from the maker to the commitment service. The secret for the hashlock is the hash of the offer (see above).

Data:

commitment = rlp(offer.offer_id, offer.timeout, amount)
hashlock = sha3(offer)  # or offer.hash

If the transfer completes, the CS is expected to counter the commitment with a commitment proof (if the commitment is accepted) or to refund the committing party:
Data:

commitment_proof = sign(commitment_sig, cs)

Swap Execution Notification

After completing a swap, the traders can request the commitment to be released from the CS.
Data:

swap_execution = rlp([order_id, timestamp])
swap_execution_sig = sign(sha3(swap_confirmation), maker/taker))

Message Content:

swap_execution_notification = rlp([swap_execution, swap_execution_sig])

This needs to be send by both parties. After the CS received both swap_execution_notifications it is expected to return the commitment (minus fees) to both trading parties.

Swap Completed

Status: okay
A CS is expected to broadcast confirmed swaps in order to build its reputation.
Data:

swap_confirmation = rlp([order_id, timestamp])
swap_confirmation_sig = sign(sha3(swap_confirmation), cs)

Broadcast:

{
  "msg": "swap_completed",
  "version": 1, 
  "data": "rlp([swap_confirmation, swap_confirmation_sig])"
}

Trading Client: User Interface

Trade History:
date|type|price|amount

Order Book:
Sell Orders | Bid Orders

Price Timeline Graph
Can be drawn based on trade history

Market Depth Graph
Can be drawn based on order book.

API provided:

• get_order_book()
• get_trade_histrory()

Order Placement
Deferred for now

Trading Bot

Implement a trading bot that will make orders and trade.
The bot mode should be a commandline option.
Probably some things from mocking can be reused

Move all scripts and css to angular-cli-json in the webapp

Elaborate if we can move all the scripts to angular-cli.json

Protocol: direct Client-Client/Client-CS communication

• add an ACK message that indicates a successful processing of incoming messages
  (e.g. the CS will respond to a takers Commitment-try with an ACK before it will determine if the taker is engaged or not)

Trading Client: OfferView

The trading client should "subscribe" to the offer publishing service and create a "usable" order book, where "usable" means:

• it can be used by the frontend #2
• it provides sufficient API methods to be used to engage in trades #3
• it updates itself on new "events", such as:
  • new offer received
  • offer timed out

An order book is specific to one market.

API:

• add_offer(offer_message)
  offer message includes the amount of both tokens (bid/ask), type (buy/sell), offer_id as well as a proof of commitment (which must be evaluated) which contains a timeout.
  The offer is added to the order book.
• remove_offer(offer_id)
  Called when the commitment service publishes that an exchange happened. offer with offer_id should be removed from the order book. Offers are also removed from the orderbook once they timeout.
• get_orderbook(limit)
• get_tradehistory(limit)
• get_lastprice()

Implement message broker

Will only be in memory, no actually network comuncation for first iteration, later flask or bottle

Use the rest api from the webapp

take a look at docs/api.md

• making a limit order
• showing trades
• showing offers in the offerbook
  later:
  show status of orders, cancel orders

Implement exchange tasks

Create TakerProof

or similar so that there is a difference between committing to a trade as a maker or taker.

commitment service in the raidex node

Implement the communication with the commitment service in the raidex node

Start implementing the REST Api

Adjust PriceChart and MarketDepthChart to enclosing dev's width

The zingchart's are not properly scaled to 100% of it's enclosing div-Element.
This should theoretically work with zing's width=100% attribute.
Also compare here.

Trading Client: Order Task

When an order is added by the user, the system spawns an OrderTask and returns an order_id which allows to interact with that task.

The task will try to fill the order by matching offers in the Offer View.

Limit Orders:

• The task will try to fill using all offers that are within the price limit
• If the task could not or only partially fill, it will create and broadcast an offer, which eventually will show up in all order views

Market Orders:

• The task will try to fill the order by picking the cheapest offers until the order is filled

Creating Orders:

• the task will constantly spawn AssetSwapRequests (ASR) with a given commitment-timeout, so that as long as the task is active and unfulfilled, there will always be an ASR open in the network corresponding to the Order. A task can be aborted, stopping it from spawning further subsequent ASRs. (Keep in mind: The latest ASR will stay active and can't be aborted due to it's commitment based nature)

Implement zoom or partial view in plots

In the price/stock/depthchart not all of the data should be plotted.
The question is if we should implement that in the webapp or in the node.

Offers of canceled orders are not removed from the UI

To reproduce: create a limit order, sleep for 5 seconds, cancel it.

Not sure if this is an issue of the UI or of the node.

Improve Exchange Task

The ExchangeTask need to take actions after a trade was done or failed.

Implement mock of Rest api for testing the webapp

Commitment Service (Raiden deposits)

The commitment service CS acts as a trusted third party that helps two parties to commit to a raiden based token swap prior to its execution. This helps to bring down the "free option" from minutes to a few seconds. In order for a client (maker or taker), to interact with the CS, it needs to open a direct raiden channel with the CS. Only the client needs to deposit into this raiden channel. The client now has an amount of credit that it can use as commitments to the CS. A direct raiden transfer from maker/taker to CS represents a commit. As an initial suggestion the commitment amount should be 1/100 of the value the maker wants to trade. When the commitment service has received commitments from both the taker and the maker, it is up to the two parties to carry out the token exchange and report back to the CS. The CS will only redeem the commitment deposits if both the maker and the taker reports back that the transfer was successful within a specified timeout period.

Note: This implementation uses a simplified approach, where the third party is trusted with all commitments. In the worst case it could run away with all of them while the in #1 proposed solution the commitment deposits would only be at risk for a limited amount of time. We've chosen above as a preliminary approach to quickly get to a PoC. Nonetheless, this setup makes it more rewarding to be a dishonest 3rd party than #1.

It works as follows:

1. The Maker creates an Offer: O
2. The Maker hashes its offer H(O)
3. The Maker sends a signed "Maker Commitment" message MC to CS, saying it is committed up to certain timestamp T to the offer O.
4. The Maker commits 1/100 of ask_amount to the CS in a raiden transfer.
5. CS confirms a successful commitment by signing a message MCS = Signed([MC, committed_amount], CS).
6. Maker broadcast the signed offer O and the commitment MCS
7. A Taker directly sends a raiden-transfer(identifier=H(O)) to the CS with the committed amount from the MCS
8. CS confirms the commitment to the sender of the first Taker-transfer by creating and signing TC (instead of MC), TC = Signed(C, CS), C = [ O, T, committer=taker_address ]
   8a) if the offer timed out or was already taken, CS responds by sending back the commitment in a raiden transfer(amount=commitment_amount - fee, identifier=H(O)) (from a backlog-queue, so maybe not immediately)
9. Taker signs the TC as a TCS = Signed(TC, Taker)
10. Taker contacts Maker proofing commitment by showing TCS. Maker and Taker eventually swap the tokens.
    11a) Taker and Maker notify of successful execution of the swap
    11b) Only one or none of the parties reports execution of the swap
11. CS refunds the commitments through raiden transfers (minus some fee) to each of the parties
12. CS sends the a signed message indicating that the deal completed to the broadcast channel (this is used by the clients to learn which of the orders was executed, so they can have last prices).

Notes:

• 1: O = [bid_token, bid_amount, ask_token, ask_amount, offer_id]
• 3: Commitment Message: CM = Signed([H(O), timeout, commitment_id], maker) where commitment_id is the hashlock
• 7: Commitment Message: CM = Signed([H(MCS), timeout, commitment_id], taker) where timeout and deposit_amount must equal those in MC.
• 8: The C is created by the commitment service, so the signing is reversed compared to the MCS: TC is signed by the CS as an commitment proof, then TC is signed by the taker as TCSand broadcasted.
• 9: the additional signing by the Taker is necessary, so that a CS cannot forge a TCS for a given Taker (this may not be an attack vector at all, who can gain from that?)

** Below methods still need revision **
Methods:
Methods are implicitly called by sending messages to the service.

redeem_commitment(CM)
Send back the commitment corresponding to the CM minus small fee through raiden transfer

commit(id, timeout, deposit)
commits user to execute an

Filter incoming offers

Incoming offers should be filtered, so that only offers get in the orderbook that are

• signed by a trusted commitmentservice
• committed with enough commitment
• what else?

Implement mockup/dummy for (subset of) the raiden api

Trading Client: Architecture

• Restful API
• Signer
• Commitment Manager
• Raiden Client
• Exchange
  • Network / Offer chat room
  • Orderbook
  • OrderTask
  • Order Manager

Restful API
see #3

Signer
Has the privkey of the user and can sign messages. Q: should we support multiple accounts?

Commitment Manager
<Outdated>
commit(commitment_service_address, offer_id, commitment_deposit, timeout)
announces an incoming commitment for offer_id to the CS
does a Raiden transfer, transferring the commitment (this is low latency for direct channels)

notify_success(commitment_service_address, offer_id)
sends a message stating that the exchange executed. the CS will return the tokens if both parties say the exchange happened.

Note: we could require that both parties send the message they used to initiate the transfer, those proofing that they received the transfer, next to the secret for the hashlock. while this could be gamed (in the sense that they'd claim the deposit although they did not exchange) it would be hard to setup and not feasible unless upfront coordinated.
</Outdated>
See #10 "Commitment Announcement/Commitment Proof"

Raiden Client
see raiden api. unfortunately we need to poll for events.

Exchange
Implements the exchange logic for a token pairs, e.g. BeerCoin/ETH i.e. a market it interacts with..
Requirements: a user needs to have Raiden channels for both Tokens in the token-pair.
Needs to be listening to one ore more pub/sub services for this market.
Exchange collects the instances specific to a certain market.

instance of an exchange which features

• Message Broker #7
• Orderbook #8
• Order Manager
• Order Tasks #4

Transport: direct Client-Client/Client-CS communication

The transport will:

• use UDP/TCP
• use raiden's discovery for address -> endpoint mapping
• use this raiden endpoint where port = raiden-port+1

Bug: Offers sometimes dont get matched

If there are a bigger number of offers in the offerbook, the offers sometimes dont get matched and the offerbook remains crossed.
It might be a race condition or a n error in the limited order code. Need to look into that.

Review the architecture of the raidex code

After the commitment service is merged we should review the structure of the code and maybe reorganize it.

• Check if we can refactor some code. For example to use more of FSM and Actors
• Reorganize the code, for example commitmentservice client belongs more to raidex node if it shares code with raidex node.

Create ETH-backed ERC20 token

Find either a working version, or implement a ERC20 Token that allows locking ETH in exchange for tokens.

Include message id in rlp and sign it too

Be able to switch markets(token pairs).

We should have more than one token pair that you can trade.

Implement the Commitment service

Should react on messages over the message broker system. Doesnt need transfers for the first iteration

Add abstraction for the message_broker

Add an abstraction for the message broker on the client side that knows the senders address and does the encoding so we dont have encode_hex all over the code anymore.