hastobegood/crypto-bot-commander

Crypto BOT using custom strategies

aws-sam-binance-trading

AWS SAM template used to deploy a Binance trading application

Getting started

Requirements

• NodeJS 14
• AWS SAM
• Yarn

Project structure

.
├── coverage                  # Coverage reports generated by tests
├── dist                      # Distribution files
│   ├── tsc                   # JavaScript files generated by TypeScript
│   └── webpack               # JavaScript files generated by Webpack
├── node_modules              # Project libraries
├── src                       # Source code
│   ├── code                  # Application code
│   └── handlers              # Application handlers (entrypoints)
└── test                      # Test code
    ├── builders              # Builders for tests
    ├── integration           # Integration tests
    └── unit                  # Unit tests

Install dependencies

$ yarn install

Build

$ yarn build

Run tests

$ yarn test

Lint

$ yarn lint
$ yarn lint:fix

Bundle

$ yarn bundle

Deploy

$ yarn deploy:{test|prod}

Usage

Trading BOT

TODO

Market evolution step

This step check the price evolution for a specific period. It is evaluated as a success as soon as the evolution percentage reach the provided threshold.

Input

{
  "source": "string",
  "interval": "string",
  "period": "number",
  "percentageThreshold": "number"
}

Source

This is the source from where the evolution percentage is calculated. It can be either Market or LastOrder.

For Market, the percentage is calculated based on the market evolution specified by the interval and period attributes.

For LastOrder, the percentage is calculated based on the last sent order.

Interval

This is the interval to retrieve candlesticks. It can be either 1m, 5m, 15m, 30m, 1h, 6h, 12h or 1d.

It is a mandatory attribute for Market source.

Period

This is the number of periods for the interval attribute.

It is a mandatory attribute for Market source.

Percentage threshold

This is the percentage threshold to reach in order to evaluate the step as a success.

Examples

Market down by 1% during the last 24 hours.

{
  "source": "Market",
  "interval": "1h",
  "period": 24,
  "percentageThreshold": -0.01
}

Market up by 5% during the last 60 minutes.

{
  "source": "Market",
  "interval": "1m",
  "period": 60,
  "percentageThreshold": 0.05
}

Market up by 10% since the last order.

{
  "source": "LastOrder",
  "percentageThreshold": 0.1
}

Output

{
  "success": "boolean",
  "lastPrice": "number",
  "currentPrice": "number",
  "percentage": "number"
}

Last price

This is the last price used for the percentage evolution.

Current price

This is the current price used for the percentage evolution.

Percentage

This is the calculated percentage evolution.

Examples

Percentage threshold reached.

{
  "success": true,
  "lastPrice": 500,
  "currentPrice": 494.3,
  "percentage": -0.0114
}

Percentage threshold not reached.

{
  "success": false,
  "lastPrice": 497.8,
  "currentPrice": 426.5,
  "percentage": -0.1432
}

Send order step

This step send a buy or sell order. It is evaluated as a success as soon as the sent order is successfully executed by the exchange.

Input

{
  "source": "string",
  "side": "string",
  "type": "string",
  "percentage": "number"
}

Source

This is the source to use to calculate the amount to buy or sell depending on the provided percentage. It can be either Account or LastOrder.

For Account, the amount is calculated based on the available quantity.

For LastOrder, the amount is calculated based on the last sent order quantity.

Side

This is the order side. It can be either Buy or Sell.

Note: currently only Buy side is supported using the LastOrder source.

Type

This is the order type. Only Market is supported at the moment.

Percentage

This is the percentage to apply on the quantity available from the account or the last order.

Examples

Market buy order with 10% of account available quantity.

{
  "source": "Account",
  "side": "Buy",
  "type": "Market",
  "percentage": 0.1
}

Market sell order with 30% of account available quantity.

{
  "source": "Account",
  "side": "Sell",
  "type": "Market",
  "percentage": 0.3
}

Market sell order with 100% of last order quantity.

{
  "source": "LastOrder",
  "side": "Sell",
  "type": "Market",
  "percentage": 1
}

Output

{
  "success": "boolean",
  "id": "string",
  "externalId": "string",
  "status": "string",
  "quantity": "number",
  "price": "number"
}

ID

This is the last price used for the percentage evolution.

External ID

This is the last price used for the percentage evolution.

Status

This is the last price used for the percentage evolution.

Quantity

This is the last price used for the percentage evolution.

Price

This is the last price used for the percentage evolution.

Examples

Successful order.

{
  "success": true,
  "id": "BNB#USDT/Buy/Market/1630920397742",
  "externalId": "549617",
  "status": "FILLED",
  "quantity": 1.3,
  "price": 503.59
}

Unsuccessful order.

{
  "success": false,
  "id": "BNB#USDT/Buy/Market/1630920397742",
  "externalId": "549617",
  "status": "EXPIRED"
}

crypto-bot-commander's Issues

Handle exchange API down while updating candlesticks

If for some reason the exchange API is unavailable while updating candlesticks, when it's available again we should retrieve all the missing datapoints.

Currently, datapoints are "lost".

Add backtesting

Add the backtesting feature.

Handle "flash crash"

Huge flash crash has been detected by the market evolution step, but the send order sell price was too high.

So I bought BNB at 412.6$. The strategy was waiting for the price to go down by at least -1%.
A huge "flash crash" happened and the price dropped to 312.5$ (see below).

 "input": {
  "interval": "1m",
  "period": 60,
  "source": "Market",
  "percentageThreshold": -0.01
 },
 "output": {
  "success": true,
  "currentPrice": 312.56,
  "lastPrice": 412.6,
  "percentage": -0.2425
 },
 "creationDate": "2021-09-08T16:35:38.100Z",
 "lastExecutionDate": "2021-09-08T17:00:36.371Z"

So the next step kicks in and send a sell market order to Binance 16ms after.
Unfortunately the order has been executed at 418$... Which is much greater than my buy price that was 412$... (see below).

  "input": {
   "side": "Buy",
   "source": "Account",
   "type": "Market",
   "percentage": 0.1
  },
  "output": {
   "externalId": "983405",
   "id": "BNB#USDT/Buy/Market/1631120438292",
   "quantity": 1.41,
   "success": true,
   "price": 418.2758865248228,
   "status": "Filled"
  },
  "creationDate": "2021-09-08T17:00:37.387Z",
  "lastExecutionDate": "2021-09-08T17:00:37.387Z"

Looking at the OHLC (Open High Low Close) candlesticks during this flash crash :

    [ 1631120400000, "416.90000000", "418.50000000", "312.50000000", "418.10000000", ... ], // 2021-09-08T17:00:00.000Z
    [ 1631120460000, "416.31000000", "418.00000000", "311.77000000", "417.40000000", ... ], // 2021-09-08T17:01:00.000Z
    [ 1631120520000, "415.70000000", "417.60000000", "311.57000000", "415.30000000", ... ], // 2021-09-08T17:02:00.000Z
    [ 1631120580000, "416.70000000", "418.40000000", "311.50000000", "417.40000000", ... ], // 2021-09-08T17:03:00.000Z

So it seems that during 4 minutes the price has moved from around 418$ to 311$ every minute and closing price was always around 418$.

That explains why my order has been executed at 418$.

A few questions :

• Is it a "normal" behavior that can happen in the real world ?
• If yes, how to handle it ?

A few ideas :

• Stop using market order but for what ?
• Any other idea ?

Symbol candlesticks

Retrieve symbol candlesticks from Binance endpoint GET api/v3/klines.

Add volume to candlesticks

Currently only OHLC are supported. Add volume.

Store orders in application

We need to store orders in the database when sending order to exchange.

Update the check order step as well to read database instead of calling exchange API.

Add a Lambda that will call exchange API for all orders with a status Waiting or PartiallyFilled.

Add a strategy budget

Add an initial budget to strategy model.

The send order step, when using Account source, should use the available quantity from the strategy budget instead of the Binance spot account.

Add 'Or' step

Add a Or step between two steps where the first that is a success is the winner.

Add AWS tags to stack

Add tags to the deployed CF stack :

• Environment
• Application

Store symbols candlesticks instead of calling Binance API at each strategy step execution

To do :

• Initialize candlesticks by year & month for a specific symbol
• Update candlesticks with schedule events
• Publish active strategies IDs based on updated candlesticks instead of schedule events
• Query candlesticks from DynamoDB instead Binance API during strategy evaluation

DCA trading is not working due to invalid symbol

Since #12 the DCA trading function fails :

   {
    "symbol": "BTCBUSD",
    "requestedQuantity": 25,
    "baseAsset": "BTC",
    "message": "Unable to extract assets, symbol 'BTCBUSD' is invalid",
    "success": false,
    "quoteAsset": "BUSD"
   }

This is due to the symbol format refactoring.

Handle Binance order statuses

Below is the exhaustive list that Binance can return when sending an order :

Status	Description
NEW	The order has been accepted by the engine.
PARTIALLY_FILLED	A part of the order has been filled.
FILLED	The order has been completed.
CANCELED	The order has been canceled by the user.
PENDING_CANCEL	Currently unused
REJECTED	The order was not accepted by the engine and not processed.
EXPIRED	The order was canceled according to the order type's rules (e.g. LIMIT FOK orders with no fill, LIMIT IOC or MARKET orders that partially fill)or by the exchange, (e.g. orders canceled during liquidation, orders canceled during maintenance)

Currently the status is not check when an order is sent even if it is not FILLED (aka order has been successfully processed).

See how to handle the other statuses.

Technical analysis - Moving average calculation

Add moving average calculation to the engine.

• SMA (Simple Moving Average)
• CMA (Cumulative Moving Average)
• EMA (Exponential Moving Average)

Executed quantity is wrong when fees are paid with the base asset

When fees are paid with the base asset (for example when trading BNB#XXX and paying fees with BNB), executed quantity returned by Binance API does not include commision quantity.

We need to substract manually the commission quantity from the executed quantity in this case.

Switch to AWS SDK v3

It could be interesting to switch to AWS SDK v3 in order to improve tree-shaking.

More info here : https://github.com/aws/aws-sdk-js-v3.

Generate model from Binance API swagger

Binance has published a swagger for its API. It could be interesting to use it to generate models instead of manually creating it.

See https://github.com/binance/binance-api-swagger.

EMA calculation is incorrect

The current EMA calculation output is different that the one from Binance.

It seems it's because not enough historical data is being used. See https://stackoverflow.com/questions/65727498/different-ema-exponential-moving-average-than-on-binance#comment116237282_65727498

Proposed solution : take 3 times more candles than the long term period requested (ex: for EMA 10d, get 30d and calculate EMA 10d based on 30d historical data)

Technical analysis - Market evolution calculation

Add market evolution calculation to the engine.

Add github actions workflow for CI

Add a github action :

• Build
• Run tests
• Run linter

It should be triggered on :

• pull requests
• main branch

Add the moving average technical analysis as a strategy step

Add the moving average crossover step.

Input attributes :

• Type (SMA, CMA or EMA)
• Crossover (CurrentPrice or ShortTermPrice)
• Signal (Buy or Sell)
• Short term period
• Long term period

Output attributes :

• Current price
• Short term price
• Long term price

Details on how to detect the crossover : https://www.investopedia.com/articles/active-trading/052014/how-use-moving-average-buy-stocks.asp

Add 'And' step

Add support for stop loss & take profit limit orders

Add 2 new order types : StopLossLimit & TakeProfitLimit.

A new attribute should be added to the SendOrder step to set the percentage related to the price threshold to use.

Update dependencies versions

Remove DCA feature

A specific repository has been created for DCA : https://github.com/hastobegood/crypto-dca.

Remove it from this repository.

Strategy engine setup

Setup a strategy engine that will process all active strategies every minute.

Strategy steps will be executed in sequential order depending on the step type and input data.

Each step will produce an output with a success indicator and additional data depending on the step type.

Each step must be saved in a table as historic data and resume point for the next execution.

Strategy example :

{
  "id": "999",
  "symbol": "BNB#USDT",
  "status": "Active",
  "template": {
    "1": {
      "id": "1",
      "type": "MarketEvolution",
      "nextId": "2",
      "input": {
        "interval": "1m",
        "period": 60,
        "source": "Market",
        "percentageThreshold": -0.01
      }
    },
    "2": {
      "id": "2",
      "type": "SendOrder",
      "nextId": "3",
      "input": {
        "side": "Buy",
        "source": "Account",
        "type": "Market",
        "percentage": 0.1
      }
    },
    "3": {
      "id": "3",
      "type": "MarketEvolution",
      "nextId": "4",
      "input": {
        "percentageThreshold": 0.01,
        "source": "LastOrder"
      }
    },
    "4": {
      "id": "4",
      "type": "SendOrder",
      "nextId": "1",
      "input": {
        "side": "Sell",
        "source": "LastOrder",
        "type": "Market",
        "percentage": 1
      }
    }
  }
}

Strategy step execution example : 

```json
{
  "id": "1",
  "type": "MarketEvolution",
  "strategyId": "999",
  "nextId": "2",
  "creationDate": "2021-09-07T02:21:38.401Z",
  "lastExecutionDate": "2021-09-07T03:29:36.680Z",
  "input": {
    "source": "Market",
    "interval": "1m",
    "period": 60,
    "percentageThreshold": -0.01
  },
  "output": {
    "success": true,
    "currentPrice": 497.8,
    "lastPrice": 504.4,
    "percentage": -0.0131
  }
}

Use ARM Graviton 2 as Lambda archictecture

See https://aws.amazon.com/about-aws/whats-new/2021/09/better-price-performance-aws-lambda-functions-aws-graviton2-processor/ for more details.

Improve performance of candlesticks retrieval from DynamoDB

Currently candlesticks are stored in DynamoDB with 1 item per minute. This results having 1440 items to request for a single day of data. This become really inefficient when requesting for example 30 days.

Find a way to improve performance and reduce number of queries to DynamoDB.

Handle gaps in candlesticks

Gaps are currently not handled in candlesticks.
It means that if I request 1000 candles but there are gaps, I will receive less than 1000 candles.

It's not a big deal for the API as it depends on the expected behaviour.

But for moving average calculation, it can lead to errors. We should detect gaps and fetch more candles until number of requested candles is reached.

Note: pay attention when there is no more data in database to avoid inifinite loop.

Add limit order feature

Currently only market orders are supported. Add limit orders support.

Use @hastobegood/crypto-bot-artillery package

Use this package which contains common code for crypto bots : https://www.npmjs.com/package/@hastobegood/crypto-bot-artillery

Add an option to enable X-Ray tracing

Add an option to the stack in order to enable/disable X-Ray tracing during deployment.

Targets :

• AWS SDK
• External HTTP requests

Fix floating point error

For example 998.5632 / 2.36 result in 423.12000000000006 instead of 423.12.

Investigate what is the best way to handle this.

Sonar analysis

Add Sonar analysis on each PR.

As the project is open source, it should be free.

Update dependencies versions

API endpoint to get all candlesticks for a specific symbol

TODO