arkecosystem/core

The ARK Core Blockchain Framework. Check https://learn.ark.dev for more information.

ARK Core

Note Please checkout MainSail for the next generation of the ARK-Core blockchain protocol featuring a new DPoS consensus engine that is more reliable and efficient. Currently in development.

Introduction

This repository contains all plugins that make up the ARK Core.

Check our dedicated documentation site for information about all available plugins and How to write a Core Plugin if you want to get started with developing your own plugins.

Documentation

• Development : https://ark.dev/docs/core/installation/intro
• Docker : https://ark.dev/docs/core/development/docker

API Documentation

• API v2 : https://ark.dev/docs/api

Security

If you discover a security vulnerability within this package, please send an e-mail to [email protected]. All security vulnerabilities will be promptly addressed.

Credits

This project exists thanks to all the people who contribute.

License

MIT © ARK Ecosystem

core's Issues

Add Transaction Pool Tests

Add Forge Manager Tests

Make transaction vendor fields searchable

We basically have 2 options currently, one would require a change to the database and use more storage but grant much greater searching abilities.

1. Keep the database how it is with a hex vendor field and require people to enter vendor fields in hex format for searches. Advantage is less storage, Disadvantage is no proper search for the most important part of a transaction to the end-user.

2. Store the vendor field in plain text in the database, allows partial matching, regex pattern, full text search. Advantages are the mentioned ones, Disadvantages are more storage.

Add Webhook Manager Tests

Restucture ark-core to include ark-js as "folder"

Move ark-js into ark-core as a folder and publishing only the directory on npm.. -
https://github.com/lerna/lerna
Reason: manage the models and different versions
Starting with ark-core v2

Add remote peer/seed list

Add utility to push unconfirmed transactions

Create a Testnet Config for Jest

Add Docker Files/Preset

Add Model Tests

Move recent model changes to packages/client

Add Webhook API Tests

Add Wallet model tests

Part of #182

Think about implementing hapijs/lout for API docs

https://github.com/hapijs/lout

Add Fixtures or Jest Snapshots to API Tests

https://facebook.github.io/jest/docs/en/snapshot-testing.html

Refactor and DRY API tests

Add Block models tests

Part of #182

Basic DDOS Transaction pool protection

• limit by the sender (i.e. max active transaction in pool)
• add option for whitelisting addresses

Switch GET /search endpoints to POST /search for nested queries

This requires to either get a PR accepted at https://github.com/fknop/hapi-pagination or fix the issue itself by including the package manually and modified.

Add Transaction model tests

Part of #182

Proper and Meaningful English Logging Messages

Should also consider removing emojis as logging is meant to provide plain informations and nothing else.

Implement bin executable

If we want people to allow to type things like ark-core start:mainnet we need to add a global bin executable that can handle all those commands we currently have inside the package.json as scripts.

    "bin": {
        "ark-core": "bin/index.js"
    },

Extract Deployer Utility into its own package

Add Logger Tests

Add Redis Manager Tests

Implement terkelg/prompts for prompts

We are currently using https://github.com/SBoudrias/Inquirer.js for the bip38 encryption but I found https://github.com/terkelg/prompts today which is more lightweight and offers much more for building interactive experiences.

A few things I could come up with that we could do with interative prompts are:

• Webhook Setup
• BIP38 Setup (Replace Inquirer)
• Node Setup
  • Database Driver
  • Logging
  • Redis Config

This would make the process of a node setup really easy.

fix issue on not picking up the right delegate in the round compared to for ark-node

Transaction Pool never removes transactions if not forging

This would eventually lead to a tonne of almost redundant data being stored in Redis, due to the broadcast of transactions to/from other peers.

Add Queue Manager Tests

Implement snapshot system

Regular export block and transactions table.
Unfinished work is here: https://github.com/fix/ark-core/blob/master/packages/core/lib/snapshot.js
Cross-distribute snaps for faster download

Upon restore:

• Just verify blocks.id (previous.id)...
• Verify Hashes and signatures… -> if is in the block
• Verify the transactions (proof of the blockchain).. Signature and hash check to the block hash

Update core-commander to accommodate plugin configuration

The issue with the new plugin system is that the configuration file is a plain javascript object that gets exported and not a JSON object like previously which makes it hard (maybe impossible without a hacky solution) to read the configuration and write it back into the file as things like path.resolve or anyFunction() are resolved when you require a file that contains module.exports.

There are basically 2 solutions to this:

1. Implement hacky solutions using reflection to provide configuration through the CLI
2. Only copy the config into ~/.ark/config with the commander and require configuration to be done manually but offer "test" functionality in the commander to make sure the configuration is valid and working
   2.1 "test" functions are things like checking if a database or redis connection can be established, if the log file path exists, etc.

Implement jeffijoe/awilix as container

Currently we already have a bunch of singletons and include files over and over again in all kind of places which can be solved by implementing a container like https://github.com/jeffijoe/awilix.

We would register all classes and globally used variables (like configs) on application start in the container and they would be auto injected and accessible from everywhere.

Also we could get rid of the hacky singletons as the container can maintain the singleton state for us by only instantiating a class once if specified and retrieving the initial instance on consecutive calls.

Revert UNSIGNED changes to database

Based on this link, sequelize does not support UNSIGNED values for PostgreSQL. Need to revert the changes that were made.

Add State Machine Tests

P2P Multibroadcast transaction

Implement transaction broadcast, upon receiving and verifying the transaction - broadcast it to XX peers.

Create a Testnet SQLite Database

Add Wallet Manager Tests

remove empty wallets from memory

In order to optimise in memory wallets, empty wallets can be removed from wallet manager and be considered as cold wallet.
Needed rules :

• If a new incoming tx, a check against the database for public key will rebuild complete state of wallet
• for API, by default if no wallet is found in memory, it means balance is 0

Implement pm2 ecosystem files

Implement http://pm2.keymetrics.io/docs/usage/pm2-api/#programmatic-api to start the relay and forger with custom configuration through the commander.

Add Forging Tests

add save dirty wallets on ctrl+c events

• catch ctrl+c event
• save dirty wallets
• indicates somehow the wallets has been saved on shutdown
• shutdown

when restart:

• check if the wallets has been saved on shutdown recently (like a few rounds back)
• if yes skip SPV build, load wallets from db and start normal sync
• if no rebuild/SPV/sync (normal behaviour)

Implement config & argument validation

We could use https://github.com/sindresorhus/ow for 2 things.

I. Config Validation

We should make sure on application start that all important configuration like the network configuration is valid.

II. Argument Validation

We should make sure that arguments passed onto methods have the right type, length and format to avoid any unexpected behaviour that could cause harm or create corrupted data.

Plug Fork Management to state machine

Add Strict Mode to v1 /api/transactions

Integrate ark.js v2 once released

Create light transaction memory pool plugin - i.e. no redis

Performance Hit of Block Endpoints

We currently have API endpoints both in v1 and v2 that have a major impact on performance because the database has to be hit for a count of millions of records and then retrieve 50-100 records based on an offset.

This performance hit in a normal environment would be easy to fix by sharding, getting more hardware or optimizing the implementation of the database layer by using postgres specific functions like https://wiki.postgresql.org/wiki/Count_estimate which is not an option as we use Sequelize.

https://github.com/fix/ark-core/blob/master/app/api/public/versions/1/index.js#L11
https://github.com/fix/ark-core/blob/master/app/api/public/versions/2/index.js#L15
https://github.com/fix/ark-core/blob/master/app/api/public/versions/2/index.js#L39
https://github.com/fix/ark-core/blob/master/app/api/public/versions/2/index.js#L40

The only real solution without major changes or spec changes for people that want to run a node I can think of is that we load all blocks and transactions into memory on application start and push new blocks into memory after they are stored in the database.

This way we avoid hitting the database for millions of records as we could simply use filter, sortBy or length to count records instead of having to do heavy database queries after the application is booted.

Currently the requests on the block endpoints can take anywhere from 10 to 20 seconds (up to 3 minutes on stat endpoints) and this request time will only increase over time as we are not optimizing around postgres or something like cassandra which is designed for larger datasets.

The reality is people run their nodes on the cheapest possible option so we sadly have to work around that.

API v1 issues

http://localhost:4003/api/transactions?orderBy=timestamp:desc&offset=0&limit=50&recipientId=DTRdbaUW3RQQSL5By4G43JVaeHiqfVp9oh&senderId=DTRdbaUW3RQQSL5By4G43JVaeHiqfVp9oh

{
  "statusCode": 500,
  "error": "Internal Server Error",
  "message": "An internal server error occurred"
}

need to remove the senderId:

http://localhost:4003/api/transactions?orderBy=timestamp:desc&offset=0&limit=50&recipientId=DTRdbaUW3RQQSL5By4G43JVaeHiqfVp9oh

{
  "transactions": []
}

This was pulled directly from the Desktop Wallet, where I added my devnet v2 node to.

Extract Commander Utility into its own package

Add Cache Manager Tests