comit-network / create-comit-app Goto Github PK
View Code? Open in Web Editor NEWSet up a local development environment for COMIT apps with one command.
License: Apache License 2.0
Set up a local development environment for COMIT apps with one command.
License: Apache License 2.0
User has to:
Reduce the steps above by publishing create-comit-app on npmjs.com as it is the one guaranteed package manager that a TS/JS dev uses.
I check https://www.npmjs.com/policies/open-source-terms#acceptable-use and https://www.npmjs.com/package/create-react-app and it should be fine.
Upload various binaries on npm (one for each platform) so that a user can just do:
npx create-comit-app
# or
yarn add create-comit-app && create-comit-app
The bitcoincore-rpc client we are using is not async and hence blocks the current thread during execution. This is not good because the remaining code is based on futures and hence the executor cannot do its work.
Don't block the executor thread.
We only use the client to generate blocks and fund.
Remove the dependency on the client and handcraft the JSON-rpc request with an async HTTP client like reqwest.
Look into building an image on the fly using shiplift crate that:
new
then cd
the new folder and then run start-env
. Failure to do so will return unclear errors.start-env
twice, he gets unclear errors./tmp/create-comit-app
are not cleaned-up.cd
step.start-env
twice on the same machine.As per the Linux Information Project:
Among the various subdirectories within /var are /var/cache (contains cached data from application programs), /var/games (contains variable data relating to games in /usr), /var/lib (contains dynamic data libraries and files), /var/lock (contains lock files created by programs to indicate that they are using a particular file or device), /var/log (contains log files), /var/run (contains PIDs and other system information that is valid until the system is booted again) and /var/spool (contains mail, news and printer queues).
Hence, the proposal would be to replace ./.env
with /var/run/create-comit-app/env
.
This could then be used as a lock
file, meaning, if the file exists and a user runs start-env
then an error should explain that it seems like an instance is already running:
▶ create-comit-app start-env > /dev/null 2>&1 &
[1] 71840
▶ create-comit-app start-env
An environment is already running!
If you are sure that no other `create-comit-app start-env` process is running, delete the lock file at `/var/run/create-comit-app/env`.
/var/run/create-comit-app/
so start-env can delete the whole folder at the end (instead of just the env file).Problem: nothing is tested automatically. To check if something works we have to try it out manually.
Goal: to be able to verify that the code that we write works as intended.
Ideas:
start_env
and new
in isolation.start_env
and new
work together.No message is displayed when new
is successfully done.
new
is successfully done (or failed).start-env
)start-env
.Print a message once new
is done, for example:
> create-comit-app new foo
Your project `foo` has been created!
Now, start your development environment by running `create-comit-app start-env`
Currently, the serialization of the env
file is not automatically tested. This causes regressions that are only caught once you try to run it against an example or an application.
Partly, this issue is caused by us not making good use of Rust's type system, hence the compiler cannot check certain things for us (like writing all the variables to the env file).
Model the env file as a struct that has a dedicated method for serializing. Write tests for serializing this struct and compare it.
By using a struct instead of the methods on EnvFile
directly, we already get feedback at compile-time instead of cryptic errors during the e2e tests with the examples.
We need to compile the Mac OS Binary locally when releasing a new version of create-comit-app
Avoid manual steps for a smoother devex.
Use GitHub Action on a mac os environment to produce and attach the binary.
Copy-pasta/see #33.
The ERC20-BTC example cannot be easily tested as part of the CI as it requires shell interaction.
Automate run of all examples to ensure our user can always run them
yes |
command during tests, @D4nte.
As it is not tested, we could end up with a non-working example in the repository which could impact COMIT App devs or comit workshops.
@bonomat commented on Wed Oct 02 2019
Hello-swap currently only supports BTC-ETH (or vice versa). The assets
are taken from the offer but the ledgers
are hardcoded.
Try to do a swap between ERC20-BTC (or vice versa) and modify hello-swap accordingly to make this easier. Keep ETH-BTC as the main example.
We need to think about how we want hello-swap to evolve and how to support different use cases in the future.
@da-kami proposed to create something similar to truffle unbox
https://www.trufflesuite.com/docs/truffle/advanced/creating-a-truffle-box
Imho, ERC20-BTC is an essential feature and I would not create an extra box for this but would build this directly into the main example, however, keep the flow as is, i.e. still have the swap of BTC-ETH swap hardcoded.
If a previous run of start-env
did not clean up properly the docker network create-comit-app
then there may be issues.
If only one network is remaining, that is fine.
But if 2 or more are remaining (the network name do not have to be unique) then some strange error such as:
Starting Bitcoin node...Error encountered when starting container: Fault { code: 400, message: "network create-comit-app is ambiguous (3 matches found on name)" }
is displayed to the user (and start-env
fails).
Make it seamless for the user if a previous start-env
run did not clean-up the docker network.
Up to you!
A. abort and tell the user how to clean-up the networks: docker network prune
if the user has no other relevant network or clean-up manually any create-comit-app
network if the user knows what to do.
B. Go in a failure path where the app will try to clean-up all the create-comit-app
networks.
When releasing, once the binary are attached, we need to run npm publish
locally to upload the latest binary to npmjs.com
Reduce the number of manual steps for smoother devex.
Add a GitHub Action to automatically publish on npm once mac and linux binary are attached to the release.
Problem: we have hard-coded username and password used to instantiate a bitcoin RPC client.
Goal: have no hard-coded credentials inside code.
Suggestion: use the cookie file instead.
For now, create-comit-app allows setting up regtest nodes for Ethereum and Bitcoin. While this is suitable for a developer environment it is far away from a real-world setup.
Especially for workshops a testnet setup is more suitable as it enables participants to trade against each other.
Enable create-comit-app to setup a testnet environment with:
At the moment, the error handling in create-comit-app is pretty verbose.
Minimize the code that is needed for robust error handling.
Out of the experimentation here, I recommend using anyhow
and thiserror
.
After CTRL-C
or kill
the btc-eth example, the command returns code 130
.
It should return code 0
if no issue was encountered as it is the expected usage.
This creates issues to recognise healthy behaviour.
new_project/examples/btc-eth 53-e2e-tests ✗ 1h2m ⚑
▶ yarn run start
yarn run v1.19.1
warning ../../package.json: No license field
$ DOTENV_CONFIG_PATH=$HOME/.create-comit-app/env ts-node -r dotenv/config ./src/index.ts
[maker] Bitcoin balance: 10. Ether balance: 100
[taker] Bitcoin balance: 10. Ether balance: 100
[maker] Creating offer: sell {"coin":"ether","amount":10}, buy {"coin":"bitcoin","amount":1}
[taker] Taking offer: sell {"coin":"bitcoin","amount":0.5}, buy {"coin":"ether","amount":5}
[taker] Performing fund action for swap fd7c2de7-5ab8-4e92-8e29-89b7ca600b16
[maker] Performing fund action for swap fd7c2de7-5ab8-4e92-8e29-89b7ca600b16
[taker] Performing redeem action for swap fd7c2de7-5ab8-4e92-8e29-89b7ca600b16
[maker] Performing redeem action for swap fd7c2de7-5ab8-4e92-8e29-89b7ca600b16
[maker] Swap finished with status SWAPPED: fd7c2de7-5ab8-4e92-8e29-89b7ca600b16
[taker] Swap finished with status SWAPPED: fd7c2de7-5ab8-4e92-8e29-89b7ca600b16
^C
[maker] Bitcoin balance: 10.5. Ether balance: 95
[taker] Bitcoin balance: 9.5. Ether balance: 105
new_project/examples/btc-eth 53-e2e-tests ✗ 1h3m ⚑ ⍉
▶ echo $?
130
This also means we have to hack around it in the test as we do set -e
:
create-comit-app/tests/run_example.sh
Line 101 in 8e60adf
Not return bad code if everything is good.
Remove this hack:
create-comit-app/tests/run_example.sh
Line 101 in 8e60adf
Check the signal handling and fix it.
CCA has a quite some complexity because it is basically a composition of a lot asynchronous tasks (Futures).
An easy to maintain and understand codebase.
Move the whole codebase to async/await and async-std
.
As part of this re-design, consider problems like:
Image
trait is implemented for both blockchain nodes and cnd, but it has a method new
which instantiates clients for the blockchain nodes yet does nothing for cnd
:
create-comit-app/src/docker/cnd.rs
Lines 28 to 30 in 57fe0c7
This is dead, useless code.
To not have dead, useless code.
Move new
function from Image
to BlockchainImage
as @D4nte suggested below.
Problem: btsieve is hard to explain - now that it is integrated in cnd we should not have to mention it.
Goal: Not have to explain btsieve in Osaka
Proposal: After cnd remove btsieve from startup. Adapt the cnd config creation if needed.
@D4nte commented on Fri Sep 27 2019
During a workshop/live coding, the linter is cumbersome, especially around unused imports and variables/arguments.
However, we want to keep the hello-swap codebase to a high quality level.
Only apply linter when we modify the hello-swap codebase, not when it is used as part of a new project
Steps:
tslint.json
.tslint-strict.json
file that is used during the CI and commit hooks@D4nte commented on Wed Oct 16 2019
Findings: none because tsc
uses options in tsconfig.json
so both tsconfig
and tslint
files should be modified.
Currently, the application doesn't print any output until the blockchain nodes have started up.
Problem: The user has no feedback for about 5 seconds on whether or not anything has happened.
Goal: It should be clear to the user, what the application is doing.
Recommendation:
Light-weight solution: Log something before and after an asynchronous task, like:
> Starting bitcoin node ...
> Done.
> Starting ethereum node ...
> Done.
etc ...
Long-term solution: Provide interactive output of what is happening, with animations etc.
Goal: Make it easier for users to run the application
Problem: The binary name is very long.
Recommendation: Rename the binary name to cca
so that people only have to type cca start-env
.
Problem: Create-comit-app can be started twice.
a) This takes up unnecessary resources
b) It can be confusing to the user
c) It might actually cause problems because we use static names for docker containers and networks
Recommendation: Fail if we find a .env
file in the current directory.
Currently, the name
of the project that is initialized for the user is hardcoded in the package.json:
Create a project that is consistent with the users input.
Search and replace all uses of new_project
after it is has been initialized with the name the user passed on the commandline.
If the user types: yarn create comit-app my-cool-app
, the package.json should say: "name": "my-cool-app"
.
Problem: btsieve is confusing for people to see it without explanation.
Goal: Don't have to mention btsieve during presentation (it is re-integrated in cnd now anyway)
Recommendation: Just remove the log output until we actually remove it from the startup.
create-comit-app usage is still a bit cumbersome as there is the new
and start-env
command. Knowing that start-env
needs to be done in a different repo.
User types the following commands to run an example:
yarn create comit-app foo
cd foo/examples/bar
yarn # Install dependencies
yarn run env # Does create-comit-app start-env and hog the terminal
--- open new terminal
yarn run swap # Run the example, name of the command is example specific
# OR
yarn run taker
...
CTRL-C
--- back to "env" terminal
CTRL-C
User types the following commands to run their app:
yarn create comit-app foo
cd foo
yarn # Install dependencies
yarn run env
--- open new terminal
touch bar.js
yarn add some-deps
<update package.json with a start command>
yarn start
...
--- back to "env" terminal
CTRL-C
When we do a workshop people cannot communicate with each other at the moment. Everything is just on the local machine.
When we did our first workshop in Singapore using comit-i we had a lot of dynamic resulting from the fact that people could interact with each other on the UI. The current workshops don't allow that.
Be able to have interaction between participants for workshops.
Grab a mate and a room and come up with an interactive example first. Then decide on the technical requirements.
Network ideas:
--network
flag to create-comit-app's start-env
command.Current instructions tells user to download binary from github, there is a more practical way to install create-comit-app
The standard way of using create-comit-app should be descrived as using yarn create
or npx
Reproduce the README.md changes done in https://github.com/comit-network/create-comit-app/pull/64/files
Be sure to test them before opening the PR.
At the moment Hello-Swap contains both maker and taker and the user sees how they interact but cannot control much. It is difficult to understand
Split the example so one has to run it in two terminals. One terminal is the maker and one the taker.
Additionally it would be cool to, once/if we have testnet support, be able to run the example with another person. (One person running maker and one person running taker side.)
Currently the environment is static, there is no configuration possible to add funds to a certain account/address.
Be able to fund somehow.
Add commands in create-comit-app to allow funding of a running environment. (configuration files are an alternative, but make things even more complicated).
In the long run we should work on the infrastructure in general (like providing faucet, testnet setup and similar for workshops - this needs more thought though), see
Problem: We always fetch the images. I ran into several timeouts with 500
errors against https://registry-1.docker.io
- just could not start the environment then.
Goal: Be able to reuse the images already pulled and not pull on every startup.
Proposal:
We have to build release binaries locally to when releasing a new version.
Have a binary built automatically when releasing
Try to do it with GitHub actions.
Ensure there are no unwrap()
left.
panic!()
and expect()
are ok but should only be done in start_env.rs
and new.rs
files. Bubble up errors elsewhere to start_env
and new
.
Create-comit-app only supports native assets on Bitcoin and Ethereum at the moment. There is a large amount of non-native assets (aka tokens) on Ethereum.
Cnd supports ERC20 tokens already but it's a non-trivial job to setup a test environment for executing an atomic swaps from ERC20 on Ethereum to BTC on Bitcoin.
Enable create-comit-app to setup a (reg)test environment with:
I would recommend to build this directly into the existing start-env
command and not create a new one specifically for ERC20.
When developing on create-comit-app
and often running the start-env
, the new changes may lead to a panic
.
Which means that they may be some artefacts not properly cleaned up such as the ~/.create-comit-app
folder, the docker network or containers
Make it possible to run start-env
after a panic!
that is due to new code when working on create-comit-app
A. Add a panic hook that catches any panic and process with clean up
B. Somehow make it possible to start-env
on a dirty environment (while yet not impacting the case where a user do 2 start-env by mistake)
When running the btc_eth
examples the output differs quite a bit from the erc20_btc
example. This might be confusing when using create-comit-app
outside a workshop where this can be explained.
When the user runs any example the output is similar so the user can relate to it.
For now I would only fix the output, the behaviour of the examples CAN (should?) be different so that one can see what is possible.
In btc_eth
:
In erc20_btc
:
btc_eth
example), move the transaction IDs to the data level.With the introduction of the ERC20 environment (#79), we are also now locally signing transaction through emerald. Unfortunately, emerald brings in a compile-time dependency on the shared library libusb-1.0
.
Have CCA be as easy to compile as possible: no dependencies on shared libraries.
Make a PR to emerald-vault-core to feature-gate whatever code depends on libusb.
CCA is a user-facing CLI and should therefore have good error messages. Currently, we simply Debug-print our error enum variants to the user if something fails.
cnd
.Take a look at snafu and use it in CCA.
Note: This is the outcome of the discussion with @bonomat and @thomaseizinger 04.10.2019.
Having multiple repositories as starting point is difficult to understand.
The idea is to integrate all examples into cca so there is only one go-to project as starting point. There will be multiple ones. The examples should be functionally driven (e.g. eth-btc, erc20-btc, ...).
When creating a new comit-app you get a template and all examples to look at. The examples can be run through create-comit-app. The examples are not meant as templates to build an application on, but help to understand functional parts.
For a workshop it has to be able to write a new examples (basically re-create one of the simple ones in the example folder) within 30 minutes. We can measure how good the sdk is by that measure.
cca
|- src (1)
|- new_project
--|- index.ts (2)
|- package.json (3)
|- examples
--|- eth-btc (4)
--|- src
|- ...
(1) cca rust code
(2) start point for creating a comit-app, an "empty template"
(3) presets all dependencies (comit-js-sdk ...)
(4) contains one self-contained examples, in the future we can create more packages to share between examples (like e.g. wallets)
new
run app
run example [example_name]
To be discussed:
start-env
still separate?Once comit-rs:0.3.0
is released there will be no need to run btsieve
as a separate binary. This will speed up start-env
and simplify the code.
Currently, many threads try to write to .env file, which can lead to race conditions if not handled correctly.
Instead, those threads should bubble up the information that needs to be written and the writing should be by the main thread. This would also remove the .env
file dependency on starting Docker containers for cnd
and blockchain nodes.
Definition of Done: env file path stays in start_env and not be passed
When starting create-comit-app start-env
via the npm package, when CTRL-C is received, node gives back the terminal handle before create-comit-app finishes to shutdown:
▶ cd create-comit-app/.npm
▶ yarn install
▶ yarn start start-env
yarn run v1.17.0
$ node cca.js start-env
Creating Docker network (create-comit-app)...✓
Starting Bitcoin node...✓
Starting Ethereum node...✓
Writing configuration in env file...✓
Starting two cnds...✓
🎉 Environment is ready, time to create a COMIT app!
^C
Signal received, terminating...
src/create-comit-app/.npm npm-packaging ✔ 21m ⍉
▶ ✓ # This tick is emitted by create-comit-app
# as is this new line
Make it so that node does not terminate until create-comit-app has
Not sure, the signal handling code is in .npm/cca.js
Running create-comit-app start-env
creates an env
file somewhere in the file system, but the location is not indicated when running this command. If a user wants to locate the file in order to look at values such as comit node HTTP API endpoints for manual debugging they have to look at the source code.
Not require users to read the source code if they want to manually debug or inspect the environment.
Log its location after start-env
has finished writing to it.
For the Hackathon in Dez. we might have Windows users. Windows is currently not supported.
Release create-comit-app for native Windows.
Time-box this. Since create-comit-app is reasonably small we don't have to use github-actions.
during the workshop we had one participant running into this issue:
./create-comit-app new my-cool-app
thread 'main' panicked at 'Failed to create initial commit: Error { code: -3, klass: 7, message: "config value \'user.name\' was not found" }', src/new.rs:25:22
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace.
seems to be related to the git configuration
Strong recommendation: Do git init
but do not do an initial commit.
The user knows that an image in being pulled but does not know the progress.
Some docker images being bigger than other, it would be great to have some feedback.
Give useful feedback to the user of create-comit-app start-env
.
This can be done by adding an .inspect
on the stream pulling the image:
create-comit-app/src/docker/mod.rs
Line 121 in 2878009
That prints some progress.
Or something better/more clever :)
Two ways to go:
A. Have a generic visual aid that moves everytime there is some progress (for example, something like |/-\|/-|
... so it looks like it rotates when it's pulling
B. Actually decode the data provided by the docker api to show percentage completion.
ATM, we have to update dependencies manually, which we practically never do.
Don't be bitten by a lot of painful dependency updates at the same time.
Setup dependabot.
We work a lot on create-comit-app and merging PRs manually is cumbersome.
Make work on CCA as efficient as possible.
Setup bors.
Improve the consistency of the examples to improve the learning speed.
With 2 different examples we can already see that examples can be quite different in behaviour and implementation detail. At the moment there is no strategy for adding more examples.
Multiple example testing against the same environment:
Output of the example is similar
hello swap
Behaviour proposal:
Example naming:
btc-eth
to peer-to-peer
(or hello-swap
)erc20-btc
to using-orderbook
(or simple-orderbook
or auto-execute-with-orderbook
)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.