Giter Site home page Giter Site logo

Comments (21)

brenzi avatar brenzi commented on July 30, 2024 1

It does complete the playbook now! However, the nodes are not producing nor finalizing blocks!

Side remark: It did complete, but now it's not obvious to me how to play with my network. The console only tells me the IP's

what I did then is

ssh -i ~/gantree-work/ssh/ssh_id_rsa_validator [email protected]
 journalctl -f -u substrate

now, unfortunately I have to realize that my testnet does not work, because I'm still at block zero:

Apr 23 13:43:49 nctr-test-1 encointer-node[3393]: 2020-04-23 13:43:49 Idle (0 peers), best: #0 (0x2f25…a760), finalized #0 (0x2f25…a760), ⬇ 0 ⬆ 0
Apr 23 13:43:54 nctr-test-1 encointer-node[3393]: 2020-04-23 13:43:54 Idle (0 peers), best: #0 (0x2f25…a760), finalized #0 (0x2f25…a760), ⬇ 0 ⬆ 0
Apr 23 13:43:59 nctr-test-1 encointer-node[3393]: 2020-04-23 13:43:59 Idle (0 peers), best: #0 (0x2f25…a760), finalized #0 (0x2f25…a760), ⬇ 0 ⬆ 0
Apr 23 13:44:04 nctr-test-1 encointer-node[3393]: 2020-04-23 13:44:04 Idle (0 peers), best: #0 (0x2f25…a760), finalized #0 (0x2f25…a760), ⬇ 0 ⬆ 0
Apr 23 13:44:09 nctr-test-1 encointer-node[3393]: 2020-04-23 13:44:09 Idle (0 peers), best: #0 (0x2f25…a760), finalized #0 (0x2f25…a760), ⬇ 0 ⬆ 0
Apr 23 13:44:14 nctr-test-1 encointer-node[3393]: 2020-04-23 13:44:14 Idle (0 peers), best: #0 (0x2f25…a760), finalized #0 (0x2f25…a760), ⬇ 0 ⬆ 0
Apr 23 13:44:19 nctr-test-1 encointer-node[3393]: 2020-04-23 13:44:19 Idle (0 peers), best: #0 (0x2f25…a760), finalized #0 (0x2f25…a760), ⬇ 0 ⬆ 0
Apr 23 13:44:24 nctr-test-1 encointer-node[3393]: 2020-04-23 13:44:24 Idle (0 peers), best: #0 (0x2f25…a760), finalized #0 (0x2f25…a760), ⬇ 0 ⬆ 0

are validator keys for babe and grandpa injected by gantree?

btw., I've extended my config to have two validators:

{
    "metadata":{
        "version":"2.0",
        "project":"nctr-test"
    },
    "binary":{
        "chain":false,
        "fetch":{
            "url":"https://github.com/encointer/encointer-node/releases/download/v0.2.0/encointer-node",
            "sha256":"0c0cda03410bff227c693a140094e2c8da6d9192238f5fe4e190ce7fab962af1"
        },
        "filename":"encointer-node",
        "useBinChainSpec":"true"
    },
    "nodes":[
        {
            "validator":true,
            "instance":{
				"sshPrivateKeyPath": "/gantree/ssh/ssh_id_rsa_validator",
                "provider":"do",
                "dropletSize":"s-2vcpu-4gb",
                "region":"fra1",
				"sshPublicKey": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg4KTwEVbIG5FjmfX0E/Ip0HPOiOKKPAc+8yS4KO1fIJMMdHBybnRVrMHZL22DmQnih/Q1O6Zw4SPV5Ou+mZ4Ojz4JKv3VQ8HdIp1qS5AEqEOwAovkXhTTjnB/y5Mnoha1Pm5tC/15MOSjwclYHEegDBRosbBewOg6Or2ieeRbE1jYW4Sp8HWxhLAybQauqgKqCty8E78ZjG5cPdR54QmDQsypV918cBnKLWabV6UlZ67AprBmSeUo2QUmhOnPLVW4IsmpJuzuiJR9fBuXREk52zcRiXZoIrZqGzknR6xv5AnED2eIKjwciZL34cyOvVTNoX7vL1dM4qfsXzFGFWnX root@3f6b799bd119"
            }
        },
        {
            "validator":true,
            "instance":{
                "sshPrivateKeyPath": "/gantree/ssh/ssh_id_rsa_validator",
                "provider":"do",
                "dropletSize":"s-2vcpu-4gb",
                "region":"fra1",
                "sshPublicKey": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg4KTwEVbIG5FjmfX0E/Ip0HPOiOKKPAc+8yS4KO1fIJMMdHBybnRVrMHZL22DmQnih/Q1O6Zw4SPV5Ou+mZ4Ojz4JKv3VQ8HdIp1qS5AEqEOwAovkXhTTjnB/y5Mnoha1Pm5tC/15MOSjwclYHEegDBRosbBewOg6Or2ieeRbE1jYW4Sp8HWxhLAybQauqgKqCty8E78ZjG5cPdR54QmDQsypV918cBnKLWabV6UlZ67AprBmSeUo2QUmhOnPLVW4IsmpJuzuiJR9fBuXREk52zcRiXZoIrZqGzknR6xv5AnED2eIKjwciZL34cyOvVTNoX7vL1dM4qfsXzFGFWnX root@3f6b799bd119"
            }
        }
    ]
}

from gantree-cli-docker.

morelazers avatar morelazers commented on July 30, 2024 1

It shouldn't matter whether it's Babe or Aura.

"useBinChainSpec":"true" - Delete this, as well as the "chain": false and you should be good to go, they're finalising on my setup.

image

from gantree-cli-docker.

DrTexx avatar DrTexx commented on July 30, 2024

Hi @brenzi, we're having a look into this now.

The most recent documentation re. configurations can be found in the gantree-lib repository here, it's accurate to gantree-cli v0.8.0 and gantree-lib v0.6.0.

As a sidenote, configurations no longer require a public key as this is now extracted from the pem private key.

from gantree-cli-docker.

DrTexx avatar DrTexx commented on July 30, 2024

@brenzi Could you please let us know your:

  • OS and version
  • Output from id -u
  • Output from id -g
  • Your docker version

from gantree-cli-docker.

brenzi avatar brenzi commented on July 30, 2024
  • ubuntu 18.04
  • id -u ->1004
  • id -g -> 1004
  • Docker version 18.09.7, build 2d0083d

...and I did chown -R 1004:1004 gantree-work

from gantree-cli-docker.

DrTexx avatar DrTexx commented on July 30, 2024

Thanks for that 👍
I'll try reproducing the error.

from gantree-cli-docker.

DrTexx avatar DrTexx commented on July 30, 2024

@brenzi I haven't had any luck reproducing this error.

Would you mind trying the same Gantree configuration with the latest release of the dockerized cli to see if this issue is persistent?

from gantree-cli-docker.

brenzi avatar brenzi commented on July 30, 2024

Just tried, but I think your documentation is just not ready (and the abovementioned error is still there):

I followed the instructions here
So I did everything from scratch

git clone https://github.com/flex-dapps/gantree-cli-docker.git
cd gantree-cli-docker
docker build -t gantree-cli-docker .
mkdir ~/gantree-work/
cd gantree-work
mkdir config
mkdir ssh
nano config/encointer-testnet.json
cd ssh
ssh-keygen -f ./my_validator_key -t rsa -m PEM -q -N ""
mkdir ~/gantree-env
nano ~gantree-env/envfile.encointer

docker run -v ~/gantree-work/:/gantree --env-file ~/gantree-env/envfile.encointer --user $(id -u):$(id -g) --rm -ti gantree-cli-docker sync

But then I get

FAIL:[8] bad config: ENOENT: no such file or directory, open '/gantree/ssh/ssh_id_rsa_validator'

So your keygen instructions should actually read

ssh-keygen -f ./ssh_id_rsa_validator -t rsa -m PEM -q -N ""

but then I launch it again and get:

docker run -v ~/gantree-work/:/gantree --env-file ~/gantree-env/envfile.encointer --user $(id -u):$(id -g) --rm -ti gantree-cli-docker sync

Identity added: /gantree/ssh/ssh_id_rsa_validator (/gantree/ssh/ssh_id_rsa_validator)
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/gantree/ssh/ssh_id_rsa_validator.pub' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.



 ██████╗  █████╗ ███╗   ██╗████████╗██████╗ ███████╗███████╗
██╔════╝ ██╔══██╗████╗  ██║╚══██╔══╝██╔══██╗██╔════╝██╔════╝
██║  ███╗███████║██╔██╗ ██║   ██║   ██████╔╝█████╗  █████╗  
██║   ██║██╔══██║██║╚██╗██║   ██║   ██╔══██╗██╔══╝  ██╔══╝  
╚██████╔╝██║  ██║██║ ╚████║   ██║   ██║  ██║███████╗███████╗
 ╚═════╝ ╚═╝  ╚═╝╚═╝  ╚═══╝   ╚═╝   ╚═╝  ╚═╝╚══════╝╚══════╝

[!] Using override for Gantree config path

[!] Using override for inventory path

2020-04-21T07:05:17.428Z [Gantree] (lib/gantree) info: reading gantree configuration
2020-04-21T07:05:17.455Z [Gantree] (lib/config/validate) info: Gantree configuration validated successfully
2020-04-21T07:05:17.456Z [Gantree] (lib/config/preprocessors/boolToString) info: converting booleans in config to strings
2020-04-21T07:05:17.456Z [Gantree] (lib/config/preprocessors/injectEnvVars) info: injecting environment variables into config
2020-04-21T07:05:17.457Z [Gantree] (lib/config/inject) info: injecting defaults
2020-04-21T07:05:17.457Z [Gantree] (lib/ansible/inventory) info: creating namespace
2020-04-21T07:05:17.457Z [Gantree] (lib/ansible/inventory) info: namespace created
2020-04-21T07:05:17.458Z [Gantree] (lib/ansible/inventory) info: creating Gantree inventory
2020-04-21T07:05:17.505Z [Gantree] (lib/ansible/inventory) info: Gantree config hash written (7d498a53ed95c3cd58a2105d591fec475170053f15b43987b59a8cb9013fb926)
2020-04-21T07:05:17.505Z [Gantree] (lib/ansible/inventory) info: Gantree inventory created
2020-04-21T07:05:17.505Z [Gantree] (lib/ansible/commands) info: running playbook: /usr/local/lib/node_modules/gantree-cli/node_modules/gantree-lib/ansible/infra.yml
2020-04-21T07:05:17.506Z [Gantree] (cmd) info: Executing: ansible-playbook -i /gantree/inventory/nctr-test/gantree -i /gantree/inventory/nctr-test/active /usr/local/lib/node_modules/gantree-cli/node_modules/gantree-lib/ansible/infra.yml, {"verbose":true}
2020-04-21T07:05:18.057Z [Gantree] (cmd) error: Error occured during command execution 'ansible-playbook -i /gantree/inventory/nctr-test/gantree -i /gantree/inventory/nctr-test/active /usr/local/lib/node_modules/gantree-cli/node_modules/gantree-lib/ansible/infra.yml': 
Unhandled error:
 Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/ansible/utils/path.py", line 85, in makedirs_safe
    os.makedirs(b_rpath, mode)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: b'/.ansible'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/ansible/config/manager.py", line 554, in update_config_data
    value, origin = self.get_config_value_and_origin(config, configfile)
  File "/usr/lib/python3.8/site-packages/ansible/config/manager.py", line 498, in get_config_value_and_origin
    value = ensure_type(value, defs[config].get('type'), origin=origin)
  File "/usr/lib/python3.8/site-packages/ansible/config/manager.py", line 121, in ensure_type
    makedirs_safe(value, 0o700)
  File "/usr/lib/python3.8/site-packages/ansible/utils/path.py", line 90, in makedirs_safe
    raise AnsibleError("Unable to create local directories(%s): %s" % (to_native(rpath), to_native(e)))
ansible.errors.AnsibleError: Unable to create local directories(/.ansible/tmp): [Errno 13] Permission denied: b'/.ansible'



2020-04-21T07:05:18.066Z [Gantree] (cmd) error: Error occured during command execution 'ansible-playbook -i /gantree/inventory/nctr-test/gantree -i /gantree/inventory/nctr-test/active /usr/local/lib/node_modules/gantree-cli/node_modules/gantree-lib/ansible/infra.yml': 
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/ansible/utils/path.py", line 85, in makedirs_safe
    os.makedirs(b_rpath, mode)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: b'/.ansible'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/ansible/config/manager.py", line 554, in update_config_data
    value, origin = self.get_config_value_and_origin(config, configfile)
  File "/usr/lib/python3.8/site-packages/ansible/config/manager.py", line 498, in get_config_value_and_origin
    value = ensure_type(value, defs[config].get('type'), origin=origin)
  File "/usr/lib/python3.8/site-packages/ansible/config/manager.py", line 121, in ensure_type
    makedirs_safe(value, 0o700)
  File "/usr/lib/python3.8/site-packages/ansible/utils/path.py", line 90, in makedirs_safe
    raise AnsibleError("Unable to create local directories(%s): %s" % (to_native(rpath), to_native(e)))
ansible.errors.AnsibleError: Unable to create local directories(/.ansible/tmp): [Errno 13] Permission denied: b'/.ansible'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/ansible-playbook", line 62, in <module>
    import ansible.constants as C
  File "/usr/lib/python3.8/site-packages/ansible/constants.py", line 174, in <module>
    config = ConfigManager()
  File "/usr/lib/python3.8/site-packages/ansible/config/manager.py", line 290, in __init__
    self.update_config_data()
  File "/usr/lib/python3.8/site-packages/ansible/config/manager.py", line 566, in update_config_data
    raise AnsibleError("Invalid settings supplied for %s: %s\n" % (config, to_native(e)), orig_exc=e)
ansible.errors.AnsibleError: Invalid settings supplied for DEFAULT_LOCAL_TMP: Unable to create local directories(/.ansible/tmp): [Errno 13] Permission denied: b'/.ansible'


2020-04-21T07:05:18.073Z [Gantree] (cmd) error: Execution failed with code 1: ansible-playbook -i /gantree/inventory/nctr-test/gantree -i /gantree/inventory/nctr-test/active /usr/local/lib/node_modules/gantree-cli/node_modules/gantree-lib/ansible/infra.yml
(node:17) UnhandledPromiseRejectionWarning: Error: 1
    at childCloseCallback (/usr/local/lib/node_modules/gantree-cli/node_modules/gantree-lib/src/lib/cmd.js:117:18)
    at ChildProcess.<anonymous> (/usr/local/lib/node_modules/gantree-cli/node_modules/gantree-lib/src/lib/cmd.js:55:5)
    at ChildProcess.emit (events.js:323:22)
    at maybeClose (internal/child_process.js:1021:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)
(node:17) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:17) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

from gantree-cli-docker.

DrTexx avatar DrTexx commented on July 30, 2024

Hi @brenzi,
This information should be very helpful, thank you. I appreciate you breaking down each of the steps you took.
@rozifus is investigating the issue now.

from gantree-cli-docker.

rozifus avatar rozifus commented on July 30, 2024

Hi @brenzi,

Thanks for raising this issue

I believe there was a problem with provided user-ids other than 1000 not existing on the docker image. We were changing the user-id so that files created by gantree in the host-folder matched the account on the host machine. I've updated the image to now run as root inside the container and fix the ownership of files after they're created.

Could you try the following branch:
allow-users-other-than-1000

You'll need to rebuild the docker image
The run command also has a small change

--user $(id -u):$(id -g)

is now

-e HOST_USER=$(id -u)

Let me know if this resolves the problem on your end

from gantree-cli-docker.

morelazers avatar morelazers commented on July 30, 2024

Glad it deployed!

We do inject generated Babe/Grandpa keys into each of the nodes, so they should produce and finalise blocks. You can see an example of this working if you deploy the substrate-node-template binary.

We'll make some changes to the tutorial that make it a little more clear how to get into the boxes. What you did is correct 👍

Nodes currently don't report to the default telemetry endpoint, however you can ensure that it does by adding

"binary": {
  ...,
  "substrateOptions": [
    "--telemetry-url ws://telemetry.polkadot.io:1024"
  ]
}
...

from gantree-cli-docker.

brenzi avatar brenzi commented on July 30, 2024

wait, babe? My chain is Aura. Is that the problem? But the node-template is aura too....

from gantree-cli-docker.

brenzi avatar brenzi commented on July 30, 2024

I just tried with another chain of ours:
https://github.com/scs/substraTEE-node/releases/download/v0.6.0/substratee-node-0.6.0

same result. no blocks produced.

Something seems odd: both nodes show 0 peers and that kinda makes sense because:

Apr 26 17:56:09 substratee-test-0 substratee-node-0.6.0[6380]: 2020-04-26 17:56:09 🏷  Local node identity is: QmToVsXxWFZz1UmjPYWjxMc7jDsygmzyf6kqnsDXzTH58H
Apr 26 17:58:14 substratee-test-1 substratee-node-0.6.0[6069]: 2020-04-26 17:58:14 🏷  Local node identity is: QmfWuGphwxb4aMJY5SFDiszjRQ7ma1ra7QcaQ7QmgFF3JG

These node identities differ from what is stored in `/home/subuser/session.....yaml

  • node-0: QmXpnUZzYZZTs8ZMxahufFMkGhv51MpgVfcRAznhR9rNMW
  • node-1: QmddHxmWfEfXRXeKEVjxv3nghqAVxe2DLG2kC7C8C9Je3q

So I guess the bootnode setting is wrong.

Then I checked the service script and found another oddity:

[Unit]
Description=Substrate Node

[Service]
User=subuser
Group=subgroup
ExecStart=/usr/local/bin/substratee-node-0.6.0 \
                   --name substratee-test-1-1 \
                  --validator \
                                             --rpc-port=9933 \
                                                 --telemetry-url="ws://127.0.0.1:8000/submit 0"
                    
Restart=always

[Install]
WantedBy=multi-user.target

there is no --chain argument. And no --bootnode. (and my custom telemetry endpoint doesn't show up neither)

So the node log says:

Apr 26 17:56:09 substratee-test-0 substratee-node-0.6.0[6380]: 2020-04-26 17:56:09 📋 Chain specification: Local Testnet

but that's not what I would expect to be launched. I thought what you do is along the tutorial for a private network?

As you have created secrets for each node I guess you also created a chainSpec.json and injected the public keys there? But where is it?

from gantree-cli-docker.

brenzi avatar brenzi commented on July 30, 2024

not yet.

Apr 27 05:53:54 substratee-test-1 substratee-node-0.6.0[9711]: 2020-04-27 05:53:54 👤 Role: AUTHORITY
Apr 27 05:53:54 substratee-test-1 substratee-node-0.6.0[9711]: 2020-04-27 05:53:54 ⛓  Native runtime: substratee-node-runtime-1:1(substratee-node-runtime-2)
Apr 27 05:53:54 substratee-test-1 substratee-node-0.6.0[9711]: 2020-04-27 05:53:54 📦 Highest known block at #0
Apr 27 05:53:54 substratee-test-1 substratee-node-0.6.0[9711]: 2020-04-27 05:53:54 Using default protocol ID "sup" because none is configured in the chain specs
Apr 27 05:53:54 substratee-test-1 substratee-node-0.6.0[9711]: 2020-04-27 05:53:54 🏷  Local node identity is: QmfWuGphwxb4aMJY5SFDiszjRQ7ma1ra7QcaQ7QmgFF3JG
Apr 27 05:53:54 substratee-test-1 substratee-node-0.6.0[9711]: 2020-04-27 05:53:54 〽️ Prometheus server started at 127.0.0.1:9615
Apr 27 05:53:54 substratee-test-1 substratee-node-0.6.0[9711]: 2020-04-27 05:53:54 💔 Invalid peer ID from bootnode, expected `QmXpnUZzYZZTs8ZMxahufFMkGhv51MpgVfcRAznhR9rNMW` at address `/ip4/157.230.120.26/tcp/30333`.
Apr 27 05:53:54 substratee-test-1 substratee-node-0.6.0[9711]: 2020-04-27 05:53:54 💔 Invalid peer ID from bootnode, expected `QmXpnUZzYZZTs8ZMxahufFMkGhv51MpgVfcRAznhR9rNMW` at address `/ip4/157.230.120.26/tcp/30333`.
Apr 27 05:53:55 substratee-test-1 substratee-node-0.6.0[9711]: 2020-04-27 05:53:55 💔 Invalid peer ID from bootnode, expected `QmXpnUZzYZZTs8ZMxahufFMkGhv51MpgVfcRAznhR9rNMW` at address `/ip4/157.230.120.26/tcp/30333`.
Apr 27 05:53:55 substratee-test-1 substratee-node-0.6.0[9711]: 2020-04-27 05:53:55 💔 Invalid peer ID from bootnode, expected `QmXpnUZzYZZTs8ZMxahufFMkGhv51MpgVfcRAznhR9rNMW` at address `/ip4/157.230.120.26/tcp/30333`.
Apr 27 05:53:57 substratee-test-1 substratee-node-0.6.0[9711]: 2020-04-27 05:53:57 💔 Invalid peer ID from bootnode, expected `QmXpnUZzYZZTs8ZMxahufFMkGhv51MpgVfcRAznhR9rNMW` at address `/ip4/157.230.120.26/tcp/30333`.
Apr 27 05:53:57 substratee-test-1 substratee-node-0.6.0[9711]: 2020-04-27 05:53:57 💔 Invalid peer ID from bootnode, expected `QmXpnUZzYZZTs8ZMxahufFMkGhv51MpgVfcRAznhR9rNMW` at address `/ip4/157.230.120.26/tcp/30333`.
Apr 27 05:53:59 substratee-test-1 substratee-node-0.6.0[9711]: 2020-04-27 05:53:59 💤 Idle (0 peers), best: #0 (0xda99…aca4), finalized #0 (0xda99…aca4), ⬇ 1.2kiB/s ⬆ 1.3kiB/s

and on the other

Apr 27 05:55:12 substratee-test-0 substratee-node-0.6.0[12117]: 2020-04-27 05:55:12 📋 Chain specification: Local Testnet
Apr 27 05:55:12 substratee-test-0 substratee-node-0.6.0[12117]: 2020-04-27 05:55:12 🏷  Node name: substratee-test-0-0
Apr 27 05:55:12 substratee-test-0 substratee-node-0.6.0[12117]: 2020-04-27 05:55:12 👤 Role: AUTHORITY
Apr 27 05:55:12 substratee-test-0 substratee-node-0.6.0[12117]: 2020-04-27 05:55:12 ⛓  Native runtime: substratee-node-runtime-1:1(substratee-node-runtime-2)
Apr 27 05:55:12 substratee-test-0 substratee-node-0.6.0[12117]: 2020-04-27 05:55:12 📦 Highest known block at #0
Apr 27 05:55:12 substratee-test-0 substratee-node-0.6.0[12117]: 2020-04-27 05:55:12 Using default protocol ID "sup" because none is configured in the chain specs
Apr 27 05:55:12 substratee-test-0 substratee-node-0.6.0[12117]: 2020-04-27 05:55:12 🏷  Local node identity is: QmToVsXxWFZz1UmjPYWjxMc7jDsygmzyf6kqnsDXzTH58H
Apr 27 05:55:12 substratee-test-0 substratee-node-0.6.0[12117]: 2020-04-27 05:55:12 〽️ Prometheus server started at 127.0.0.1:9615
Apr 27 05:55:12 substratee-test-0 substratee-node-0.6.0[12117]: 2020-04-27 05:55:12 💔 Invalid peer ID from bootnode, expected `QmddHxmWfEfXRXeKEVjxv3nghqAVxe2DLG2kC7C8C9Je3q` at address `/ip4/134.122.93.76/tcp/30333`.
Apr 27 05:55:13 substratee-test-0 substratee-node-0.6.0[12117]: 2020-04-27 05:55:13 💔 Invalid peer ID from bootnode, expected `QmddHxmWfEfXRXeKEVjxv3nghqAVxe2DLG2kC7C8C9Je3q` at address `/ip4/134.122.93.76/tcp/30333`.
Apr 27 05:55:13 substratee-test-0 substratee-node-0.6.0[12117]: 2020-04-27 05:55:13 💔 Invalid peer ID from bootnode, expected `QmddHxmWfEfXRXeKEVjxv3nghqAVxe2DLG2kC7C8C9Je3q` at address `/ip4/134.122.93.76/tcp/30333`.
Apr 27 05:55:14 substratee-test-0 substratee-node-0.6.0[12117]: 2020-04-27 05:55:14 💔 Invalid peer ID from bootnode, expected `QmddHxmWfEfXRXeKEVjxv3nghqAVxe2DLG2kC7C8C9Je3q` at address `/ip4/134.122.93.76/tcp/30333`.
Apr 27 05:55:15 substratee-test-0 substratee-node-0.6.0[12117]: 2020-04-27 05:55:15 💔 Invalid peer ID from bootnode, expected `QmddHxmWfEfXRXeKEVjxv3nghqAVxe2DLG2kC7C8C9Je3q` at address `/ip4/134.122.93.76/tcp/30333`.
Apr 27 05:55:16 substratee-test-0 substratee-node-0.6.0[12117]: 2020-04-27 05:55:16 💔 Invalid peer ID from bootnode, expected `QmddHxmWfEfXRXeKEVjxv3nghqAVxe2DLG2kC7C8C9Je3q` at address `/ip4/134.122.93.76/tcp/30333`.

from gantree-cli-docker.

brenzi avatar brenzi commented on July 30, 2024

I tried gantree-cli clean, then sync. To no success. same issue. Here's my exact configuration

{
    "metadata":{
        "version":"2.0",
        "project":"substratee-test"
    },
    "binary":{
        "fetch":{
            "url":"https://github.com/scs/substraTEE-node/releases/download/v0.6.0/substratee-node-0.6.0",
            "sha256":"6b48d692bfb6ecec60dba537ab79e0cbb50ac0ebd8076505222fe92980928e63"
        },
        "filename":"substratee-node-0.6.0",
        "substrateOptions": ["--telemetry-url ws://telemetry.polkadot.io:1024"]
    },
    "nodes":[
        {
            "validator":true,
            "instance":{
                "sshPrivateKeyPath": "/gantree/ssh/ssh_id_rsa_validator",
                "provider":"do",
                "dropletSize":"s-2vcpu-4gb",
                "region":"fra1",
                "sshPublicKey": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg4KTwEVbIG5FjmfX0E/Ip0HPOiOKKPAc+8yS4KO1fIJMMdHBybnRVrMHZL22DmQnih/Q1O6Zw4SPV5Ou+mZ4Ojz4JKv3VQ8HdIp1qS5AEqEOwAovkXhTTjnB/y5Mnoha1Pm5tC/15MOSjwclYHEegDBRosbBewOg6Or2ieeRbE1jYW4Sp8HWxhLAybQauqgKqCty8E78ZjG5cPdR54QmDQsypV918cBnKLWabV6UlZ67AprBmSeUo2QUmhOnPLVW4IsmpJuzuiJR9fBuXREk52zcRiXZoIrZqGzknR6xv5AnED2eIKjwciZL34cyOvVTNoX7vL1dM4qfsXzFGFWnX root@3f6b799bd119"
            }
        },
        {
            "validator":true,
            "instance":{
                "sshPrivateKeyPath": "/gantree/ssh/ssh_id_rsa_validator",
                "provider":"do",
                "dropletSize":"s-2vcpu-4gb",
                "region":"fra1",
                "sshPublicKey": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg4KTwEVbIG5FjmfX0E/Ip0HPOiOKKPAc+8yS4KO1fIJMMdHBybnRVrMHZL22DmQnih/Q1O6Zw4SPV5Ou+mZ4Ojz4JKv3VQ8HdIp1qS5AEqEOwAovkXhTTjnB/y5Mnoha1Pm5tC/15MOSjwclYHEegDBRosbBewOg6Or2ieeRbE1jYW4Sp8HWxhLAybQauqgKqCty8E78ZjG5cPdR54QmDQsypV918cBnKLWabV6UlZ67AprBmSeUo2QUmhOnPLVW4IsmpJuzuiJR9fBuXREk52zcRiXZoIrZqGzknR6xv5AnED2eIKjwciZL34cyOvVTNoX7vL1dM4qfsXzFGFWnX root@3f6b799bd119"
            }
        }
    ]
}

from gantree-cli-docker.

rozifus avatar rozifus commented on July 30, 2024

Could you give this config a shot and let us know how it goes

Changes are:

  • moved substrateOptions to binaryOptions on the node (apologies, docs were not up to date on this)
  • updated telemetry-url value with verbosity level - 0, and wrapped in quotes
  • public key is no longer needed as it gets extracted from the private key

We've also pushed an update to gantree-cli that fixed an internal issue we hadn't seen before, so please make sure you're running >= 0.8.4 of gantree-cli on npm, or force a full rebuild of the docker image if you're using gantree-cli-docker

{
	"metadata": {
		"version": "2.0",
		"project": "substratee-test"
	},
	"binary": {
		"fetch": {
			"url": "https://github.com/scs/substraTEE-node/releases/download/v0.6.0/substratee-node-0.6.0",
			"sha256": "6b48d692bfb6ecec60dba537ab79e0cbb50ac0ebd8076505222fe92980928e63"
		},
		"filename": "substratee-node-0.6.0"
	},
	"nodes": [{
			"validator": true,
			"binaryOptions": {
				"substrateOptions": ["--telemetry-url=\"ws://telemetry.polkadot.io:1024 0\""]
			},
			"instance": {
				"sshPrivateKeyPath": "/gantree/ssh/ssh_id_rsa_validator",
				"provider": "do",
				"dropletSize": "s-2vcpu-4gb",
				"region": "fra1"
			}
		},
		{
			"validator": true,
			"binaryOptions": {
				"substrateOptions": ["--telemetry-url=\"ws://telemetry.polkadot.io:1024 0\""]
			},
			"instance": {
				"sshPrivateKeyPath": "/gantree/ssh/ssh_id_rsa_validator",
				"provider": "do",
				"dropletSize": "s-2vcpu-4gb",
				"region": "fra1"
			}
		}
	]
}

from gantree-cli-docker.

brenzi avatar brenzi commented on July 30, 2024

well....

PLAY RECAP *********************************************************************
104.248.35.51              : ok=59   changed=28   unreachable=0    failed=0    skipped=8    rescued=0    ignored=0
68.183.216.242             : ok=31   changed=19   unreachable=0    failed=1    skipped=1    rescued=0    ignored=0
localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

2020-04-29T08:09:18.023Z [Gantree] (cmd) error: Execution failed with code 2: ansible-playbook -i /gantree/inventory/substratee-test/gantree -i /gantree/inventory/substratee-test/active /usr/local/lib/node_modules/gantree-cli/node_modules/gantree-lib/ansible/operation.yml
(node:12) UnhandledPromiseRejectionWarning: Error: 2
    at childCloseCallback (/usr/local/lib/node_modules/gantree-cli/node_modules/gantree-lib/src/lib/cmd.js:117:18)
    at ChildProcess.<anonymous> (/usr/local/lib/node_modules/gantree-cli/node_modules/gantree-lib/src/lib/cmd.js:55:5)
    at ChildProcess.emit (events.js:323:22)
    at maybeClose (internal/child_process.js:1021:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)
(node:12) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:12) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

gantree-cli 0.8.4
gantree-lib 0.6.6
your config from above ^^

Can you run it and it works, producing blocks?

from gantree-cli-docker.

morelazers avatar morelazers commented on July 30, 2024

Is there any more data from the output? There should be a fail reason above the Play Recap table.

from gantree-cli-docker.

brenzi avatar brenzi commented on July 30, 2024

Here is the entire log

from gantree-cli-docker.

rozifus avatar rozifus commented on July 30, 2024

Hi @brenzi,

I can confirm I've copied the config back to my machine and was able to use it to get nodes finalizing.

Apr 29 14:09:12 substratee-test-1 substratee-node-0.6.0[3641]: 2020-04-29 14:09:12 🎁 Prepared block for proposing at 18 [hash: 0xc76623758ade30e0344b836dce197f857a9e2bd2738555f309f964559337d788; parent_hash: 0x64c2…36e
Apr 29 14:09:12 substratee-test-1 substratee-node-0.6.0[3641]: 2020-04-29 14:09:12 🔖 Pre-sealed block for proposal at 18. Hash now 0x296ce70093379605e618d2534275d930d32a25de406fb647f0464f4657b82f7a, previously 0xc76623
Apr 29 14:09:12 substratee-test-1 substratee-node-0.6.0[3641]: 2020-04-29 14:09:12 ✨ Imported #18 (0x296c…2f7a)
Apr 29 14:09:15 substratee-test-1 substratee-node-0.6.0[3641]: 2020-04-29 14:09:15 💤 Idle (1 peers), best: #18 (0x296c…2f7a), finalized #16 (0xaf9d…7b1a), ⬇ 1.1kiB/s ⬆ 1.0kiB/s
Apr 29 14:09:18 substratee-test-1 substratee-node-0.6.0[3641]: 2020-04-29 14:09:18 ✨ Imported #19 (0x07b4…6a1d)
Apr 29 14:09:20 substratee-test-1 substratee-node-0.6.0[3641]: 2020-04-29 14:09:20 💤 Idle (1 peers), best: #19 (0x07b4…6a1d), finalized #17 (0x64c2…36e6), ⬇ 1.1kiB/s ⬆ 1.1kiB/s

Looking at the log you provided I suspect we may have a race condition with the substrate service starting up, I haven't seen this happen before but it looks like the service/rpc wasn't quite running at the point we tried to grab the aura key, but was running immediately after when we grabbed the gran key. I'll add a ticket to investigate this further and potentially add a wait-check for the substrate service.

If you run the sync a 2nd time I believe it will work.

Hopefully this error is a rare occurrence and isn't something we've added recently, but I'll need to investigate further

from gantree-cli-docker.

brenzi avatar brenzi commented on July 30, 2024

yes, indeed!
Second try succeeded. race conditions seems plausible. I'll close this as it's gone off-topic for a while already.....

from gantree-cli-docker.

Related Issues (2)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.