poanetwork / blockscout-terraform Goto Github PK

An automation framework for spinning up cloud infrastructure to run BlockScout

License: GNU General Public License v3.0

Shell 21.48% HCL 64.10% JavaScript 14.42%

blockscout-terraform's Introduction

About

This repo contains scripts designed to automate Blockscout deployment builds. It currently only supports AWS as a cloud provider.

Ansible Playbooks are located in the root folder. These will create all necessary infrastructure and deploy BlockScout.

Deployment details, prerequisites and other information is available in the BlockScout docs.

Sections include:

Prerequisites: Infrastructure and BlockScout prerequisites
AWS Permissions: AWS setup
Variables: Configuration, Infra, BlockScout & Common variables
Deploying the Infrastructure. Describes all steps to deploy the virtual hardware required for a production instance of BlockScout. Skip this section if you already have an infrastructure and simply want to install or update your BlockScout instance.
Deploying BlockScout. Install or update your BlockScout.
Destroying Provisioned Infrastructure. Destroy your BlockScout installation.
Common Additional Tasks: Cleaning the cache, migrating the deployer, attaching an existing db.
Common Errors and Questions: Troubleshooting provisioning or server errors.

In addition, refer to the lambda folder which contains a set of scripts that may be useful in setting up your BlockScout infrastructure.

License

This project is licensed under the GNU General Public License v3.0. See the LICENSE file for details.

blockscout-terraform's People

Contributors

Stargazers

Watchers

blockscout-terraform's Issues

Restarting explorer on Erlang Crash- permissions

@bitwalker Do you know how I would restart the explorer service if Erlang crashes? I've tried systemctl start explorer.service but I do not have the correct permissions since I'm logged in as ec2-user.

Error when attempting to view the smart contract verification page

Jul 04 16:32:15 ip-10-0-0-187.poa.internal explorer[4044]: 16:32:15.260 request_id=rp69cppqmrg9ass9tenq6ahutqm9t2ji [info] Sent 500 in 398ms
Jul 04 16:32:15 ip-10-0-0-187.poa.internal explorer[4044]: 16:32:15.261 [error] #PID<0.15563.2> running ExplorerWeb.Endpoint terminated
Jul 04 16:32:15 ip-10-0-0-187.poa.internal explorer[4044]: Server: sokd-explorer-sokol-elb-1087081350.us-east-1.elb.amazonaws.com:80 (http)
Jul 04 16:32:15 ip-10-0-0-187.poa.internal explorer[4044]: Request: GET /en/addresses/0xc75f6d207328257d16467cfc17c6009c8511d2e4/contract_verifications/new
Jul 04 16:32:15 ip-10-0-0-187.poa.internal explorer[4044]: ** (exit) an exception was raised:
Jul 04 16:32:15 ip-10-0-0-187.poa.internal explorer[4044]: ** (ArgumentError) cookie store expects conn.secret_key_base to be at least 64 bytes
Jul 04 16:32:15 ip-10-0-0-187.poa.internal explorer[4044]: (plug) lib/plug/session/cookie.ex:160: Plug.Session.COOKIE.validate_secret_key_base/1
Jul 04 16:32:15 ip-10-0-0-187.poa.internal explorer[4044]: (plug) lib/plug/session/cookie.ex:153: Plug.Session.COOKIE.derive/3
Jul 04 16:32:15 ip-10-0-0-187.poa.internal explorer[4044]: (plug) lib/plug/session/cookie.ex:101: Plug.Session.COOKIE.put/4
Jul 04 16:32:15 ip-10-0-0-187.poa.internal explorer[4044]: (plug) lib/plug/session.ex:93: anonymous fn/3 in Plug.Session.before_send/2
Jul 04 16:32:15 ip-10-0-0-187.poa.internal explorer[4044]: (elixir) lib/enum.ex:1899: Enum."-reduce/3-lists^foldl/2-0-"/3
Jul 04 16:32:15 ip-10-0-0-187.poa.internal explorer[4044]: (plug) lib/plug/conn.ex:1145: Plug.Conn.run_before_send/2
Jul 04 16:32:15 ip-10-0-0-187.poa.internal explorer[4044]: (plug) lib/plug/conn.ex:393: Plug.Conn.send_resp/1
Jul 04 16:32:15 ip-10-0-0-187.poa.internal explorer[4044]: (explorer_web) lib/explorer_web/controllers/address_contract_verification_controller.ex:1: ExplorerWeb.AddressContractVerificationController.action/2```

g++ command not found

[stdout]===> Compiling zipper
[stdout]===> Compiling katana_code
[stdout]===> Compiling rebar3_elvis_plugin
[stdout]===> Compiling rebar3_archive_plugin
[stdout]===> Compiling prometheus_process_collector
[stdout]make: Entering directory `/opt/app/deps/prometheus_process_collector/c_src'
[stdout]g++ -O3 -finline-functions -fPIC -I /usr/lib/erlang/erts-10.0.5/include/ -I /usr/lib/erlang/lib/erl_interface-3.10.3/include -std=c++11 -Wall -c -o prometheus_process_collector_nif.o prometheus_process_collector_nif.cc
[stdout]make: g++: Command not found
[stdout]make: *** [prometheus_process_collector_nif.o] Error 127
[stdout]make: Leaving directory `/opt/app/deps/prometheus_process_collector/c_src'
[stdout]===> Hook for compile failed!
[stdout]
[stderr]** (Mix) Could not compile dependency :prometheus_process_collector, "/home/ec2-user/.mix/rebar3 bare compile --paths "/opt/app/_build/prod/lib/*/ebin"" command failed. You can recompile this dependency with "mix deps.compile prometheus_process_collector", update it with "mix deps.update prometheus_process_collector" or clean it with "mix deps.clean prometheus_process_collector"

(Feature) List prerequisites

Could you list all requires prerequisites in the README (New Relic license, etc?), so that the user could prepare them before provisioning.

bin/infra provision did not provide a privkey

I've never seen this issue before where the terraform script deployed successfully but when I was attempting to ssh into the EC2 instance a .privkey file was not generated during the deployment stage.

No action should be taken at this time but I would like to keep note that this has happened.

Error applying plan

After a couple of hit-and-miss rounds, I got to this stage, where my provision script fails with an error, which appears as if it was expected to happen:

===========
Error: Error applying plan:

1 error(s) occurred:

module.stack.aws_autoscaling_group.explorer: aws_autoscaling_group.explorer: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue.

Please include the following information in your report:

Terraform Version: 0.11.7
Resource ID: aws_autoscaling_group.explorer
Mismatch reason: attribute mismatch: availability_zones.1978419061

...

Not sure what the next steps should be. NOTE: Am kinda glad to be taking this challenge - been curious about Terraform for the last couple of months, but prioritized learning Ansible above it, since it can help me doing things I'm already doing.

diffOne.txt

Health check for ALB needs to remove internationalization

With the removal of internationalization from blockscout we need to edit the health check of the load balancer to /blocks instead of /en/blocks.

Not complete set of parameters is returned by aws ssm

The first call to aws ssm does not return a complete list of parameters.
The rest should be fetched by subsequent calls with --next-token cli option using NextToken returned by the previous call and results should be concatenated.

log "Fetching configuration from Parameter Store..."
parameters_json=$(aws ssm get-parameters-by-path --region "$REGION" --path "/$PREFIX/$CHAIN")
params=$(echo "$parameters_json" | jq '.Parameters[].Name' --raw-output)

Default storage should be increased to at least 100GB

The sokol chain runs out of space towards the end of indexing. This variable can be defined in a custom tfvars file using the variable db_storage, but the current 20GB is low for any chain that would deploy explorer.

mix ecto.drop is not dropping the database during build

I recently attempted to drop the database by adding mix ecto.drop to bin/deployment/migrate but the database did not drop. Is there something else I should be adding to this file?

bin/infra -v provision hangs, if no .tfvars provided in repo

If I clone clear repo, and no .tfvars file provided, the script hangs after prefix generation process.

I expected it to ask for necessary variables.

I investigated a little and looks like this will hangs forewer, if $EXTRA_VARS empty.
https://github.com/poanetwork/poa-explorer-infra/blob/master/bin/infra#L165
because we will have cat without arguments in this case.

I'n not sure about proper fix, but probably $EXTRA_VARS should be checked, and some another flow provided, if it's empty.

Code-deploy agent failing

with the following error (from new instance `/var/log/aws/codedeploy-agent/codedeploy-agent.log):

2018-05-30 13:05:15 INFO  [codedeploy-agent(12040)]: [Aws::CodeDeployCommand::Client 200 0.023102 0 retries] put_host_command_complete(command_status:"Failed",diagnostics:{format:"JSON",payload:"{\"error_code\":5,\"script_name\":\"\",\"message\":\"Concurrent::RejectedExecutionError\",\"log\":\"\"}"},host_command_identifier:"WyJjb20uYW1hem9uLmFwb2xsby5kZXBsb3ljb250cm9sLmRvbWFpbi5Ib3N0Q29tbWFuZElkZW50aWZpZXIiLHsiZGVwbG95bWVudElkIjoiQ29kZURlcGxveS91cy1lYXN0LTIvUHJvZC9hcm46YXdzOnNkczp1cy1lYXN0LTI6NzU4MDExMTI3ODMyOmRlcGxveW1lbnQvZC05VVQ4RjI3MFNfUmV2aXNpb24iLCJob3N0SWQiOiJhcm46YXdzOmVjMjp1cy1lYXN0LTI6NzU4MDExMTI3ODMyOmluc3RhbmNlL2ktMGY3ZTE0NGYyMjZjZDFhYjciLCJjb21tYW5kTmFtZSI6IkFwcGxpY2F0aW9uU3RvcCIsImNvbW1hbmRQb3NpdGlvbiI6MSwiY29tbWFuZEF0dGVtcHQiOjF9XQ==")  

2018-05-30 13:05:15 ERROR [codedeploy-agent(12040)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Error during perform: Concurrent::RejectedExecutionError - Concurrent::RejectedExecutionError - /opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/abstract_executor_service.rb:76:in `handle_fallback'

Limit prefix length to 5 chars

When prefix is too long (including auto-generated), cluster_id exceeds max allowed length

Error: module.stack.aws_elasticache_cluster.default: "cluster_id" ("yzx6jjxl-explorer-redis") must contain from 1 to 20 alphanumeric characters or hyphens

Can't exec "libtoolize": No such file or directory at

With the addition of the read smart contract functionality just merged on the poa-explorer repo, we are running into issues on the deploy script.

I had a similar error on my local machine. I'll try and add libtool to the build script.

[stdout]Can't exec "libtoolize": No such file or directory at /usr/share/autoconf/Autom4te/FileUtils.pm line 345, <GEN3> line 5.
[stdout]autoreconf: failed to run libtoolize: No such file or directory
[stdout]autoreconf: libtoolize is needed because this package uses Libtool
[stdout]===> Hook for compile failed!
[stdout]
[stderr]** (Mix) Could not compile dependency :libsecp256k1, "/home/ec2-user/.mix/rebar3 bare compile --paths "/opt/app/_build/prod/lib/*/ebin"" command failed. You can recompile this dependency with "mix deps.compile libsecp256k1", update it with "mix deps.update libsecp256k1" or clean it with "mix deps.clean libsecp256k1"

(Bug) Using default prefix with s3 bucket names

When starting the infra, user is asked to enter a prefix

What prefix should be used? (default is 6fzYlvFh):

If left as a default value, s3 bucket creation sometimes fail, probably due to prefix containing upper-case letters https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-s3-bucket-naming-requirements.html

Error: Error applying plan:

1 error(s) occurred:

* module.backend.aws_s3_bucket.terraform_state: 1 error(s) occurred:

* aws_s3_bucket.terraform_state: Error creating S3 bucket: InvalidBucketName: The specified bucket is not valid.
    status code: 400, request id: ..., host id: ...

It's probably best to allow only lower-case prefixes, since this error may happen with other resources too.

Access to s3 denied for codedeploy agent

(assuming #23 fixed)

Getting the following error in /var/log/aws/codedeploy-agent/codedeploy-agent.log:

2018-05-31 20:10:55 INFO  [codedeploy-agent(12373)]: [Aws::CodeDeployCommand::Client 200 0.029551 0 retries] put_host_command_complete(command_status:"Failed",diagnostics:{format:"JSON",payload:"{\"error_code\":5,\"script_name\":\"\",\"message\":\"Access Denied\",\"log\":\"\"}"},host_command_identifier:"WyJjb20uYW1hem9uLmFwb2xsby5kZXBsb3ljb250cm9sLmRvbWFpbi5Ib3N0Q29tbWFuZElkZW50aWZpZXIiLHsiZGVwbG95bWVudElkIjoiQ29kZURlcGxveS91cy1lYXN0LTIvUHJvZC9hcm46YXdzOnNkczp1cy1lYXN0LTI6NzU4MDExMTI3ODMyOmRlcGxveW1lbnQvZC1ZU1paWDE4MVNfUmV2aXNpb24iLCJob3N0SWQiOiJhcm46YXdzOmVjMjp1cy1lYXN0LTI6NzU4MDExMTI3ODMyOmluc3RhbmNlL2ktMDYyYzJhNGNjMGNiMDNjZWMiLCJjb21tYW5kTmFtZSI6IkRvd25sb2FkQnVuZGxlIiwiY29tbWFuZFBvc2l0aW9uIjoyLCJjb21tYW5kQXR0ZW1wdCI6MX1d")  

2018-05-31 20:10:55 ERROR [codedeploy-agent(12373)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Error during perform: Aws::S3::Errors::AccessDenied - Access Denied - /opt/codedeploy-agent/vendor/gems/aws-sdk-core-2.10.104/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
/opt/codedeploy-agent/vendor/gems/aws-sdk-core-2.10.104/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19:in `call'
/opt/codedeploy-agent/vendor/gems/aws-sdk-core-2.10.104/lib/aws-sdk-core/plugins/s3_dualstack.rb:24:in `call'
/opt/codedeploy-agent/vendor/gems/aws-sdk-core-2.10.104/lib/aws-sdk-core/plugins/s3_accelerate.rb:34:in `call'
/opt/codedeploy-agent/vendor/gems/aws-sdk-core-2.10.104/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
/opt/codedeploy-agent/vendor/gems/aws-sdk-core-2.10.104/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in `call'
/opt/codedeploy-agent/vendor/gems/aws-sdk-core-2.10.104/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
/opt/codedeploy-agent/vendor/gems/aws-sdk-core-2.10.104/lib/seahorse/client/plugins/response_target.rb:21:in `call'
/opt/codedeploy-agent/vendor/gems/aws-sdk-core-2.10.104/lib/seahorse/client/request.rb:70:in `send_request'
/opt/codedeploy-agent/vendor/gems/aws-sdk-core-2.10.104/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:274:in `block in download_from_s3'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:269:in `open'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:269:in `download_from_s3'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:82:in `block in <class:CommandExecutor>'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:68:in `execute_command'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_poller.rb:143:in `process_command'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_poller.rb:76:in `block in perform'
/opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `call'
/opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `run_task'
/opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in `block (3 levels) in create_worker'
/opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `loop'
/opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (2 levels) in create_worker'
/opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `catch'
/opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `block in create_worker'
/opt/codedeploy-agent/vendor/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call'
/opt/codedeploy-agent/vendor/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'

Not clear what is it trying to access

Error when attempting to verify smart contract

I'm currently receiving the following error when attempting to verify a smart contract:

Jul 05 14:01:21 ip-10-0-0-159.poa.internal explorer[4039]: 14:01:21.249 request_id=gujis2lcqv0ol5bf6gcdk64uq5vohe1u [info] Sent 500 in 7ms
Jul 05 14:01:21 ip-10-0-0-159.poa.internal explorer[4039]: 14:01:21.251 [error] #PID<0.8411.4> running ExplorerWeb.Endpoint terminated
Jul 05 14:01:21 ip-10-0-0-159.poa.internal explorer[4039]: Server: sokg-explorer-sokol-elb-682326660.us-east-1.elb.amazonaws.com:80 (http)
Jul 05 14:01:21 ip-10-0-0-159.poa.internal explorer[4039]: Request: POST /en/addresses/0x8974cd6822ff825ea9e47bef6416cbf73fd328b6/contract_verifications
Jul 05 14:01:21 ip-10-0-0-159.poa.internal explorer[4039]: ** (exit) an exception was raised:
Jul 05 14:01:21 ip-10-0-0-159.poa.internal explorer[4039]: ** (ErlangError) Erlang error: :enoent
Jul 05 14:01:21 ip-10-0-0-159.poa.internal explorer[4039]: (elixir) lib/system.ex:622: System.cmd("node", ["/opt/app/_build/prod/lib/explorer/priv/compile_solc.js", "contract SimpleStorage {\r\n    uint storedData;\r\n\r\n    function set(uint x) public {\r\n        storedData = x;\r\n    }\r\n\r\n    function get() public constant returns (uint) {\r\n        return storedData;\r\n    }\r\n}", "v0.4.24+commit.e67f0147", "0"], [])
Jul 05 14:01:21 ip-10-0-0-159.poa.internal explorer[4039]: (explorer) lib/explorer/smart_contract/solidity/code_compiler.ex:64: Explorer.SmartContract.Solidity.CodeCompiler.run/4
Jul 05 14:01:21 ip-10-0-0-159.poa.internal explorer[4039]: (explorer) lib/explorer/smart_contract/verifier.ex:24: Explorer.SmartContract.Verifier.evaluate_authenticity/2
Jul 05 14:01:21 ip-10-0-0-159.poa.internal explorer[4039]: (explorer) lib/explorer/smart_contract/publisher.ex:27: Explorer.SmartContract.Publisher.publish/2
Jul 05 14:01:21 ip-10-0-0-159.poa.internal explorer[4039]: (explorer_web) lib/explorer_web/controllers/address_contract_verification_controller.ex:24: ExplorerWeb.AddressContractVerificationController.create/2
Jul 05 14:01:21 ip-10-0-0-159.poa.internal explorer[4039]: (explorer_web) lib/explorer_web/controllers/address_contract_verification_controller.ex:1: ExplorerWeb.AddressContractVerificationController.action/2
Jul 05 14:01:21 ip-10-0-0-159.poa.internal explorer[4039]: (explorer_web) lib/explorer_web/controllers/address_contract_verification_controller.ex:1: ExplorerWeb.AddressContractVerificationController.phoenix_controller_pipeline/2
Jul 05 14:01:21 ip-10-0-0-159.poa.internal explorer[4039]: (explorer_web) lib/explorer_web/endpoint.ex:1: ExplorerWeb.Endpoint.instrument/4```

(Bug) Rerouting Traffic to replacement instance

Server and indexer start on the new instance but when I reroute traffic to this instance I receive a permissions error

Message

Role does not have correct permissions. role arn:aws:iam::290679793836:role/tel-deployer-role sessionName dHRQW18JHT. for activityId="5" of activityType={Name: ExecuteCentralizedCommandOnInstanceActivity.runCentralizedCommand,Version: 1.00}

Log Tail

com.amazonaws.services.simpleworkflow.flow.ActivityTaskFailedException: Role does not have correct permissions. role arn:aws:iam::290679793836:role/tel-deployer-role sessionName dHRQW18JHT.
 for activityId="5" of activityType={Name: ExecuteCentralizedCommandOnInstanceActivity.runCentralizedCommand,Version: 1.00}

Error message: Initialization required

The following error message is displayed: "Failed to load backend: Initialization required.", but the script continues to run, not sure if it's an actual error.
More detailed log:

...
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
+ grep 'main$'
+ terraform workspace list
Failed to load backend: Initialization required. Please see the error message above.
+ terraform workspace new main main
Created and switched to workspace "main"!
...

Change RPC Endpoint names

blockscout/blockscout#1006 changes the environment variables for the ws_url, ethereum_url, and trace_url.

Error during deployment via circleci

When deploying via circleCI I got the following error on AfterInstall step:

Error Code: ScriptFailed
Script Name: bin/deployment/build
Message: Script at specified location: bin/deployment/build run as user ec2-user failed with exit code 1
Log Tail:
[stdout]  idna 5.1.1
[stdout]  jason 1.0.0
[stdout]  jsx 2.8.3
[stdout]  junit_formatter 2.1.0
[stdout]  math 0.3.0
[stdout]  meck 0.8.9
[stdout]  metrics 1.0.1
[stdout]  mime 1.2.0
[stdout]  mimerl 1.0.2
[stdout]  mochiweb 2.15.0
[stdout]  mock 0.3.1
[stdout]  mox 0.3.2
[stdout]  parallel_stream 1.0.6
[stdout]  parse_trans 3.2.0
[stdout]  phoenix 1.3.0
[stdout]  phoenix_ecto 3.3.0
[stdout]  phoenix_html 2.10.5
[stdout]  phoenix_live_reload 1.1.3
[stdout]  phoenix_pubsub 1.0.2
[stdout]  plug 1.4.4
[stdout]  poison 3.1.0
[stdout]  poolboy 1.5.1
[stdout]  postgrex 0.13.5
[stdout]  qrcode 0.1.1
[stdout]  ranch 1.3.2
[stdout]  scrivener 2.5.0
[stdout]  scrivener_ecto 1.3.0
[stdout]  scrivener_html 1.7.1
[stdout]  sobelow 0.7.0
[stdout]  ssl_verify_fun 1.1.1
[stdout]  timex 3.1.25 RETIRED!
[stdout]  (invalid) Incorrect Elixir version requirement
[stdout]  timex_ecto 3.2.1
[stdout]  tzdata 0.5.16
[stdout]  unicode_util_compat 0.3.1
[stdout]  wallaby 0.20.0
[stderr]** (File.Error) could not make directory (with -p) "/opt/app/_build/prod/lib/parse_trans": no such file or directory
[stderr]    (elixir) lib/file.ex:271: File.mkdir_p!/1
[stderr]    (mix) lib/mix/tasks/deps.compile.ex:190: Mix.Tasks.Deps.Compile.do_rebar3/2
[stderr]    (mix) lib/mix/tasks/deps.compile.ex:85: anonymous fn/4 in Mix.Tasks.Deps.Compile.compile/2
[stderr]    (elixir) lib/enum.ex:1294: Enum."-map/2-lists^map/1-0-"/2
[stderr]    (mix) lib/mix/tasks/deps.compile.ex:65: Mix.Tasks.Deps.Compile.compile/2
[stderr]    (mix) lib/mix/tasks/deps.loadpaths.ex:90: Mix.Tasks.Deps.Loadpaths.deps_check/2
[stderr]    (mix) lib/mix/tasks/deps.loadpaths.ex:27: Mix.Tasks.Deps.Loadpaths.run/1
[stderr]    (mix) lib/mix/task.ex:314: Mix.Task.run_task/3

Question on access rights

README states that following permissions are required

VPCs and associated networking resources (subnets, routing tables, etc.)
Security Groups
EC2
S3
SSM
DynamoDB
Route53
RDS

however it looks like there are some more:

* aws_iam_role.deployer: Error creating IAM Role t1234-deployer-role: AccessDenied: User: arn:aws:iam::758011127832:user/test-poa-explorer is not authorized to perform: iam:CreateRole on resource: arn:aws:iam::758011127832:role/t1234-deployer-role

* aws_elasticache_subnet_group.redis: Error creating CacheSubnetGroup: AccessDenied: User: arn:aws:iam::758011127832:user/test-poa-explorer is not authorized to perform: elasticache:CreateCacheSubnetGroup
    status code: 403, request id: 29db6b78-4c8f-11e8-91ec-d7b7dc1ccbae

So that in total seems to give more than 10 permissions per user, which is not allowed by AWS, I had to give my use full access.

I'm opening this just to clarify if my thoughts are correct or there's another way to provide required permissions.

Add Prometheus and Grafana to cluster

Prometheus and Grafana have been added to blockscout in blockscout/blockscout#841, but the Prometheus and Grafana servers need to be setup in the cluster and configured to talk to each other and the BlockScout /metrics endpoint.

(Feature) Add optional support for aws profiles

awscli supports having multiple profiles in config, for each aws command the selected profile can be specified with --profile flag.
Please add an option to select the profile to use throughout infra run.

aws_autoscaling_group.explorer' not found for variable 'aws_autoscaling_group.explorer.name'

When trying to utilize:

chains = {
    "mychain" = "url/to/endpoint"
}
chain_trace_endpoints = {
    "mychain" = "url/to/debug/endpoint/or/the/main/chain/endpoint"
}

to deploy multiple chains you receive the following errors

* module.stack.aws_autoscaling_policy.explorer-up: Resource 'aws_autoscaling_group.explorer' not found for variable 'aws_autoscaling_group.explorer.name'
* module.stack.aws_autoscaling_policy.explorer-down: 1 error(s) occurred:

* module.stack.aws_autoscaling_policy.explorer-down: Resource 'aws_autoscaling_group.explorer' not found for variable 'aws_autoscaling_group.explorer.name'

To work around this you need to update the variables chains and chain_trace_endpoints in main/variables.tf manually. This only allows you to build a single chain at a time per terraform deploy.

infra destroy doesn't seem to work

So after creating the terraform.tfvars file, which allowed to skip all the prompts, I got past the previous problem, the one which is mentioned in the README:

The installer will prompt during its initial run to ask if you want to migrate the Terraform state to S3, this is a necessary step

That immediatey fails. But before I get into that, the second issue, which I believe is more serious, manifested itself:

➜ poa-explorer-infra git:(master) ✗ ./bin/infra destroy
Error: Error loading modules: module stack: not found, may need to run 'terraform init'

As now but the S3 bucket and the DinamoDB table are being successfully created, the cleanup is longer. Those two, and the local state:

akamac.home ➜ poa-explorer-infra git:(master) ✗ rm -rf .terraform
akamac.home ➜ poa-explorer-infra git:(master) ✗ rm -rf terraform.tfstate.d

destroy was supposed to take care of all 4, correct?

Now, the failure to migrate. The bucket does exist; I was able to copy the file there with aws s3 cp <s3_url> <local_file> - presumably nothing should be in the way of the 'provision' script, right?

The dialog is being very polite and explains everything that needs to happen. "Local" backend to the S3 one, which was just created. However,

Enter a value: yes

Error migrating the workspace "base" from the previous "local" backend to the newly
configured "s3" backend:
Error loading state:
failed to lock s3 state: 2 error(s) occurred:

ResourceNotFoundException: Requested resource not found
status code: 400, request id: F06O603P6T3AJPDNLBPAHQS60FVV4KQNSO5AEMVJF66Q9ASUAAJG

ResourceNotFoundException: Requested resource not found
status code: 400, request id: RATGBBS4B7FP1CJUNVCI4TQSMFVV4KQNSO5AEMVJF6

Very strange. Two errors, not one. One for a local, and one for remote? Neither is found? Well, "locally" I wasn't asked to create any state manually, so I assume the script and Terraform did everything; and remote bucket was created just fine..

Increase the size of the EC2 root_block_device

A few of the chains have run out of space. This volume is currently set at 8GB. I'm going to test setting this at 100GB.

Build Microsoft Azure Marketplace Deployment and Infrastructure

Create a replica of blockscout-terraform in the Microsoft Azure marketplace
Currently, at the end of the blockscout-terraform script we do not actually deploy blockscout, this should be changed in the Azure build script.

libtinfo.so.5()(64bit) is needed by esl-erlang-21.0-1.x86_64

All of a sudden while trying to deploy today all deployments were failing, even previous deployments that have already been successful in the past. After taking a look at the logs, I noticed that there is a dependency is now required to install erlang.

 warning: esl-erlang_21.0-1~centos~7_amd64.rpm: Header V4 RSA/SHA1 Signature, key ID a14f4fca: NOKEY
Oct 06 01:37:28 ip-10-0-0-158.poa.internal cloud-init[3276]: error: Failed dependencies:
Oct 06 01:37:28 ip-10-0-0-158.poa.internal cloud-init[3276]: libtinfo.so.5()(64bit) is needed by esl-erlang-21.0-1.x86_64

Clarify about migrating workspaces

During the setup, user is asked to migrate local workspace to s3

Do you want to migrate all workspaces to "s3"?
  Both the existing "local" backend and the newly configured "s3" backend support
  workspaces. When migrating between backends, Terraform will copy all
  workspaces (with the same names). THIS WILL OVERWRITE any conflicting
  states in the destination.
  
  Terraform initialization doesn't currently migrate only select workspaces.
  If you want to migrate a select number of workspaces, you must manually
  pull and push those states.
  
  If you answer "yes", Terraform will migrate all states. If you answer
  "no", Terraform will abort.

could you clarify what will happen during migration, what are possible downsides?

Validate db password length

db password must be at least 8 chars long

* aws_db_instance.default: Error creating DB Instance: InvalidParameterValue: The parameter MasterUserPassword is not a valid password because it is shorter than 8 characters.

explorer.service missing Hex dependency

(assuming #24 fixed)

deployment fails on ValidateService step. CodeDeploy agent logs:

Error Code: ScriptFailed
Script Name: bin/deployment/health_check
Message: Script at specified location: bin/deployment/health_check failed with exit code 124
Log Tail:
[stderr]curl: (7) Failed to connect to localhost port 80: Connection refused
[stderr]curl: (7) Failed to connect to localhost port 80: Connection refused

the following error is reported by explorer.service:

May 31 20:39:56 ip-10-0-0-56.poa.internal systemd[1]: Started POA Explorer.
May 31 20:39:56 ip-10-0-0-56.poa.internal systemd[1]: Starting POA Explorer...
May 31 20:39:57 ip-10-0-0-56.poa.internal explorer[12404]: warning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it exp
May 31 20:39:57 ip-10-0-0-56.poa.internal explorer[12404]: Could not find Hex, which is needed to build dependency :ex_doc
May 31 20:39:57 ip-10-0-0-56.poa.internal explorer[12404]: Shall I install Hex? (if running non-interactively, use "mix local.hex --force") [Yn] ** (Mix) Could not find 
May 31 20:39:57 ip-10-0-0-56.poa.internal systemd[1]: explorer.service: main process exited, code=exited, status=1/FAILURE
May 31 20:39:57 ip-10-0-0-56.poa.internal systemd[1]: Unit explorer.service entered failed state.
May 31 20:39:57 ip-10-0-0-56.poa.internal systemd[1]: explorer.service failed.
May 31 20:39:56 ip-10-0-0-56.poa.internal systemd[1]: Started POA Explorer.
May 31 20:39:56 ip-10-0-0-56.poa.internal systemd[1]: Starting POA Explorer...
May 31 20:39:57 ip-10-0-0-56.poa.internal explorer[12404]: warning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it exp
May 31 20:39:57 ip-10-0-0-56.poa.internal explorer[12404]: Could not find Hex, which is needed to build dependency :ex_doc
May 31 20:39:57 ip-10-0-0-56.poa.internal explorer[12404]: Shall I install Hex? (if running non-interactively, use "mix local.hex --force") [Yn] ** (Mix) Could not find 
May 31 20:39:57 ip-10-0-0-56.poa.internal systemd[1]: explorer.service: main process exited, code=exited, status=1/FAILURE
May 31 20:39:57 ip-10-0-0-56.poa.internal systemd[1]: Unit explorer.service entered failed state.
May 31 20:39:57 ip-10-0-0-56.poa.internal systemd[1]: explorer.service failed.

(Bug) native name encoding of latin1 which may cause Elixir to malfunction as it expects utf8

After explorer app starts I receive the following error:

Jun 01 22:19:46 ip-10-0-0-160.poa.internal explorer[13055]: warning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it expects utf8. Please ensure your locale is set to UTF-8 (which can be verified by running "locale" in your shell)
Jun 01 22:19:46 ip-10-0-0-160.poa.internal rsyslogd[2223]: imjournal: journal reloaded... [v8.24.0 try http://www.rsyslog.com/e/0 ]
Jun 01 22:19:47 ip-10-0-0-160.poa.internal explorer[13055]: 22:19:47.823 [info] Running ExplorerWeb.Endpoint with Cowboy using http://:::4000
Jun 01 22:19:47 ip-10-0-0-160.poa.internal explorer[13055]: 22:19:47.869 [error] Could not find static manifest at "/opt/app/_build/prod/lib/explorer_web/priv/static/cache_manifest.json". Run "mix phx.digest" after building your static files or remove the configuration from "config/prod.exs".
Jun 01 22:19:50 ip-10-0-0-160.poa.internal explorer[13055]: 22:19:50.531 [info] tzdata release in place is from a file last modified Mon, 20 Mar 2017 18:53:44 GMT. Release file on server was last modified Thu, 03 May 2018 23:55:14 GMT.
Jun 01 22:19:51 ip-10-0-0-160.poa.internal explorer[13055]: 22:19:51.247 [info] Tzdata has updated the release from 2017b to 2018e

Bug with terraform

I'm using

terraform --version
Terraform v0.11.7
+ provider.aws v1.16.0

and got the following error (I did a clean start, so it should be unrelated to other issues I had before):

module.stack.aws_elasticache_cluster.default: Creation complete after 5m43s (ID: a1234-explorer-redis)
module.stack.aws_ssm_parameter.redis_url: Creating...
  arn:    "" => "<computed>"
  key_id: "" => "<computed>"
  name:   "" => "/a1234/sokol/redis_url"
  type:   "" => "String"
  value:  "<sensitive>" => "<sensitive>"
module.stack.aws_ssm_parameter.redis_url: Creation complete after 7s (ID: /a1234/sokol/redis_url)
Releasing state lock. This may take a few moments...

Error: Error applying plan:

1 error(s) occurred:

* module.stack.aws_autoscaling_group.explorer: aws_autoscaling_group.explorer: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue.

Please include the following information in your report:

    Terraform Version: 0.11.7
    Resource ID: aws_autoscaling_group.explorer
    Mismatch reason: attribute mismatch: availability_zones.1252502072

403 Error during websocket handshake when deployed behind ELB

The following error is displayed in the browser console when the application is deployed behind ELB:
WebSocket connection to 'ws://sokaj-explorer-sokol-elb-1723354604.us-east-1.elb.amazonaws.com/socket/websocket?locale=en&vsn=2.0.0' failed: Error during WebSocket handshake: Unexpected response code: 403

journalctl shows the following error:

Jul 27 17:36:00 ...[16337]: 17:36:00.450 [error] Could not check origin for Phoenix.Socket transport.
Jul 27 17:36:00 ...[16337]: This happens when you are attempting a socket connection to
Jul 27 17:36:00 ...[16337]: a different host than the one configured in your config/
Jul 27 17:36:00 ...[16337]: files. For example, in development the host is configured
Jul 27 17:36:00 ...[16337]: to "localhost" but you may be trying to access it from
Jul 27 17:36:00 ...[16337]: "127.0.0.1". To fix this issue, you may either:
Jul 27 17:36:00 ...[16337]: 1. update [url: [host: ...]] to your actual host in the
Jul 27 17:36:00 ...[16337]: config file for your current environment (recommended)
Jul 27 17:36:00 ...[16337]: 2. pass the :check_origin option when configuring your
Jul 27 17:36:00 ...[16337]: endpoint or when configuring the transport in your
Jul 27 17:36:00 ...[16337]: UserSocket module, explicitly outlining which origins
Jul 27 17:36:00 ...[16337]: are allowed:
Jul 27 17:36:00 ...[16337]: check_origin: ["https://example.com",
Jul 27 17:36:00 ...[16337]: "//another.com:888", "//other.com"]

Add more parameters to configure each chain

(Bug) S3 bucket not found

The following error appears during infra run:

...
+ terraform init -backend-config=backend.tfvars base
Initializing modules...
- module.backend

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
Error loading state: NoSuchBucket: The specified bucket does not exist
	status code: 404, request id: ..., host id: ...

Indeed, no bucket was created.

(Feature) backend.tfvars and main.tfvars overriding terraform.tfvars file

If a user provides the terraform.tfvars file, backend.tfvars and main.tfvars should not override variables found in terraform.tfvars.

Add Heart Env Variables

Heartbeat is an Erlang monitoring service that will restart explorer if it becomes unresponsive. We need to add HEART_BEAT_TIMEOUT and HEART_COMMAND to Terraform to activate heart.

Error when provisioning multiple chains

When provisioning infra with multiple chains, I got the following error:

data.aws_iam_policy_document.instance-assume-role-policy: Refreshing state...
data.aws_iam_policy_document.deployer-assume-role-policy: Refreshing state...
data.aws_availability_zones.available: Refreshing state...
data.aws_ami.explorer: Refreshing state...
data.aws_iam_policy.AmazonEC2RoleForAWSCodeDeploy: Refreshing state...
data.aws_iam_policy.AmazonEC2RoleForSSM: Refreshing state...
data.aws_iam_policy_document.config-policy: Refreshing state...
data.aws_iam_policy.AWSCodeDeployRole: Refreshing state...

------------------------------------------------------------------------
Releasing state lock. This may take a few moments...

Error: Error running plan: 2 error(s) occurred:

* module.stack.aws_autoscaling_policy.explorer-up: 1 error(s) occurred:

* module.stack.aws_autoscaling_policy.explorer-up: Resource 'aws_autoscaling_group.explorer' not found for variable 'aws_autoscaling_group.explorer.name'
* module.stack.aws_autoscaling_policy.explorer-down: 1 error(s) occurred:

* module.stack.aws_autoscaling_policy.explorer-down: Resource 'aws_autoscaling_group.explorer' not found for variable 'aws_autoscaling_group.explorer.name'

To include multiple chains I added the following lines to terraform.tfvars:

chains = {
    "sokol" = "https://sokol.poa.network"
    "core" = "https://core.poa.network"
}

chain_trace_endpoints = {
    "sokol" = "https://sokol-trace.poa.network"
    "core" = "https://core-trace.poa.network"
}

Add a Listener on Port 443

In order to create a secure connection, we must add a listener on port 443.

Add Validator Variables for POA chains

Metadata Contract
Validators Contract

Increase the number of connections to the RDS instance

Create a custom variable for pool_size in modules/stack/config.tf.

Add hostname to the ec2 instance to access observer

We need to be able to identify where certain chains are hanging. Observer is a great monitoring tool and needs to be accessed remotely. A hostname is required to accomplish this.

Add Deployed Version env var

This environment variable is used in the footer of BlockScout and should be set during deployment.

More rights required by -explorer-role

Getting the following errors in /var/log/amazon/ssm/error.log:

2018-05-31 17:57:41 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-05939a0faa0bbb4d3] [MessagingDeliveryService] error when calling AWS APIs. error details - GetMessages Error: AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-05939a0faa0bbb4d3 is not authorized to perform: ec2messages:GetMessages on resource: *
    status code: 400, request id: 1d0c1fbb-64fc-11e8-9d96-95f629114d32
2018-05-31 17:57:41 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-05939a0faa0bbb4d3] [HealthCheck] error when calling AWS APIs. error details - AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-05939a0faa0bbb4d3 is not authorized to perform: ssm:UpdateInstanceInformation on resource: arn:aws:ec2:us-east-2:758011127832:instance/i-05939a0faa0bbb4d3
    status code: 400, request id: 6681f3a3-7c19-4982-9816-52498e9ddb32
2018-05-31 17:57:41 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-05939a0faa0bbb4d3] [HealthCheck] error when calling AWS APIs. error details - AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-05939a0faa0bbb4d3 is not authorized to perform: ssm:UpdateInstanceInformation on resource: arn:aws:ec2:us-east-2:758011127832:instance/i-05939a0faa0bbb4d3
    status code: 400, request id: 6681f3a3-7c19-4982-9816-52498e9ddb32
2018-05-31 17:57:43 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-05939a0faa0bbb4d3] [MessagingDeliveryService] error when calling AWS APIs. error details - GetMessages Error: AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-05939a0faa0bbb4d3 is not authorized to perform: ec2messages:GetMessages on resource: *
    status code: 400, request id: 1e40ceee-64fc-11e8-9d96-95f629114d32
2018-05-31 17:57:45 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-05939a0faa0bbb4d3] [MessagingDeliveryService] error when calling AWS APIs. error details - GetMessages Error: AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-05939a0faa0bbb4d3 is not authorized to perform: ec2messages:GetMessages on resource: *
    status code: 400, request id: 1f61f67b-64fc-11e8-9d96-95f629114d32
2018-05-31 17:57:47 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-05939a0faa0bbb4d3] [MessagingDeliveryService] error when calling AWS APIs. error details - GetMessages Error: AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-05939a0faa0bbb4d3 is not authorized to perform: ec2messages:GetMessages on resource: *
    status code: 400, request id: 20b8d43d-64fc-11e8-9d96-95f629114d32
2018-05-31 17:57:50 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-05939a0faa0bbb4d3] [MessagingDeliveryService] error when calling AWS APIs. error details - GetMessages Error: AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-05939a0faa0bbb4d3 is not authorized to perform: ec2messages:GetMessages on resource: *
    status code: 400, request id: 2223ae5c-64fc-11e8-9d96-95f629114d32
2018-05-31 17:57:52 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-05939a0faa0bbb4d3] [MessagingDeliveryService] error when calling AWS APIs. error details - GetMessages Error: AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-05939a0faa0bbb4d3 is not authorized to perform: ec2messages:GetMessages on resource: *
    status code: 400, request id: 235e2a1d-64fc-11e8-9d96-95f629114d32
2018-05-31 17:57:54 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-05939a0faa0bbb4d3] [MessagingDeliveryService] error when calling AWS APIs. error details - GetMessages Error: AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-05939a0faa0bbb4d3 is not authorized to perform: ec2messages:GetMessages on resource: *
    status code: 400, request id: 24dc3ea2-64fc-11e8-9d96-95f629114d32
2018-05-31 17:57:57 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-05939a0faa0bbb4d3] [MessagingDeliveryService] error when calling AWS APIs. error details - GetMessages Error: AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-05939a0faa0bbb4d3 is not authorized to perform: ec2messages:GetMessages on resource: *
    status code: 400, request id: 264d5ad6-64fc-11e8-9d96-95f629114d32
2018-05-31 17:57:57 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-05939a0faa0bbb4d3] [MessagingDeliveryService] [Association] error when calling AWS APIs. error details - AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-05939a0faa0bbb4d3 is not authorized to perform: ssm:ListInstanceAssociations on resource: arn:aws:ec2:us-east-2:758011127832:instance/i-05939a0faa0bbb4d3
    status code: 400, request id: 59f7a442-a381-4007-beb6-12c6a431cee9
2018-05-31 17:57:57 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-05939a0faa0bbb4d3] [MessagingDeliveryService] [Association] error when calling AWS APIs. error details - AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-05939a0faa0bbb4d3 is not authorized to perform: ssm:ListAssociations on resource: arn:aws:ssm:us-east-2:758011127832:*
    status code: 400, request id: 507f7991-6804-479f-b274-140fe0c6db18
2018-05-31 17:57:57 ERROR [ProcessAssociation @ processor.go.157] [instanceID=i-05939a0faa0bbb4d3] [MessagingDeliveryService] [Association] Unable to load instance associations, unable to retrieve associations unable to retrieve associations AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-05939a0faa0bbb4d3 is not authorized to perform: ssm:ListAssociations on resource: arn:aws:ssm:us-east-2:758011127832:*
    status code: 400, request id: 507f7991-6804-479f-b274-140fe0c6db18
2018-05-31 17:57:59 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-05939a0faa0bbb4d3] [MessagingDeliveryService] error when calling AWS APIs. error details - GetMessages Error: AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-05939a0faa0bbb4d3 is not authorized to perform: ec2messages:GetMessages on resource: *
    status code: 400, request id: 27af832a-64fc-11e8-9d96-95f629114d32
2018-05-31 17:58:01 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-05939a0faa0bbb4d3] [MessagingDeliveryService] error when calling AWS APIs. error details - GetMessages Error: AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-05939a0faa0bbb4d3 is not authorized to perform: ec2messages:GetMessages on resource: *
    status code: 400, request id: 2929a047-64fc-11e8-9d96-95f629114d32
2018-05-31 17:58:04 ERROR [loop @ scheduler.go.56] [instanceID=i-05939a0faa0bbb4d3] [MessagingDeliveryService] MessagingDeliveryService stopped temporarily due to internal failure. We will retry automatically after 15 minutes
2018-05-31 18:00:17 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-05939a0faa0bbb4d3] [HealthCheck] error when calling AWS APIs. error details - AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-05939a0faa0bbb4d3 is not authorized to perform: ssm:UpdateInstanceInformation on resource: arn:aws:ec2:us-east-2:758011127832:instance/i-05939a0faa0bbb4d3
    status code: 400, request id: e70201bf-2b67-44f2-818a-4663ee5ff5c3
2018-05-31 18:00:17 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-05939a0faa0bbb4d3] [HealthCheck] error when calling AWS APIs. error details - AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-05939a0faa0bbb4d3 is not authorized to perform: ssm:UpdateInstanceInformation on resource: arn:aws:ec2:us-east-2:758011127832:instance/i-05939a0faa0bbb4d3
    status code: 400, request id: e70201bf-2b67-44f2-818a-4663ee5ff5c3

This is the policy that worked for me eventually:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": "ssm:DescribeParameters",
            "Resource": "*"
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "ssm:GetParametersByPath",
                "ssm:GetParameters",
                "ssm:GetParameter"
            ],
            "Resource": [
                "arn:aws:ssm:*:*:parameter/$PREFIX/*/*",
                "arn:aws:ssm:*:*:parameter/$PREFIX/*"
            ]
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": "ec2:DescribeTags",
            "Resource": "*"
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": "ec2messages:GetMessages",
            "Resource": "*"
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "ssm:UpdateInstanceInformation",
                "ssm:ListInstanceAssociations"
            ],
            "Resource": "arn:aws:ec2:*"
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": "ssm:ListAssociations",
            "Resource": "arn:aws:ssm:*"
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::aws-codedeploy-us-west-2/*",
                "arn:aws:s3:::aws-codedeploy-us-west-1/*",
                "arn:aws:s3:::aws-codedeploy-us-east-2/*",
                "arn:aws:s3:::aws-codedeploy-us-east-1/*",
                "arn:aws:s3:::aws-codedeploy-sa-east-1/*",
                "arn:aws:s3:::aws-codedeploy-eu-west-1/*",
                "arn:aws:s3:::aws-codedeploy-eu-central-1/*",
                "arn:aws:s3:::aws-codedeploy-ap-southeast-2/*",
                "arn:aws:s3:::aws-codedeploy-ap-southeast-1/*",
                "arn:aws:s3:::aws-codedeploy-ap-south-1/*",
                "arn:aws:s3:::aws-codedeploy-ap-northeast-2/*",
                "arn:aws:s3:::aws-codedeploy-ap-northeast-1/*"
            ]
        }
    ]
}

Insufficient right for -explorer-role

*-explorer-role doesn't have enough access rights, this is error from /var/log/amazon/ssm/errors.log:

2018-05-30 12:43:59 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-092dff920f2db8f96] [HealthCheck] error when calling AWS APIs. error details - AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-092dff920f2db8f96 is not authorized to perform: ssm:UpdateInstanceInformation on resource: arn:aws:ec2:us-east-2:758011127832:instance/i-092dff920f2db8f96
    status code: 400, request id: cf26d018-6e15-4073-bef6-2cf07464d6ba
2018-05-30 12:43:59 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-092dff920f2db8f96] [HealthCheck] error when calling AWS APIs. error details - AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-092dff920f2db8f96 is not authorized to perform: ssm:UpdateInstanceInformation on resource: arn:aws:ec2:us-east-2:758011127832:instance/i-092dff920f2db8f96
    status code: 400, request id: cf26d018-6e15-4073-bef6-2cf07464d6ba
2018-05-30 12:43:59 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-092dff920f2db8f96] [MessagingDeliveryService] error when calling AWS APIs. error details - GetMessages Error: AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-092dff920f2db8f96 is not authorized to perform: ec2messages:GetMessages on resource: *
    status code: 400, request id: 1fc9d6af-6407-11e8-af36-31d19929b99f
2018-05-30 12:44:01 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-092dff920f2db8f96] [MessagingDeliveryService] error when calling AWS APIs. error details - GetMessages Error: AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-092dff920f2db8f96 is not authorized to perform: ec2messages:GetMessages on resource: *
    status code: 400, request id: 2132b4c6-6407-11e8-af36-31d19929b99f
2018-05-30 12:44:04 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-092dff920f2db8f96] [MessagingDeliveryService] error when calling AWS APIs. error details - GetMessages Error: AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-092dff920f2db8f96 is not authorized to perform: ec2messages:GetMessages on resource: *
    status code: 400, request id: 228302da-6407-11e8-af36-31d19929b99f
2018-05-30 12:44:06 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-092dff920f2db8f96] [MessagingDeliveryService] error when calling AWS APIs. error details - GetMessages Error: AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-092dff920f2db8f96 is not authorized to perform: ec2messages:GetMessages on resource: *
    status code: 400, request id: 23eca594-6407-11e8-af36-31d19929b99f
2018-05-30 12:44:08 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-092dff920f2db8f96] [MessagingDeliveryService] error when calling AWS APIs. error details - GetMessages Error: AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-092dff920f2db8f96 is not authorized to perform: ec2messages:GetMessages on resource: *
    status code: 400, request id: 2543a9cb-6407-11e8-af36-31d19929b99f
2018-05-30 12:44:10 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-092dff920f2db8f96] [MessagingDeliveryService] error when calling AWS APIs. error details - GetMessages Error: AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-092dff920f2db8f96 is not authorized to perform: ec2messages:GetMessages on resource: *
    status code: 400, request id: 267ceca8-6407-11e8-af36-31d19929b99f
2018-05-30 12:44:12 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-092dff920f2db8f96] [MessagingDeliveryService] error when calling AWS APIs. error details - GetMessages Error: AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-092dff920f2db8f96 is not authorized to perform: ec2messages:GetMessages on resource: *
    status code: 400, request id: 27bf0a00-6407-11e8-af36-31d19929b99f
2018-05-30 12:44:15 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-092dff920f2db8f96] [MessagingDeliveryService] error when calling AWS APIs. error details - GetMessages Error: AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-092dff920f2db8f96 is not authorized to perform: ec2messages:GetMessages on resource: *
    status code: 400, request id: 2908051c-6407-11e8-af36-31d19929b99f
2018-05-30 12:44:17 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-092dff920f2db8f96] [MessagingDeliveryService] error when calling AWS APIs. error details - GetMessages Error: AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-092dff920f2db8f96 is not authorized to perform: ec2messages:GetMessages on resource: *
    status code: 400, request id: 2a6e9969-6407-11e8-af36-31d19929b99f
2018-05-30 12:44:18 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-092dff920f2db8f96] [MessagingDeliveryService] [Association] error when calling AWS APIs. error details - AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-092dff920f2db8f96 is not authorized to perform: ssm:ListInstanceAssociations on resource: arn:aws:ec2:us-east-2:758011127832:instance/i-092dff920f2db8f96
    status code: 400, request id: 6e892baf-be10-4bb3-b5ae-6324f2ee3500
2018-05-30 12:44:18 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-092dff920f2db8f96] [MessagingDeliveryService] [Association] error when calling AWS APIs. error details - AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-092dff920f2db8f96 is not authorized to perform: ssm:ListAssociations on resource: arn:aws:ssm:us-east-2:758011127832:*
    status code: 400, request id: c7898419-d08f-490b-9935-040ca04a35f3
2018-05-30 12:44:18 ERROR [ProcessAssociation @ processor.go.157] [instanceID=i-092dff920f2db8f96] [MessagingDeliveryService] [Association] Unable to load instance associations, unable to retrieve associations unable to retrieve associations AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-092dff920f2db8f96 is not authorized to perform: ssm:ListAssociations on resource: arn:aws:ssm:us-east-2:758011127832:*
    status code: 400, request id: c7898419-d08f-490b-9935-040ca04a35f3
2018-05-30 12:44:19 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-092dff920f2db8f96] [MessagingDeliveryService] error when calling AWS APIs. error details - GetMessages Error: AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-092dff920f2db8f96 is not authorized to perform: ec2messages:GetMessages on resource: *
    status code: 400, request id: 2bcbb8a1-6407-11e8-af36-31d19929b99f
2018-05-30 12:44:22 ERROR [loop @ scheduler.go.56] [instanceID=i-092dff920f2db8f96] [MessagingDeliveryService] MessagingDeliveryService stopped temporarily due to internal failure. We will retry automatically after 15 minutes
2018-05-30 12:44:31 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-092dff920f2db8f96] [HealthCheck] error when calling AWS APIs. error details - AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-092dff920f2db8f96 is not authorized to perform: ssm:UpdateInstanceInformation on resource: arn:aws:ec2:us-east-2:758011127832:instance/i-092dff920f2db8f96
    status code: 400, request id: cf6d1db4-8b9f-41b7-94c0-33bd4d72013e
2018-05-30 12:44:31 ERROR [HandleAwsError @ awserr.go.48] [instanceID=i-092dff920f2db8f96] [HealthCheck] error when calling AWS APIs. error details - AccessDeniedException: User: arn:aws:sts::758011127832:assumed-role/stag0-explorer-role/i-092dff920f2db8f96 is not authorized to perform: ssm:UpdateInstanceInformation on resource: arn:aws:ec2:us-east-2:758011127832:instance/i-092dff920f2db8f96
    status code: 400, request id: cf6d1db4-8b9f-41b7-94c0-33bd4d72013e

(Mix) The task "phx.server" could not be found.

Hello, during deployment of blockscout-terraform on a fresh AWS account I noticed the web application was not starting. I confirmed that the userdata was ran on the EC2 instance, the explorer.service was created, but the service will not start due to the error. "(Mix) The task "phx.server" could not be found." I looked at the unit-file and it indicated:

WorkingDirectory=/opt/app
ExecStart=/opt/elixir/bin/mix phx.server

/opt/app was an empty folder, and i was not able to find the file phx.server anywhere on the local file system. I was able to get blockscout working by manually installing it on the ec2 instance. I followed the guide here. https://github.com/poanetwork/blockscout/blob/master/README.md I would like to know if anyone else has encountered this error when trying to deploy blockscout via terraform.

Could not compile dependency :keccakf1600

While testing #42 I ran into this problem while CodeDeploy was installing dependencies.

[stdout]===> Compiling keccakf1600
[stdout]make: Entering directory `/opt/app/deps/keccakf1600/c_src'
[stdout] C      keccakf1600_nif.c
[stdout] LD     keccakf1600.so
[stdout]/opt/app/deps/keccakf1600/c_src/decaf-utils.o: file not recognized: File format not recognized
[stdout]collect2: error: ld returned 1 exit status
[stdout]make: *** [/opt/app/deps/keccakf1600/c_src/../priv/keccakf1600.so] Error 1
[stdout]make: Leaving directory `/opt/app/deps/keccakf1600/c_src'
[stdout]===> Hook for compile failed!
[stdout]
[stderr]** (Mix) Could not compile dependency :keccakf1600, "/home/ec2-user/.mix/rebar3 bare compile --paths "/opt/app/_build/prod/lib/*/ebin"" command failed. You can recompile this dependency with "mix deps.compile keccakf1600", update it with "mix deps.update keccakf1600" or clean it with "mix deps.clean keccakf1600"

bug is mentioned in README, but

...but the solution proposed doesn't make sense:

Error inspecting states in the "s3" backend:
    NoSuchBucket: The specified bucket does not exist
    status code: 404, request id: xxxxxxxx, host id: xxxxxxxx

Prior to changing backends, Terraform inspects the source and destination
states to determine what kind of migration steps need to be taken, if any.
Terraform failed to load the states. The data in both the source and the
destination remain unmodified. Please resolve the above error and try again.

This is due to mismatched variables in terraform.tfvars and main.tfvars files. Update the terraform.tfvars file to match the main.tfvars file.

what is this terraform.tfvar file, then? It doesn't exist anywhere on the box, and specifically in the cloned repo, before of after the provisioning (that fails with this error). That said, I did inspect the base/terraform.tfstate, and found a suspicious line (3rd below):

"bucket": "akar-poa-state",
"bucket_domain_name": "akar-poa-state.s3.amazonaws.com",
"bucket_regional_domain_name": "akar-poa-state.s3.us-east-2.amazonaws.com",

This is indeed the prefix I've provided, and my bucket name. But this region has nothing to do with my account; the bin/infra script never asked me for my region, neither did the instructions specify any region as required. My ~/.aws/config has this region specified:

[default]
region = ca-central-1
output = json

and indeed, the provision step reports bucket and ddb tables being successfully created, and I do see them (in my real region).

poanetwork / blockscout-terraform Goto Github PK

blockscout-terraform's Introduction

About

License

blockscout-terraform's People

Contributors

Stargazers

Watchers

Forkers

blockscout-terraform's Issues

Recommend Projects

Recommend Topics

Recommend Org