Comments (9)
Fixed in cq-provider-aws v0.2.1 cloudquery/cq-provider-aws#8
from cloudquery.
Hello, I am not able to perform a fetch on my environnement, it fails on random operations.
I assume this is due to a rate limit, is there any way to show more debug output ?
2021/03/31 14:06:35 rpc error: code = Unknown desc = operation error IAM: GenerateCredentialReport, request canceled, context deadline exceeded
ERROR: 1
Error: rpc error: code = Unknown desc = operation error S3: GetBucketCors, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Error: rpc error: code = Unknown desc = operation error IAM: GetAccountAuthorizationDetails, https response error StatusCode: 0, RequestID: , canceled, context deadline exceeded
Used : cloudquery_Linux_x86_64-v0.11.6 / cq-provider-aws_linux_amd64-v0.2.17
from cloudquery.
@Rackme can you try change the following settings:
aws_debug: true
max_retries: 5
max_backoff: 30
timeout: 30
maybe change timeout to 300 ?
from cloudquery.
I tried with 30 and 300s for the timeout, with the following configuration, it's still failing on random operation :
providers:
- name: aws
version: latest
accounts:
- id: production
- id: sharedstorage
- id: test-us
- id: test-eu
- id: test-rds
- id: prod-kms
- id: uballot
- id: mmkpm
# regions: # Optional. if commented out assumes all regions
# - us-east-1
# - us-west-2
log_level: debug # Optional. if commented out will enable AWS SDK debug logging. possible values: debug, debug_with_signing, debug_with_http_body, debug_with_request_retries, debug_with_request_error, debug_with_event_stream_body
aws_debug: true
max_retries: 5
max_backoff: 30
timeout: 300
resources: # You can comment resources your are not interested in for faster fetching.
*
Unfortunately, I don't have the associated failed request-id :
11:21AM INF Fetched resources @module=aws account_id=12345678912 count=0 region=sa-east-1 resource=fsx.backups timestamp=2021-04-01T11:21:06.750Z
11:21AM INF Fetched resources @module=aws account_id=12345678912 count=1 region=sa-east-1 resource=ec2.security_groups timestamp=2021-04-01T11:21:06.751Z
SDK 2021/04/01 11:21:06 DEBUG Response
HTTP/2.0 200 OK
Content-Length: 14
Content-Type: application/x-amz-json-1.1
Date: Thu, 01 Apr 2021 11:21:06 GMT
X-Amzn-Requestid: aa6764e5-de84-4321-87ed-dec999a19428
SDK 2021/04/01 11:21:06 DEBUG Response
HTTP/1.1 200 OK
Content-Length: 210
Content-Type: application/x-amz-json-1.1
Date: Thu, 01 Apr 2021 11:21:06 GMT
X-Amzn-Requestid: 647c8634-da01-4ca4-bf66-2ddc95df1233
11:21AM INF Fetched resources @module=aws account_id=12345678912 count=1 region=sa-east-1 resource=ec2.network_acls timestamp=2021-04-01T11:21:06.768Z
11:21AM INF Fetched resources @module=aws account_id=12345678912 count=1 region=sa-east-1 resource=cloudtrail.trails timestamp=2021-04-01T11:21:06.776Z
11:21AM INF Fetched resources @module=aws account_id=12345678912 count=1 region=sa-east-1 resource=ec2.vpcs timestamp=2021-04-01T11:21:06.783Z
11:21AM INF Fetched resources @module=aws account_id=12345678912 count=0 region=sa-east-1 resource=sns.subscriptions timestamp=2021-04-01T11:21:06.841Z
SDK 2021/04/01 11:21:06 DEBUG Response
HTTP/1.1 200 OK
Content-Length: 291
Content-Type: text/xml
Date: Thu, 01 Apr 2021 11:21:05 GMT
X-Amzn-Requestid: 53c9f70e-63ad-55e9-8a85-afd1cd4fa10a
Error: rpc error: code = Unknown desc = operation error IAM: ListRoleTags, https response error StatusCode: 0, RequestID: , canceled, context deadline exceeded
11:21AM INF Fetched resources @module=aws account_id=12345678912 count=0 region=sa-east-1 resource=ecs.cluster timestamp=2021-04-01T11:21:06.901Z
SDK 2021/04/01 11:21:06 DEBUG Response
HTTP/1.1 200
Content-Length: 108
Content-Type: application/x-amz-json-1.1
Date: Thu, 01 Apr 2021 11:21:06 GMT
X-Amzn-Requestid: 042f8c02-e5e8-4546-8952-c225be8da1b1
Usage:
cloudquery fetch [flags]
Flags:
--driver string database driver postgresql/neo4j (env: CQ_DRIVER) (default "postgresql")
--dsn string database connection string (env: CQ_DSN) (example: 'host=localhost user=postgres password=pass DB.name=postgres port=5432')
-h, --help help for fetch
--path string path to configuration file. can be generated with 'gen config' command (env: CQ_CONFIG_PATH) (default "./config.yml")
--version version for fetch
Global Flags:
--enableConsoleLog Enable console logging (default true)
--enableFileLogging enableFileLogging makes the framework logging to a file (default true)
--encodeLogsAsJson EncodeLogsAsJson makes the logging framework logging JSON
--logDirectory string Directory to logging to to when file logging is enabled (default ".")
--logFile string Filename is the name of the logfile which will be placed inside the directory (default "cloudquery.log")
--maxAge int MaxAge the max age in days to keep a logfile (default 3)
--maxBackups int MaxBackups the max number of rolled files to keep (default 3)
--maxSize int MaxSize the max size in MB of the logfile before it's rolled (default 30)
--plugin-dir string Directory to save and load Cloudquery plugins from (env: CQ_PLUGIN_DIR) (default "/root")
-v, --verbose Enable Verbose logging
2021/04/01 11:21:06 rpc error: code = Unknown desc = operation error IAM: ListRoleTags, https response error StatusCode: 0, RequestID: , canceled, context deadline exceeded
root@59f586c902df:~#
from cloudquery.
After how long it fails? if it fails after 5min then I think you are hitting the timeout (can you try increasing to 1200?) if it's throwing the error before 5min are passed then it might be a different issue.
from cloudquery.
Started at :
11:20AM INF No regions specified in config.yml. Assuming all 22 regions @module=aws timestamp=2021-04-01T11:20:00.526Z
11:20AM INF Configuring SDK retryer @module=aws max_backoff=30 retry_attempts=5 timestamp=2021-04-01T11:20:00.526Z
SDK 2021/04/01 11:20:00 DEBUG Request
Failed at :
SDK 2021/04/01 11:21:06 DEBUG Response
: '(
from cloudquery.
Got it. Can you please try increasing the timeout to 3000 and also commenting out some of the accounts? just to understand if it's a specific account or not. for example try running on 1 account only?
from cloudquery.
All good with the 8 accounts and a timeout at 3000 👍
Total execution time : 2021-04-01T12:02:28.017Z -> 2021-04-01T12:04:13.869Z = 104s
from cloudquery.
@Rackme I also fixed some bugs with how contexts are being passed as well as a better default for timeouts so this should be solved with https://github.com/cloudquery/cq-provider-aws/releases/tag/v0.2.18
from cloudquery.
Related Issues (20)
- bug: Running Sequential Syncs with MySQL results in error
- bug: Build: `CGO_ENABLED` doesn't seem to have an effect in the release process HOT 1
- feat: Document a "rich" docker image is needed to run the SQLite, DuckDB and Snowflake plugins
- Feedback for Official Stripe Plugin. HOT 12
- feat: Support more Reserved Instances tables HOT 2
- bug: Azure Compliance pack - Issue with Model / View in Snowflake HOT 5
- bug: `snyk_sbom` table only syncs data from last organisation HOT 6
- feat: Update CLI docs for addon downloads HOT 2
- feat(github-source): Make issues table incremental based on `updated` field HOT 3
- feat(gcp): Support GCS HMAC Keys HOT 1
- bug: `azure_compute_restore_point_collections` is missing list of restore points
- bug: Oracle: missing defined_tags on oracle_compute_instances resource HOT 7
- feat: Support Azure storage account access keys resource
- Validate Snowflake DSN
- bug: aws_s3_buckets does not specify Server Access log format HOT 3
- bug: Github Plugin not returning repo custom properties HOT 3
- bug: Jira, not showing issue parentage HOT 4
- feat: PostgreSQL destination delete stale requires sequential scan HOT 3
- bug: Postgres source is not recognizing partitioned tables
- feat: Log failed message on column resolver exception in Plugin SDK
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cloudquery.