aws/amazon-ecs-logs-collector

The script will be used to collect general os logs as well as Docker and ecs-agent logs, it also support to enable debug mode for docker and ecs-agent in Amazon Linux.

amazon-ecs-logs-collector's Issues

Change variable infodir to use instanceId rather than the word collect

To make it easier to associate an extracted zip file with a specific instance, if we can make:
infodir="${curdir}/collect"
to something more like:
infodir="${curdir}/${instanceId}"
this would help a lot when viewing multiple instances logs. Perhaps even with the collection date in-case multiple logs from the one instance over multiple times of collection.

Feature Request: dmesg logs Human Readable timestamps

Summary

The dmesg logs can be collected with Human readable timestamps with a simple flag. "-T"

Description

In the "get_kernel_logs()" function, change "dmesg" to "dmesg -T"

Expected Behavior

Observed Behavior

Update log collector to work with AL2 (and other systemd based things)

Summary

The logs collector today is focused on Amazon Linux 1. However, Amazon Linux 2 is systemd based which means log collection should happen via journald rather than via /var/log/.

Consider collecting aws-appmesh.log

The recent App Mesh CNI integration with proxyConfiguration logs to /var/log/ecs/aws-appmesh.log.yyy-mm-dd-hh

Raising this to create awareness and determine if this is useful to collect in the ecs-logs-collector

Creates wrong archive type

This may be a minor issue, but, this script produces a tgz, which can not be uploaded to GitHub. So the user has to uncompress and create a zip file.

Add DateTime to the collect-i-*.tgz file name.

Summary

Add the Date Time YYYYMMDDHHmm to the file name, to if automated differ the time of the snapshot that the logs was collected from the same Container Instance

Description

• Add the Date Time YYYYMMDDHHmm to the file name, to if automated differ the time of the snapshot that the logs was collected from the same Container Instance

Detect required tools and optionally install if missing

Summary

Command line tools required by this script are checked at the time of the invocation, it would be more fitting to check their presence at the beginning and either notify the user what is missing and optionally offer to install them. Additional checks for certain functionalities (flag support, rudimentary version requirements) may be added based on usages but not necessary for consideration with this issue.

Unquoted paths could result in incorrect output

Summary

The unquoted paths in the ecs-logs-collector.sh could result in the script breaking due to unanticipated spaces or could place files in unexpected locations.

Script didnt generate tarball..

Summary

ecs-logs-collector script did not create tarball.

Description

Script Documentation details require update with right tarball location.

Expected Behavior

collect.tgz file is exepcted in the path where script is executed.

Observed Behavior

tarball with instancename.tgz is observed in collect folder.

Below are additional details..

Trying to collect Docker daemon logs ... ok
Trying to collect Amazon ECS Container Agent logs ... ok
Trying to collect Amazon ECS Container Agent state and config ... ok
Trying to collect Amazon ECS Container Agent engine data ... ok
Trying to collect Amazon ECS init logs ... ok
Trying to check if the script is running as root ... ok
Trying to resolve instance-id ... ecs-logs-collector.sh: line 175: ./collect/i-0cf25496945da6295/i-0cf25496945da6295/system/instance-id.txt: No such file or directory
ok
Trying to collect system information ... ok
Trying to enable debug mode for the Docker daemon ... Trying to restart Docker daemon to enable debug mode ... Stopping docker: [ OK ]
Starting docker: . [ OK ]
ok
Trying to enable debug mode for the Amazon ECS Container Agent ... Trying to restart the Amazon ECS Container Agent to enable debug mode ... stop: Job has already been stopped: ecs
ecs start/running, process 3087
ok
Trying to archive gathered log information ... ok

ls

collect ecs-logs-collector.sh

The script is generating tarball with instance id in the collect folder. Please update the Documentation page.

Trying to archive gathered log information ... + local tar_bin
++ which tar

• tar_bin=/bin/tar
• '[' -z /bin/tar ']'
• cd .
• /bin/tar -czf ./collect/i-0cf25496945da6295.tgz ./collect/i-0cf25496945da6295
• ok
• echo ok

Addition of disable-debug mode to ECS Logs Collector

Summary

Addition of disable-debug mode to ECS Logs Collector will help in disabling debug mode previously enabled for the Docker daemon and the Amazon ECS Container Agent.

Description

When troubleshooting issues in Amazon ECS, it's often necessary to enable debug logging for the Docker daemon and the Amazon ECS Container Agent to gather detailed logs. However, once the troubleshooting is completed, it's crucial to disable the debug mode as the verbose logging can consume a significant amount of storage space and disk I/O operations, leading to unnecessary resource consumption and potential performance issues.

The proposed feature is to add a "disable-debug" mode to the ECS Logs Collector, which would allow users to disable the debug logging for the Docker daemon and the Amazon ECS Container Agent with a single command or configuration setting. This would streamline the process of reverting to normal logging levels after troubleshooting, ensuring efficient resource utilization and preventing potential issues caused by excessive logging.

Expected Behavior

After enabling the "disable-debug" mode, the Docker daemon and the Amazon ECS Container Agent should be reverted to their default logging levels, reducing the storage space and disk I/O operations consumed by debug logging.

Observed Behavior

Currently, there is no built-in mechanism or convenient way to disable the debug logging for the Docker daemon and the Amazon ECS Container Agent after troubleshooting. Users either forget or have to manually modify the configuration files or execute multiple commands to revert to the default logging levels, which can be time-consuming and error-prone.

Benefits

• Improved resource efficiency by reducing unnecessary storage space and disk I/O operations consumed by debug logging.
• Streamlined process for disabling debug logging after troubleshooting, reducing the risk of human error and saving time.
• Potential performance improvements by eliminating the overhead of excessive logging.
• Increased operational efficiency and ease of use for Amazon ECS users.

Proposed Implementation Approach

Proposal is to extend the existing ECS Logs Collector functionality to include a "disable-debug" as third supported mode, This can be achieved by:

1. Adding a new command-line mode (e.g., `--mode=disable-debug) to the ECS Logs Collector tool.
2. Implementing the necessary logic within the ECS Logs Collector to disable the debug logging for the Docker daemon and the Amazon ECS Container Agent when the "disable-debug" mode is used.
3. Updating the ECS Logs Collector documentation to include instructions on how to use the new "disable-debug" mode.

[Feature Request] Collect GPU Info for GPU Container Instances

Summary

Added a routine to collect GPU details for GPU Container Instances

Description

Currently script doesn't collects GPU related information GPU Container Instances, this change will add output of:

1. nvidia-smi -L listing GPUs
2. nvidia-smi -q Details of all GPUs

Expected Behavior

Observed Behavior

Improve the Usage Documentation

It's really common when requesting logs customers send the output of the .sh execution but tho the collect.tgz/collect.tar.gz depending on the version.

Creating this ISSUE to update do the Pull request improving the README.md to have some information on how to collect the logs.

collect lspci output

we should collect lspci output as it would be useful for debugging eni attachment related issues.

[Feature Request] Automatically mask credential information when running the ECS Logs Collector

Summary

Description

When customers run Logs Collector and submit the resulting log, it is difficult to mask the values obtained from the SSM parameter store one by one.

support engineers don't need to look at secrets information unnecessarily.

Can anything be done about this?

Expected Behavior

Observed Behavior

Collect and probe based on environment, not on detected system

Summary

Instead of limiting the functionality based on the detected OS, this tool could be made more flexible by working with well known locations to collect what is possible based on what is present.

"wget" didn't come on my Amazon Linux ECS box

After ssh'ing into my server, I found out that it didn't come with wget.

I had to install it using:

• sudo yum install wget

---

I was wondering:

• why my server didn't come with wget preinstalled
• wether that yum line should be added to the readme

log collection can wait indefinitely

Summary

If start ecs fails, the log collector can hang indefinitely. We should change this to retry, and also report logs even if this command fails.

script hangs on collecting Docker information

Hi,

I'm trying to execute this script on a post-mortem EC2 instance (stopped and started again). It hangs on the line:

Trying to gather Docker daemon information...

Here is the full output:

[ec2-user@ip-172-31-21-122 ~]$ sudo bash ecs-logs-collector.sh --mode=debug
Trying to check if the script is running as root... ok
Trying to check disk space usage... ok
Trying to collect system information... Amazon Linux AMI release 2016.09
ok
Trying to collect common operating system logs... ok
Trying to get mount points and volume information... ok
Trying to check SELinux status... ok
Trying to get iptables list... ok
Trying to detect installed packages... ok
Trying to detect active system services list... ok
Trying to gather Docker daemon information... 

Switching modes doesn't help.