lucagrulla/cw

The best way to tail AWS CloudWatch Logs from your terminal

cw

The best way to tail AWS CloudWatch Logs from your terminal.

Author - Luca Grulla - https://www.lucagrulla.com

• Features
• Installation
• Commands and options
• Examples
• AWS credentials and configuration
• Miscellaneous
• Release notes

Features

• No external dependencies
  • cw is a native executable targeting your OS. No pip, npm, rubygems.
• Fast.
  • cw is written in golang and compiled against your architecture.
• Flexible date and time parser.
  • Work with either Local timezone or UTC (default).
  • Flexible parsing.
    • Human friendly formats, i.e. 2d1h20m to indicate 2 days, 1 hour and 20 minutes ago.
    • a specific hour, i.e. 13:10 to indicate 13:10 of today.
    • a full timestamp 2018-10-20T8:53.
• Multi log groups tailing
  • tail multiple log groups in parallel: cw tail my-auth-service my-web.
• Powerful built-in grep (--grep) and grepv (--grepv).
• JMESPath support for JSON queries (matching the AWS CLI --query flag)
• Pipe operator supported
  • echo my-group | cw tail and cat groups.txt | cw tail.
• Redirection operator >> supported
  • cw tail -f my-stream >> myfile.txt.
• Coloured output
  • --no-color flag to disable if needed.
• Flexible credentials control.
  • By default the AWS .aws/credentials and .aws/profile files are used. Overrides can be achieved with the --profile and --region flags.

Installation

Mac OSX

using Homebrew

brew tap lucagrulla/tap
brew install cw

Linux

using Linuxbrew

brew tap lucagrulla/tap
brew install cw

.deb/.rpm

Download the .deb or .rpm from the releases page and install with dpkg -i and rpm -i respectively.

using Snapcraft.io

Note: If you upgrade to 3.3.0 please note the new alias command. This is required to comply with snapcraft new release rules.

snap install cw-sh
sudo snap connect cw-sh:dot-aws-config-credentials
sudo snap alias cw-sh.cw cw

cw runs with strict confinement; the dot-aws-config-credentials interface connection is required to have access to .aws/config and .aws/credentials files

On Windows

using Scoop.sh

scoop bucket add cw https://github.com/lucagrulla/cw-scoop-bucket.git
scoop install cw

Go tools

go get github.com/lucagrulla/cw

Commands and options

Global flags

• --profile=profile-name Override the AWS profile used for connection.
• --region=aws-region Override the target AWS region.
• --no-color Disable coloured output.
• --endpoint The target AWS endpoint url. By default cw will use the default aws endpoints.
• --no-version-check Ignore checks if a newer version of the module is available.

Commands

• cw ls list all the log groups/log streams within a group

  Usage: cw ls <command>
  
  show an entity
  
  Flags:
    -h, --help               Show context-sensitive help.
        --endpoint=URL       The target AWS endpoint url. By default cw will use the default aws endpoints. NOTE: v4.0.0
                            dropped the flag short version.
        --profile=PROFILE    The target AWS profile. By default cw will use the default profile defined in the
                            .aws/credentials file. NOTE: v4.0.0 dropped the flag short version.
        --region=REGION      The target AWS region. By default cw will use the default region defined in the
                            .aws/credentials file. NOTE: v4.0.0 dropped the flag short version.
        --no-color           Disable coloured output.NOTE: v4.0.0 dropped the flag short version.
        --version            Print version information and quit
        --no-version-check   Ignore checks if a newer version of the module is available.
  
  Commands:
    ls groups
      Show all groups.
  
    ls streams <group>
      Show all streams in a given log group.
  
  cw: error: expected one of "groups",  "streams"

• cw tail tail a given log group/log stream

  Usage: cw tail <groupName[:logStreamPrefix]> ...
  
  Tail log groups/streams.
  
  Arguments:
    <groupName[:logStreamPrefix]> ...    The log group and stream name, with group:prefix syntax. Stream name can be just the prefix. If no stream name is specified all stream names in the given
                                        group will be tailed. Multiple group/stream tuple can be passed. e.g. cw tail group1:prefix1 group2:prefix2 group3:prefix3.
  
  Flags:
    -h, --help                           Show context-sensitive help.
        --endpoint=URL                   The target AWS endpoint url. By default cw will use the default aws endpoints. NOTE: v4.0.0 dropped the flag short version.
        --profile=PROFILE                The target AWS profile. By default cw will use the default profile defined in the .aws/credentials file. NOTE: v4.0.0 dropped the flag short version.
        --region=REGION                  The target AWS region. By default cw will use the default region defined in the .aws/credentials file. NOTE: v4.0.0 dropped the flag short version.
        --no-color                       Disable coloured output.NOTE: v4.0.0 dropped the flag short version.
        --version                        Print version information and quit
        --no-version-check               Ignore checks if a newer version of the module is available.
  
    -f, --follow                         Don't stop when the end of streams is reached, but rather wait for additional data to be appended.
    -t, --timestamp                      Print the event timestamp.
    -i, --event-id                       Print the event Id.
    -s, --stream-name                    Print the log stream name this event belongs to.
    -n, --group-name                     Print the log group name this event belongs to.
    -r, --retry                          Keep trying to open a log group/log stream if it is inaccessible.
    -b, --start="2021-04-11T08:21:52"    The UTC start time. Passed as either date/time or human-friendly format. The human-friendly format accepts the number of days, hours and minutes prior to
                                        the present. Denote days with 'd', hours with 'h' and minutes with 'm' i.e. 80m, 4h30m, 2d4h. If just time is used (format: hh[:mm]) it is expanded to
                                        today at the given time. Full available date/time format: 2017-02-27[T09[:00[:00]].
    -e, --end=STRING                     The UTC end time. Passed as either date/time or human-friendly format. The human-friendly format accepts the number of days, hours and minutes prior to the
                                        present. Denote days with 'd', hours with 'h' and minutes with 'm' i.e. 80m, 4h30m, 2d4h. If just time is used (format: hh[:mm]) it is expanded to today at
                                        the given time. Full available date/time format: 2017-02-27[T09[:00[:00]].
    -l, --local                          Treat date and time in Local timezone.
    -g, --grep=STRING                    Pattern to filter logs by. See http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html for syntax.
    -v, --grepv=STRING                   Equivalent of grep --invert-match. Invert match pattern to filter logs by.
    -q, --query=STRING                   Equivalent of the --query flag in AWS CLI. Takes a JMESPath expression to filter JSON logs by. If the query fails (e.g. the log message was not JSON) then the original line is returned.

Examples

• list of the available log groups

  • cw ls groups

• list of the log streams in a given log group

  • cw ls streams my-log-group

• tail and follow given log groups/streams

  • cw tail -f my-log-group
  • cw tail -f my-log-group:my-log-stream-prefix
  • cw tail -f my-log-group:my-log-stream-prefix my-log-group2
  • cw tail -f my-log-group:my-log-stream-prefix -b2017-01-01T08:10:10 -e2017-01-01T08:05:00
  • cw tail -f my-log-group:my-log-stream-prefix -b7d to start from 7 days ago.
  • cw tail -f my-log-group:my-log-stream-prefix -b3h to start from 3 hours ago.
  • cw tail -f my-log-group:my-log-stream-prefix -b100m to start from 100 minutes ago.
  • cw tail -f my-log-group:my-log-stream-prefix -b2h30m to start from 2 hours and 30 minutes ago.
  • cw tail -f my-log-group -b9:00 -e9:01

• query JSON logs using JMESPath syntax

  • cw tail -f my-log-group --query "machines[?state=='running'].name"

Time and Dates

Time and dates are treated as UTC by default. Use the --local flag if you prefer to use Local zone.

AWS credentials and configuration

cw uses the default credentials profile (stored in ./aws/credentials) for authentication and shared config (.aws/config) for identifying the target AWS region. Both profile and region are overridable via the profile and region global flags.

AWS SSO

AWS SSO is supported if you:

• use a CLI profile (either default or an alternate named profile) that includes the various SSO properties
  • sso_start_url, sso_account_id, sso_role_name, etc
• have a valid, active SSO session
  • via aws sso login

If you get an error message that includes ...failed to sign request: failed to retrieve credentials: the SSO session has expired or is invalid... then you should renew your SSO session via aws sso login (and specify the named profile, if appropriate).

Miscellaneous

Use cw behind a proxy

Please use HTTP_PROXY environment variable as required by AWS cli: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-proxy.html

Breaking changes notes

Read here

cw's Issues

panic because close of closed chanel

Hit the panic scenario which Tail method was called multiple times, or you have a little more time delay so that the Ticker fired more signal to trigger the page handler, which then try to close the closed chanel.

Using the Ticker is a little problematic here as it has no exit but rely on the chanel being closed and also does not allow caller to reconsume the Tail method.

I have used a fix of using the removed Ticker, it just worked fine.

Error in fetchLatestVersion

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x80 pc=0x142790f]

goroutine 21 [running]:
main.fetchLatestVersion.func1(0xc0001f2cc0)
	/Users/lucagrulla/Projects/cw/main.go:113 +0x3f
created by main.fetchLatestVersion
	/Users/lucagrulla/Projects/cw/main.go:110 +0x58

This might be because the network request to check for a new version is failing?

Add godownloader distribution

https://github.com/goreleaser/godownloader

Human-friendly format not works

Thanks for 2.0.0 release 🎉

Log not found

$ date -u
Wed Oct 17 03:32:52 UTC 2018

It was correctly outputted by the way used until. (FYI: I am in the time zone of JST so some logs below are displayed at +0900)

$ cw -p my-profile tail -ft /ecs/dev-my-server
2018-10-17T12:32:42 - [2018-10-17 03:32:42.533]    0 ms -- /health/check [:GET]   200 [] "-" "-" ::ffff:10.0.1.137
2018-10-17T12:32:49 - [2018-10-17 03:32:49.065]    0 ms -- /health/check [:GET]   200 [] "-" "-" ::ffff:10.0.0.174

But it is not output as a new way.

$ cw -p my-profile tail -ft /ecs/dev-my-server \* 5m

It was not found even if full log stream name was specified.

$ cw -p my-profile tail -ft /ecs/dev-my-server ecs/my-server/bcc9fc24-7184-4064-aad7-921dd0bf32fc 5m
No such log stream(s).

$ cw -p my-profile ls streams /ecs/dev-my-server | grep bcc9fc24
ecs/my-server/bcc9fc24-7184-4064-aad7-921dd0bf32fc

panic: send on closed channel

About the case where wrong args is specified, I sometimes encounter panic error.

$ cw -p my-profile tail -ft /ecs/dev-my-server 2m
No such log stream(s).
panic: send on closed channel

goroutine 55 [running]:
github.com/lucagrulla/cw/cloudwatch.(*CW).Tail.func4(0xc000056680, 0xc000056501, 0x1012bd2)
	/Users/lucagrulla/Projects/cw/cloudwatch/tail.go:150 +0x175
github.com/aws/aws-sdk-go/service/cloudwatchlogs.(*CloudWatchLogs).FilterLogEventsPagesWithContext(0xc0001ae050, 0x1fb1128, 0xc0000240b0, 0xc0001000f0, 0xc000056580, 0x0, 0x0, 0x0, 0xc0001a8b01, 0xc0001000f0)
	/Users/lucagrulla/go/pkg/mod/github.com/aws/[email protected]/service/cloudwatchlogs/api.go:2288 +0x116
github.com/aws/aws-sdk-go/service/cloudwatchlogs.(*CloudWatchLogs).FilterLogEventsPages(0xc0001ae050, 0xc0001000f0, 0xc000056580, 0x0, 0x0)
	/Users/lucagrulla/go/pkg/mod/github.com/aws/[email protected]/service/cloudwatchlogs/api.go:2260 +0x90
github.com/lucagrulla/cw/cloudwatch.(*CW).Tail.func5(0xc0001cc2d0, 0xc00004d480, 0xc0001a8b40, 0xc0001c45d8, 0x0, 0xc00004d450, 0xc000024ad7, 0xc0001ae058, 0xc000056580)
	/Users/lucagrulla/Projects/cw/cloudwatch/tail.go:175 +0x235
created by github.com/lucagrulla/cw/cloudwatch.(*CW).Tail
	/Users/lucagrulla/Projects/cw/cloudwatch/tail.go:171 +0x40d

AWS SSO CLI v2 does not work

Describe the bug

The new AWS CLI v2 provides Single Sign On:

https://aws.amazon.com/blogs/developer/aws-cli-v2-is-now-generally-available/

But doesn't seem to play well with cw:

$ cw ls -p dev groups
no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

To Reproduce

$ aws configure sso
SSO start URL [https://yourcompany.awsapps.com/start]:
SSO Region [ap-southeast-2]:
Attempting to automatically open the SSO authorization page in your default browser.
If the browser does not open or you wish to use a different device to authorize this request, open the following URL:

https://device.sso.ap-southeast-2.amazonaws.com/

Then enter the code:

CQCS-BFPZ
There are 8 AWS accounts available to you.
Using the account ID <ACCOUNT_ID>
There are 3 roles available to you.
Using the role name "default"
CLI default client Region [ap-southeast-2]:
CLI default output format [None]:
CLI profile name [default]: default

$ aws s3 ls
<LIST OF BUCKETS>

$ cw ls -p dev groups
no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

SIGSEGV on APIgateway log group ls

Marks-MacBook-Pro[13:32:55] cw --profile=zjdevacct --region=us-east-2 ls streams "API-Gateway-Execution-Logs_72oi6rv36b/Alpha"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x13d6c8b]

goroutine 8 [running]:
github.com/lucagrulla/cw/cloudwatch.(*CW).LsStreams.func1.1(0x1, 0x0, 0xc0000c3db8)
/Users/lucagrulla/Projects/cw/cloudwatch/lsstreams.go:24 +0x2b
sort.insertionSort_func(0xc0000c3e38, 0xc000487c00, 0x0, 0x14)
/usr/local/Cellar/go/1.11.5/libexec/src/sort/zfuncversion.go:12 +0x93
sort.stable_func(0xc0000c3e38, 0xc000487c00, 0x32)
/usr/local/Cellar/go/1.11.5/libexec/src/sort/zfuncversion.go:167 +0x51
sort.SliceStable(0x146ace0, 0xc000487be0, 0xc0000c3e38)
/usr/local/Cellar/go/1.11.5/libexec/src/sort/slice.go:31 +0x106
github.com/lucagrulla/cw/cloudwatch.(*CW).LsStreams.func1(0xc00000b460, 0xc000030a00, 0xc0000556b8)
/Users/lucagrulla/Projects/cw/cloudwatch/lsstreams.go:23 +0x81
github.com/aws/aws-sdk-go/service/cloudwatchlogs.(*CloudWatchLogs).DescribeLogStreamsPagesWithContext(0xc00000c270, 0x1a74188, 0xc0000260b0, 0xc000183320, 0xc000061820, 0x0, 0x0, 0x0, 0x0, 0x0)
/Users/lucagrulla/go/pkg/mod/github.com/aws/[email protected]/service/cloudwatchlogs/api.go:1670 +0x116
github.com/aws/aws-sdk-go/service/cloudwatchlogs.(*CloudWatchLogs).DescribeLogStreamsPages(0xc00000c270, 0xc000183320, 0xc000061820, 0x0, 0x0)
/Users/lucagrulla/go/pkg/mod/github.com/aws/[email protected]/service/cloudwatchlogs/api.go:1642 +0x90
github.com/lucagrulla/cw/cloudwatch.(*CW).LsStreams.func2(0xc000061810, 0xc000183320, 0xc000061820)
/Users/lucagrulla/Projects/cw/cloudwatch/lsstreams.go:37 +0x49
created by github.com/lucagrulla/cw/cloudwatch.(*CW).LsStreams
/Users/lucagrulla/Projects/cw/cloudwatch/lsstreams.go:36 +0xf1

Hit AWS rate limit

From time to time we hit AWS req limit.

Panic when tailing old stream

Hi. Nice project, but I've hit a bug.

These streams exist:

$ cw ls streams /var/log/stuff/uat
access/instance1
access/instance2
catalina/instance1
catalina/instance2
localhost/instance1
localhost/instance2

But I can't tail some of them:

$ cw tail /var/log/stuff/uat localhost/instance2
No such log stream(s).
panic: send on closed channel

goroutine 42 [running]:
github.com/lucagrulla/cw/cloudwatch.(*CW).Tail.func4(0xc00001c780, 0xc00001c701, 0x412da2)
	/Users/lucagrulla/Projects/cw/cloudwatch/tail.go:149 +0x175
github.com/aws/aws-sdk-go/service/cloudwatchlogs.(*CloudWatchLogs).FilterLogEventsPagesWithContext(0xc00000c2a8, 0x7f6299adb130, 0xc000026120, 0xc00007d590, 0xc00001c6c0, 0x0, 0x0, 0x0, 0xc00016f101, 0xc00007d590)
	/Users/lucagrulla/go/pkg/mod/github.com/aws/[email protected]/service/cloudwatchlogs/api.go:2288 +0x116
github.com/aws/aws-sdk-go/service/cloudwatchlogs.(*CloudWatchLogs).FilterLogEventsPages(0xc00000c2a8, 0xc00007d590, 0xc00001c6c0, 0x0, 0x0)
	/Users/lucagrulla/go/pkg/mod/github.com/aws/[email protected]/service/cloudwatchlogs/api.go:2260 +0x90
github.com/lucagrulla/cw/cloudwatch.(*CW).Tail.func5(0xc00007e910, 0xc0000593c0, 0xc00016f1d0, 0xc000026ff8, 0x0, 0xc000059390, 0xc000026b47, 0xc00000c2b0, 0xc00001c6c0)
	/Users/lucagrulla/Projects/cw/cloudwatch/tail.go:174 +0x235
created by github.com/lucagrulla/cw/cloudwatch.(*CW).Tail
	/Users/lucagrulla/Projects/cw/cloudwatch/tail.go:170 +0x3b3

Unless I specify the date.

$ cw tail /var/log/stuff/uat localhost/instance2 2018-11-09T14:52
09-Nov-2018 14:52:17.320 INFO [main] org.apache.catalina.core.ApplicationContext.log ContextListener: contextInitialized()
09-Nov-2018 14:52:17.320 INFO [main] org.apache.catalina.core.ApplicationContext.log SessionListener: contextInitialized()
09-Nov-2018 14:52:17.321 INFO [main] org.apache.catalina.core.ApplicationContext.log ContextListener: attributeAdded('StockTicker', 'async.Stockticker@62379589')
09-Nov-2018 14:52:20.921 INFO [main] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath
09-Nov-2018 14:52:21.208 INFO [main] org.apache.catalina.core.ApplicationContext.log Initializing Spring root WebApplicationContext

This stream hasn't had any activity for a few days, and that's the only difference I can think of between the broken streams and the working ones.

I'm using the latest release.

$ cw --version
2.1.1

New version message is presented even if no new version available

View Cloudwatch metrics

Has anyone thought to use this tool to view cloudwatch metrics combining with something like gnuplot or https://github.com/holman/spark ?

Hide fields such as timestamp, group-name, stream-name with --no- option

--no-timestamp        Don't print the event timestamp.
--no-event-id           Don't print the event Id.
--no-stream-name      Don't print the log stream name this event belongs to.
--no-group-name       Don't print the log log group name this event belongs to.

Not able to connect may be because of proxy

Is your feature request related to a problem? Please describe.
Not able to connect. Getting send request failed

Describe the solution you'd like
Add support for proxy

Make installation Linux friendly

Which package manager to target?
Look into snapcraft

Allow parallel tail of multiple log groups

Add LinuxBrew distribution

Brew recipe works only for Mac.

Currently depending on:
goreleaser/goreleaser#428

brew caveats wrong message

Once installed it tells you to add to source this file
/usr/local/share/zsh/site-functions/_cw

instead of

/usr/local/share/zsh/site-functions/cw

==> Caveats

In order to get cw completion,
  [bash] you need to install `bash-completion` with brew.
  OR
  [zsh], add the following line to your ~/.zshrc:
    source /usr/local/share/zsh/site-functions/_cw

Bash completion has been installed to:
  /usr/local/etc/bash_completion.d

zsh functions have been installed to:
  /usr/local/share/zsh/site-functions

inconsistency bewtween long and short format for tail --start flag

Describe the bug
the long format of the --start flag should be begin rather than start for consistency

Time options

I am satisfied with the features of cw already, but if you want to more better to improve, I propose the Time options feature implementation.

I used awslogs until recently, but the problem with this tool is that it depends on python 2.x version. (In that respect, cw implemented with golang is very smart!)

awslogs has a feature called Time options, which is useful for viewing the most recent log.

• https://github.com/jorgebastida/awslogs#time-options
  • --start=2m
  • --start=1h

The tail command of cw can specify UTC start time as the third argument, but in my environment the timezone is GMT+09:00, which is a bit confusing.

$ cw -p my_other_profile tail -f -t /ecs/my-server ecs/my-server/f276cdff-375d-4c65-a4d6-29afd025f221 2018-10-01T10:13:00
2018-10-01T19:13:13 - [2018-10-01 10:13:13.488]    0 ms -- /health/check [:GET]   200 [] "-" "-" ::ffff:10.0.0.174
2018-10-01T19:13:17 - [2018-10-01 10:13:17.282]    0 ms -- /health/check [:GET]   200 [] "-" "-" ::ffff:10.0.1.137
2018-10-01T19:14:13 - [2018-10-01 10:14:13.518]    0 ms -- /health/check [:GET]   200 [] "-" "-" ::ffff:10.0.0.174
2018-10-01T19:14:17 - [2018-10-01 10:14:17.312]    0 ms -- /health/check [:GET]   200 [] "-" "-" ::ffff:10.0.1.137

This is because JST(GMT+09:00) time is output with -t flag although I have specified time in UTC.
Therefore it is convenient to specify like Time option.

$ cw tail --follow --timestamp my-log-group my-log-stream-prefix 2m

or

$ cw tail --follow --timestamp --start 2m my-log-group my-log-stream-prefix

Double newline

Describe the bug
I'm running the latest cw release (3.2.2) and there is an superfluous linebreak after every log line.

To Reproduce
Happens to me with all tail commands

Expected behavior
There shouldn't be an extra newline between each log line

Screenshots

Desktop (please complete the following information):

• OS: macos 10.15.1
• Terminal: zsh 5.7.1 in iTerm2 3.3.7

Best regards,
Felix

Package full name on Homebrew

I don't know if this is a mistake or not... I report you just in case.

lucagrulla/cw/cw -> lucagrulla/cw ?

$ brew info cw
lucagrulla/cw/cw: stable 2.1.1

$ brew upgrade cw
==> Upgrading 1 outdated package:
lucagrulla/cw/cw 2.1.0 -> 2.1.1
==> Upgrading lucagrulla/cw/cw
==> Downloading https://github.com/lucagrulla/cw/releases/download/v2.1.1/cw_2.1.1_Darwin_x86_64.tar.gz

Fix timezone issue

The app doesn't support Daylight saving properly

Duplicate print of events for high volume logs

When running in -f follow mode tail.go polls cloudwatch every ~205ms for logs and checks against a cache to ignore seen events. Most of the time the cache is hit and the event correctly ignored, but when the cache is emptied (every ~70 secs) we get a repeat of any events that occurred at the same time as the last event (for events with timestamp diff > 1ms this manifests as logging the last event over and over again). This isn't ideal.

scoop windows install in being block by antivirus

PS C:\Users\Narsi Nallamilli> Get-Host | Select-Object Version
Version

5.1.18362.752

PS C:\Users\Narsi Nallamilli> iwr -useb get.scoop.sh | iex
iex : At line:1 char:1

• #Requires -Version 5

This script contains malicious content and has been blocked by your antivirus software.
At line:1 char:26

• iwr -useb get.scoop.sh | iex

•                      ~~~
  

  • CategoryInfo : ParserError: (:) [Invoke-Expression], ParseException
  • FullyQualifiedErrorId : ScriptContainedMaliciousContent,Microsoft.PowerShell.Commands.InvokeExpressionCommand

Snap not able to read default AWS config/credentials files

Snapcraft security sandbox blocks access to the config files.

too many streams error

CW doesn't cap the number of tailed streams and hit the API limits of 50.

Allow parameter for errors "Rate exceeded"

Sometimes, when the application is running multiple times, the error "Rate exceeded" is thrown and the application exits. It would be a good idea to have a parameter to continue, or add a delay, when that error happens.

feature request: colorise message output

Other utilities like https://github.com/jorgebastida/awslogs colorise the output, it would be nice to have the same feature here

Make installation Windows friendly

Configure goreleaser for scoop.sh based installation

Retry if log group does not exist (like tail -F/--retry)

Is your feature request related to a problem? Please describe.
I'm working on a script that starts an ECS task and then tails the log from Cloudwatch. I wait until the task is actually in RUNNING state before kicking off cw, but there's still a few seconds between that state transition and the creation of the log group, so cw
errors out with "The specified log group does not exist."

Describe the solution you'd like
tail has the following options:

-F     same as --follow=name --retry
...
--retry keep trying to open a file if it is inaccessible

Basically I'd like to see -F replicated in cw. If the log group doesn't exist, retry until it does. Emit STDERR messages if the initial log group check fails, if the log group appears, and if the log group disappears.
My script already kills cw when the task has completed, so there's no concern about the log group never appearing.

Describe alternatives you've considered
I can test if the log group exists myself, and wait to invoke cw until it does exist.

Additional context
As inspiration for the type of output I'd like to see here's tail's STDERR output when following a file that does not exist on invocation, appears while tail is running, and is then deleted while tail is running:

$ tail -F /tmp/logfile > /dev/null
tail: cannot open '/tmp/logfile' for reading: No such file or directory
tail: '/tmp/logfile' has appeared;  following new file
tail: '/tmp/logfile' has become inaccessible: No such file or directory

-- following from time to time repeats content

the --following feature randomly repeats logs.

AssumeRoleTokenProviderNotSetError

Describe the bug
My AWS profile has enabled MFA, When I ssue the following command, I get Authentication errors as below:

cw ls groups

panic: AssumeRoleTokenProviderNotSetError: assume role with MFA enabled, but AssumeRoleTokenProvider session option not set.
goroutine 1 [running]:
github.com/aws/aws-sdk-go/aws/session.Must(...)
/Users/lucagrulla/go/pkg/mod/github.com/aws/[email protected]/aws/session/session.go:281
github.com/lucagrulla/cw/cloudwatch.New(0xc00005d3a0, 0xc00005d3b0, 0xc0000946e0, 0x0)
/Users/lucagrulla/Projects/cw/cloudwatch/client.go:34 +0x284
main.main()
/Users/lucagrulla/Projects/cw/main.go:176 +0x251

To Reproduce
Steps to reproduce the behavior:

1. Which command
   cw ls groups
2. Which flags/arguments are used
   ls

Expected behavior
Expected to list groups

Screenshots
N/A

Desktop (please complete the following information):

• MacOS 10.14.2
• Terminal (bash)

Additional context
N/A

Make cw stdin and pipe friendly

Describe the bug
stdin pipe in currently won't work

To Reproduce
echo "mygroup:*" | cw tail

Expected behavior
shoul tail "mygroup".

fix help text for starttime and endtime

The help text for the 2 tail params is misleading in the identification of which parts or optional

linuxbrew: support arm

Describe the bug
Current linuxbrew manifest doesn't support ARM architecture.

--profile flag

Thanks for nice tool 😃
I want to use the --profile flag like below to specify other aws credential profile.

$ cw --profile second ls groups

[default]
aws_access_key_id = ****
aws_secret_access_key = ****

[second]
aws_access_key_id = ****
aws_secret_access_key = ****

bash autocompletion is not working

Actually it's not even installed properly

Improve scheduler management

Describe the bug
when a Tail is completed the trigger channel should be removed from the round robin scheduler

Package name conflict with Ubuntu package

Describe the bug
The package name that you're using for the debian package, cw conflicts with this package name in Ubuntu : https://launchpad.net/ubuntu/bionic/amd64/cw

As a result, if a user has both packages they replace each other depending on which one is installed last.

To Reproduce
Steps to reproduce the behavior:

1. On an Ubuntu system run sudo apt install cw
2. run wget https://github.com/lucagrulla/cw/releases/download/v3.3.0/cw_amd64.deb && sudo dpkg -i cw_amd64.deb

Expected behavior
The existing cw package wouldn't be replaced and the CloudWatch tool would have a non conflicting name

Desktop (please complete the following information):

• OS: Ubuntu 18.04
• Terminal : any

Support 2FA

Is your feature request related to a problem? Please describe.
see #45

Describe the solution you'd like
Support authentication with 2FA.

Additional context
Accounts with 2FA enabled won't work.

CW region issue

Describe the bug
On ubuntu (18.04) server I have installed aws cli and it works without issue, but with cw I have region issue. When try to run cw ls groups getting this error: could not find region configuration. Also I tried run it like this cw ls groups --region="us-east-2", but in that case getting this error no valid providers in chain. Deprecated.

To Reproduce
Steps to reproduce the behavior:

1. install cw to ubuntu 18.04 server
2. cw ls groups --region="us-east-2"
3. cw ls groups

Expected behavior
I expect to get output like this, what works fine on my local Mac machine
/aws/rds/cluster/item-importer-sneki-test-cluster/error
/aws/rds/cluster/snekitest-cluster/error

Screenshots
If applicable, add screenshots to help explain your

• problem.
  

Desktop (please complete the following information):

• OS: Ubuntu 18.04

Works awesomely well with fzf

First of all, thanks for the project! It works amazingly well.

Today I realized it can be integrated with fzf quite easily too. Here is my little snippet that I am using:

cw tail -f "$(cw ls groups | fzf)"

Probably could be a useful doc addition. Or even a feature inside cw itself.

Cheers!

Improve log caching purge strategy

Current purge strategy for log id cache is naive.

zsh completion not working

`go get` installs version 1.7.2

Describe the bug
if you install cs by go get github.com/lucagrulla/cw or go get -u github.com/lucagrulla/cw you get version 1.7.2 and will be stuck at that version.

To Reproduce
just do go get -u github.com/lucagrulla/cw
I have "go version go1.13.4 darwin/amd64" installed , on a mac with latests macos, in zsh,

Expected behavior
able to get the latest version of cw which is 3.2.3

Screenshots
NA

Desktop (please complete the following information):

• OS: MacOS 10.15.2
• Terminal zsh, in iTerm

Add flag to specify aws endpoint url

Is your feature request related to a problem? Please describe.
We develop locally with localstack. I would like use the same tooling locally as on aws. I.e. I would like point cw to the url of my localstack and tail logs via cw as if it where the real aws.

I currently see no way how to point cw to localstack.

Describe the solution you'd like
Add a command line flag "--endpoint-url" analogous to the flag of aws cli with same name. So the call to tail logs from localstack would be:

./cw --endpoint-url=http://localhost:4566 tail -f /aws/lambda/input-processor-lambda

Describe alternatives you've considered
No alternatives considered yet.

Additional context
No additional information yet.

Add flag for disable coloured output

Tail multiple log groups

A feature request for this tool is the ability to tail multiple groups.

This would be extremely useful for Lambda, where an "app" is composed of many Lambda functions, and each function has its own log groups.

I'd like to be able to cw tail /aws/lambda/*:

Would you consider a PR with the feature?

Pretty Print JSON

Is your feature request related to a problem? Please describe.

One of the amazing features of CloudWatch is the ability to log to JSON, and then build intelligant searches based upon the JSON output.

This means logs often endup as long single lines of JSON output:

{"error": "this was an error", "level": "error", "data": {"key1": 1, "key2": 2}}

To make this better cloud watch takes the JSON and pretty prints it. cw outputs it as the line was put into cloudwatch.

It would be nice if cw detected if a line was JSON or not, and if JSON would pretty print it for easier reading. Possibly as a flag to enable/disable.

Describe alternatives you've considered
I'm not sure if there is an easy "fix". Normally I'd pipe the output to jq but that doesn't work since some automatic lambda and other input is not JSON.

Additionally pretty printing the JSON to cloudwatch doesn't work becasue then cloudwatch doesn't consider a JSON blob as a single JSON event but instead each line is a different text event.

Add --limit flag as per aws logs

Describe the solution you'd like
In aws logs CLI get-log-events has a --limit flag to define how many logs you want to receive.

List IAM permissions needed to use CW

• After downloading cw, I was not able to execute any of the command listed in the documentation, including cw ls groups. I ultimately want to run cw tail.

• I'm getting a The security token included in the request is invalid. error.

• I was allowed full access to Cloudwatch logs, and my aws creds are correct since I was able to use aws cli.

We provide our IAM permissions based on needs, and I can't figure out the permissions needed to use cw.

It'd be great if there was some guidance for the needed IAM roles somewhere. In the documentation somewhere like in the Readme, or as a cw -list-permissions or a cw -list-iam command.

I went through the codebase and found some mention of ec2 roles, but again I can't tell which ones.

Thank you!

Update: Rotating my access keys allowed me access. Why though