clearcodehq/go-forward

AWS Cloudwatch Logs forwarder

go-forward's Issues

Variables in stream name.

AWS does not limit number of streams.
Possible variables:

• instance-id
• hostname

This solution will prevent #12

Add SIGINT handling.

When SIGINT is received, program should:

• stop listening for new messages
• send all messages in buffers to their destinations

upload to group: * stream: * failed ThrottlingException Rate exceeded

Caused when multiple clients send at the same time to same log group and stream. Can be fixed by exponential backoff

Dependency pinning

We need to pin dependencies in some way in order to have same dependency version across builds.

One uploader at a time.

Cloudwatch logs have certain constraints for putLogEvents action:

• 200ms delay between putLogEventsDelay call.
• sequenceToken must change in order to successfully send messages. Otherwise DataAlreadyAcceptedException is returned as an error.
• sequenceToken is assigned for each log stream.

Problem:
How to run a goroutine and know when it finished ?

Configurable message format.

As syslog messages are sent / forwarded with different fields and formats, config option providing message format is essential.
Assumptions:

• message format is known
• program is only interested in user defined fields
• fields are parsed

This may be implemented using parser.

Clarify the default config file location

In README.md:

      Config file location. (default "/etc/logs_agent.cfg")

In main.go:

const defaultConfigFile = "/etc/awslogs.conf"
[...]
	flag.StringVar(&cfgfile, "c", defaultConfigFile, "Config file location.")

Which should be used?

I have seen /etc/awslogs.conf used by a different package with a different content, probably /etc/logs_agent.cfg is more unique. Or name it /etc/go_forward.cfg?

Include a copy of GNU GPL version 3

I don't see where we specify that the LGPL applies, but we should include the GPL too: LGPL3 works as an additional permission and refers to the GPL. See https://www.gnu.org/licenses/gpl-howto.en.html.

Configurable queue size.

Queue size should be configurable.

• size is a maximum number of messages in queue that can be buffered.
• defaults to 50000 (max messages in batch is 10000)
• when limit is reached, new messages are discarded

Message formatters.

Formatting a message before sending it may be useful e.g. remove facility and severity because there is no need to store it. Timestamps should always be removed since they are already stored in cloudwatch logs.

Possible implementations:

• templates

Config file support

Config file should be readable, simple and commentable.

Possible implementations:

• go-ini
• go-yaml

counters and stats

• number of upload calls
• number of errors

Multiple parsers support

Depending on receiver, messages may arrive in different formats. We should support formats for different kinds of messages.