Clarify documentation on start_position / sincedb

[lukewaite]

Clarify documentation on start_position / sincedb

[lukewaite]

Hi @danielmcquillen - yes, perhaps clearer documentation is in order.

As it currently stands, start_position is only respected if there is no entry for a given log group in your sincedb.

This allows you to specify how "newly seen" log groups will begin ingesting (from the start, from the end, or via a step-back from the current time).

The default, if you do not specify a start_position, when it sees a new log group, is to start from the first entry in the log group, and work forward. This will update your since-db as it ingests, and subsequent start-ups of ingestion on this log-group will work from the since-db's start point, ignoring the start_position parameter.

Does this clear things up?

[danielmcquillen]

Hi @lukewaite . Ok cool. So if at anytime I want to completely reload everything, I need to remove sincedb. Is that right?

[lukewaite]

Yep. That’s correct.

…

On Mon, Oct 30, 2017 at 11:55 PM Daniel McQuillen ***@***.***> wrote: Hi @lukewaite <https://github.com/lukewaite> . Ok cool. So if at anytime I want to completely reload everything, I need to remove sincedb. Is that right? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#44 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAlukixyvnKpwGCkrLMUQQAdgurhTWwXks5sxpowgaJpZM4QFNUc> .

[mikejr83]

Does the path for sincedb need to be created prior to run? At the moment I'm using the default but am not seeing any logs or the .sincedb being created.

[kabinfh]

Hello @lukewaite

While trying to set start_position => 86400, it is not getting data from one day past.

How can I only ask for today data and onwards from CW data to input into logstash?

Thanking u in advance...

It is super important for me to setup as old data are too big to move and we dont need as well

[william-amaral]

Is there any solution for it? I have the same question of @kabinfh
start_position seems not work

[jeyakumar8]

@lukewaite
Thanks for this awesome plugin.
I also have the same question as of @kabinfh

The logstash pipeline seems to be pulling very older logs even though the start_position => 86400 set.

Is there any work around for this?

Thanks in advance!!