sincedb file not created, files from bucket not deleted

[niekosau]

sincedb file not created, files from bucket not deleted

[froheik]

@niekosau can you show your logstash log of input.s3? May be i have a same problem

[zeroad]

I'm using logstash 7.17.4 with plugin version 3.8.3

I think the comparison logic is the culprit here. In my case it is comparing timestamps which end with different local formats.

Before this I added

::File.open('/tmp/debug.log', 'a') { |file| file.write("object.last_modified: ", object.last_modified, ",  log.last_modified: ", log.last_modified, "\n") }

Which gives me:

object.last_modified: 2021-06-25 16:38:23 +0000,  log.last_modified: 2021-06-25 16:38:23 UTC

I'm not familiar with ruby but when comparing the dates it probably casts them to the above string representations which are not the same.

In the first step I fixed it on my side by comparing the unix time stamps

if object.last_modified.to_i == log.last_modified.to_i

Still this works not 100% as expected. The logic applied here is to save the latest timestamp seen in s3 to the sincedb. If you do not delete your files in the s3 you will always import all files which have the same timestamp as stored in the sincedb path. Which means you will import the latest file again and again.

To fix this issue I always add 1s to the timestamp written to the sincedb, after this line

since = since + 1

I'm not sure if there will be any side effects when simply adding +1 to the timestamp (at least it works for me now as expected).
One should probably fix the comparising logic.