Old events not properly cleared

[federico-zangani]

Old events not properly cleared

[jasonish]

What version are you using? If you add -vv you should get some extra debug information and should see something like this once a minute:

2023-09-14 04:23:10 TRACE evebox::sqlite::retention: Deleting events older than 7 days
2023-09-14 04:23:10 DEBUG evebox::sqlite::retention: Deleted 39 events older than 2023-09-07 4:23:10.810783427 +00:00:00 (7 days) in 9 ms

[federico-zangani]

I'm running version 0.17.2 in a docker container, I see

INFO evebox::sqlite::retention: Database retention settings: days=7, size=0
DEBUG evebox::sqlite::retention: Deleted 1000 events older than 2023-09-07 7:40:22.656996778 +00:00:00 in 2661 ms

I assume my config file is not being read because the size constraint that I put is not there and was probably just matched by chance when I checked, but in the web page I still see older events, aswell as some alerts dating more than 7 days

Thanks for the help

[jasonish]

For the config file, make sure your Docker volumes are correct and you are specifying the path to the configuration file. if you still run into issues there, let me know. But this is how I run EveBox as well.

It may take a while to delete all the old events. Write activity in SQLite locks the database, so we only delete 1000 events at a time. If 1000 events were actually deleted, we reschedule the next purge in 1 second rather than 1 minute. So if you are dropping from a retention of 30 days to 7, in can take a while. There is also an issue if you happen to be ingesting more than 1000 events per second.

[federico-zangani]

I specified the config file in a better way and it's now working properly

I went into the debug logs to see and it seems like it is now slowly deleting old events, but since I get more than 1000 per second at some times they still showed up

I'm also experiencing some lag (evebox loads events very slowly and falls behind compared to the suricata logs), could those be related?

Thanks :)

[jasonish]

Looks like this issue is resolvd. Detecting lag would be a good to add tho.