V3: Storage is no longer emptied properly.

[Csardelacal]

V3: Storage is no longer emptied properly.

[adamcooke]

This is now fixed in 3.3.4.

[willpower232]

It may be that the code is biased towards the days but I can not remember off the top of my head.

Have you tried connecting to the database to see how many raw tables you have?

FWIW Postal was not intended to store your messages long term, you should be doing that in whatever application you are using to create the emails and then you could also listen to webhooks and store the information for as long as you need.

[Csardelacal]

It used to work just fine. It would (at least I understood that) delete all the messages older than, and then continue until it would meet the storage limit. This seems to no longer be the case:

MariaDB [postal-server-1]> SHOW TABLES;
+---------------------------+
| Tables_in_postal-server-1 |
+---------------------------+
| clicks                    |
| deliveries                |
| links                     |
| live_stats                |
| loads                     |
| messages                  |
| migrations                |
| raw-2023-11-03            |
| raw-2023-11-04            |
| raw-2023-11-05            |
| raw-2023-11-06            |
| raw-2023-11-07            |
| raw-2023-11-08            |
| raw-2023-11-09            |
| raw-2023-11-10            |
| raw-2023-11-11            |
| raw-2023-11-12            |
| raw-2023-11-13            |
| raw-2023-11-14            |
| raw-2023-11-15            |
| raw-2023-11-16            |
| raw-2023-11-17            |
| raw-2023-11-18            |
| raw-2023-11-19            |
| raw-2023-11-20            |
| raw-2023-11-21            |
| raw-2023-11-22            |
| raw-2023-11-23            |
| raw-2023-11-24            |
| raw-2023-11-25            |
| raw-2023-11-26            |
| raw-2023-11-27            |
| raw-2023-11-28            |
| raw-2023-11-29            |
| raw-2023-11-30            |
| raw-2023-12-01            |
| raw-2023-12-02            |
| raw-2023-12-03            |
| raw-2023-12-04            |
| raw-2023-12-05            |
| raw-2023-12-06            |
| raw-2023-12-07            |
| raw-2023-12-08            |
| raw-2023-12-09            |
| raw-2023-12-10            |
| raw-2023-12-11            |
| raw-2023-12-12            |
| raw-2023-12-13            |
| raw-2023-12-14            |
| raw-2023-12-15            |
| raw-2023-12-16            |
| raw-2023-12-17            |
| raw-2023-12-18            |
| raw-2023-12-19            |
| raw-2023-12-20            |
| raw-2023-12-21            |
| raw-2023-12-22            |
| raw-2023-12-23            |
| raw-2023-12-24            |
| raw-2023-12-25            |
| raw-2023-12-26            |
| raw-2023-12-27            |
| raw-2023-12-28            |
| raw-2023-12-29            |
| raw-2023-12-30            |
| raw-2023-12-31            |
| raw-2024-01-01            |
| raw-2024-01-02            |
| raw-2024-01-03            |
| raw-2024-01-04            |
| raw-2024-01-05            |
| raw-2024-01-06            |
| raw-2024-01-07            |
| raw-2024-01-08            |
| raw-2024-01-09            |
| raw-2024-01-10            |
| raw-2024-01-11            |
| raw-2024-01-12            |
| raw-2024-01-13            |
| raw-2024-01-14            |
| raw-2024-01-15            |
| raw-2024-01-16            |
| raw-2024-01-17            |
| raw-2024-01-18            |
| raw-2024-01-19            |
| raw-2024-01-20            |
| raw-2024-01-21            |
| raw-2024-01-22            |
| raw-2024-01-23            |
| raw-2024-01-24            |
| raw-2024-01-25            |
| raw-2024-01-26            |
| raw-2024-01-27            |
| raw-2024-01-28            |
| raw-2024-01-29            |
| raw-2024-01-30            |
| raw-2024-01-31            |
| raw-2024-02-01            |
| raw-2024-02-02            |
| raw-2024-02-03            |
| raw-2024-02-04            |
| raw-2024-02-05            |
| raw-2024-02-06            |
| raw-2024-02-07            |
| raw-2024-02-08            |
| raw-2024-02-09            |
| raw-2024-02-10            |
| raw-2024-02-11            |
| raw-2024-02-12            |
| raw-2024-02-13            |
| raw-2024-02-14            |
| raw-2024-02-15            |
| raw-2024-02-16            |
| raw-2024-02-17            |
| raw-2024-02-18            |
| raw-2024-02-19            |
| raw-2024-02-20            |
| raw-2024-02-21            |
| raw-2024-02-22            |
| raw-2024-02-23            |
| raw-2024-02-24            |
| raw-2024-02-25            |
| raw-2024-02-26            |
| raw-2024-02-27            |
| raw-2024-02-28            |
| raw-2024-02-29            |
| raw-2024-03-01            |
| raw-2024-03-02            |
| raw-2024-03-03            |
| raw-2024-03-04            |
| raw-2024-03-05            |
| raw-2024-03-06            |
| raw-2024-03-07            |
| raw-2024-03-08            |
| raw-2024-03-09            |
| raw-2024-03-10            |
| raw-2024-03-11            |
| raw-2024-03-12            |
| raw-2024-03-13            |
| raw-2024-03-14            |
| raw-2024-03-15            |
| raw-2024-03-16            |
| raw-2024-03-17            |
| raw-2024-03-18            |
| raw-2024-03-19            |
| raw-2024-03-20            |
| raw-2024-03-21            |
| raw-2024-03-22            |
| raw-2024-03-23            |
| raw-2024-03-24            |
| raw-2024-03-25            |
| raw-2024-03-26            |
| raw-2024-03-27            |
| raw-2024-03-28            |
| raw-2024-03-29            |
| raw-2024-03-30            |
| raw-2024-03-31            |
| raw-2024-04-01            |
| raw-2024-04-02            |
| raw-2024-04-03            |
| raw-2024-04-04            |
| raw-2024-04-05            |
| raw-2024-04-06            |
| raw-2024-04-07            |
| raw-2024-04-08            |
| raw-2024-04-09            |
| raw-2024-04-10            |
| raw-2024-04-11            |
| raw-2024-04-12            |
| raw-2024-04-13            |
| raw-2024-04-14            |
| raw-2024-04-15            |
| raw-2024-04-16            |
| raw-2024-04-17            |
| raw-2024-04-18            |
| raw-2024-04-19            |
| raw-2024-04-20            |
| raw-2024-04-21            |
| raw-2024-04-22            |
| raw-2024-04-23            |
| raw-2024-04-24            |
| raw-2024-04-25            |
| raw-2024-04-26            |
| raw-2024-04-27            |
| raw-2024-04-28            |
| raw-2024-04-29            |
| raw-2024-04-30            |
| raw-2024-05-01            |
| raw-2024-05-02            |
| raw-2024-05-03            |
| raw-2024-05-04            |
| raw-2024-05-05            |
| raw-2024-05-06            |
| raw-2024-05-07            |
| raw-2024-05-08            |
| raw-2024-05-09            |
| raw-2024-05-10            |
| raw-2024-05-11            |
| raw-2024-05-12            |
| raw-2024-05-13            |
| raw-2024-05-14            |
| raw-2024-05-15            |
| raw-2024-05-16            |
| raw_message_sizes         |
| spam_checks               |
| stats_daily               |
| stats_hourly              |
| stats_monthly             |
| stats_yearly              |
| suppressions              |
| webhook_requests          |
+---------------------------+
211 rows in set (0.001 sec)

I've been doing my best to reduce the amount of storage time, since we've had issues with it being slow in the UI. But it's really useful to have a good backlog since users sometimes won't report that they're not receiving emails for several months.

Regardless, the storage cap feature is not working any more.

[willpower232]

With nearly 200 days of raw messages, none of the settings seem to be working for you.

Version 3 changed the cron to use scheduled tasks instead, I think there should be a scheduled_tasks table in your main postal database which may or may not shed more light on the issue?

[ndTEC]

I can confirm the problem.

[Csardelacal]

Looks fine to me:

MariaDB [postal]> SELECT * FROM scheduled_tasks;
+----+--------------------------------------+---------------------+
| id | name                                 | next_run_after      |
+----+--------------------------------------+---------------------+
|  1 | ActionDeletionsScheduledTask         | 2024-05-21 12:15:00 |
|  2 | CheckAllDNSScheduledTask             | 2024-05-21 12:15:00 |
|  3 | CleanupAuthieSessionsScheduledTask   | 2024-05-21 12:15:00 |
|  4 | ExpireHeldMessagesScheduledTask      | 2024-05-21 12:15:00 |
|  5 | ProcessMessageRetentionScheduledTask | 2024-05-22 03:00:00 |
|  6 | PruneSuppressionListsScheduledTask   | 2024-05-22 03:00:00 |
|  7 | PruneWebhookRequestsScheduledTask    | 2024-05-21 12:45:00 |
|  8 | SendNotificationsScheduledTask       | 2024-05-21 11:47:21 |
|  9 | TidyQueuedMessagesTask               | 2024-05-21 12:45:00 |
+----+--------------------------------------+---------------------+
9 rows in set (0.000 sec)

[ndTEC]

@willpower232 could you share how can we manually delete old emails like the task is doing? We are running out of space until a new release.

[willpower232]

You should be able to safely remove the raw-* tables yourself until you're retaining the amount of raw message contents you desire

[jmdunsing]

I'm also having the same problem. From my logs running "postal logs worker | grep ProcessMessageRetentionScheduledTask, I get the following:

worker_1 | 2024-05-27 03:00:28 +0000 INFO running task component=worker thread=tasks task=ProcessMessageRetentionScheduledTask
worker_1 | 2024-05-27 03:00:28 +0000 INFO scheduling task to next run at 2024-05-28 03:00:00 UTC component=worker thread=tasks task=ProcessMessageRetentionScheduledTask
worker_2 | 2024-05-28 03:00:48 +0000 INFO running task component=worker thread=tasks task=ProcessMessageRetentionScheduledTask
worker_2 | 2024-05-28 03:00:48 +0000 INFO scheduling task to next run at 2024-05-29 03:00:00 UTC component=worker thread=tasks task=ProcessMessageRetentionScheduledTask

This looks like the task is completed within a second. I couldn't find any logged errors associated with this task.

[LotisHunters]

I’m having the same issue.
I’ve been manually clearing out tables but it’s really inconvenient and requires regular attention.

[ugurtam]

I have the same issue, too

[chrisdeeming]

Ditto here as mentioned in another discussion.

Can we get @adamcooke's attention on this? It's not ideal and should be fixed fairly urgently IMO.

I unexpectedly ran out of space to such a point where of course mails stopped sending but even worse I couldn't actually get MySQL to run any sort of operation, even deleting tables.

I had to double our running costs by increasing the disk space before manually deleting raw messages.

[ugurtam]

Yes, I have the same out of space problem. Please @adamcooke you should handle it

[chrisdeeming]

Can confirm it is resolved. Thank you @adamcooke!