Memory leaks due to libvirtd eating a lot of memory over the time by day by day when using glusterfs driver.

[gluster]

Memory leaks due to libvirtd eating a lot of memory over the time by day by day when using glusterfs driver.

[vbellur]

@pkalever can you please help here?

[pkalever]

Yes, there is known bug/leak from glfs_fini() around 7-10MB per call, to do a leak less fini, gluster has to implement the cleanup methods per xlator, which is time taking but hopefully should be done in next couple of releases.

Work done so far to minimize the leaks:
Currently, a start of a VM (with only one disk, no snapshots) via libvirt will call 2 glfs_new/glfs_init (which will create glfs object, once for stat, read headers and next to chown) and then will fork qemu process which will call glfs once again (for actual read write IO).

Not that all, in case if we are have 4 extra attached disks, then the total calls to glfs_init() and friends will be (4+1)*2 in libvirt and (4+1)*1 in qemu space i.e 15 calls. Since we don't have control over qemu process as that executes in a different process environment, and killed on shutdown, lets do not bother much about it.

Additionally snapshot(external) scenario will further complex the situation, it does more recursive calls to each overlay on each disk.

I have submitted a patch [1] which shrinks these 10 + Snapshots (libvirt) calls (i.e objects from above example) to just one, by maintaining a cache of glfs objects. Note this patch is still under review.

Also [2] will address the memory allocation/consumption and minimize the leaks in QEMU, this patch is already in master and will appear in 2.8 release of qemu. Please go through the commit message for more details.

[1] https://www.spinics.net/linux/fedora/libvir/msg140475.html
[2] https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg07087.html

Cheers!

[neriberto]

I am with the same issue here, in two days my server memory usage raise next to maximum (32GB)

I am using Ubuntu 16.04.3 LTS

[stale]

Thank you for your contributions.
Noticed that this issue is not having any activity in last ~6 months! We are marking this issue as stale because it has not had recent activity.
It will be closed in 2 weeks if no one responds with a comment here.

[stale]

Closing this issue as there was no update since my last update on issue. If this is an issue which is still valid, feel free to open it.