[BUG] "Too much memory used (42Gb)" after a day of the conductor running

[guillemcordoba]

[BUG] "Too much memory used (42Gb)" after a day of the conductor running

[guillemcordoba]

1. Yes, I just rebooted my macmini node, the one that gave the error. The conductor booted fine and you should be able to talk to it with your own instances.
2. Here you go: https://github.com/darksoil-studio/gather/releases/download/v0.0.4/gather.happ
3. I didn't observe the behavior no... But I did find strange that holochain was consuming like 100% of one of the cores for the first ~30 minutes of booting up the node, with almost no network activity.

[abe-njama]

I ran Gather for approximately one hour and the application got to 35GB of memory.

[guillemcordoba]

This issue was just recently identified and fixed in wasmer v4.2.4, seems likely that it's related!

[guillemcordoba]

Yeah that makes sense... Let's see if we can fix this!

[ThetaSinner]

Oh dear, if it's not CPU it's memory! I of course agree that Holochain should be able to run for longer periods of time. I have got a mini PC running just for the purpose of debugging these issues. They are difficult to trace back to a single cause but my hope is that each time we find a cause that's one fewer way it can go wrong :)

1. Do you still have nodes running so that if I try to reproduce this on my mini PC I'll have something to talk to that already has data?
2. Is gather available as a happ so I can run it headless? I can unpack the webhapp if not
3. Did you observe the behaviour at all? Did the memory jump up at some point or did it continuously grow? I should be able to reproduce this but it's useful to know we're seeing the same thing.

[ThetaSinner]

Great, I have my node running it with Holochain 0.2.3-rc.0. Just to be certain, the DNA hash I see when I list installed apps is uhC0kdkl1h8BwUDwn4H8LirVnfB8-yDXWZhBBBO0s7Ei-aKrTkfta?

I have performance stats recording with sysstats which I haven't used before but can hopefully give me some feedback without having to watch top.

[ThetaSinner]

This doesn't happen headless it seems. After a few hours the memory usage is still sitting at 1gb.

Running on my desktop with the launcher the memory does grow out of control. So I'm going to say that interaction is the driver for memory growth initially. I'm seeing a lot of

Another zome function has triggered the `init()` callback, which has been blocking this zome call for longer than 30 seconds. Giving up.

being logged. That would explain why other zome calls aren't going through but doesn't directly explain the memory usage. I am thinking maybe this is a problem that was already here but the problem with zome calls is revealing it...

More investigation to be done yet :)

[maackle]

The init errors seem interesting. Perhaps there is something about the UI startup pattern that Holochain init can't handle. I would like to look at the actual zome functions being called. @guillemcordoba can you tell us which zome function(s) are being called when gather starts up for the first time? Or I can read the UI code, but asking is easier.

[guillemcordoba]

This one is the only one that's executed at first: get_agent_profile

[ThetaSinner]

Are we okay to close this issue and focus on #3028? I believe this is a consequence of the other issue. It'd be better if we didn't claim this much memory at once but for now I'd rather not impose limitations on Holochain's scaling just because it uses an unreasonable amount of memory when overloaded