Comments (24)
WuBingzheng
commented on May 18, 2024
There must be something wrong.
Would you like to give more information?
What is your OS and CPU?
Can I run your target program?
Thanks,
Wu
from memleax.
jonesmz
commented on May 18, 2024
admin@localhost:/ $ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz
stepping : 2
microcode : 0xffffffff
cpu MHz : 2596.941
cache size : 25600 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm fsgsbase bmi1 avx2 smep bmi2 erms xsaveopt
bugs :
bogomips : 5193.88
clflush size : 64
cache_alignment : 64
address sizes : 42 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz
stepping : 2
microcode : 0xffffffff
cpu MHz : 2596.941
cache size : 25600 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm fsgsbase bmi1 avx2 smep bmi2 erms xsaveopt
bugs :
bogomips : 5192.70
clflush size : 64
cache_alignment : 64
address sizes : 42 bits physical, 48 bits virtual
power management:
admin@localhost:/etc $ cat os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
The process in question is a proprietary program, and I don't have the authority to give others access, I'm sorry.
When I launched memleax, total memory usage was about 1.82 GB (according to htop) and growing at about 0.01 GB per minute. Within 60 seconds of launching memleax memory usage went up to 3.7, with 800MB of swap. The OOM killer kicked in after about 2 minutes and memleax was terminated.
Is there other information I can give you?
from memleax.
jonesmz
commented on May 18, 2024
To be honest with you, I'm not entirely sure the full list of things this application does. I'm investigating to take some load off of a co-worker.
I know that there is a lot of network traffic involved.
How long does memleax hold onto the information that it collects? Maybe there's too much traffic and memleax is getting new data faster than it releases it.
from memleax.
jonesmz
commented on May 18, 2024
Does memleax load all of the debug information from the process it attaches to?
I just noticed that it's taking a good 1-2 minutes before memleaxs outputs anything to the console.
The process I'm debugging pulls in about a dozen very-very large shared libraries. One of which is 500 MB (which includes debug symbols. The debug symbols aren't stripped out to a seperate file), and that's not all that huge compared to the smaller ones sizes.
from memleax.
WuBingzheng
commented on May 18, 2024
Could you run memleax again with arguments:
./memleax $target-pid -l 10 -m 100 -c 100
from memleax.
WuBingzheng
commented on May 18, 2024
Yes, memleax load all debug information at beginning.
So could you strip the debug symbols and try it again?
from memleax.
WuBingzheng
commented on May 18, 2024
I add some debug print for memory usage.
Could you try the attachment?
memleax.gz
from memleax.
jonesmz
commented on May 18, 2024
I'm seeing these debug statistics :
./memleax 40568 -l 10 -m 100 -c 100
- memleax start, with memory: 1232KB
- start build address maps...
- end build address maps, with memory: 812KB, total: 2044KB
- start build ptrace maps...
- end build ptrace maps, with memory: 0KB, total: 2044KB
- start build symtab...
- end build symtab with memory: 158824KB, total: 160868KB
- start build debug line...
- end build debug line, with memory: 84868KB, total: 245736KB
from memleax.
WuBingzheng
commented on May 18, 2024
So memleax allocates 245MB memory at beginning. So this may not be the problem.
Did memleax still get killed with -l10 -m100 -c100 arguments?
These arguments limit the memory usage when memleax running, and memleax will quit if getting the threshold.
from memleax.
WuBingzheng
commented on May 18, 2024
Maybe I find a memory leak in memleax. How satirical...
I am not sure about it now.
I build a bugfix version. You can try it.
memleax.gz
from memleax.
jonesmz
commented on May 18, 2024
The new version reports this:
- memleax start, with memory: 1068KB
- start build address maps...
- end build address maps, with memory: 940KB, total: 2008KB
- start build ptrace maps...
- end build ptrace maps, with memory: 0KB, total: 2008KB
- start build symtab...
- end build symtab with memory: 158824KB, total: 160832KB
- start build debug line...
- end build debug line, with memory: 84864KB, total: 245696KB
== Begin monitoring process 3286...
But I see an increase of the overall system's memory usage of 3GB+ before the program under test crashes.
I've moved my debugging environment to a VM with 12 GB of ram. The program under test, when started for a minute or two, uses about 1.5 % of total ram, and the overall system ram usage is shown by htop as 1.25GB. Obviously this will grow over time, as the process has some kind of memory leak. But the growth is measured in 100MB per day, not per minute.
When I start memleax, I can watch the memleax process grow to 30%+ of system ram, while the program under test stays at about the same amount.
Within 20 seconds of starting memleax, the total system memory goes from 1.25GB, to 2.5GB, and within a minute or two, the total usage goes to 5GB and higher.
Using the "-l 10 -m 100 -c 100" doesn't appear to change this behavior.
I'll admit that the program under test crashing could likely be caused by a bug in the program under test, and not necessarily a bug in memleax. I imagine it's probably being triggered by previously unknown timing bugs in the code for that program, with memleaxs injecting debug breakpoints and such, it wouldn't surprise me that it's just barfing over some weird timing problem.
But the memory usage of memleax is still confusing to me.
Is it normal for memleax to use a lot of CPU, even after saying that it's started processing the PID I gave it?
from memleax.
jonesmz
commented on May 18, 2024
Here's the latest run.
Sorry, had to remove the callstacks before copying.
bash# ./memleax -l 10 -m 100 -c 100 4011
- memleax start, with memory: 1140KB
- start build address maps...
- end build address maps, with memory: 812KB, total: 1952KB
- start build ptrace maps...
- end build ptrace maps, with memory: 0KB, total: 1952KB
- start build symtab...
- end build symtab with memory: 158824KB, total: 160776KB
- start build debug line...
- end build debug line, with memory: 84864KB, total: 245640KB
== Begin monitoring process 4011...
CallStack[1]: memory expires with 56 bytes, backtrace:
CallStack[2]: memory expires with 24 bytes, backtrace:
CallStack[2]: memory expires with 32 bytes, 2 times again
CallStack[3]: memory expires with 30 bytes, backtrace:
CallStack[3]: memory expires with 32 bytes, 2 times again
CallStack[3]: memory expires with 6 bytes, 3 times again
CallStack[3]: memory expires with 32 bytes, 4 times again
CallStack[3]: memory expires with 28 bytes, 5 times again
CallStack[3]: memory expires with 32 bytes, 6 times again
CallStack[4]: memory expires with 56 bytes, backtrace:
CallStack[5]: memory expires with 16 bytes, backtrace:
CallStack[6]: memory expires with 48 bytes, backtrace:
CallStack[5]: memory expires with 16 bytes, 2 times again
CallStack[3]: memory expires with 24 bytes, 7 times again
CallStack[3]: memory expires with 66 bytes, 8 times again
CallStack[2]: memory expires with 64 bytes, 3 times again
CallStack[3]: memory expires with 32 bytes, 9 times again
CallStack[2]: memory expires with 64 bytes, 4 times again
CallStack[3]: memory expires with 48 bytes, 10 times again
CallStack[3]: memory expires with 64 bytes, 11 times again
CallStack[3]: memory expires with 4 bytes, 12 times again
CallStack[3]: memory expires with 32 bytes, 13 times again
CallStack[3]: memory expires with 48 bytes, 14 times again
CallStack[3]: memory expires with 64 bytes, 15 times again
CallStack[3]: memory expires with 2 bytes, 16 times again
CallStack[3]: memory expires with 32 bytes, 17 times again
CallStack[3]: memory expires with 64 bytes, 18 times again
CallStack[3]: memory expires with 2 bytes, 19 times again
CallStack[3]: memory expires with 32 bytes, 20 times again
CallStack[3]: memory expires with 48 bytes, 21 times again
CallStack[3]: memory expires with 64 bytes, 22 times again
CallStack[3]: memory expires with 32 bytes, 23 times again
CallStack[3]: memory expires with 48 bytes, 24 times again
CallStack[3]: memory expires with 64 bytes, 25 times again
CallStack[3]: memory expires with 2 bytes, 26 times again
CallStack[3]: memory expires with 32 bytes, 27 times again
CallStack[3]: memory expires with 48 bytes, 28 times again
CallStack[3]: memory expires with 64 bytes, 29 times again
CallStack[3]: memory expires with 7 bytes, 30 times again
CallStack[3]: memory expires with 32 bytes, 31 times again
CallStack[3]: memory expires with 48 bytes, 32 times again
CallStack[3]: memory expires with 64 bytes, 33 times again
CallStack[3]: memory expires with 4 bytes, 34 times again
CallStack[3]: memory expires with 32 bytes, 35 times again
CallStack[3]: memory expires with 48 bytes, 36 times again
CallStack[3]: memory expires with 64 bytes, 37 times again
CallStack[3]: memory expires with 4 bytes, 38 times again
CallStack[3]: memory expires with 32 bytes, 39 times again
CallStack[3]: memory expires with 48 bytes, 40 times again
CallStack[3]: memory expires with 64 bytes, 41 times again
CallStack[3]: memory expires with 6 bytes, 42 times again
CallStack[3]: memory expires with 32 bytes, 43 times again
CallStack[3]: memory expires with 48 bytes, 44 times again
CallStack[3]: memory expires with 64 bytes, 45 times again
CallStack[3]: memory expires with 3 bytes, 46 times again
CallStack[3]: memory expires with 48 bytes, 47 times again
CallStack[3]: memory expires with 64 bytes, 48 times again
CallStack[3]: memory expires with 6 bytes, 49 times again
CallStack[3]: memory expires with 48 bytes, 50 times again
CallStack[3]: memory expires with 64 bytes, 51 times again
CallStack[3]: memory expires with 66 bytes, 52 times again
CallStack[3]: memory expires with 24 bytes, 53 times again
CallStack[3]: memory expires with 37 bytes, 54 times again
CallStack[2]: memory expires with 144 bytes, 5 times again
CallStack[7]: memory expires with 120 bytes, backtrace:
CallStack[2]: memory expires with 24 bytes, 6 times again
CallStack[2]: memory expires with 120 bytes, 7 times again
CallStack[3]: memory expires with 32 bytes, 55 times again
CallStack[3]: memory expires with 24 bytes, 56 times again
CallStack[2]: memory expires with 64 bytes, 8 times again
CallStack[2]: memory expires with 40 bytes, 9 times again
CallStack[3]: memory expires with 264 bytes, 57 times again
CallStack[3]: memory expires with 64 bytes, 58 times again
CallStack[3]: memory expires with 16 bytes, 59 times again
CallStack[2]: memory expires with 40 bytes, 10 times again
CallStack[3]: memory expires with 64 bytes, 60 times again
CallStack[3]: memory expires with 20 bytes, 61 times again
CallStack[3]: memory expires with 64 bytes, 62 times again
CallStack[3]: memory expires with 64 bytes, 63 times again
CallStack[3]: memory expires with 32 bytes, 64 times again
CallStack[3]: memory expires with 64 bytes, 65 times again
CallStack[3]: memory expires with 28 bytes, 66 times again
CallStack[3]: memory expires with 64 bytes, 67 times again
CallStack[3]: memory expires with 36 bytes, 68 times again
CallStack[3]: memory expires with 64 bytes, 69 times again
CallStack[3]: memory expires with 20 bytes, 70 times again
CallStack[2]: memory expires with 64 bytes, 11 times again
CallStack[3]: memory expires with 32 bytes, 71 times again
CallStack[3]: memory expires with 32 bytes, 72 times again
CallStack[3]: memory expires with 32 bytes, 73 times again
CallStack[3]: memory expires with 32 bytes, 74 times again
CallStack[3]: memory expires with 32 bytes, 75 times again
CallStack[3]: memory expires with 20 bytes, 76 times again
CallStack[3]: memory expires with 20 bytes, 77 times again
CallStack[3]: memory expires with 20 bytes, 78 times again
CallStack[3]: memory expires with 128 bytes, 79 times again
CallStack[2]: memory expires with 16 bytes, 12 times again
CallStack[3]: memory expires with 128 bytes, 80 times again
CallStack[3]: memory expires with 264 bytes, 81 times again
CallStack[3]: memory expires with 128 bytes, 82 times again
CallStack[8]: memory expires with 768 bytes, backtrace:
CallStack[9]: memory expires with 1024 bytes, backtrace:
CallStack[8]: memory expires with 768 bytes, 2 times again
CallStack[10]: memory expires with 1024 bytes, backtrace:
CallStack[3]: memory expires with 128 bytes, 83 times again
CallStack[3]: memory expires with 6 bytes, 84 times again
CallStack[3]: memory expires with 96 bytes, 85 times again
CallStack[11]: memory expires with 50 bytes, backtrace:
CallStack[2]: memory expires with 24 bytes, 13 times again
CallStack[2]: memory expires with 48 bytes, 14 times again
CallStack[3]: memory expires with 144 bytes, 86 times again
CallStack[3]: memory expires with 136 bytes, 87 times again
CallStack[3]: memory expires with 32 bytes, 88 times again
CallStack[2]: memory expires with 40 bytes, 15 times again
CallStack[2]: memory expires with 96 bytes, 16 times again
CallStack[2]: memory expires with 48 bytes, 17 times again
CallStack[3]: memory expires with 256 bytes, 89 times again
CallStack[2]: memory expires with 24 bytes, 18 times again
CallStack[2]: memory expires with 80 bytes, 19 times again
CallStack[2]: memory expires with 48 bytes, 20 times again
CallStack[3]: memory expires with 144 bytes, 90 times again
CallStack[11]: memory expires with 20 bytes, 2 times again
CallStack[2]: memory expires with 40 bytes, 21 times again
CallStack[3]: memory expires with 256 bytes, 91 times again
CallStack[3]: memory expires with 18 bytes, 92 times again
CallStack[3]: memory expires with 8 bytes, 93 times again
CallStack[3]: memory expires with 15 bytes, 94 times again
CallStack[3]: memory expires with 89 bytes, 95 times again
CallStack[3]: memory expires with 48 bytes, 96 times again
CallStack[2]: memory expires with 176 bytes, 22 times again
CallStack[3]: memory expires with 75 bytes, 97 times again
CallStack[3]: memory expires with 64 bytes, 98 times again
CallStack[3]: memory expires with 8 bytes, 99 times again
CallStack[3]: memory expires with 32 bytes, 100 times again
== 100 memory blocks leaked at CallStack[3].
== Callstack statistics: (in ascending order)
CallStack[6]: may-leak=1 (48 bytes)
expired=1 (48 bytes), free_expired=0 (0 bytes)
alloc=1 (48 bytes), free=0 (0 bytes)
freed memory live time: min=0 max=0 average=0
un-freed memory live time: max=10
CallStack[4]: may-leak=1 (56 bytes)
expired=1 (56 bytes), free_expired=0 (0 bytes)
alloc=1 (56 bytes), free=0 (0 bytes)
freed memory live time: min=0 max=0 average=0
un-freed memory live time: max=10
CallStack[1]: may-leak=1 (56 bytes)
expired=1 (56 bytes), free_expired=0 (0 bytes)
alloc=1 (56 bytes), free=0 (0 bytes)
freed memory live time: min=0 max=0 average=0
un-freed memory live time: max=10
CallStack[9]: may-leak=1 (1024 bytes)
expired=1 (1024 bytes), free_expired=0 (0 bytes)
alloc=918 (945152 bytes), free=902 (928768 bytes)
freed memory live time: min=0 max=0 average=0
un-freed memory live time: max=10
CallStack[7]: may-leak=1 (120 bytes)
expired=1 (120 bytes), free_expired=0 (0 bytes)
alloc=1 (120 bytes), free=0 (0 bytes)
freed memory live time: min=0 max=0 average=0
un-freed memory live time: max=10
CallStack[10]: may-leak=1 (1024 bytes)
expired=1 (1024 bytes), free_expired=0 (0 bytes)
alloc=57 (58368 bytes), free=55 (56320 bytes)
freed memory live time: min=0 max=0 average=0
un-freed memory live time: max=10
CallStack[11]: may-leak=2 (70 bytes)
expired=2 (70 bytes), free_expired=0 (0 bytes)
alloc=1281 (367994 bytes), free=1164 (359184 bytes)
freed memory live time: min=2 max=2 average=0
un-freed memory live time: max=10
CallStack[5]: may-leak=2 (32 bytes)
expired=2 (32 bytes), free_expired=0 (0 bytes)
alloc=2 (32 bytes), free=0 (0 bytes)
freed memory live time: min=0 max=0 average=0
un-freed memory live time: max=10
CallStack[8]: may-leak=2 (1536 bytes)
expired=2 (1536 bytes), free_expired=0 (0 bytes)
alloc=1269 (974592 bytes), free=1238 (950784 bytes)
freed memory live time: min=0 max=2 average=0
un-freed memory live time: max=10
CallStack[2]: may-leak=22 (1320 bytes)
expired=22 (1320 bytes), free_expired=0 (0 bytes)
alloc=4427 (1553691 bytes), free=3954 (1520634 bytes)
freed memory live time: min=0 max=5 average=0
un-freed memory live time: max=10
CallStack[3]: may-leak=100 (5376 bytes)
expired=100 (5376 bytes), free_expired=0 (0 bytes)
alloc=18001 (2538588 bytes), free=16721 (2157239 bytes)
freed memory live time: min=0 max=5 average=0
un-freed memory live time: max=10
from memleax.
jonesmz
commented on May 18, 2024
Perhaps a debug build that prints out how much memory memleaxs thinks it used at the time the program under test exits?
from memleax.
jonesmz
commented on May 18, 2024
Actually, now that I've run memleax a few dozen times, I can say categorically that it's always crashing the program under test when memleax stops.
If I control+c the program after memleaxs prints
== Begin monitoring process 4011...
Then the program under test always exits and gets restarted by my service manager.
Is that expected behavior?
Initially I had assumed that the program was crashing, but it now appears that memleaxs is automatically terminating it when too many memory blocks are used. I drastically increased the number of memory blocks to a few million, and the program stays alive significantly longer. When I do that, memleaxs stays at around 30% of system memory for a long while.
from memleax.
WuBingzheng
commented on May 18, 2024
Thanks for so many details.
Yes, memleax may use much CPU after attaching the target process. It depends on how much malloc/free() the target process calls.
No, memleax crashing the target process is not expected behavior. It must be a bug....
I will build another debug version with more debug logs today.
from memleax.
jonesmz
commented on May 18, 2024
I haven't seriously investigated the code of memleax to find this out myself, but how exactly are the results captured?
Does memleax do a lot of processing on every breakpoint hit? Or does it get the information from the program under test as quickly as possible and continue the program?
I ask because I noticed that when I attach memleaxs to my program, some transactions that typically only take a few milliseconds (like, less than 10 for the most part), start taking over a second or two when memleax is running.
Unfortunately, since I'm trying to debug a single process in a multi-process distributed system, that means that as soon as I attach memleax to my process, the rest of the system marks it as misbehaving and stops interacting with it.
Obviously I can't expect you to re-engineer your tool for me, but are there any architectural hurdles to speeding up the capture process? Perhaps something I can help with, time permitting?
from memleax.
jonesmz
commented on May 18, 2024
Btw, memleax is currently running at about 50% of system memory, with 12GB of ram in the system total. It's been running for a handful of hours now.
I have it set to -m 1000000000 and -c 1000000000 and -e 7200 (2 hours).
We have many types of transactions that can be very long lived, so I don't have a huge amount of choice in the duration that I track for.
That being said, reducing those parameters doesn't seem to have a big effect on the used system memory.
from memleax.
WuBingzheng
commented on May 18, 2024
I add a debug log to print the current memleax memory, and a 'possible' huge-memory-alloc point count. Please try the new version memleax at the attachment.
The job that memleax does on breakpoint is that,
- at malloc-breakpoint, it record the memory-pointer and callstack;
- at free-breakpoint, it release the according record by memory-pointer.
It's a little slow to get the callstack at malloc-breakpoint, because it has to read another process's (the target process) memory by ptrace() syscalls some times.
I cache some result to speed up this, but it's still not very quick.
If your program calls malloc() very often, memleax will slow it down much.
If your program manages memory itself, like many big programs, memleax will slow it down slightly.
from memleax.
WuBingzheng
commented on May 18, 2024
The last bug-fix version I gave you, is wrong. I fix the memory leak but brought another bug.
Please do not use it again.
from memleax.
jonesmz
commented on May 18, 2024
Heres the output
I filtered it using:
cat debug-memleaxs.txt | grep -v "0x0000" | grep -v "times again" | grep -v "will not show" | grep -v "expired-memory frees" | grep -v "memory expires" > debug-memleax-stripped.txt
and truncated at == Callstack statistics: (in ascending order)
- memleax start, with memory: 1160KB
- start build address maps...
- end build address maps, with memory: 876KB, total: 2036KB
- start build ptrace maps...
- end build ptrace maps, with memory: 0KB, total: 2036KB
- start build symtab...
- end build symtab with memory: 158824KB, total: 160860KB
- start build debug line...
- end build debug line, with memory: 84800KB, total: 245660KB
== Begin monitoring process 18488... - [debug] memory: unw_info:1KB (28), total:3805924KB
- [debug] memory: unw_info:14KB (210), total:3806404KB
- [debug] memory: unw_info:14KB (210), total:3806424KB
- [debug] memory: unw_info:14KB (210), total:3806428KB
- [debug] memory: unw_info:14KB (210), total:3806428KB
- [debug] memory: unw_info:20KB (292), total:3806592KB
- [debug] memory: unw_info:27KB (387), total:3806804KB
- [debug] memory: unw_info:40KB (577), total:3806640KB
- [debug] memory: unw_info:40KB (577), total:3806640KB
- [debug] memory: unw_info:40KB (577), total:3806640KB
- [debug] memory: unw_info:40KB (577), total:3806640KB
- [debug] memory: unw_info:40KB (577), total:3806640KB
- [debug] memory: unw_info:40KB (577), total:3806640KB
- [debug] memory: unw_info:40KB (579), total:3806828KB
- [debug] memory: unw_info:40KB (579), total:3806640KB
- [debug] memory: unw_info:40KB (579), total:3806640KB
- [debug] memory: unw_info:40KB (579), total:3806640KB
- [debug] memory: unw_info:40KB (579), total:3806640KB
- [debug] memory: unw_info:40KB (579), total:3806640KB
- [debug] memory: unw_info:40KB (579), total:3806640KB
- [debug] memory: unw_info:54KB (772), total:3806896KB
- [debug] memory: unw_info:61KB (880), total:3807404KB
- [debug] memory: unw_info:61KB (880), total:3807404KB
- [debug] memory: unw_info:61KB (880), total:3806640KB
- [debug] memory: unw_info:61KB (881), total:3806640KB
- [debug] memory: unw_info:61KB (881), total:3806640KB
- [debug] memory: unw_info:61KB (881), total:3806640KB
- [debug] memory: unw_info:62KB (887), total:3806828KB
- [debug] memory: unw_info:63KB (896), total:3807020KB
- [debug] memory: unw_info:63KB (903), total:3807084KB
- [debug] memory: unw_info:63KB (903), total:3807208KB
- [debug] memory: unw_info:63KB (903), total:3807208KB
- [debug] memory: unw_info:63KB (903), total:3807208KB
- [debug] memory: unw_info:63KB (903), total:3806640KB
- [debug] memory: unw_info:63KB (903), total:3806640KB
- [debug] memory: unw_info:63KB (903), total:3806640KB
- [debug] memory: unw_info:64KB (913), total:3807020KB
- [debug] memory: unw_info:64KB (913), total:3807020KB
- [debug] memory: unw_info:64KB (913), total:3807020KB
- [debug] memory: unw_info:64KB (913), total:3807020KB
- [debug] memory: unw_info:64KB (913), total:3807020KB
- [debug] memory: unw_info:64KB (913), total:3807020KB
- [debug] memory: unw_info:64KB (913), total:3807020KB
- [debug] memory: unw_info:64KB (913), total:3806640KB
- [debug] memory: unw_info:64KB (913), total:3806640KB
- [debug] memory: unw_info:79KB (1131), total:3806832KB
- [debug] memory: unw_info:79KB (1131), total:3806832KB
- [debug] memory: unw_info:79KB (1131), total:3806832KB
- [debug] memory: unw_info:79KB (1131), total:3806832KB
- [debug] memory: unw_info:79KB (1131), total:3806832KB
- [debug] memory: unw_info:79KB (1131), total:3806832KB
- [debug] memory: unw_info:79KB (1131), total:3806832KB
- [debug] memory: unw_info:79KB (1131), total:3806832KB
- [debug] memory: unw_info:79KB (1131), total:3806640KB
- [debug] memory: unw_info:79KB (1131), total:3806640KB
- [debug] memory: unw_info:80KB (1139), total:3806640KB
- [debug] memory: unw_info:80KB (1139), total:3806640KB
- [debug] memory: unw_info:87KB (1246), total:3807208KB
- [debug] memory: unw_info:87KB (1246), total:3807208KB
- [debug] memory: unw_info:87KB (1246), total:3807208KB
- [debug] memory: unw_info:87KB (1246), total:3807208KB
- [debug] memory: unw_info:87KB (1246), total:3807208KB
- [debug] memory: unw_info:87KB (1249), total:3806640KB
- [debug] memory: unw_info:87KB (1249), total:3806640KB
- [debug] memory: unw_info:87KB (1249), total:3806640KB
- [debug] memory: unw_info:87KB (1249), total:3806640KB
- [debug] memory: unw_info:87KB (1249), total:3806640KB
- [debug] memory: unw_info:87KB (1249), total:3806640KB
- [debug] memory: unw_info:87KB (1249), total:3806640KB
- [debug] memory: unw_info:87KB (1249), total:3806640KB
- [debug] memory: unw_info:87KB (1249), total:3806640KB
- [debug] memory: unw_info:87KB (1249), total:3806640KB
- [debug] memory: unw_info:87KB (1249), total:3806640KB
- [debug] memory: unw_info:87KB (1250), total:3806640KB
- [debug] memory: unw_info:87KB (1250), total:3806640KB
- [debug] memory: unw_info:87KB (1250), total:3806640KB
- [debug] memory: unw_info:87KB (1250), total:3806640KB
- [debug] memory: unw_info:87KB (1250), total:3806640KB
- [debug] memory: unw_info:87KB (1250), total:3806640KB
- [debug] memory: unw_info:87KB (1251), total:3806640KB
- [debug] memory: unw_info:87KB (1251), total:3806640KB
- [debug] memory: unw_info:87KB (1251), total:3806640KB
- [debug] memory: unw_info:87KB (1251), total:3806640KB
- [debug] memory: unw_info:87KB (1251), total:3806640KB
- [debug] memory: unw_info:87KB (1251), total:3806640KB
- [debug] memory: unw_info:87KB (1251), total:3806640KB
- [debug] memory: unw_info:87KB (1251), total:3806640KB
- [debug] memory: unw_info:87KB (1251), total:3806640KB
- [debug] memory: unw_info:87KB (1251), total:3806640KB
- [debug] memory: unw_info:118KB (1683), total:3807192KB
- [debug] memory: unw_info:118KB (1683), total:3807220KB
- [debug] memory: unw_info:118KB (1683), total:3807220KB
- [debug] memory: unw_info:118KB (1683), total:3807220KB
- [debug] memory: unw_info:118KB (1683), total:3807220KB
- [debug] memory: unw_info:118KB (1683), total:3807220KB
- [debug] memory: unw_info:118KB (1683), total:3807220KB
- [debug] memory: unw_info:118KB (1683), total:3807224KB
- [debug] memory: unw_info:118KB (1683), total:3807224KB
- [debug] memory: unw_info:118KB (1683), total:3806844KB
- [debug] memory: unw_info:118KB (1683), total:3806852KB
- [debug] memory: unw_info:118KB (1684), total:3806884KB
- [debug] memory: unw_info:118KB (1684), total:3806884KB
- [debug] memory: unw_info:118KB (1684), total:3806884KB
- [debug] memory: unw_info:118KB (1684), total:3806884KB
- [debug] memory: unw_info:118KB (1684), total:3806884KB
- [debug] memory: unw_info:118KB (1684), total:3806904KB
- [debug] memory: unw_info:118KB (1684), total:3806960KB
- [debug] memory: unw_info:118KB (1684), total:3806960KB
- [debug] memory: unw_info:118KB (1684), total:3806960KB
- [debug] memory: unw_info:118KB (1684), total:3806960KB
- [debug] memory: unw_info:118KB (1684), total:3806960KB
- [debug] memory: unw_info:118KB (1684), total:3806960KB
- [debug] memory: unw_info:118KB (1684), total:3806960KB
- [debug] memory: unw_info:118KB (1684), total:3806960KB
- [debug] memory: unw_info:118KB (1684), total:3806960KB
- [debug] memory: unw_info:118KB (1689), total:3807188KB
- [debug] memory: unw_info:121KB (1721), total:3807188KB
- [debug] memory: unw_info:121KB (1721), total:3807188KB
- [debug] memory: unw_info:121KB (1721), total:3807188KB
- [debug] memory: unw_info:121KB (1721), total:3807188KB
- [debug] memory: unw_info:121KB (1721), total:3807188KB
- [debug] memory: unw_info:121KB (1721), total:3807188KB
- [debug] memory: unw_info:121KB (1721), total:3807192KB
- [debug] memory: unw_info:121KB (1721), total:3807192KB
- [debug] memory: unw_info:121KB (1721), total:3807000KB
- [debug] memory: unw_info:121KB (1721), total:3807044KB
- [debug] memory: unw_info:121KB (1721), total:3807044KB
- [debug] memory: unw_info:121KB (1722), total:3807064KB
- [debug] memory: unw_info:121KB (1722), total:3807072KB
- [debug] memory: unw_info:121KB (1722), total:3807072KB
- [debug] memory: unw_info:121KB (1722), total:3807072KB
- [debug] memory: unw_info:121KB (1722), total:3807072KB
- [debug] memory: unw_info:121KB (1722), total:3807076KB
- [debug] memory: unw_info:121KB (1722), total:3807076KB
- [debug] memory: unw_info:121KB (1722), total:3807096KB
- [debug] memory: unw_info:121KB (1722), total:3807176KB
- [debug] memory: unw_info:121KB (1722), total:3807176KB
- [debug] memory: unw_info:121KB (1722), total:3807176KB
- [debug] memory: unw_info:121KB (1722), total:3807176KB
- [debug] memory: unw_info:121KB (1722), total:3807176KB
- [debug] memory: unw_info:121KB (1722), total:3807176KB
- [debug] memory: unw_info:121KB (1722), total:3807176KB
- [debug] memory: unw_info:121KB (1722), total:3807176KB
- [debug] memory: unw_info:121KB (1722), total:3807176KB
- [debug] memory: unw_info:121KB (1722), total:3807176KB
- [debug] memory: unw_info:121KB (1730), total:3807176KB
- [debug] memory: unw_info:121KB (1730), total:3807176KB
- [debug] memory: unw_info:121KB (1730), total:3807176KB
- [debug] memory: unw_info:121KB (1730), total:3807176KB
- [debug] memory: unw_info:121KB (1730), total:3807176KB
- [debug] memory: unw_info:121KB (1730), total:3807224KB
- [debug] memory: unw_info:122KB (1747), total:3807632KB
- [debug] memory: unw_info:122KB (1747), total:3807632KB
- [debug] memory: unw_info:122KB (1747), total:3807632KB
- [debug] memory: unw_info:122KB (1747), total:3807632KB
- [debug] memory: unw_info:122KB (1747), total:3807636KB
- [debug] memory: unw_info:122KB (1747), total:3807300KB
- [debug] memory: unw_info:122KB (1747), total:3807320KB
- [debug] memory: unw_info:122KB (1747), total:3807324KB
- [debug] memory: unw_info:122KB (1747), total:3807324KB
- [debug] memory: unw_info:122KB (1747), total:3807324KB
- [debug] memory: unw_info:122KB (1747), total:3807332KB
- [debug] memory: unw_info:122KB (1747), total:3807372KB
- [debug] memory: unw_info:122KB (1748), total:3807396KB
- [debug] memory: unw_info:122KB (1748), total:3807408KB
- [debug] memory: unw_info:122KB (1748), total:3807408KB
- [debug] memory: unw_info:122KB (1748), total:3807412KB
- [debug] memory: unw_info:122KB (1748), total:3807416KB
- [debug] memory: unw_info:122KB (1748), total:3807424KB
- [debug] memory: unw_info:122KB (1748), total:3807428KB
- [debug] memory: unw_info:122KB (1748), total:3807440KB
- [debug] memory: unw_info:122KB (1748), total:3807440KB
- [debug] memory: unw_info:122KB (1748), total:3807440KB
- [debug] memory: unw_info:122KB (1748), total:3807440KB
- [debug] memory: unw_info:122KB (1748), total:3807524KB
- [debug] memory: unw_info:122KB (1748), total:3807544KB
- [debug] memory: unw_info:123KB (1754), total:3807548KB
- [debug] memory: unw_info:123KB (1754), total:3807588KB
- [debug] memory: unw_info:123KB (1755), total:3807628KB
- [debug] memory: unw_info:123KB (1755), total:3807628KB
- [debug] memory: unw_info:123KB (1757), total:3807820KB
- [debug] memory: unw_info:123KB (1757), total:3807820KB
- [debug] memory: unw_info:123KB (1757), total:3807820KB
- [debug] memory: unw_info:124KB (1777), total:3807884KB
- [debug] memory: unw_info:124KB (1777), total:3807912KB
- [debug] memory: unw_info:124KB (1777), total:3807912KB
- [debug] memory: unw_info:124KB (1777), total:3807912KB
- [debug] memory: unw_info:124KB (1777), total:3807720KB
- [debug] memory: unw_info:124KB (1777), total:3807720KB
- [debug] memory: unw_info:124KB (1777), total:3807720KB
- [debug] memory: unw_info:124KB (1777), total:3807720KB
- [debug] memory: unw_info:124KB (1777), total:3807728KB
- [debug] memory: unw_info:124KB (1777), total:3807728KB
- [debug] memory: unw_info:124KB (1777), total:3807728KB
- [debug] memory: unw_info:124KB (1777), total:3807728KB
- [debug] memory: unw_info:124KB (1777), total:3807728KB
- [debug] memory: unw_info:124KB (1777), total:3807728KB
- [debug] memory: unw_info:124KB (1777), total:3807728KB
- [debug] memory: unw_info:124KB (1777), total:3807728KB
- [debug] memory: unw_info:124KB (1777), total:3807732KB
- [debug] memory: unw_info:124KB (1777), total:3807764KB
- [debug] memory: unw_info:124KB (1777), total:3807812KB
- [debug] memory: unw_info:124KB (1777), total:3807812KB
- [debug] memory: unw_info:124KB (1777), total:3807812KB
- [debug] memory: unw_info:124KB (1777), total:3807812KB
- [debug] memory: unw_info:124KB (1777), total:3807812KB
- [debug] memory: unw_info:125KB (1784), total:3808004KB
- [debug] memory: unw_info:125KB (1784), total:3808004KB
- [debug] memory: unw_info:125KB (1784), total:3808004KB
- [debug] memory: unw_info:125KB (1784), total:3807820KB
- [debug] memory: unw_info:125KB (1784), total:3807828KB
- [debug] memory: unw_info:125KB (1784), total:3807844KB
- [debug] memory: unw_info:125KB (1785), total:3807848KB
- [debug] memory: unw_info:125KB (1785), total:3807860KB
- [debug] memory: unw_info:125KB (1785), total:3807860KB
- [debug] memory: unw_info:125KB (1785), total:3807860KB
- [debug] memory: unw_info:125KB (1785), total:3807868KB
- [debug] memory: unw_info:125KB (1785), total:3807868KB
- [debug] memory: unw_info:125KB (1785), total:3807868KB
- [debug] memory: unw_info:125KB (1785), total:3807872KB
- [debug] memory: unw_info:125KB (1786), total:3807880KB
- [debug] memory: unw_info:125KB (1786), total:3807888KB
- [debug] memory: unw_info:125KB (1786), total:3807888KB
- [debug] memory: unw_info:125KB (1786), total:3807888KB
- [debug] memory: unw_info:126KB (1798), total:3808080KB
- [debug] memory: unw_info:126KB (1798), total:3808080KB
- [debug] memory: unw_info:126KB (1798), total:3808080KB
- [debug] memory: unw_info:126KB (1798), total:3807888KB
- [debug] memory: unw_info:126KB (1798), total:3807888KB
- [debug] memory: unw_info:126KB (1798), total:3807888KB
- [debug] memory: unw_info:126KB (1798), total:3807888KB
- [debug] memory: unw_info:126KB (1798), total:3807888KB
- [debug] memory: unw_info:126KB (1798), total:3807888KB
- [debug] memory: unw_info:126KB (1798), total:3807888KB
- [debug] memory: unw_info:126KB (1798), total:3807888KB
- [debug] memory: unw_info:126KB (1798), total:3807888KB
- [debug] memory: unw_info:126KB (1798), total:3807888KB
- [debug] memory: unw_info:126KB (1798), total:3807888KB
- [debug] memory: unw_info:126KB (1798), total:3807912KB
- [debug] memory: unw_info:126KB (1798), total:3807912KB
- [debug] memory: unw_info:126KB (1798), total:3807912KB
- [debug] memory: unw_info:126KB (1798), total:3807912KB
- [debug] memory: unw_info:126KB (1798), total:3807912KB
- [debug] memory: unw_info:126KB (1798), total:3807912KB
- [debug] memory: unw_info:126KB (1798), total:3807912KB
- [debug] memory: unw_info:126KB (1798), total:3807912KB
- [debug] memory: unw_info:126KB (1801), total:3808104KB
- [debug] memory: unw_info:126KB (1801), total:3807912KB
- [debug] memory: unw_info:126KB (1801), total:3807912KB
- [debug] memory: unw_info:126KB (1801), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3808104KB
- [debug] memory: unw_info:126KB (1804), total:3808104KB
- [debug] memory: unw_info:126KB (1804), total:3808104KB
- [debug] memory: unw_info:126KB (1804), total:3808104KB
- [debug] memory: unw_info:126KB (1804), total:3808104KB
- [debug] memory: unw_info:126KB (1804), total:3808104KB
- [debug] memory: unw_info:126KB (1804), total:3808104KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807912KB
- [debug] memory: unw_info:126KB (1804), total:3807984KB
- [debug] memory: unw_info:126KB (1804), total:3807984KB
- [debug] memory: unw_info:126KB (1804), total:3807984KB
- [debug] memory: unw_info:126KB (1804), total:3807984KB
- [debug] memory: unw_info:126KB (1804), total:3807984KB
- [debug] memory: unw_info:126KB (1804), total:3807984KB
- [debug] memory: unw_info:126KB (1804), total:3807984KB
- [debug] memory: unw_info:126KB (1804), total:3807984KB
- [debug] memory: unw_info:126KB (1804), total:3807996KB
- [debug] memory: unw_info:126KB (1804), total:3807996KB
- [debug] memory: unw_info:126KB (1804), total:3807996KB
- [debug] memory: unw_info:126KB (1804), total:3807996KB
- [debug] memory: unw_info:126KB (1804), total:3807996KB
- [debug] memory: unw_info:126KB (1804), total:3807996KB
- [debug] memory: unw_info:126KB (1804), total:3808004KB
- [debug] memory: unw_info:126KB (1804), total:3808008KB
- [debug] memory: unw_info:126KB (1804), total:3808008KB
- [debug] memory: unw_info:126KB (1804), total:3808008KB
- [debug] memory: unw_info:126KB (1804), total:3808008KB
- [debug] memory: unw_info:126KB (1804), total:3808008KB
- [debug] memory: unw_info:126KB (1804), total:3808008KB
- [debug] memory: unw_info:126KB (1804), total:3808008KB
- [debug] memory: unw_info:126KB (1804), total:3808008KB
- [debug] memory: unw_info:126KB (1804), total:3808012KB
- [debug] memory: unw_info:126KB (1804), total:3808012KB
- [debug] memory: unw_info:126KB (1804), total:3808012KB
- [debug] memory: unw_info:126KB (1804), total:3808012KB
- [debug] memory: unw_info:126KB (1804), total:3808012KB
- [debug] memory: unw_info:126KB (1804), total:3808012KB
- [debug] memory: unw_info:126KB (1804), total:3808012KB
- [debug] memory: unw_info:126KB (1804), total:3808012KB
- [debug] memory: unw_info:126KB (1804), total:3808012KB
- [debug] memory: unw_info:126KB (1804), total:3808012KB
- [debug] memory: unw_info:126KB (1804), total:3808016KB
- [debug] memory: unw_info:126KB (1804), total:3808064KB
- [debug] memory: unw_info:126KB (1804), total:3808064KB
- [debug] memory: unw_info:126KB (1804), total:3808064KB
- [debug] memory: unw_info:126KB (1804), total:3808064KB
- [debug] memory: unw_info:126KB (1804), total:3808064KB
- [debug] memory: unw_info:126KB (1804), total:3808064KB
- [debug] memory: unw_info:126KB (1804), total:3808064KB
- [debug] memory: unw_info:126KB (1804), total:3808064KB
- [debug] memory: unw_info:126KB (1804), total:3808064KB
- [debug] memory: unw_info:126KB (1804), total:3808064KB
- [debug] memory: unw_info:126KB (1804), total:3808076KB
- [debug] memory: unw_info:126KB (1804), total:3808080KB
- [debug] memory: unw_info:126KB (1804), total:3808080KB
- [debug] memory: unw_info:126KB (1804), total:3808080KB
- [debug] memory: unw_info:126KB (1804), total:3808080KB
- [debug] memory: unw_info:126KB (1804), total:3808080KB
- [debug] memory: unw_info:126KB (1804), total:3808080KB
- [debug] memory: unw_info:126KB (1804), total:3808080KB
- [debug] memory: unw_info:126KB (1804), total:3808080KB
- [debug] memory: unw_info:126KB (1805), total:3808088KB
- [debug] memory: unw_info:126KB (1805), total:3808088KB
- [debug] memory: unw_info:126KB (1805), total:3808088KB
- [debug] memory: unw_info:126KB (1805), total:3808088KB
- [debug] memory: unw_info:126KB (1805), total:3808088KB
- [debug] memory: unw_info:126KB (1805), total:3808088KB
- [debug] memory: unw_info:126KB (1805), total:3808088KB
- [debug] memory: unw_info:126KB (1805), total:3808088KB
- [debug] memory: unw_info:127KB (1807), total:3808280KB
- [debug] memory: unw_info:127KB (1807), total:3808280KB
- [debug] memory: unw_info:127KB (1807), total:3808088KB
- [debug] memory: unw_info:127KB (1807), total:3808112KB
- [debug] memory: unw_info:127KB (1808), total:3808120KB
- [debug] memory: unw_info:127KB (1808), total:3808144KB
- [debug] memory: unw_info:127KB (1808), total:3808144KB
- [debug] memory: unw_info:127KB (1808), total:3808144KB
- [debug] memory: unw_info:127KB (1808), total:3808144KB
- [debug] memory: unw_info:127KB (1808), total:3808144KB
- [debug] memory: unw_info:127KB (1808), total:3808144KB
- [debug] memory: unw_info:127KB (1808), total:3808144KB
- [debug] memory: unw_info:127KB (1808), total:3808144KB
- [debug] memory: unw_info:127KB (1808), total:3808152KB
- [debug] memory: unw_info:127KB (1808), total:3808172KB
- [debug] memory: unw_info:127KB (1808), total:3808172KB
- [debug] memory: unw_info:127KB (1808), total:3808172KB
- [debug] memory: unw_info:127KB (1808), total:3808172KB
- [debug] memory: unw_info:127KB (1808), total:3808172KB
- [debug] memory: unw_info:127KB (1808), total:3808172KB
- [debug] memory: unw_info:127KB (1808), total:3808172KB
- [debug] memory: unw_info:127KB (1808), total:3808172KB
- [debug] memory: unw_info:127KB (1808), total:3808172KB
- [debug] memory: unw_info:127KB (1808), total:3808172KB
- [debug] memory: unw_info:127KB (1808), total:3808172KB
- [debug] memory: unw_info:127KB (1808), total:3808172KB
- [debug] memory: unw_info:127KB (1808), total:3808172KB
- [debug] memory: unw_info:127KB (1808), total:3808172KB
- [debug] memory: unw_info:127KB (1808), total:3808172KB
- [debug] memory: unw_info:127KB (1808), total:3808172KB
- [debug] memory: unw_info:127KB (1808), total:3808172KB
- [debug] memory: unw_info:127KB (1808), total:3808172KB
- [debug] memory: unw_info:127KB (1808), total:3808172KB
- [debug] memory: unw_info:127KB (1808), total:3808192KB
- [debug] memory: unw_info:127KB (1808), total:3808192KB
- [debug] memory: unw_info:127KB (1808), total:3808192KB
- [debug] memory: unw_info:127KB (1808), total:3808192KB
- [debug] memory: unw_info:127KB (1808), total:3808192KB
- [debug] memory: unw_info:127KB (1808), total:3808208KB
- [debug] memory: unw_info:127KB (1808), total:3808208KB
- [debug] memory: unw_info:127KB (1808), total:3808208KB
- [debug] memory: unw_info:127KB (1808), total:3808208KB
- [debug] memory: unw_info:127KB (1808), total:3808208KB
- [debug] memory: unw_info:127KB (1808), total:3808208KB
- [debug] memory: unw_info:127KB (1808), total:3808208KB
- [debug] memory: unw_info:127KB (1808), total:3808208KB
- [debug] memory: unw_info:127KB (1808), total:3808208KB
- [debug] memory: unw_info:127KB (1808), total:3808208KB
- [debug] memory: unw_info:127KB (1808), total:3808232KB
- [debug] memory: unw_info:127KB (1808), total:3808236KB
- [debug] memory: unw_info:127KB (1808), total:3808236KB
- [debug] memory: unw_info:127KB (1808), total:3808236KB
- [debug] memory: unw_info:127KB (1808), total:3808236KB
- [debug] memory: unw_info:127KB (1808), total:3808236KB
- [debug] memory: unw_info:127KB (1808), total:3808236KB
- [debug] memory: unw_info:127KB (1808), total:3808236KB
- [debug] memory: unw_info:127KB (1808), total:3808236KB
- [debug] memory: unw_info:127KB (1808), total:3808244KB
- [debug] memory: unw_info:127KB (1808), total:3808256KB
- [debug] memory: unw_info:127KB (1808), total:3808264KB
- [debug] memory: unw_info:127KB (1808), total:3808272KB
- [debug] memory: unw_info:127KB (1808), total:3808272KB
- [debug] memory: unw_info:127KB (1808), total:3808272KB
- [debug] memory: unw_info:127KB (1808), total:3808280KB
- [debug] memory: unw_info:127KB (1808), total:3808284KB
- [debug] memory: unw_info:127KB (1808), total:3808292KB
- [debug] memory: unw_info:127KB (1808), total:3808296KB
- [debug] memory: unw_info:127KB (1808), total:3808332KB
- [debug] memory: unw_info:127KB (1808), total:3808336KB
- [debug] memory: unw_info:127KB (1808), total:3808340KB
- [debug] memory: unw_info:127KB (1808), total:3808340KB
- [debug] memory: unw_info:127KB (1808), total:3808340KB
- [debug] memory: unw_info:127KB (1808), total:3808372KB
- [debug] memory: unw_info:127KB (1808), total:3808372KB
- [debug] memory: unw_info:127KB (1808), total:3808372KB
- [debug] memory: unw_info:127KB (1808), total:3808372KB
- [debug] memory: unw_info:127KB (1808), total:3808376KB
- [debug] memory: unw_info:127KB (1808), total:3808384KB
- [debug] memory: unw_info:127KB (1808), total:3808388KB
- [debug] memory: unw_info:127KB (1808), total:3808484KB
- [debug] memory: unw_info:127KB (1808), total:3808484KB
- [debug] memory: unw_info:127KB (1808), total:3808484KB
- [debug] memory: unw_info:127KB (1815), total:3808568KB
- [debug] memory: unw_info:145KB (2071), total:3808776KB
- [debug] memory: unw_info:145KB (2074), total:3808976KB
- [debug] memory: unw_info:145KB (2074), total:3808976KB
- [debug] memory: unw_info:145KB (2074), total:3808824KB
- [debug] memory: unw_info:145KB (2074), total:3808824KB
- [debug] memory: unw_info:145KB (2074), total:3808824KB
- [debug] memory: unw_info:145KB (2074), total:3808824KB
- [debug] memory: unw_info:145KB (2074), total:3808824KB
- [debug] memory: unw_info:145KB (2075), total:3808632KB
- [debug] memory: unw_info:145KB (2075), total:3808636KB
- [debug] memory: unw_info:145KB (2075), total:3808636KB
- [debug] memory: unw_info:145KB (2075), total:3808636KB
- [debug] memory: unw_info:145KB (2075), total:3808636KB
- [debug] memory: unw_info:145KB (2075), total:3808636KB
- [debug] memory: unw_info:145KB (2075), total:3808636KB
- [debug] memory: unw_info:145KB (2075), total:3808636KB
- [debug] memory: unw_info:145KB (2075), total:3808636KB
- [debug] memory: unw_info:145KB (2075), total:3808636KB
- [debug] memory: unw_info:145KB (2075), total:3808640KB
- [debug] memory: unw_info:145KB (2075), total:3808672KB
- [debug] memory: unw_info:145KB (2075), total:3808672KB
- [debug] memory: unw_info:145KB (2075), total:3808672KB
- [debug] memory: unw_info:145KB (2075), total:3808672KB
- [debug] memory: unw_info:145KB (2075), total:3808672KB
- [debug] memory: unw_info:145KB (2075), total:3808672KB
- [debug] memory: unw_info:145KB (2075), total:3808672KB
- [debug] memory: unw_info:145KB (2075), total:3808672KB
- [debug] memory: unw_info:145KB (2075), total:3808672KB
- [debug] memory: unw_info:145KB (2075), total:3808672KB
- [debug] memory: unw_info:146KB (2080), total:3809060KB
- [debug] memory: unw_info:146KB (2080), total:3808676KB
- [debug] memory: unw_info:146KB (2080), total:3808676KB
- [debug] memory: unw_info:146KB (2080), total:3808676KB
- [debug] memory: unw_info:146KB (2080), total:3808676KB
- [debug] memory: unw_info:146KB (2080), total:3808676KB
- [debug] memory: unw_info:146KB (2080), total:3808676KB
- [debug] memory: unw_info:146KB (2080), total:3808676KB
- [debug] memory: unw_info:146KB (2080), total:3808676KB
- [debug] memory: unw_info:146KB (2080), total:3808676KB
- [debug] memory: unw_info:146KB (2080), total:3808676KB
- [debug] memory: unw_info:146KB (2080), total:3808676KB
- [debug] memory: unw_info:146KB (2080), total:3808676KB
- [debug] memory: unw_info:146KB (2080), total:3808676KB
- [debug] memory: unw_info:146KB (2080), total:3808676KB
- [debug] memory: unw_info:146KB (2080), total:3808676KB
- [debug] memory: unw_info:146KB (2080), total:3808676KB
- [debug] memory: unw_info:146KB (2080), total:3808676KB
- [debug] memory: unw_info:146KB (2080), total:3808676KB
- [debug] memory: unw_info:146KB (2080), total:3808676KB
- [debug] memory: unw_info:146KB (2080), total:3808676KB
- [debug] memory: unw_info:146KB (2080), total:3808676KB
- [debug] memory: unw_info:146KB (2083), total:3808676KB
- [debug] memory: unw_info:146KB (2083), total:3808676KB
- [debug] memory: unw_info:146KB (2083), total:3808676KB
- [debug] memory: unw_info:146KB (2083), total:3808676KB
- [debug] memory: unw_info:146KB (2085), total:3808868KB
== Terminate monitoring.
== Callstack statistics: (in ascending order)
from memleax.
WuBingzheng
commented on May 18, 2024
Thanks for your wonderful output.
It seems that 3.8G memory is allocated between "== Begin monitoring process 18488..." and the first "[debug] memory: unw_info" .
I add some more debug print. Please try it again.
Thanks very much for your time.
from memleax.
jonesmz
commented on May 18, 2024
- memleax start, with memory: 1224KB
- start build address maps...
- end build address maps, with memory: 864KB, total: 2088KB
- start build ptrace maps...
- end build ptrace maps, with memory: 0KB, total: 2088KB
- start build symtab...
- end build symtab with memory: 158824KB, total: 160912KB
- start build debug line...
- end build debug line, with memory: 84804KB, total: 245716KB
- [debug] o0. memory: 3805456KB
- [debug] o0. memory: 3805544KB
- [debug] o1. memory: 3805544KB
- [debug] o0. memory: 3805544KB
- [debug] o1. memory: 3805544KB
- [debug] o0. memory: 3805544KB
- [debug] o1. memory: 3805544KB
Additionally, memleax segfaulted when I hit ctrl+c. I imagine as a sideeffect of the new debug code. So no big deal.
Need other info from that run? I stripped it the same way, but there were many thousands of lines, so I only pasted the first chunk.
from memleax.
WuBingzheng
commented on May 18, 2024
This is enough. Thanks.
It seems that huge memory is allocated between "end build debug line, with memory" and "[debug] o0. memory".
As you can see in the memleax.c , there is only one function proc_task() there.
memleax.c.txt
Could you add some debug print in proc_task()?
You can just print getMemory() some places.
I think it's more convenient you add debug print, than I add and send it to you.
Thanks
from memleax.
WuBingzheng
commented on May 18, 2024
fixed in v1.1.1.
from memleax.
Related Issues (20)
- Version for i386/32 bits HOT 7
- aarch64 support HOT 1
- Script to mimic valgrind behaviour? HOT 5
- Future supported architectures? HOT 3
- How to compile it on embeded linux HOT 1
- [fix failure to find libunwind on armhf] switch to cmake or autotools HOT 7
- 32-bit app crashes with segmentation fault on CentOS 7 x64 HOT 2
- Can't compile on CentOS 6.9 HOT 7
- does memleax handle allocations from free list correctly? HOT 11
- BUG: miss breakpoint for multi-thread program HOT 6
- BUG: crash on exit when debugging multi-thread program HOT 1
- == Attach process error:: Operation not permitted HOT 2
- not found api: malloc HOT 13
- centos 6x64 and 7x64 HOT 4
- memleax crashes the monitored process too - using version 1.1.1-1 HOT 10
- Does this support ARM platform HOT 2
- Include CMakeLists.txt in tar.gz release file HOT 5
- Memleax is killed after sometime while debugging process memory usage HOT 3
- memleax does not report for Photon OS HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from memleax.