Packaged game : Drone camera freezing inplace

[orig74]

Packaged game : Drone camera freezing inplace

[geevargs]

Ensure condition failed: TextureRHI [File:/home/airspace/UnrealEngine/Engine/Source/Runtime/OpenGLDrv/Private/OpenGLRenderTarget.cpp] [Line: 855]

[2018.03.23-21.29.16:602][592]LogStats: FPlatformStackWalk::StackWalkAndDump - 0.018 s
[2018.03.23-21.29.16:602][592]LogOutputDevice: Error: === Handled ensure: ===
[2018.03.23-21.29.16:602][592]LogOutputDevice: Error:
[2018.03.23-21.29.16:602][592]LogOutputDevice: Error: Ensure condition failed: TextureRHI [File:/home/airspace/UnrealEngine/Engine/Source/Runtime/OpenGLDrv/Private/OpenGLRenderTarget.cpp] [Line: 855]
[2018.03.23-21.29.16:602][592]LogOutputDevice: Error:
[2018.03.23-21.29.16:602][592]LogOutputDevice: Error: Stack:
[2018.03.23-21.29.16:602][592]LogOutputDevice: Error: [Callstack] 00 0x000000000259e0cf FLinuxPlatformStackWalk::CaptureStackBackTrace(unsigned long long*, unsigned int, void*)
[2018.03.23-21.29.16:602][592]LogOutputDevice: Error: [Callstack] 01 0x0000000002472d05 FGenericPlatformStackWalk::StackWalkAndDump(char*, unsigned long, int, void*)
[2018.03.23-21.29.16:602][592]LogOutputDevice: Error: [Callstack] 02 0x000000000259e089 FLinuxPlatformStackWalk::StackWalkAndDumpEx(char*, unsigned long, int, unsigned int, void*)
[2018.03.23-21.29.16:602][592]LogOutputDevice: Error: [Callstack] 03 0x00000000025e7148 FDebug::EnsureFailed(char const*, char const*, int, wchar_t const*)
[2018.03.23-21.29.16:602][592]LogOutputDevice: Error: [Callstack] 04 0x00000000025d344f FDebug::OptionallyLogFormattedEnsureMessageReturningFalse(bool, char const*, char const*, int, wchar_t const*, ...)
[2018.03.23-21.29.16:602][592]LogOutputDevice: Error: [Callstack] 05 0x0000000003942ba6 FOpenGLDynamicRHI::RHIReadSurfaceData(FRHITexture*, FIntRect, TArray<FColor, FDefaultAllocator>&, FReadSurfaceDataFlags)
[2018.03.23-21.29.16:602][592]LogOutputDevice: Error: [Callstack] 06 0x000000000501dffe PX4/Binaries/Linux/PX4() [0x501dffe]
[2018.03.23-21.29.16:602][592]LogOutputDevice: Error: [Callstack] 07 0x0000000005078543 PX4/Binaries/Linux/PX4() [0x5078543]
[2018.03.23-21.29.16:602][592]LogOutputDevice: Error: [Callstack] 08 0x0000000002498056 FNamedTaskThread::ProcessTasksNamedThread(int, bool)
[2018.03.23-21.29.16:602][592]LogOutputDevice: Error: [Callstack] 09 0x00000000024975ae FNamedTaskThread::ProcessTasksUntilQuit(int)
[2018.03.23-21.29.16:602][592]LogOutputDevice: Error: [Callstack] 10 0x00000000039cd32a RenderingThreadMain(FEvent*)
[2018.03.23-21.29.16:602][592]LogOutputDevice: Error: [Callstack] 11 0x00000000039db992 FRenderingThread::Run()
[2018.03.23-21.29.16:602][592]LogOutputDevice: Error: [Callstack] 12 0x00000000024cdbf7 FRunnableThreadPThread::Run()
[2018.03.23-21.29.16:602][592]LogOutputDevice: Error: [Callstack] 13 0x00000000024ab440 FRunnableThreadPThread::_ThreadProc(void*)
[2018.03.23-21.29.16:602][592]LogOutputDevice: Error: [Callstack] 14 0x00007ff02bf5d184 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7ff02bf5d184]
[2018.03.23-21.29.16:602][592]LogOutputDevice: Error: [Callstack] 15 0x00007ff02b0e7ffd /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7ff02b0e7ffd]
[2018.03.23-21.29.16:602][592]LogOutputDevice: Error:
[2018.03.23-21.29.16:602][592]LogStats: SubmitErrorReport - 0.000 s
[2018.03.23-21.29.16:639][592]LogStats: SendNewReport - 0.020 s
[2018.03.23-21.29.16:640][592]LogStats: FDebug::EnsureFailed - 0.038 s
Traceback (most recent call last):
File "", line 1, in
File "/home/airspace/DroneSimLab/UE4PyhtonBridge/src/Wrappers/pyinit.py", line 43, in PyTick
next(main_loop_iter)
File "../../DroneSimLab/demos/unreal_proxies/two_drones/unreal_proxy.py", line 117, in main_loop
img=ph.GetTextureData(drone_textures[drone_index])
File "/home/airspace/DroneSimLab/UE4PyhtonBridge/src/Wrappers/phandlers.py", line 102, in GetTextureData
tmp_capture_mem=np.zeros(req_mem_sz,'uint8')
MemoryError
ending main loop!!

[geevargs]

I ran a few test cases to diagnose the error:
It appears that the numpy array is full beyond its capacity and thus giving this error.
It only seems to be a problem in packaged games in my 2nd PC it crashes regardless of data being sent to the FDM but this problem is not seen in my dev platform.
after 1min of running.
tmp_capture_mem=np.zeros(req_mem_sz,'uint8')
MemoryError

I believe this error has occurred because I ran it for weeks continuously

I have linux 14,04 and have 12 gb free space on my PC.
Unreal 4.17.2

[orig74]

Try to add 2 lines between 101 and 102 in the file "/home/airspace/DroneSimLab/UE4PyhtonBridge/src/Wrappers/phandlers.py",
del tmp_capture_mem # free memory
print('requested size id:',req_mem_sz) #for debug

[geevargs]

I did the print('requested size id:',req_mem_sz) #for debug
requested size id: 4
requested size id: 8294400

[geevargs]

del tmp_capture_mem # free memory didnt work...should i redo set_path.py...
This occurs only in binary mode on one Pc but works fine on my dev platform.

[orig74]

Can you track the memory usage during the simulation on both systems (the dev platform and the other PC)? You can do it with tools like system monitor. It will help us understand the difference.

[geevargs]

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13978 airspace 20 0 6954920 2.059g 195052 R 208.4 13.2 3:34.98 PX4 ----- production platform
19340 sujith 20 0 6126620 2.213g 205272 S 177.4 7.1 1:02.32 MyProject2+ ------dev platform

[geevargs]

The %memory usage is more than 200% and is 170+ in the dev platform.
Everything else is similar.

[geevargs]

Production platform:

[geevargs]

Dev platform

[orig74]

can you look if the memory usage is increasing?
show me the resource graphs from the moment you open the application until it fails on both stations.

[geevargs]

Production platform:

[geevargs]

Dev Platform:

[orig74]

I don't really understand why it fails since I can't see any memory issue. but let's try and bypass the problem for now.
try and change line 82 in "phandlers.py" to something like:
tmp_capture_mem=np.zeros(8294400,'uint8')
this hopefully will eliminate the need to reallocating the memory (line 102).

[geevargs]

I tried tmp_capture_mem=np.zeros(8294400,'uint8') didnt work but i tried the same line in line 102 and the simulator lasted longer but crashed completely after 2 iterations.
thats 1 more than before

[orig74]

Can you show me what it was printing when it crashed (the debug lines we added)?

[geevargs]

Just the debug lines

[geevargs]

With the tmp_capture_mem=np.zeros(8294400,'uint8')

[orig74]

I can see the problem now, the requested size is huge. This means some kind of memory corruption somewhere. Try and add some debug prints of the size before the allocation.
print(txt_index,'req_mem_sz is ',sz,ret)
you should see more the one print of this output. send me all of the outputs of that line

[geevargs]

print(tex_ptr,'req_mem_sz is ',sz,ret) at line 101

[geevargs]

it doesnt print txt_index..its not declared before line 168
and the program fails before reaching it.

[geevargs]

I declared txt_index earlier and did a few prints.

[orig74]

I can see the problem now.
I think it is not directly related to the framework. in the output it says:
Ensure condition fails: TextureRHI: ....

I had this issue before I don't quite remember how I solved it :<
I found a reference to it:
https://answers.unrealengine.com/questions/718835/418h-must-be-first-header-when-packaging-plugin.html?sort=oldest

In any case, I will have a look later this week at my code and let you know if and when I changed anything.

[orig74]

have you tried the solution in the link?
adding to the *.build.cs file the line:

PCHUsage = ModuleRules.PCHUsageMode.UseExplicitOrSharedPCHs;

[geevargs]

PCHUsage = ModuleRules.PCHUsageMode.UseExplicitOrSharedPCHs;

[geevargs]

Didnt work but I am gonna keep working on it...do you have any other suggestions?

[orig74]

Since it is PCH related I would try to change the order of the include files in the Source of the game. I think this is how I solved it when had this error.