🐛 [BUG] Soundfont - SF2 file > 1gb will fail to load

[guizmox]

🐛 [BUG] Soundfont - SF2 file > 1gb will fail to load

[guizmox]

I used a HDMI out kernel version to check the LOG, and here's the result :
It seems that the zoneallocator is fixed to 940 megabytes (line 4)

[dwhinham]

Thank you for the detailed bug report and providing the logs, it's much appreciated.

The reason for this is because of the way Circle's memory map/allocation is implemented. Unfortunately, we don't get access to the whole 4GB (or 8GB on those models) at the moment.

On Pi 4, the first 1GB is called the "low heap", and Circle's internal memory allocator uses this. The next 2GB is the "high heap" and is handled by mt32-pi's custom allocator - this is what FluidSynth uses. The remaining 1GB on Pi 4/4GB (or 5GB on Pi4/8GB) is not usable right now.

That said, we are only getting 940MB of "high heap" according to your log, when it should be closer to 2GB, so this needs investigating.

(On Pi 2/3/02W there is no "high heap", only a "low heap". Here, 32MB is reserved for Circle's internal allocator, and the remainder is handled by ours).

Circle's internal memory allocator cannot keep track of large allocations, hence the need for a custom allocator, otherwise FluidSynth cannot unload SoundFonts and free the memory for when the user wants to load another one.

There are a few things that need to happen so that we can use the rest of the memory on Pi 4:

• Treat the "low heap" like we do on Pi 2/3/02W; reserve 32MB for the internal allocator, then give the remainder to a custom allocator so that FluidSynth can use it - this will give us (almost) an additional 1GB. This shouldn't be too difficult.
• Figure out why the custom allocator isn't getting the full 2GB that Circle should be able to provide from the upper heap.
• Investigate how we can extend the range of the upper heap so that the rest of the RAM can be used. The reasons for the memory range being limited are given in the Circle documentation but it's not clear if we need to worry about the "specific DMA-handling" or not for this project.

I will need some time to figure out the best way to solve this.

P.S. thank you very much for the donation, I really appreciate it! 🙂

[guizmox]

Hello and thank you very much for the detailled answer, your project is really great, and useful for a lot of people including me.
That's seems to match with what I read from the "zoneallocator.cpp" file

const size_t nHighHeapSize = pMemorySystem->GetHeapFreeSpace(HEAP_HIGH);
	if (nHighHeapSize)
	{
		// >1GB RAM Pi 4 - allocate all of the remaining HIGH region
		m_nHeapSize = nHighHeapSize - sizeof(THeapBlockHeader);
		m_pHeap     = pMemorySystem->HeapAllocate(m_nHeapSize, HEAP_HIGH);
	}

I would first try to set a fixed value for "nHighHeapSize" and see what happens, but I'm unfortunately not able to fork your code, I'm a C# dev. with no experience with C++
An additionnal 1gb would be great, I'm fully aware that bigger SF2 files does not mean better sound, but I'm not using MT32-pi as a GM/GS machine but more as a VST instrument to extend my sound palette :)

[guizmox]

I read the memory.h Circle code, and this may explain things :

In case of Pi4, Your custom allocator does the following :
pMemorySystem->GetHeapFreeSpace(HEAP_HIGH);

According to Circle, using HEAP_ANY should also add the HEAP_LOW Free Space and return an addition of both, even if it's not clear why the HEAP_HIGH zone only returns 940mb.

I now have to figure out how I can compile your code :D

[dwhinham]

It's not that simple. The custom allocator needs to make one large, single contiguous allocation from either one of those heaps, so using the size of "any" (the two heap sizes added together) doesn't make sense. Changing the GetHeapFreeSpace() call to use HEAP_ANY will give it a bigger number, but it will be incorrect.

The best solution (which I have wanted to do for a long time due to other related issues) will be to remove Circle's allocator completely and use the Zone allocator system-wide. There is a branch (allocator) where I have been working on this in the past, but there are some show-stopping bugs that need fixing before it can be used.

[guizmox]

Interesting ! I'll look into that branch and see what I can find. Thank you :)

[guizmox]

I was able to compile the code and made a slight change :

#if RASPI >= 4
	const size_t nHighHeapSize = pMemorySystem->GetHeapFreeSpace(HEAP_HIGH);
	if (nHighHeapSize)
	{
		// >1GB RAM Pi 4 - allocate all of the remaining HIGH region
		m_nHeapSize = nHighHeapSize - sizeof(THeapBlockHeader);
		m_pHeap     = pMemorySystem->HeapAllocate(m_nHeapSize, HEAP_HIGH);
	}
	else
	{
#endif
		// Allocate the majority of the remaining LOW region; leave some space for Circle/libc malloc()
		m_nHeapSize = pMemorySystem->GetHeapFreeSpace(HEAP_LOW) - MallocHeapSize;
		m_pHeap     = pMemorySystem->HeapAllocate(m_nHeapSize, HEAP_LOW);
#if RASPI >= 4
	}
#endif

HEAP_HIGH part of the code dedicated to RPI4 was always ignored. I believe because of the RASPI variable that is never initialized in the compilation process.

My change :

	const size_t nHighHeapSize = pMemorySystem->GetHeapFreeSpace(HEAP_HIGH);
	if (nHighHeapSize)
	{
		// >1GB RAM Pi 4 - allocate all of the remaining HIGH region
		m_nHeapSize = nHighHeapSize - sizeof(THeapBlockHeader);
		m_pHeap     = pMemorySystem->HeapAllocate(m_nHeapSize, HEAP_HIGH);
	}
	else
	{
		// Allocate the majority of the remaining LOW region; leave some space for Circle/libc malloc()
		m_nHeapSize = pMemorySystem->GetHeapFreeSpace(HEAP_LOW) - MallocHeapSize;
		m_pHeap     = pMemorySystem->HeapAllocate(m_nHeapSize, HEAP_LOW);
	}

As you explained to me, the HIGH_HEAP zone was able to allocate 2048mb :)
I was able to successfully load a 1.5gb SF2 file (in 83 seconds)