Out of memory for 100+ GB map

[technicjelle]

Out of memory for 100+ GB map

[TechnicJelle]

Ah, I'm glad to hear that it's working now!

And wow, that's a long time and a big world, then!
I feel like I should mention that this plugin extracts the signs from all your world files every time BlueMap loads, which is at least every server startup... (But also when you do /bluemap reload)
I have been hoping that won't turn out to be a problem, but with worlds as large as this, it probably will...

And this plugin doesn't support chunks older than 1.13. I am pretty sure it'll just skip the old chunks, though; it shouldn't stop the whole process.
But please open another bug report if it does end up stopping the whole process.

[TechnicJelle]

Thanks for the bug report!

I've been hard at work this evening with refactoring the plugin more.
There are still a couple of bugs, but if you're willing to try it out already, I'll attach a preview version that you can use.

If you do try it, please let me know if it fixes the issue!
It seems to at least already be much faster, if anything.

BlueMapSignExtractor-1.3-PREVIEW.jar.zip

[ShDis]

With 1.3-PREVIEW.jar

[14:48:18] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMap] Loading textures...
[14:48:22] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMap] Baking resources...
[14:48:22] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMap] Resources loaded.
[14:48:22] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMap] Loading world '97a8f878-4494-43d1-950e-9178e5d0e1d6' (C:\2s2d\world)...
[14:48:22] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMap] Initializing Storage: 'file' (Type: FILE)
[14:48:22] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMap] Loading map 'world'...
[14:48:23] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMap] Loading world '314ebe7f-8190-424a-8fb8-7c5f8324dba3' (C:\2s2d\world_the_end\DIM1)...
[14:48:23] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMap] Loading map 'world_the_end'...
[14:48:23] [Craft Scheduler Thread - 5 - BlueMap/WARN]: [BlueMapSignExtractor] New version available: v1.2 (current: v1.3-PREVIEW)
[14:48:23] [Craft Scheduler Thread - 5 - BlueMap/WARN]: [BlueMapSignExtractor] Download it at https://github.com/TechnicJelle/BlueMapSignExtractor/releases/latest
[14:48:23] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMapAreaControl] BlueMapAreaControl enabled!
[14:48:23] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMapAreaControl] Loading existing configs
[14:48:23] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMapAreaControl] Loading config for map: world
[14:48:23] [Craft Scheduler Thread - 3 - BlueMapSignExtractor/INFO]: [BlueMapSignExtractor] World "world": Extracting signs into markers...
[14:48:23] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMapAreaControl] Loading config for map: world_the_end
[14:48:23] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMap] Loaded!
[14:49:05] [Craft Scheduler Thread - 3 - BlueMapSignExtractor/ERROR]: [BlueMapSignExtractor] Error reading region file
java.io.EOFException: null
	at com.technicjelle.bluemapsignextractor.common.MCA.readFully(MCA.java:153) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
	at com.technicjelle.bluemapsignextractor.common.MCA.readFully(MCA.java:141) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
	at com.technicjelle.bluemapsignextractor.common.MCA.loadChunk(MCA.java:111) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
	at com.technicjelle.bluemapsignextractor.common.MCA.getBlockEntities(MCA.java:70) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
	at com.technicjelle.bluemapsignextractor.common.Core.fillMarkerSetFromRegionFile(Core.java:57) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
	at com.technicjelle.bluemapsignextractor.common.Core.lambda$loadMarkerSetFromWorld$2(Core.java:42) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) ~[?:?]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) ~[?:?]
	at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ~[?:?]
	at com.technicjelle.bluemapsignextractor.common.Core.loadMarkerSetFromWorld(Core.java:42) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
	at com.technicjelle.bluemapsignextractor.common.Core.addMarkersToBlueMapWorld(Core.java:24) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
	at com.technicjelle.bluemapsignextractor.BlueMapSignExtractor.loadMarkersFromWorlds(BlueMapSignExtractor.java:67) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
	at com.technicjelle.bluemapsignextractor.BlueMapSignExtractor.lambda$new$0(BlueMapSignExtractor.java:50) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
	at org.bukkit.craftbukkit.v1_20_R2.scheduler.CraftTask.run(CraftTask.java:101) ~[purpur-1.20.2.jar:git-Purpur-2095]
	at org.bukkit.craftbukkit.v1_20_R2.scheduler.CraftAsyncTask.run(CraftAsyncTask.java:57) ~[purpur-1.20.2.jar:git-Purpur-2095]
	at com.destroystokyo.paper.ServerSchedulerReportingWrapper.run(ServerSchedulerReportingWrapper.java:22) ~[purpur-1.20.2.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
[14:49:05] [Craft Scheduler Thread - 3 - BlueMapSignExtractor/ERROR]: [BlueMapSignExtractor] Error reading region file
java.io.EOFException: null
	at com.technicjelle.bluemapsignextractor.common.MCA.readFully(MCA.java:153) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
	at com.technicjelle.bluemapsignextractor.common.MCA.readFully(MCA.java:141) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
	at com.technicjelle.bluemapsignextractor.common.MCA.loadChunk(MCA.java:111) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
	at com.technicjelle.bluemapsignextractor.common.MCA.getBlockEntities(MCA.java:70) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
	at com.technicjelle.bluemapsignextractor.common.Core.fillMarkerSetFromRegionFile(Core.java:57) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
	at com.technicjelle.bluemapsignextractor.common.Core.lambda$loadMarkerSetFromWorld$2(Core.java:42) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) ~[?:?]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) ~[?:?]
	at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ~[?:?]
	at com.technicjelle.bluemapsignextractor.common.Core.loadMarkerSetFromWorld(Core.java:42) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
	at com.technicjelle.bluemapsignextractor.common.Core.addMarkersToBlueMapWorld(Core.java:24) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
	at com.technicjelle.bluemapsignextractor.BlueMapSignExtractor.loadMarkersFromWorlds(BlueMapSignExtractor.java:67) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
	at com.technicjelle.bluemapsignextractor.BlueMapSignExtractor.lambda$new$0(BlueMapSignExtractor.java:50) ~[BlueMapSignExtractor-1.3-PREVIEW.jar:?]
	at org.bukkit.craftbukkit.v1_20_R2.scheduler.CraftTask.run(CraftTask.java:101) ~[purpur-1.20.2.jar:git-Purpur-2095]
	at org.bukkit.craftbukkit.v1_20_R2.scheduler.CraftAsyncTask.run(CraftAsyncTask.java:57) ~[purpur-1.20.2.jar:git-Purpur-2095]
	at com.destroystokyo.paper.ServerSchedulerReportingWrapper.run(ServerSchedulerReportingWrapper.java:22) ~[purpur-1.20.2.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]

[TechnicJelle]

Ah dang, I was hoping this was a rare error, but it seems like it's more common...
Thanks for giving it a shot!
I'll try to diagnose this issue further, but I couldn't really figure it out yesterday.
Maybe today I'll have some fresh thoughts...

[TechnicJelle]

Thanks to Blue, I was able to solve the EOF Exception!
Could you please try out this new version?

BlueMapSignExtractor-1.3-PREVIEW2.jar.zip

[ShDis]

Cool! It seems to be working. I think it'll take some time to complete. Very curious to see the result (last time I collected the signs with Overviewer and with one working core of E8600 it took about three days). The map also contains some old chunks from around 1.12 or earlier