Exception while collecting markers

[technicjelle]

Exception while collecting markers

[TechnicJelle]

Aha, this was surprisingly simple to diagnose! :D

This is the internal data of the chunk:

As you can see, it's inside a Level section.

And only inside that Level section, is the actual Status:

This Level section is how it used to be before 1.18:

BlueMapSignExtractor/src/main/java/com/technicjelle/bluemapsignextractor/versions/MC_1_17_1/MC_1_17_1_Chunk.java

Lines 8 to 18 in 1e1d96a

	public class MC_1_17_1_Chunk extends ChunkWithVersion implements Chunk {
	static class Level {
	@NBTName("TileEntities")
	public MC_1_17_1_Sign[] tileEntities;
	@NBTName("Status")
	private String status;
	}
	@NBTName("Level")
	private Level level;

My 1.17 chunk parser takes the Level section into account, while my 1.18 parser does not:

BlueMapSignExtractor/src/main/java/com/technicjelle/bluemapsignextractor/versions/MC_1_18_2/MC_1_18_2_Chunk.java

Lines 9 to 14 in 1e1d96a

	public class MC_1_18_2_Chunk extends ChunkWithVersion implements Chunk {
	@NBTName("block_entities")
	private MC_1_17_1_Sign[] blockEntities; //The actual sign format hasn't changed, only the chunk format has
	@NBTName("Status")
	private String status;

But the Data Version of this chunk (2840) was making the plugin use the 1.18 parser, instead of the 1.17 one:

BlueMapSignExtractor/src/main/java/com/technicjelle/bluemapsignextractor/common/ChunkClass.java

Lines 39 to 53 in 1e1d96a

	if (dataVersion >= 3463) {
	return new ChunkClass(MC_1_20_4_Chunk.class, dataVersion);
	} else if (intInRange(dataVersion, 2825, 3337)) {
	//For versions:
	// - 1.18.2
	// - 1.19.4
	return new ChunkClass(MC_1_18_2_Chunk.class, dataVersion);
	} else if (intInRange(dataVersion, 2724, 2730)) {
	return new ChunkClass(MC_1_17_1_Chunk.class, dataVersion);
	} else if (intInRange(dataVersion, 1901, 2586)) {
	//For versions:
	// - 1.14.4
	// - 1.15.2
	// - 1.16.5
	return new ChunkClass(MC_1_14_4_Chunk.class, dataVersion);

It is extremely tedious to go through every single snapshot changelog to see where exactly things changed, so I just eyeballed these cut-off points.
Clearly they need more tweaking, which I shall do!
Give me a moment...

[ShDis]

Thanks for the detailed explanation!
I'll test the new build but it'll take about two hours to collect markers, so there's a small chance I won't be back until tomorrow.

[TechnicJelle]

I think I'll add a config option to suppress warnings about old/incompatible chunks...

[TechnicJelle]

Huh...
Somewhere, there is a chunk that does not have a status.

The problem lies here:

default boolean isFinished() {
	return FINISHED_STATUSES.contains(getStatus());
}

It's trying to see if the chunk's status is finished, but the getStatus() call is returning null for some reason!

I'll attach a version of the plugin with extra logging, so we can find the exact region file and the exact chunk inside the region file that is this broken.
Could you please send the region file that is causing this problem here, so I can inspect it and think up a good solution for cases like these?

BlueMapSignExtractor-1.3-PREVIEW3.jar.zip

[ShDis]

Launched. I'll be back tomorrow with the results. Thanks for your work (7 minutes to patch since opening issue o-0)

[TechnicJelle]

I'm eagerly awaiting the result!
And keep in mind this isn't a patch, this is just extra logging, so we may find out more information about the cause.

[ShDis]

There were a lot of errors in the log (almost 1mb), mostly with chunks older than 1.13.2.
But after two hours 6000+ signs were observable via site, that's awesome. Beside framerate in browser, it's like 4-5 frames/second :)

The error with null status could be connected with part of the world contained some old 1.18 snapshot chunks:

[04:03:32] [Craft Scheduler Thread - 10 - BlueMapSignExtractor/INFO]: [BlueMapSignExtractor] World "world": Extracting signs into markers...
[04:03:32] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMapAreaControl] Loading config for map: world_the_end
[04:03:32] [Craft Scheduler Thread - 5 - BlueMap/INFO]: [BlueMap] Loaded!
[05:40:57] [Craft Scheduler Thread - 10 - BlueMapSignExtractor/ERROR]: [BlueMapSignExtractor] Error reading region file
java.io.IOException: NullPointerException in chunk 20, 11 in region file C:\2s2d\.\world\region\r.282.253.mca
		Chunk class: ChunkClass { DataVersion: 2840 -> Loader: MC_1_18_2_Chunk }
	at com.technicjelle.bluemapsignextractor.common.MCARegion.getBlockEntities(MCARegion.java:115) ~[BlueMapSignExtractor-1.3-PREVIEW3.jar:?]
	at com.technicjelle.bluemapsignextractor.common.Core.fillMarkerSetFromRegionFile(Core.java:57) ~[BlueMapSignExtractor-1.3-PREVIEW3.jar:?]
	at com.technicjelle.bluemapsignextractor.common.Core.lambda$loadMarkerSetFromWorld$2(Core.java:42) ~[BlueMapSignExtractor-1.3-PREVIEW3.jar:?]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) ~[?:?]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) ~[?:?]
	at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ~[?:?]
	at com.technicjelle.bluemapsignextractor.common.Core.loadMarkerSetFromWorld(Core.java:42) ~[BlueMapSignExtractor-1.3-PREVIEW3.jar:?]
	at com.technicjelle.bluemapsignextractor.common.Core.addMarkersToBlueMapWorld(Core.java:24) ~[BlueMapSignExtractor-1.3-PREVIEW3.jar:?]
	at com.technicjelle.bluemapsignextractor.BlueMapSignExtractor.loadMarkersFromWorlds(BlueMapSignExtractor.java:67) ~[BlueMapSignExtractor-1.3-PREVIEW3.jar:?]
	at com.technicjelle.bluemapsignextractor.BlueMapSignExtractor.lambda$new$0(BlueMapSignExtractor.java:50) ~[BlueMapSignExtractor-1.3-PREVIEW3.jar:?]
	at org.bukkit.craftbukkit.v1_20_R2.scheduler.CraftTask.run(CraftTask.java:101) ~[purpur-1.20.2.jar:git-Purpur-2095]
	at org.bukkit.craftbukkit.v1_20_R2.scheduler.CraftAsyncTask.run(CraftAsyncTask.java:57) ~[purpur-1.20.2.jar:git-Purpur-2095]
	at com.destroystokyo.paper.ServerSchedulerReportingWrapper.run(ServerSchedulerReportingWrapper.java:22) ~[purpur-1.20.2.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.lang.NullPointerException
	at java.util.Objects.requireNonNull(Objects.java:208) ~[?:?]
	at java.util.ImmutableCollections$SetN.contains(ImmutableCollections.java:937) ~[?:?]
	at com.technicjelle.bluemapsignextractor.common.Chunk.isFinished(Chunk.java:20) ~[BlueMapSignExtractor-1.3-PREVIEW3.jar:?]
	at com.technicjelle.bluemapsignextractor.common.Chunk.isGenerated(Chunk.java:31) ~[BlueMapSignExtractor-1.3-PREVIEW3.jar:?]
	at com.technicjelle.bluemapsignextractor.common.MCARegion.getBlockEntities(MCARegion.java:113) ~[BlueMapSignExtractor-1.3-PREVIEW3.jar:?]
	... 22 more

[ShDis]

Region file causing the error
r.282.253.zip

[TechnicJelle]

Thanks! I'll check it out soon!

[TechnicJelle]

Oh, I actually did find the exact snapshot in which they did the cleanup just now!
It's 21w34a, with Data Version 2844!

This means I can now pinpoint the exact cut-off point for at least this change!
Thank you very much for pointing me in the right direction!

[TechnicJelle]

Regarding these points you had:

There were a lot of errors in the log (almost 1mb), mostly with chunks older than 1.13.2.

I'm planning on making the way it handles unsupported chunks a bit more seamless and spammy,
by making it differentiate Chunk Parsing Exceptions and Region File IO Exceptions.
And I'll also add that it only logs an error once, and then mutes the next times it happens.

Beside framerate in browser, it's like 4-5 frames/second :)

I can't do much about framerate, sadly, as that's just what happens where you have a lot of BlueMap markers. Though I expect that when I get the filtering (#5) in, this will be easier to handle by just not adding unimportant signs.

[TechnicJelle]

In e800afc, I changed the way I log stuff, which should hopefully reduce the large amount of logs.
I haven't implemented that it only logs an error/warning once, yet, though.
(I am kind of hoping it's not necessary any more, with the change I just made)

If you're up for it, could you try this new build?

BlueMapSignExtractor-1.3-PREVIEW4.jar.zip

[ShDis]

Looks good, but it's still about ~23mb log of warnings.