querz/nbt

A java implementation of the NBT protocol, including a way to implement custom tags.

nbt's Issues

StringOutOfBoundsIndexException issue, and ParseException suggestion

The SNBTUtil.fromString() method throws a StringIndexOutOfBoundsException if an empty string is passed to it, so it would be convenient if that was fixed so one would not have to add catch statements for it whenever they invoke the method.

For the ParseException class, I think that it should have a method for getting the index at which the error was found. In my case, I want it so I can create a (java.text.)ParseException from it for a JFormattedTextField's formatter.

Edit: Alternatively, you could simply use the java.text.ParseException class instead of creating your own.

Only implement Comparable where needed

It might be good to implement Comparable only where needed and not for all Tags. Currently the behavior is also a little bit inconsistent since some implementations can throw exceptions while others, such as CompoundTag just return 0.

.0 is ignored in doubles in toSNBT

Example, tag has a double value: 20.0
After using SNBTUtil.toSNBT(tag), the value turns to 20.
After trying to convert it back to Tag it will throw an error.
Example:

 net.querz.nbt.io.ParseException: cannot add IntTag to ListTag<DoubleTag> at: ...ore":0,"Pos":[-301.5592469090256,72<--[HERE]

Populating a new chunk from scratch results in an ArrayIndexOutOfBoundsException

I am using this library to create an entirely new chunk copied from an existing chunk from the bottom up that I've loaded from elsewhere. I plan on making changes to the chunk as it copies in the future, but for now it's just a straight copy where I have to manually create the blockState for each x, y, and z. As it builds up the copied chunk, it seems to work fine until it hits a certain number of items in the palette. I get this exception:

java.lang.ArrayIndexOutOfBoundsException: Index 320 out of bounds for length 320
[20:15:52 WARN]:        at com.briarcraft.shadow.querz.mca.Section.setPaletteIndex(Section.java:199)
[20:15:52 WARN]:        at com.briarcraft.shadow.querz.mca.Section.adjustBlockStateBits(Section.java:294)
[20:15:52 WARN]:        at com.briarcraft.shadow.querz.mca.Section.setBlockStateAt(Section.java:138)
[20:15:52 WARN]:        at com.briarcraft.shadow.querz.mca.Chunk.setBlockStateAt(Chunk.java:305)

I believe this is happening when it has to resize the blockdata palette index byte size because we've added one too many palette items. In Section on line 137 I see some logic that looks like it's meant to address the issue.

I'm not sure if there is an issue with me manually creating the blockState or something else causing this issue. Thanks for your assistance!

20w17a BlockState format change issue

Hello,
I have seen that you already fixed this new blockstate issue in mcaselector and it would be nice to also do it here. I use this library for a Vanilla WebMap and since yesterday the generated map is broken at a lot of places, due to this change.

Charset problem while recreating Minecraft servers.dat file

My code: {servers:[{name:Server,ip:"0.0.0.0"}]}
Minecraft Generated code: {servers:[{name:Servidor,ip:"0.0.0.0"}]}

My tests

        File f = new File("servers.dat");
        System.out.println(SNBTUtil.toSNBT(NBTUtil.read(f).getTag())); // Testing with the original file

        CompoundTag compoundTag = new CompoundTag();
        CompoundTag serversCT = new CompoundTag();

        ListTag<CompoundTag> servers = new ListTag<>(CompoundTag.class);

        CompoundTag server = new CompoundTag();
        server.putString("ip", "0.0.0.0");
        server.putString("name", "Server");

        servers.add(server);

        compoundTag.put("servers", servers);
        System.out.println(SNBTUtil.toSNBT(compoundTag)); // Testing with generated code
        NBTUtil.write(compoundTag, "output.dat");

But the binary is completely different:

MC Generated:

Generated by my code:

Thanks!

Remove Tag.getEmptyValue()

In my opinion it might be good to replace Tag.getEmptyValue() with public constants (or factories for ListTag and CompoundTag). Additionally the ability to provide null when constructing tags which results in the default value being used might be error-prone. Instead the argument should be non-null and the no-arg constructors could pass the default value to it.
If it is unknown whether a value is null, maybe constructors with an Optional parameter could be added in case it is needed.

The method Tag.getEmptyValue() encourages bad performing code, see

NBT/src/main/java/net/querz/nbt/CompoundTag.java

Lines 124 to 172 in 74e37d4

	public byte getByte(String key) {
	ByteTag t = getByteTag(key);
	return t == null ? new ByteTag().getEmptyValue() : t.asByte();
	}
	public short getShort(String key) {
	ShortTag t = getShortTag(key);
	return t == null ? new ShortTag().getEmptyValue() : t.asShort();
	}
	public int getInt(String key) {
	IntTag t = getIntTag(key);
	return t == null ? new IntTag().getEmptyValue() : t.asInt();
	}
	public long getLong(String key) {
	LongTag t = getLongTag(key);
	return t == null ? new LongTag().getEmptyValue() : t.asLong();
	}
	public float getFloat(String key) {
	FloatTag t = getFloatTag(key);
	return t == null ? new FloatTag().getEmptyValue() : t.asFloat();
	}
	public double getDouble(String key) {
	DoubleTag t = getDoubleTag(key);
	return t == null ? new DoubleTag().getEmptyValue() : t.asDouble();
	}
	public String getString(String key) {
	StringTag t = getStringTag(key);
	return t == null ? new StringTag().getEmptyValue() : t.getValue();
	}
	public byte[] getByteArray(String key) {
	ByteArrayTag t = getByteArrayTag(key);
	return t == null ? new ByteArrayTag().getEmptyValue() : t.getValue();
	}
	public int[] getIntArray(String key) {
	IntArrayTag t = getIntArrayTag(key);
	return t == null ? new IntArrayTag().getEmptyValue() : t.getValue();
	}
	public long[] getLongArray(String key) {
	LongArrayTag t = getLongArrayTag(key);
	return t == null ? new LongArrayTag().getEmptyValue() : t.getValue();
	}

Retrieving local palette from a section

How am I supposed to retrieve local palette from a section? Should't be available a palette getter within Section class?

Old region files crash

I'm trying to use your library to read some old region files and the next error is being thrown.

java.lang.ClassCastException: Cannot cast net.querz.nbt.tag.LongArrayTag to net.querz.nbt.tag.ListTag
at java.lang.Class.cast(Class.java:3369) ~[?:1.8.0_161]
at net.querz.nbt.tag.CompoundTag.get(CompoundTag.java:74) ~[?:?]
at net.querz.nbt.tag.CompoundTag.getListTag(CompoundTag.java:124) ~[?:?]
at net.querz.mca.Chunk.initReferences(Chunk.java:79) ~[?:?]
at net.querz.mca.Chunk.deserialize(Chunk.java:178) ~[?:?]
at net.querz.mca.MCAFile.deserialize(MCAFile.java:61) ~[?:?]
at net.querz.mca.MCAUtil.read(MCAUtil.java:60) ~[?:?]
at net.querz.mca.MCAUtil.read(MCAUtil.java:26) ~[?:?]

It is caused by Entities being an array of bytes in versions prior 1.10 (as can be seen in https://minecraft.gamepedia.com/Chunk_format).
Fixing this might lead to other errors since other fields were arrays in older versions (TileEntities, TileTicks and LiquidTicks).
Is there any way to read it? My intention is to update those fields to the new format since using forceUpgrade didn't work, so only adding a way to load the chunk with the current data structure would be fine. If that can't be done, just loading everything but those values is fine for me since I don't need them for anything.

Use actual element type for empty ListTag

Currently for empty ListTags the type EndTag is used, even if the actual type is a different one. This introduces all kind of type-safety problems, see #14.

According to https://wiki.vg/NBT#Specification it is not "required" that EndTag is used:

The notchian implementation uses TAG_End in that situation, but another reference implementation by Mojang uses 1 instead; parsers should accept any type if the length is <= 0

Therefore it might be good to at least have this library use the correct type for serialization (but keep deserialization the same for legacy support). The methods getTypeID() and getTypeClass() should be changed as well.

Also ListTag.typeClass could be Class<? extends T> and an non-type-safe list would have null as type, or maybe rather Optional<Class<? extends T>> to take advantage of some of Optional's methods.
This woud prevent some casts which are currently necessary.

Let me know if I should write a pull-request for it. Wanted to know your opinion about this first.

How do I serialize an NBT compound?

All in the title

Heightmap is always null in 1.15

It seems like heightmap is not loaded correctly. It's null in all 1.15 worlds files I've been testing. Is this a bug in your code or did Minecraft stop storing heightmaps at some point?

NBT classes do not implement hashCode when implementing equals

The NBT classes implement equals, but not hashCode. The java-doc says that both should be implemented:

Note that it is generally necessary to override the hashCode method whenever this method is overridden, so as to maintain the general contract for the hashCode method, which states that equal objects must have equal hash codes.

CompoundTag .get method returning null with valid key

i am trying to get the LevelName from a level.dat file but whenever I try to do that it always returns null

CompoundTag namedTag = (CompoundTag)NBTUtil.read(levelDat).getTag();
this.WORLD_NAME = namedTag.getStringTag("LevelName").getValue();

LOGGER.info(WORLD_NAME);

Caused by: java.lang.NullPointerException: Cannot invoke "net.querz.nbt.tag.StringTag.getValue()" because the return value of "net.querz.nbt.tag.CompoundTag.getStringTag(String)" is null

Strings with the literal values "true" and "false" are mistaken with bytes when written and parsed from SNBT.

If you have a string tag with the value "true" or "false" and you write it to SNBT, it's read as a byte when parsed again. This is problematic when trying to manipulate gamerules using SNBT.
Potential fix is to check if a string matches "true" or "false" when writing to SNBT and surround it with quotes if it does.

This repo doesnt write valid NBT for MCA

Is there any way to output it as nbt?

Example MCA file
https://drive.google.com/file/d/1KElnUSyT9TUQ85dDNqFkCkbf-5EG28Qi/view?usp=sharing

NBTUtil.write(NamedTag, File) does not save to .dat file properly

I used this lib to change player location before it logins (AsyncPlayerPreLoginEvent), but looks like it does not save the file properly, even without changes. Simplified sample I tested:

// find file
UUID uuid = UUID.nameUUIDFromBytes(("OfflinePlayer:" + playerName).getBytes(StandardCharsets.UTF_8));
File folderRoot = plugin.getDataFolder().getParentFile().getParentFile();
File folderWorld = new File(folderRoot, "world");
File folderPlayerData = new File(folderWorld, "playerdata");
File filePlayer = new File(folderPlayerData, uuid.toString() + ".dat");

// read & write
NamedTag tagFile = NBTUtil.read(filePlayer);
CompoundTag tagRoot = (CompoundTag) tagFile.getTag();
NBTUtil.write(tagFile, filePlayer);

And that's how file saved:
c0e79713-eb16-3c1f-a9fa-726bd3382b9b.zip

Used version 5.3 from https://jitpack.io/.

P.S. I read location from file earlier, so no problems with file reading.

Limited height on write region file

When we (a friend of mine and me) are working on project (using your library).
We need to write blocs in a mcafile, using the 'setBlockStateAt' function. But, when the second argument (y) go above 17, there is an error :

-> Exception in thread "main" java.lang.NullPointerException: Cannot invoke "net.querz.nbt.tag.ListTag.size()" because "this.palette" is null
at net.querz.mca.Section.setBlockStateAt(Section.java:132)
at net.querz.mca.Chunk.setBlockStateAt(Chunk.java:313)
at net.querz.mca.MCAFile.setBlockStateAt(MCAFile.java:265)
at magicbox.Test.main(Test.java:31)

There is the code involved :

Is this an error, or is there something we're doing wrong ?

Anyway, this library is game-changing, thanks a lot !

Cannot cast net.querz.nbt.ByteArrayTag to net.querz.nbt.IntArrayTag

When i executed this:
MCAFile mcaFile = MCAUtil.readMCAFile("r.0.0.mca");
this error occurs:
java.lang.ClassCastException: Cannot cast net.querz.nbt.ByteArrayTag to net.querz.nbt.IntArrayTag

The file r.0.0.mca exists

Anyone help me

Move deserialization to static method or separate classes

It might be good to move the deserialization to static methods or have separate classes (can be nested) such as TagDeserializer (base), StringTagDeserializer, etc.. Though I think using static methods and registering them using method references might be better.

The problem with the current design is that it is not consistent (e.g. ListTag might or might not replace the existing value) and probably irritating to the user.

Let me know what you think about this.

data does not contain Level tag fix

The Level tag does no longer exist everything is nor directly in the root tag.
Change Chunk.java L73 from CompoundTag level; to CompoundTag level = data; and remove Chunk.java L74-L76

ListTag Double issue

I'm trying to get player's position from .dat file inside playerdata folder

    public double[] getPosition(CompoundTag tag){
        double result[] = new double[3];
        ListTag list = tag.getListTag("Pos");
        for(int i=0;i<list.size();i++){
            result[result.length-1] = Double.parseDouble(list.get(i).valueToString());
        }
        return result;
    }

        CompoundTag tag = nbt.parseFile(utils.baseDir+"/world/playerdata/<UUID>");
        double[] pos = nbt.getPosition(tag); // <--- this
        for(int i=0;i<pos.length;i++){
            plugin.getLogger().info(String.valueOf(pos[i]));
        }

This function returns {0,0,0} but when I check the file with NBTEditor the values are correct with the actual position

1.18 Support?

I think mojang changed the nbt format for biomes. Now the biomes are stored per section, using their own data palette instead of the global id palette. That and the height changes are actually being used now (negative sections). Are there any plans to support 1.18?

Deserialize crashes on empty sections

With Minecraft 1.14, there are sometimes empty sections in the chunks. They look something like this:

Not sure why they appear, but they should be handled gracefully.

The code in Chunk.deserialize() tries to create a new Section here:

NBT/src/main/java/net/querz/nbt/mca/Chunk.java

Line 76 in 8cedf1e

this.sections[section.getByte("Y")] = new Section(section);

...but fails on a NullPointerException in the Section constructor since there's no Palette member here:

NBT/src/main/java/net/querz/nbt/mca/Section.java

Line 21 in 8cedf1e

palette = sectionRoot.getListTag("Palette").asCompoundTagList();

"cannot cast ListTag<ByteTag> to ListTag<CompoundTag>" when open Entities tag

The map version is 1.5.2. I‘m trying to port some old classic maps to higher version. However I encountered this error when trying to open a .mca file. In fact, there is no ByteTag under Entities. So I want to ask if it works for 1.5.2.

Cast exception while reading mca file

Hi!
When trying to read an mca file, it throws a class cast exception, could you show a correct example of reading mca regions?
For the sake of completeness, I am using Minecraft 1.12 version. Can your API work with this version of minecraft?

I look forward to your response, I will provide the project code upon request in private messages (private repo) :)

Cannot cast net.querz.nbt.tag.ByteArrayTag to net.querz.nbt.tag.IntArrayTag

Suggestion: Block State Parser

Basically, a utility what would parse block state strings (E.G. "[facing=north,waterlogged=false]") and convert them to compound tags, and vice versa.

GZIP streams are not closed, causing errors when extracting.

Minecraft (and some other apps) don't seem to care if the stream is terminated correctly, but those that do won't open nbt files created with this lib.

Testing the file results in "Unexpected end of data"

Use Supplier for TagFactory instead of requiring no-args constructor

It might be better to change the method TagFactory.registerCustomTag(int, Class<? extends Tag>) to TagFactory.registerCustomTag(int, Supplier<? extends Tag<?>>).

This forces the user to either have a no-args constructor and pass it as supplier, or to have a factory method. Either way prevents the case where the user forgot to add a no-args constructor.

Doesn't work with >= 1.15.1

Biome data was changed and now you can get biome data based on (x,z) see my super simple example and code comments:

package com.company;

import net.querz.nbt.mca.MCAFile;
import net.querz.nbt.mca.MCAUtil;
import java.io.IOException;

public class Main {
    public static void main(String[] args) throws IOException {
        int biome;
        MCAFile mcaFile1 = MCAUtil.readMCAFile("FD-000010/input/region/r.0.0.mca"); // 1.14.4 file
        biome = mcaFile1.getBiomeAt(72,26);
        System.out.println(biome); // should output 17 and does
        MCAFile mcaFile2 = MCAUtil.readMCAFile("FD-000011/input/region/r.0.0.mca"); // 1.15.1 file
        biome = mcaFile2.getBiomeAt(72,26);
        System.out.println(biome); //should output 3 but outputs -1
        MCAFile mcaFile3 = MCAUtil.readMCAFile("FD-000012/input/region/r.0.0.mca"); // 1.15.2 file
        biome = mcaFile3.getBiomeAt(28,53);
        System.out.println(biome); //should output 162 but outputs -1
    }
}

Get Biomes 1.15.1.zip

Improve overall type safety

As the main issue is type safety, I think it will be best to combine #20 and #14 into one pr (#18 ).

Summary:

• Fix excessive use of raw types especially for Tag and ListTag
• You should not be able to change the type of a ListTag after a type has been specified
• ListTags of type EndTag should not be allowed
• Handle ListTag#equals(), ListTag#clone() and ListTag#compareTo() correctly
• Handle serialisation and deserialisation correctly with the above changes
• Adjust Unit Tests accordingly

Upload releases to maven central

Hi! We want to start using this library but unfortunately we cannot add Jitpack to our repositories. Would it be possible to get this deployed to maven central instead?

According to https://maven.apache.org/repository/guide-central-repository-upload.html you can use com.github.Querz.NBT as groupid.artifactid, no extra domain required

Thanks!

Distribute via Maven?

Would love to pull this down through gradle.

Casting of empty ListTag is error-prone

You can change the type of an empty ListTag by adding an element of a different type. This appears to be "intended", see ListTagTest.testCasting(), but in my opinion this is pretty error-prone.

It makes it pretty easy to change the list type by accident and when you then use the list at a completely unrelated place you suddenly get an exception.

I would suggest adding a Class<? extends Tag> to the constructor or using some new class to store both class and typeID and then validate every added element against them.

Selective/partial chunk loading

Some applications, like mappers, benefit from caching chunks and regions locally. However, a fully loaded region can take up significant space in memory, which limits the usefulness of caches.

I'm proposing a mechanism for loading only the portions of a chunk or region that an application intends to use. Then, the internal reference to data in the chunks could be released to allow it to be garbage collected.

I propose something like this:

MCAFile::public void deserialize(RandomAccessFile raf, int fieldFlags) throws IOException

It would be used like this:

mca.deserialize(file, BLOCK_IDS|HEIGHTMAPS|BLOCKLIGHT);

A chunk loaded this way cannot be written to disk and any attempt to do so should throw an exception.

If this seems OK, I will prepare a PR with the changes.

Support for POI mca files?

It seems that neither NBTUtil nor MCAUtil can read Minecraft POI files, as found in foo/world/poi.
I am expecting to find data as described in this fandom wiki
MCAUtil fails with "java.lang.IllegalArgumentException: data does not contain "Level" tag" at "net.querz.mca.Chunk.initReferences(Chunk.java:71)"
I tried this method, and I have not been able to get any tag content.

    public static void readPOIMCAs() {
        final String POI_DIR_STR = "L:\\bin\\minecraft-server\\server_instance\\latest\\world\\poi";
        java.nio.file.Path poi_dir = Path.of(POI_DIR_STR);
        if (!Files.exists(poi_dir)) {
            IOException inner = new IOException("Could not find directory " + poi_dir);
            throw new RuntimeException(inner);
        }
        LOGGER.info("Scanning files in " + poi_dir);
        Set<Path> mcaPaths = new HashSet<>();
        try (DirectoryStream<Path> stream = Files.newDirectoryStream(poi_dir)) {
            for (Path path : stream) {
                String fileName = path.getFileName().toString();
                if (fileName.matches("r\\..+\\.mca")) {
                    mcaPaths.add(path);
                }
            }
            LOGGER.info(String.format("POI dir \"%s\" has %d mca files", poi_dir, mcaPaths.size()));
            for(Path mcaPath : mcaPaths){
                String mcaFileString = mcaPath.toAbsolutePath().toString();
                // MCAFile mcaFile = MCAUtil.read(mcaFileString);
                NamedTag namedTag = NBTUtil.read(mcaFileString);
                String tagName= namedTag.getName();
                Tag<?> innerTag = namedTag.getTag();
                String snbt = SNBTUtil.toSNBT(innerTag);
                LOGGER.info(String.format("File: %s; Tag \"%s\"; SNBT: \"%s\"; JSON: %s", mcaFileString, tagName, snbt,innerTag.toString()));
            }
        }
        catch (IOException ioe) {
            throw new RuntimeException(ioe);
        }
    }
    

There is no content from snbt nor json. The log output here looks like:

INFO: File: L:\bin\minecraft-server\server_instance\latest\world\poi\r.1.2.mca; Tag ""; SNBT: ""; JSON: {"type":"EndTag","value":"end"}

Excessive usage of raw types

Often generic types such as Tag do not have the type parameters specified (i.e. raw type), even in recent commits.

Raw types can become pretty dangerous, see this StackOverflow answer for a good explanation.

Version 4.1 is still labeled as 4.0

After building v4.1, resulting files in build/libs are still labeled as 4.0.
On the occasion, change version = '4.0' to '4.1' in NBT-4.1/build.gradle

Biomes should be ByteArrayTag

Minecraft version 1.14.4. ClassCastException net.querz.nbt.mca.Chunk:65

Reverse nesting depth

This is only an idea, might not be worth to implement it.

It might be good to reverse the nesting depth. Currently it is incremented until it reaches MAX_DEPTH, however it might be more user friendly if instead you provide the max depth (or the default is used) and it is then decremented instead. This makes it easier for the user to specify the maximum nesting depth instead of having to calculate it based on the MAX_DEPTH.

Update README

Gradle and the Maven README are different in versions and both are not up-to-date.

Please, edit the "Tag" version into "6.0", and the graddle one too.
The "Tag" version doesn't even work.

Return of the empty sections crash

Within some of your commits, the empty section fix was removed somehow. This of course leads to the return of #31.

Please merge the little-endian-io-SNAPSHOT branch over the recommended public branch(es)

There's at least one fix in your experimental branch that was the difference between a corrupted set of world files and working world files.

https://github.com/Querz/mcaselector/blob/master/build.gradle

Your other (thank you it works well!) program is able to trim chunks without eating my world files and in this project pulls in: implementation 'com.github.Querz:NBT:little-endian-io-SNAPSHOT'

I suspect that it's a combination of the logical and for the loadflags (part of commit ef9433b ) and also the next patch after that made everything reference the static class variable (rather than a mix of instance and static class variables), but quite a bit ( 90646ab ) which fixed the chunk math ( return biomeY * 16 + biomeZ * 4 + biomeX ); Though that would make more sense to me as a series of bit-shifts ( return Y << 4 + Z << 2 + X << 0 ).

PS: I figured out this is what I needed to pull based on looking at the build file for your other project, since I wanted to know what I was doing wrong in trying to write a tiny java program to use the same base library to fix some shuffled mod-biomes for a mod-pack that added new biomes during updates.