civplatform/map-sync

Mod to sync map data with others instantly and privately, and update tiles for Leaflet/CivMap

Map-Sync

Share map data with your friends, live, privately. Supports Journeymap and Voxelmap.

Download

Join the Discord for announcements, discussion, and support.

Usage

Join a Minecraft server, press the GUI keybind (comma , by default), enter the address of your Sync Server, and click "Connect".

How it works

When you connect, you receive all chunks that your friends have mapped since the last time you played (and were connected to the Sync Server).

Every time any of your friends load a chunk with Map-Sync installed (even if they don't use any map mods!), it gets mapped and the map data gets sent to the Sync Server. It will then send it to everyone else, and if you have a compatible map mod installed (Journeymap or Voxelmap), the mod will display your friends' chunks.

Map-Sync tracks a timestamp per chunk, so old data will never overwrite newer data.

You can control who has access to a Sync Server by editing its allowed-users.txt. If someone connects who is not allowed access yet, their name and UUID gets written to denied-users.txt, from where you can just cut+paste it into allowed-users.txt and restart the server to grant access.

Running a server

Docker Install (recommended)

1. Install the Docker Engine, if you haven't already.
2. Install Docker Compose (We're using Docker Compose V2, so update if you haven't already done so.)
3. Open a terminal.
4. Clone our code.
   • git clone https://github.com/CivPlatform/map-sync.git
5. Change your working directory.
   • cd map-sync/
6. To run the server with interactive prompt:
   • docker compose run --rm -it -p 12312:12312 map-sync
   • To stop the interactive prompt: hit ctrl-c twice
7. To run the server headless:
   • docker compose up map-sync -d
   • To stop the headless server: docker compose down map-sync

System Install

• install recent nodejs (~17)
• clone code, cd server
• npm install
• npm run build -- this has to be run after every time the code is edited
• npm run start
• to stop, press Ctrl+C twice

Server commands

Run these inside the command-line interface after starting the server.

whitelist_load
whitelist_save
whitelist_add_ign <name> -- requires the player to have connected in the past
whitelist_remove_ign <name> -- requires the player to have connected in the past
whitelist_add <uuid>
whitelist_remove <uuid>

---

Copyright (C) 2022 Map-Sync contributors

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.

map-sync's Issues

mod: render outside main thread

maybe reuse journeymap's render thread

map data compression

compress map data clientside, so server can just store the binary blob

voxelmap integration

Terminate connection for outdated versions of the mod

To ensure older clients get their connection terminated early and don't receive garbage data

Sync chunks since last session

• on connect, server sends list of each (up to ~1260) region's newest chunk ts - so we can ignore regions that are fully synced
• client finds region where its oldest chunk ts is older than server's newest chunk ts in that region
  • use file modification date? so we don't have to read the file's contents. must set explicitly because by default it's the newest chunk ts
• if a region is too old, client sends that region's chunk ts's (1024) to server; server responds with all newer chunk data of that region
• client only requests regions when not busy rendering

consider using protobuf

• allows reusing the same packet definitions for client and server
• built into netty https://netty.io/4.1/api/io/netty/handler/codec/protobuf/ProtobufDecoder.html

Github Workflow for the server should include a prettier check

xaero world map integration

Consider limiting to a single sync connection

Currently, for each Minecraft server, you can connect to and sync with multiple MapSync servers. This is great in theory as it allows players to source their map data from multiple places, however, in practice, this is difficult to maintain.

---

It creates the means for players/servers to taint other player's/server's maps in real time

This is not made impossible by limiting clients to a single sync connection, but it does as friction. Also, the sync server itself needn't be malicious. Imagine if, during the last war, PhysicsGamer had a Temporal Isles sync server. And say some Icenians were still connected to it from when it was an Icenian state. And say Icenia also has a sync server. PhysicsGamer could then send tainted chunks to his Temporal Isles sync server and those chunks would be relayed to the Icenian sync server. This could become an attack vector to obfuscate or white-out keys areas of the map, or even the whole map.

There are ways to combat something like this, like not sending map data to sync servers that came from other sync servers, but this isn't currently tracked. Or enforced. Sync servers do not do any kind of validation. It's almost surprising this hasn't happened yet, except by accident with creative-server chunks.

---

Any memory issues are compounded

Any places where MapSync is holding data permanently in memory per-server are multiplied by however many servers people are connected to.

---

Any synchronisation/threading issues are compounded

Same as above.

Version 2.0.0 not retaining data

Using JourneyMap and version 2.0.0 and every time I run the docket script it creates a config, whitelist and uuid_cache.

Is there some setting in Docker I need to change? Regardless of how I shut it down, it creates new files every time.

Synchronize player locations

Would it be possible to add in other things such as waypoints and player locations?

The server does not send trees to the client correctly on voxelmap

access control: list of uuids

on connect, when uuid is in whitelist file, allow access
otherwise, append uuid and mojang name to a text file
server admin can copypaste lines from that file into the whitelist file and restart the server
because the client may get whitelisted later, keep receiving any chunk packets but don't forward them to others

gui show connection status

connecting, connected, disconnect error

Feature request: syncing and updating waypoints

some kind of ability to automatically sync waypoints? maybe not every single waypoint, but waypoints with a public prefix of some kind, or determined by general consensus

and then, eventually, finally, syncing all of this with the ol' civmap/dynmap

io.netty.handler.codec.TooLongFrameException: Adjusted frame length exceeds 32768: 13396197 - discarded

This has become an issue during testing due to the 4.6GB database that Gjum gave me to stress test MapSync's memory usage. At some point during the connection, the client will throw this error. I'm not entirely certain, but I think it's throwing while receiving an oversized packet, here's the full exception:

[07:30:54] [nioEventLoopGroup-2-1/WARN] (DefaultChannelPipeline) An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
 io.netty.handler.codec.TooLongFrameException: Adjusted frame length exceeds 32768: 13396197 - discarded
	at io.netty.handler.codec.LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:503) ~[netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.handler.codec.LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder.java:489) ~[netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.handler.codec.LengthFieldBasedFrameDecoder.exceededFrameLength(LengthFieldBasedFrameDecoder.java:376) ~[netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:419) ~[netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:332) ~[netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) [netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) [netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.68.Final.jar%2330!/:4.1.68.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]

This is primarily due to #71, which lowered the maximum frame size from 16,777,216 (2 ^24), down to 32,768 (2 ^15). Keep in mind though that the exception is referencing a frame length of 13,396,197, so even if we reverted that PR, it's still only just about fitting within that ridiculously large frame size. So if the database used covered even more of the world, or if the world were much larger (like Civclassics size), or if you used it for other servers like 2b2t, you run to risk of exceeding the frame size even then.

The short term fix is to revert #71, but this is fundamentally an issue with the protocol, which just dumps every region's timestamps into a single packet, followed by, in all likelihood, an even larger chunk timestamps packet, since there's far more chunks, and chunk coordinates are ints, not shorts. It may be worth going for a staggered approach, perhaps preventing the client from sending chunk data from a particular region without first having synced its timestamps with the server first.

tcp keepalive

consider switching to websockets

use binary mode

requires more setup for server hosting (certificate, eg. letsencrypt with caddy)

do filesystem operations on our own thread, not minecraft's thread

disconnect detection using keepalives

Need a command to exit the server, have to Ctrl+C twice

There should probably eventually be a way to quit or exit gracefully on the server when in interactive mode.
Presently, the only way to shut down the server is by hitting Ctrl+C at least twice, causing the program to throw a non-zero exit code.
Gjum suggested a proper implementation of process.exit

I'll look more into how to issue said exit command on the docker-compose side if this gets implemented.

view+edit whitelist through gui; admin role

• accounts can be added to a new "admins" list in addition to the "whitelist"
• admins can open a gui in-game to view the whitelisted accounts and add/remove accounts to the whitelist
• the admins list could be just editable server side or also through a gui

map data sharing server

• there needs to be some encryption as well as authentication against mojang so we can do access control later on
• when the server receives a chunk map data packet, it sends it to all clients who don't have it yet (by tracking which clients have observed which chunk map data hashes), and writes it to a database (schema here)
• eventually clients will also receive data mapped by others during the client's absence when it connects again moved to #5

support different dimensions

• track dimensions: Join Game packet, Respawn packet
• tell server which dimension client is in, restrict packets to that dimension
• catchup dimension state after entering dimension, so map mods can render it
• catchup must be interruptible when leaving dimension, see #5

gui

• configure sync server address
• connection status; button to reconnect to sync server
• info about map-sync

mod crashes client upon entering an invalid ip address

so I'm assuming domain names are not a valid input?
regardless, i don't think the entire client should crash if you type the wrong string of characters into the input field.

Just information for version

HI
I hope this message finds you well. I wanted to inquire if you might be considering upgrading your "mod" to the next versions, including the 1.19.2 release. Your work has been exceptional, and your insights could greatly enhance map functionality.

With a friend, we would appreciate being able to use your mod, and we were wondering if there was a possibility of implementing it in most of our future adventures. Your mod enriches the experience of the maps, without having to use dynmap and that journeymap no longer allows the webmap from the server and to see it available in the next versions would be fantastic.

Please feel no pressure, as I understand the demands of such endeavors. Your decision will be highly respected. If you have any thoughts or plans regarding this, I'd be eager to hear.

Thank you for your time and contributions.

Best regards,

get biome from chunk, not world

figure out what the three numeric arguments to LevelChunk.getNoiseBiome() mean

compare to net.minecraft.world.level.biome.BiomeManager#getBiome

terminate clients connecting from the wrong MC server

MC server should already be sent in the Handshake, double check it's correct and never accidentally the previously joined server.

The server should continue to accept all MC servers by default (so it's up to the client to only fill in the Sync Server for the correct MC server(s)).
In addition, an environment variable should be able to be set, to reject all MC server addresses that don't match it.

Optionally, the comparison could be done by resolved IP address instead of comparing the hostname (there could be different hostnames for the same IP/MC server).

render block pixels

• use block colors from journeymap's exported table (see comment below)
• biome tint
• water transparency
• slope shading

wip code: https://github.com/CivPlatform/map-sync/blob/master/render/src/coloring/terrain.rs#L12

Corrupted uuid_cache.json

Whoops, in my commit (cca301c) that updated metadata parsing, I failed to remove the conversation to an array during the saving process (see here), which means that the uuid_cache.json file is being saved as an array and loaded as an object.

const json = JSON.parse('[["Orinnari","ecf75ae0-4fa1-418d-b5bc-813abfac7626"]]');
const uuid_cache = new Map();
for (const [key, value] of Object.entries(json)) {
	uuid_cache.set(key, String(value));
}
// Map(1) {'0' => 'Orinnari,ecf75ae0-4fa1-418d-b5bc-813abfac7626'}

This means that, through progressive saves, each entry will duplicate and get progressively more corrupted. Here's my current development uuid_cache.json:

[["0","0,0,0,0,Orinnari,ecf75ae0-4fa1-418d-b5bc-813abfac7626"],["1","1,1,1,Orinnari,ecf75ae0-4fa1-418d-b5bc-813abfac7626"],["2","2,2,Orinnari,ecf75ae0-4fa1-418d-b5bc-813abfac7626"],["3","3,Orinnari,ecf75ae0-4fa1-418d-b5bc-813abfac7626"],["4","Orinnari,ecf75ae0-4fa1-418d-b5bc-813abfac7626"],["Orinnari","ecf75ae0-4fa1-418d-b5bc-813abfac7626"]]