Mod architecture

[citybound]

Mod architecture

[aeplay]

Alright, I made up my mind. Here is my proposal and plan of action:

For now, we go with @martinrlilja's 2nd option:

Mods = Dynamic Rust Libraries

To reiterate and comment on pro's & cons:

Pros

• Native performance
  • hugely important, since a lot of "high-level game logic" actually runs in very hot loops (especially in traffic simulation, pathfinding, road-planning - the same will be true for economy and other areas)
• Existing interfaces should be possible to re-use / mod and mainline code will be compatible.
  • this would allow us to achieve moddability very soon, interfacing reasonably with a scripting language is really non-trivial and a lot of work to get right.
• Should make odd things possible. (Running web servers, hooking up python scripts, etc.)
  • not sure yet how important that actually is

Cons

• Not safe (same access to the system as the game has).
  • I was thinking about jailing the game process anyways using a helper "launcher" process. All the game could then do would be write savegames. But also not really an issue given proper education of the user & publicly visible mod reviewing
• Relatively easy to mess up cross platform support. (Depends on what you are doing.)
  • I don't see where any game code would be platform dependent (except maybe odd stuff as defined above)
• Rust can be a bit tricky to use.
  • that is probably the biggest gotcha, but most of the game code doesn't delve deeply at all into tricky stuff like generics + lifetimes, leaving something that is "almost a scripting language"

Plan of action

In separate repos, develop:

1. A library for runtime discovery and binding of mod-dylibs

• this should define the interface conventions and scaffolding around a mod
• should allow for mods loading sub-mod-dylibs themselves (to achieve moddability at varying levels of granularity)
• try to get hot-code-reload working

2. A mod-compiler (probably a very thin layer on top of rustc) that can create dylibs from a source-form of a mod (that might be simpler than a full-blown cargo crate)

Then:

• integrate the runtime-library into the main game and repackage all game code as mods, compile them manually
• make the mod-compiler a mod itself (meta!!) that is downloaded on-demand once one made changes to the game source that ships with the game (since the mod-compiler will be at least as big as rust itself, so >100mb) that will automatically recompile changed mods, allowing local development of mods without having to setup a dev-environment.
• somehow build cross-compilation-as-a-service based on the mod-compiler

Finally:

• Create a webservice ("mod-store") for mod discovery, rating and registry.

Conclusion

The nice thing about this plan is that already the very first steps will be useful for the main game development as well and incrementally lead towards a pretty comfortable modding situation.

Let me know what you think! (Especially @mijgame, @martinrlilja)

[mijgame]

@martinrlilja that's a good idea as well. In the long run, a official 'mod' could be made that integrates a scripting language with a standard API. This would allow for a lot of mods that want to add simple functionality, like interfaces / views with info about your city, extra buildings / zones and stuff like that. Mods that really want to change the core of the game could write in Rust.

[chdudek]

Its maybe a bit late to add my two cents to the discussion (but will do it anyways :D)

I really like the idea of using a scripting language for modding, as long as there is a good api! The modding community would highly benefit from a easier to learn and/or widely used languag.

For me, I would be highly interested into modding, but I don't know if I will have enough time to learn Rust just for some modding (I've never heard of Rust before!). And I could imagine that stands for many other potential modders out there.

So, I think both variants, modding via libraries and modding with scripts would be greate. It would give an easier entry for interested modders. And if someone reaches the limits of scripting-mods, they may can switch to the rust libraries, if possible.

[mijgame]

@aeickhoff I'm not very familiar with functional programming languages, so I'll try it out and learn some things about it before I give my opinion.

[aeplay]

Yeah, Rust -> WebAssembly was the plan for that

[aeplay]

That sounds pretty good, I'll consider it!

[tsteinholz]

+1 for scripting - rust even has some scripting languages like Dyon, Ketos, and Rhai. I am also a fan of Lua which also has some rust bindings. Might be worth making an Engineering Style Decision Matrix to compare the different languages. Or maybe one even to pick between the dynamic libraries and scripting options.

[nielsmh]

Mods used must be stored in savegames. Active mods should definitely not be a global state. This can help prevent players load a saved game that requires a mod they don't have, and can let players keep multiple cities going with different mod sets without having to manually switch around.

With this approach, you would obviously need a mechanism for adding further mods to an existing save (making it a deliberate process for the player), and mods may need to have different behavior depending on whether they are loaded for a brand new city, or if they are being enabled for an existing city.
Additionally, mods should ideally support unloading/disabling them from a city. That may not be possible for some types of mods, they need to carry a flag for whether they support being taken out.

Lastly there can also be classes of mods that are never actually necessary for saved games to work, that could e.g. be a mod that affects what sound effects are played when, and those should be marked as optional in some way for savegames' mod dependency list.
Possibly there could even be a class of "presentation mods" that only affects UI and not simulation, that are loaded as globals and never listed in a saved game.

As for languages, I also like Lua a lot, and want to remind that LuaJit is a thing that exists, and it can give very good performance. It still has the limitations of having to develop a separate API for it, but that requirement is sort-of eliminated if the entirety of the game logic is written in it. (Then mods are no longer a special case but instead are the norm.)

[tsteinholz]

What type of priority does the modding system have on this game? If we wanted this game to be very easily modifiable, then the best approach would be to make the game it's self a type of mod. Then extending the game's functionality would be as easy original implementation. Then the game would be a core system with a few base "mods" and a new way to factor in new mods. For example, the lanes and the cars would each be built the same way a mod would be. Unfortunately the game is already built in a way that if we were to implement this, we would either need to backtrack, or expand the concept of the core game. But if we expand the concept of what the core of the game is too much, then none of the components of the game would actually be build the same way a mod would be.

[mijgame]

I thinks a game like Factorio is a nice example of how to do modding. The game itself is the 'Base mod' and extending the game is done using Lua. It works very nicely.

[aeplay]

My initial plans for modding support were actually more radical, let me know what you think:

Given that CB is open-source...

• A mod is just a fork with some modifications
• There is a build-service that makes it easy to cross-compile the whole game for all three platforms on-demand
• There is a UI-tool that (using the Github API or something) allows you to create combinations of mods by merging them - if there is no merge-conflicts, it will probably work
• Very popular mod-combinations could be manually maintained by the authors of the individual mods to guarantee that they will work together
• "Officially blessed" mods could be pulled into the Citybound Github organisation and thus marked as "probably not malicious" in contrast to completely unsupervised new mods

[mijgame]

That sounds way, way too complicated for the average player. How many players actually know what a 'fork' even is? Or what compiling means? I know people who actually have difficulty installing mods because the can't find the right folder, because it is in AppData and they can't find that folder.

I think the only way to make mods and modding user-friendly and developer-friendly is to use a good, dynamic scripting language like Lua or JavaScript. Which would allow for users to just drag and drop files in order to install mods. What if you wanted to switch modpacks? That would require you to either have two (or more) installs of the game or recompile the game everytime you want to switch.

[aeplay]

All this complexity would be hidden away from the player, a "launcher" UI could offer an app-store like interface for mods doing all of this in the background. You are right that it would make the transition from player->modder more difficult though. What makes me shy away a little from stuff like a modding API or a scripting language is that it imposes pretty strict limits on what is moddable and what isn't.

[mijgame]

Does it though? I think you can make the game as moddable as you want to. Does every single part of the game have to be moddable? Do you want every part of the game to be moddable? I think a well-designed modding API (that is supported!) would be better than having to learn the internals of the game in order to mod it. Scripting languages are generally easier than Rust too, which makes making mods (or the base game) easier I'd say.

[aeplay]

The more I think about it, the more I think your dynamic library idea seems feasible. I think the dissection line between core engine and mods can be exactly [kay + monet + some essentials like UI] vs. [game actor defintions]. In the long run I would love to actually develop my own scripting language on top of kay, but that's for later.

[aeplay]

And if you look at files like src/game/lanes_and_cars/mod.rs or any other file containing an actor, you can already see my convention of having a setup function that just takes the actor system as a parameter and then registers its actors and messages. This could very well become the interface for dynamic libraries.

[aeplay]

Could you elaborate on how you would handle mod combinations that affect the same dynamic library though?

[mijgame]

Yep. Don't develop your own language for use as a scripting language for modding though. Using a language that is very accessible to people allows them to be more creative with mods.

[mijgame]

How would mods affect the same dynamic library? If you are using Lua for example, the only thing the mod has access to is the modding API that you expose. I don't want to 'advertise' Factorio or something, I just know that there are very advanced mods for it that work really well. I recommend looking at their API (http://lua-api.factorio.com/latest/) in order to get an idea how they handle it.

[aeplay]

Regarding scripting again:

I just really like about Rust how it's strict Type and Borrowing system saved me from all kinds of run-time bugs, and I would hate to not have that for the game logic.

Regarding mod API:

I really want mods to be able to completely replace behaviour, not just augment it. What if there are two mods, in one cars overtake more aggressively and in the other cars keep more distance in general. Both would have to completely replace the "micro-traffic"-library - how should I know which parts of my traffic simulation should be set in stone and which shouldn't be? And how to combine these two mods for example?

[mijgame]

Scripting:

Core engine stuff should be Rust, as that should perform. Behaviour and game logic is generally defined in scripting languages because it is much easier to define complex behaviour in a dynamic language. If implemented well it can really help with your workflow too. You could make it that a Lua script that you are editing, is reloaded in the game in realtime on save. You can get realtime feedback on what you have written. I believe you have done web development, so you should know how important it is for your workflow to get direct feedback on what you have written. What kind of run-time bugs where you running in to, that were solved by having the Type and Borrowing system?

Mod API:

You don't make a mod completely replace the 'micro-traffic' library. You could make the micro-traffic library expose an API that you provide the default behaviour for. When mods initialize, they could override this behaviour. The last mod that overrides this behaviour 'wins'. You can't combine mods if they are inherently not compatible, the best you can do is generate a warning or error that alerts the user or modder about the conflicting mods. I'm not an expert on this by any means btw, so other people are free to correct me on this.

[aeplay]

Scripting: yes, I would love to have hot-code reload. Mostly bugs like use-after-free or equivalent bugs as well as two parts of the code trying to write/write or write/read the same thing in an order that will produce unexpected results. If in theory I included the Rust compiler in the engine, I could get hot-code reload and use Rust as my "scripting language".

Mod API: I know what you mean, but just by looking at my existing code, I don't see where I would want to draw this line at all. Both small and drastic changes should be possible, and I don't know how to fit that into one "API". Or are we not actually talking about an API but more like a soup of modules which can have submodules, each of which can be overridden at any level of granularity (this seems to be what factory offers from having a quick look)?

[aeplay]

Also, let me just add that I'm very happy about this discussion and that I have some people with relevant background knowledge to figure this out with! :)

I'd also love to hear @martinrlilja's thoughts about what we've covered so far, since he basically started it by laying out options.

[mijgame]

Scripting: use after free and race conditions mean something is wrong with the code in the first place. Needing a other language to not have the problem is evading it, not solving it. 'Use after free' is also a lot less destructive in a scripting language because it would simply be nil / null / 0. Regarding Rust as a scripting language: personally, I don't know Rust super well. The language is very complicated after all. I would seriously consider modding Citybound if it used a scripting language like Lua, JavaScript, Python or C# (look at Mono it is amazing) or similar. If it used Rust I would not. I don't want to fight the compiler, I'd rather have dynamic feedback on runtime. That is very much a personal thing though.

And yes, basically making the game a bunch of 'modules' is how I look at it. That is oversimplification ofcourse. You know how you can override JavaScript functions that are provided by the browser? That would be a way of looking at it. You provide the default behaviour that works 99% of the time, but allow overriding it.

And yes, I love the discussions that are already going on. I think making Citybound open-source is an amazing decision and I'm very interested in the game (been following it for like 1.5 years now I believe?)

[mijgame]

I agree with your plan. Native performance is a very good thing, especially given the nature of the game. But the following things are a must:

• Runtime discovery, binding / compiling of mods. Allowing users to drag and drop mods into a directory is a good thing. You could even make it a convention to use the .cbmod extension or something, which could make it really clear.
• Expose helper functions for various operations and things in the game to ease mod development. I think we should strive to not make the modder read the source code all the time in order to figure out what to do.
• Documentation and / or helper scripts to get up and running.
• Hot code reload should be a priority really, it speeds up development the moment you get it working smoothly!

Something you want to consider:

• 100mb downloads on-demand are not a very good idea, since there are a lot of people that don't have good internet or have caps. This is also more of a detail, we should keep the plan straightforward and see to the details later.

The webservice is something that should not be a requirement and should currently have low priority. Modding in the game should not be dependend on the web service, but be enhanced by it. Which is something to keep in mind.

What do you think of this?

[martinrlilja]

Sounds like a nice plan. Maybe it would be a good idea to implement a scripting language as an official mod as well? Or at least start debating over a language, as it's probably a bit too early for an implementation before the API's stabilise. Unless, of course, someone wants to go ahead either way. Maybe we could have a separate issue for this?

[martinrlilja]

I could get started on the first point (A library for runtime discovery and binding of mod-dylibs), should I create another issue for this?

[aeplay]

Yes please. And I'll leave it up to you if you want to do it in-tree in this repo (kinda like the kay crate) or in a completely new repo. Thank you very much, looking forward to it.

[mijgame]

@chdudek it is never too late to add to the discussion 👍 . And I completely agree with what you say: Rust for the people that want to make complex / core changes, scripting with a good API for the general case.

[aeplay]

I agree that scripting will be important, Rust mods are needed as a first step towards that anyways

[tsteinholz]

With the proposed mod system, how do we prevent any operating system from being excluded from a mod? Are there any good cross platform rust compilers, capable of compiling a game that links with native binaries? And has this been tested with CB?

If not there could potentially be a huge mod (like DayZ for Arma) that wouldn't be supported on some platform thus making a cross platform game useless. Even with minor mods, this could inconvenience the player you would want a push and play experience.

[chances]

@tsteinholz Your concerns are already being considered:

"I don't see where any game code would be platform dependent (except maybe odd stuff ...)"

"somehow build cross-compilation-as-a-service based on the mod-compiler"

[aeplay]

What do you guys think about Gluon as a potential scripting language? I'll let you know what I think later (gotta get those sweet uninfluenced opinions).

[chances]

I was casually perusing some recent updates to the WebAssembly website and an interesting point in their tooling docs piqued my interest with regard to modding/webservice integration for Citybound:

The tooling we expect to support includes:
...
Compilers for languages which can target WebAssembly (C/C++, Rust, Go, C#) should be able to run in WebAssembly themselves, emit a WebAssembly module that can then be executed.

(Emphasis added.)

I remember (like a year or more ago) @aeickhoff mentioning something about a potential embedded, smaller Citybound web client that users could place on web pages to share snapshots of their cities online. WebAssembly could help make that idea a reality in the future. If nothing else, I know I would be very interested in contributing something like it.

Perhaps a mod could be created that somehow serializes a city for this kind of embedding? But then there's the issue of trying to compile a Citybound WebAssembly/WebGL target...

Apologies if this is off topic.

[chances]

As always, the genius of @aeickhoff is way ahead of me... 😜

[martinrlilja]

I tried this, but with asm.js a couple of weeks ago, but got a lot of weird errors from glium (trying to resolve some function pointers to OpenGL). This is a good starting point if someone wants to take a stab at this: https://users.rust-lang.org/t/compiling-to-the-web-with-rust-and-emscripten/7627

[aeplay]

The modstore proposal will also heavily depend on whatever we decide about mods here.

[n1313]

A few extra points for consideration:

1. Are closed-source (precompiled) mods allowed?
2. Assuming malicious intent of the mod author, is there a way to safely compile Rust code?
3. Assuming malicious intent of the mod author, is there a way to safely run compiled Rust code?
4. How will the mod load order be managed? Let's say you have two mods, one paints all cars red, another paints all trucks blue, so depending on their load order you might end with different results.

[chances]

Expanding on @n1313's load order point:

Apart from conflicts, ensure dependencies are loaded before the mod that depends on them.

[aeplay]

@n1313: regarding 2 & 3 my plan is just manual review of published mods in the official mod store + a disclaimer for users that a mod can in the worst case do whatever a virus can do + maybe sandboxing the whole game process

[n1313]

manual review

That's not scalable. Time spent on mod reviews won't be spent on main game development.

a disclaimer for users that a mod can in the worst case do whatever a virus can do

So essentially the security of users' machines will depend on Rust skill level of a mod reviewer? :(

[aeplay]

what do you suggest? I can't think of anything that protects 100% from malicious intent

[n1313]

I'm afraid I do not have a constructive solution to this problem. We should look at how other games do this, but I suspect that the answer will be something like "don't treat mods as first-class citizens, have them crippled by default".

[aeplay]

My suggestion with sandboxing was to basically cripple the whole game by default - the only interaction it needs with the outside world is reading and writing save games. Then, mods can be a first-class citizen again (the plan is to implement all game code as mods)

[martinrlilja]

@n1313 Cities: Skylines, which also has mods which can do anything, solves this by not solving it. The game just contains a warning about potentially harmful effects of modding. There is also a reporting system in the Steam Workshop.

On sandboxing, at least on linux it's possible to use ptrace to override the functionality of system calls. It has been used to create sandboxed filesystems for processes. FreeBSD has proper sandboxing. I don't know if something similar exists for mac and windows though.

[nickelc]

Hey, i would like to bring mod.io for mod hosting to the table. The api provides everything needed and i'm working on the modio crate for the rust bindings.

They are also working on an embeddable web app for browsing mods.

[intenscia]

To join the conversation here, I'm founder of mod.io - happy to answer any questions you may have about our open cross-platform mod API. From a tech perspective though @nickelc is the rust expert and a great person to speak with. Looking forward to seeing if we can help the modstore reality come to life :D