lispysnake/serpent

Cross-platform gaming kit in the D programming language

Serpent Game Framework

The Serpent Game Framework is a brand new game framework from Lispy Snake, Ltd leveraging the latest technologies such as D, OpenGL and Vulkan, to make indie game development easier than ever.

Support It

This framework is being developed by Lispy Snake for our first games. While we would love to develop it full time, basic economics says we must reinvest any contract-work revenue to support development in any remaining time.

To accelerate development (and time-to-market) for our framework and first game, consider buying a Lifetime License from us ($20!) to have lifetime access to our games. If you just want to send a tip to help with Serpent development (and our other efforts) then please click one of the links below!

Modifications

Please note any modifications must be hygienic - compiling with neither warning nor error. Additionally you must have run scripts/update_format.sh to ensure consistent code-styling before sending in changes.

Design Considerations

Provide the best possible functionality required for simpler 2D games at minimal technical debt, both for us, the framework developers, and you, the library consumer.

Our previous engine implementation was an all-inclusive engine written in C with a WIP rendering pipeline. Long story short, way too much debt for us and for new users.

D Language

Whilst some may argue the merits of D, we've found it perfectly suited to our game development requirements. Consider the built-in concurrency support when dealing with batches of SOA entities.

Additionally, we wanted to avoid a few pitfalls (despite being C lovers)

• String issues (\0, mutability, UTF..)
• Forced to reinvent all the wheels (to avoid linking to beastly opinionated refcount libraries)
• Time to market. It hurts.

Cross-platform support

We need to support, at minimum:

• Windows (Vulkan/OpenGL)
• Linux (Vulkan/OpenGL) & X11/Wayland
• Android.

Utilities

The framework simply wraps a bunch of libraries together, and provides utilities to manage the game loop and do stuff. Thus, we'll provide utilities for lifecycle management and actually loading/drawing stuff.

2D Focus

We want to disguise the pipeline under a 2D front. This framework is currently designed for 2D games, that benefit from an accelerated 3D pipeline. To that end, we'll make it possible to make awesome 2D games with UIs, sound, tiling, etc. But you can still get slick bloom shaders..

Reuse Where Possible

Where it is feasible we will reuse other projects to save us from significant technical debt, such as rendering pipelines, etc.

Below is the current list of projects we know we WILL reuse. This may be subject to modification.

bgfx

We will reuse bgfx to power the underlying rendering pipeline in order to abstract support for various platforms and rendering APIs.

Currently we're focused on Vulkan and OpenGL, with Metal and DirectX on the cards in the future.

SDL

We'll use SDL for our basic windowing, OS integration and input handling. This allows consumers to leverage the advanced controller support found within SDL (some would say a USP).

Chipmunk2D

After investigating several options, we're probably going to use Chipmunk2D for 2D physics, and find another 3D option should the need arise. We looked into Newton Dynamics and it is too problematic for integration.

OpenAL

Need decent sound, right?

Invent Where Unavoidable

Some areas we're going to be forced to do a small bit of reinvention. This will involve basic UI support in the framework, primarily because the main available libraries are explicitly locked to OpenGL. We very much need to support OpenGL and Vulkan.

serpent's Issues

Add subtextures

We need the ability to split a texture into child textures with their own UV coordinates.
This is required so we don't keep reloading textures for the sake of a child node and will
greatly simplify sprite animations.

We need to be able to evenly split by rows, columns, returning a set of textures.
Additionally we need to be able to split by specific regions. Eventually we want a
texture cache which can sort by the parent handle to prevent unnecessary
texture switches for child textures (such as spritesheets)

Add Animation APIs

We need simple Animation APis for now that can be dirt simple (changing the texture).
These need some kind of timeline struct so that we can have multiple instances of an animation that aren't directly shared.

Animation API should step through Frames (potentially basic structs) and increment the index once per duration completion, with the ability to loop back to index 0. This will require an AnimationProcessor to step through all AnimationComponent instances for the sprites at the very least.

For Tile Maps we'll likely need a very specialised Animation class.

Improve ECS performance

A known limitation with the ECS implementation is the reliance on an associative array of dynamically allocated components. A better strategy is a greedily resizing array per component to ensure have linear access to entity data.

We don't currently support the notion of archetypes but if this becomes part of the plan then we can in theory chunk entity data across archetypes. For now lets look at large per-component datasets with a nicer memory layout.

Remove static initialisers for bgfx

The static initialisers have the codebase locked to bgfx usage and initialisation, which is a Bad Thing for the noop pipeline.

It also completely prevents us from dynamically linked to bgfx, which is very much wanted.

Add support for 3d graphics

← Arrows to the left

Basic Windows build

Sorry for not going normal fork-PR procedure. But if you insist I can do it that way.

So, anyway. In order to build on Windows you need to handle dependencies with dub and handle platform init.

dub.json - split 'posix' deps (these are your current deps) and 'windows' deps (also note you have bxRelease two times, which is probably a type or copy paste error)

	"libs-posix": [
		"SDL2", "SDL2_image",
		"bgfxRelease", "bxRelease", "bimgRelease",
		"stdc++", "GL", "x11", "bxRelease",
	],
	"libs-windows": [
		"SDL2", "SDL2_image",
		"bgfxRelease", "bxRelease", "bimgRelease",
	],

However this is not enough, LDC will happily tell you that unreachable code is detected in file
source\serpent\graphics\pipeline\bgfx\pipeline.d somewhere inside integrateWindowBgfx() function.

Let's fix this, and while we are here let's also change platform handling to proper static assert which will clearly state the problem and give a hint to potential devs what they should do before they can build and use it.

source\serpent\graphics\pipeline\bgfx\pipeline.d

    final void integrateWindowBgfx() @system
    {
        // ...
        version (Posix)
        {
            // ... 
        }
        else version (Windows)
        {
            pd.nwh = wm.info.win.window;
        }
        else
        {
            static assert(0, "Unsupported platform");
        }
        // ...
    }

Assuming you've already build SDL2, SDL2_Image and bgfx libraries and placed them inside that same folder(a hacky solution for now) you will be able to build demo scene project like the one on a repo page.

Set sampling per-texture-basis

Right now we set MAG_POINT sampling universally - however this breaks non-pixel-perfect applications when scaling.

See with MAG_POINT

And without MAG_POINT

It's a fairly quick fix, bgfx_set_texture should respect texture clamp/sampling via uint32_t.max, and in texture construction we should allow overriding the sampling. Make a clearly documented API on this one to avoid confusion.

Add Shader handles

We need a nice cache for Vertex and Fragment shaders, with double lookups (int key or slow string find)

This will allow us to cache our builtin shaders, as well as allowing the consumer to completely override the shaders. Eventually we'll also want a way to specify custom shader rules for various parts of the codebase,such as in the tiled renderer. i.e. it'd be nice to use a shader to make the
trees 'move', etc.

We'll also want this to be not-ugly, so lets use structs, not classes, and have comparable handles. This will allow us in future to sort through batches if the underlying shader or texture changes. (Read: Material)

Correctly integrate shader builds

We don't really need to ship a resource pack for Serpent's core framework, but we do need to get the shaders in. Thus it might make sense to actually compile the required shaders into the final binary, where they can be recompiled into the cache.

We absolutely need this fixed to support any other platform than Linux.

Integrate bgfx callbacks

Major ticket items here:

• Shader cache

Despite needing AoT 'compiled' shaders, they're still going to be dynamically recompiled on the target platform. Use bgfx callbacks to implement shader caching if enabled in the context. We should ideally expose a set of directories here, i.e. application name for XDG config/cache directories.

• Builtin screen capture support

Another nice feature of bgfx callbacks - we can screen-capture at will. Don't hotkey it, just expose the API. Ideally support automatic filenames here.

Split demos into separate projects

This isn't the highest priority just yet but we're going to want multiple serpent demos and dub isn't too great with parallel configurations, plus we don't want nasty git history.

Ideally we want serpent-demo-scifi and serpent-demo-rpg projects to demonstrate various platform features. We need to get a bit further along though and complete some more basic component support before we can do that (due to fluctuating APIs)

Scene makes no sense. Kill in favour of Space

Literally it serves no purpose right now. We don't have a scene graph and kinda don't want one either.

Rather, we would prefer some notion of a Space, which has its own set of entities. This would allow us to hotswitch between two complete system states by updating which set of entities we have access to.

This would require storing a list of spaces within the EntityManager, each with their own independent storage buckets. The StagingArchetype and manifest would need to come from the parent EntityManager and a clear() operation would need to nuke all spaces. However each Space should also support a clear() too...

Clip offscreen sprites

The Tiled Renderer doesn't perform any rendering of tiles outside the current viewport.
Ideally we need to do the same with Sprites so that we can handle many, many entities.

Abstraction of pipeline

Much of our code was hardcoded to bgfx, which proved an error in design as we can't do any proper
unit testing that way.

We need to continue the implementation of the NoopPipeline so that we can perform CPU side testing. This means abstracting the texture and shader implementations further until no bgfx
specific code exists outside of the pipeline.bgfx module.

Future: CameraSystem + changed support

We should implement a 'changed' modifier so that we can skip iteration of unchanged chunks.
This will allow us to do some hackery whereby we update transformed positions of components on-demand in a system only when they've been changed, and use this cached data in the renderers.

serpent-demo-rpg broken with debug build

Not entirely sure why, but the entitymanager bricks on debug build in the RPG demo, and not in the Serpent build. Likely a weird mismatch of the LDC flags.

Simplify registerComponent usage

The boilerplate to registerComponent for builtin types is cumbersome. Adding a MapRenderer/SpriteRenderer should try to register the component if not already registered, taking away yet another implementation detail.

Add a build for newton

This is not an immediate requirement but we will need to build newton in tree.
Additionally this will need packaging as libraries for SSE/SSE4/AVX2 acceleration
on specific targets.

This will be used for our physics support.

Modularize the codebase

Right now we're suffering from quite a bit of spaghettiness and maintaining everything in one codebase raises the temptation to 'change all the things' - introducing wide breakage in each
update.

We need to split everything up and ensure certain components are self dependent.

For example, there should be the following git repo/package layout at minimum:

• serpent.core
• serpent.tiled
• serpent.physics2D
• serpent.graphics2D

Unfortunately right now we have a mangly looking situation where we have interdependencies all over the place. Chiefly, serpent.core depends on serpent.graphics. This is due to the Context object owning and initializing the display. As Display is such a key factor in using Serpent it might be worth splitting that module out into core, as we're always going to be reliant on SDL.

Indeed, this introduces yet another dependency problem where the Display is dependent on the Pipeline and many graphics APIs..

Soooo...

We need to split 2D specific graphics APIs from the current graphics module, making most of the current API private. Then serpent.core can contain just enough to support the core APIs and a displayable window, with everything else being ripped out.

Conversely it may make sense to rename serpent.graphics.display to serpent.display.Window, in case we ever want to support multiwindowing in future. It is then the job of the context to tie everything together into an event loop.

Phase 1: Draw Stuff Again

Our initial bootstrap phase 1

• Add Renderable struct with position and texture handle data
• Add some kind of manager for these entities
• Add some kind of Renderer for Sprite types
• Draw them all.

Add initial build for bgfx

Initially we need to make the bgfx library usable from D.
We want to compile it as part of our greater library build, so:

Begin integrating Chipmunk2D

We began experimenting with Chipmunk2D integration in serpent-demo-paddle - which is more a proof of concept. However, we want a nice clean API around Chipmunk to make it feel tightly integrated with Serpent.

Whack out a Chipmunk wrapper as part of the core API and rebase the paddle demo.

• Add a core processor and world type
• Add basic Shape + Body types
• Add some basic properties
• Add some rotation support into sprite renderers

EntityManager crash on finalization

Hi,
I was doing some basic benchmarks using available ECS libs in D and encountered this little problem. (I'm only interested in ECS framework itself and not the whole engine as I actually plan to embed the "game" into Unreal or Unity)

DMD 2.092 / LDC 2.091 both debug and release - crash at end(well, it seems like I may need to join worker threads before calling EntityManager.clear() but I haven't found how to reach them), but it also doesn't matter if I do EntityManager.clear() or not, so no idea what's going on.

If you think I'm doing something completely retarded please let me know before closing the issue.

DMD doesn't have stack trace at all

LDC stack traces below:

Main Thread

ntdll.dll!00007ffe938e72a6() (Unknown Source:0)
ntdll.dll!00007ffe938fb576() (Unknown Source:0)
ntdll.dll!00007ffe938fb3c0() (Unknown Source:0)
serpent-bench.exe!core.sync.mutex.Mutex.lock_nothrow!(shared(Mutex))() Line 196 (c:\D\ldc2\import\core\sync\mutex.d:196)
serpent-bench.exe!serpent.core.entitymanager.EntityManager.clear() Line 866 (f:\prog\D\ecs-bench\deps\serpent\source\serpent\core\entitymanager.d:866)
serpent-bench.exe!serpent.core.entitymanager.EntityManager.~this() Line 422 (f:\prog\D\ecs-bench\deps\serpent\source\serpent\core\entitymanager.d:422)
serpent-bench.exe!rt_finalizeFromGC�() (Unknown Source:0)
serpent-bench.exe!_D2gc4impl12conservativeQw3Gcx5sweepMFNbZm�() (Unknown Source:0)
serpent-bench.exe!_D2gc4impl12conservativeQw3Gcx11fullcollectMFNbbZm�() (Unknown Source:0)
serpent-bench.exe!_D2gc4impl12conservativeQw14ConservativeGC__T9runLockedS_DQCeQCeQCcQCnQBs18fullCollectNoStackMFNbZ2goFNbPSQEaQEaQDyQEj3GcxZmTQvZQDfMFNbKQBgZm�() (Unknown Source:0)
serpent-bench.exe!gc_term�() (Unknown Source:0)
serpent-bench.exe!rt_term�() (Unknown Source:0)
serpent-bench.exe!_D2rt6dmain212_d_run_main2UAAamPUQgZiZ6runAllMFZv�() (Unknown Source:0)
serpent-bench.exe!_d_run_main2�() (Unknown Source:0)
serpent-bench.exe!_d_wrun_main�() (Unknown Source:0)
serpent-bench.exe!app.wmain() Line 32 (c:\D\ldc2\import\core\internal\entrypoint.d:32)
[Inline Frame] serpent-bench.exe!invoke_main() Line 90 (d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:90)
serpent-bench.exe!__scrt_common_main_seh() Line 288 (d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)
kernel32.dll!00007ffe920e7bd4() (Unknown Source:0)
ntdll.dll!00007ffe9394ce51() (Unknown Source:0)

there is 7 other threads like one below but they seems to be irrelevant

serpent-bench.exe!thread_start<unsigned ... blah blah>

ntdll.dll!00007ffe9397c0f4() (Unknown Source:0)
KernelBase.dll!00007ffe91528b03() (Unknown Source:0)
serpent-bench.exe!_D4core4sync5event5Event4waitMFNbNiZb�() (Unknown Source:0)
serpent-bench.exe!_D2gc4impl12conservativeQw3Gcx14scanBackgroundMFNbZv�() (Unknown Source:0)
serpent-bench.exe!_D4core6thread8osthread20createLowLevelThreadFNbNiDFNbZvkDFNbZvZ20thread_lowlevelEntryWNbPvZk�() (Unknown Source:0)
serpent-bench.exe!thread_start<unsigned int (__cdecl*)(void *),1>(void * const parameter) Line 97 (c:\Users\username\AppData\Local\Programs\Microsoft VS Code\minkernel\crts\ucrt\src\appcrt\startup\thread.cpp:97)
kernel32.dll!00007ffe920e7bd4() (Unknown Source:0)
ntdll.dll!00007ffe9394ce51() (Unknown Source:0)

Here is the code
(note that this benchmark isn't really fair as allocating entities may trigger GC pause when library relies on GC memory management while actual processing time budget isn't reached yet)

import std.stdio;
import std.random;
import std.datetime;

import serpent;


@serpentComponent
struct Position
{
    double x, y;
}

@serpentComponent
struct Velocity
{
    double x, y;
}

@serpentComponent
struct Renderer
{
    ushort padding;
}


final class MovementSystem : Processor!ReadWrite
{
    override void run(View!ReadWrite entityView)
    {
        foreach(ent, pos, vel; entityView.withComponents!(Position, Velocity))
        {
            pos.x += vel.x;
            pos.y += vel.y;
        }
    }
}


final class RendererSystem : Processor!ReadOnly
{
    override void run(View!ReadOnly entityView)
    {
        foreach(ent, pos, ren; entityView.withComponents!(Position, Renderer))
        {
            writeln("x, y: ", pos.x, ", ", pos.y);
        }
    }
}


void main()
{
    auto engine = new EntityManager();
    scope(exit) 
        engine.clear();

    auto movement = new MovementSystem();
    auto renderer = new RendererSystem();

    engine.registerComponent!Position();
    engine.registerComponent!Velocity();
    engine.registerComponent!Renderer();

    if (!engine.built)
    {
        engine.build();
    }

    Duration delta;
    Duration time;
    MonoTime before;
    MonoTime after;
    size_t numEntities = 0;

    engine.step();

    while(true)
    {
        delta = after - before;
        time = time + delta;
        before = MonoTime.currTime;
        const deltaSec = delta.total!"hnsecs"() * 1.0e-7;

        if (deltaSec < 0.016)
        {
            // add batches instead of just one
            foreach (i; 0..11)
            {	
                auto entity = engine.create();
                engine.addComponent(entity, Position(uniform01(), uniform01()));
                engine.addComponent(entity, Velocity(uniform01(), uniform01()));
                if (i == 0 && numEntities == 0)
                {
                    engine.addComponent(entity, Renderer());
                }
            }
            numEntities += 10;
        }
        else
        {
            writefln!"reached %.4f seconds loop update out of 16 ms budget with %d number of entities in %.4f seconds"( 
                deltaSec, numEntities, time.total!"hnsecs"() * 1.0e-7
            );
            break;
        }

        auto view = View!ReadWrite(engine);
        movement.run(view);

        auto viewRO = View!ReadOnly(engine);
        renderer.run(viewRO);

        engine.step();

        after = MonoTime.currTime;
    }
}