arves100/danmakufu

shh, it's all a secret

Hello!

I'm Arves100, pizza guy with a passion of programming and reverse engineer wannabe.

My stuff

• An open soruce C# emulator for GameSpy (with SDK included!): https://github.com/GameProgressive
• Game patcher for Fur Fighters PC (2000): https://github.com/arves100/ffloader
• Open source serializer and deserializer for RAD Granny2 3D file formats: https://github.com/arves100/opengr2
• A work-in-progress (possibly will never end) open source clone of Mystery Dungeon games with nCine: https://github.com/arves100/OpenMysteryToolkit
• An experimental C# emulator of Brave Frontier: https://github.com/arves100/gimufrontier

Life goals for 2023 / Q3 2024

• Complete Crysis2 Direct2Game server emulator
• Rewrite FF Loader to v2 with some updates (perhaps EVEN a modloader or SP splitscreen)
• Complete some Tools to release for the Metin2 PServer scene (liblyketo, Server linux, binerp, mispack/lyketotool)
• Offline mod for Brave Frontier (and possibly something in decompilator regards)
• The secret linux port project (not really secret)
• Crappy release of some stuff with OMT depends on where I got during that time
• Things for mobile adapter gb (libadaptergb, mgba port)
• Some contributions I am certantly forgetting

Contacts

• Discord: Arves100#1812
• Forums: Arves100 (Inforge/Metin2dev), hyperpika11 (Ragezone)
• Email: pizzaiolo100 (at) proton (dot) me

danmakufu's Issues

SDL2/C++11

A beginning porting of the code would be removing and refactoring GcLibStd to use modern c++ and optionally use SDL2 as well, this would help in the process of having the app to work with SDL2.

• Remove IO api and use better APIs (perhaps SDL_RWops or C++ stream files)
• Think of implementing the threaded loading file with either Boost.asio (async way) or something else (synced api on another thread)
• Change Window creating and handling with SDL2
• Refactor and restructure messages API in preparation of other stuff like input
• Properly fix sizes by predicting the borders
• Create a new option file format (or change version to the existing one), we don't have the option to be in conflict of the old one
• Add fullscreen and hidpi flags

Branch master and oldgen_lnxa can help.

3D Math conversion

Current code, as expected by a DX9 app, uses DirectXMath, this is a deprecated API introduced in DirectX9, the idea would be to swap it with glm (or optionally directxtk simplemath)

• Convert general code such as for projection and so on
• Convert camera code
• See how to add support for OpenGL/different APIs at the same time (we know that OGL and D3D uses different conventions)
• Make compatibility with DX9 first (glm does it with a simple define)

GUI conversions

The sources comes with two extra applications, the configuration app and the FileArchiver app, both of this applications would require a simple rewrite to Qt6 to make them work correctly under Linux

• Archiver
• Configuration

Support for 0.12m

I currently know that 0.12m should be a previous version of this engine with DirectX8, and there would be API differences with 0.12m and ph3 altrough the syntax should be similar. An interesting idea would be adding 0.12m support under this engine.
Would there be enough interest and willing to cover support for 0.12?

NOTE: I have not researched 0.12m api at the moment, so this issue is subject to change

• Figure out the api differences between 0.12m and ph3
• Figure out the same API names with 0.12m and ph3 and began testing on the differences
• Figure out a way to see if the game that is playing is 0.12m or ph3 hopefully without having to touch the games at all

JIT/AOT for scripting

An idea would be changing the way GcLibStd loads scripting file by using a more modern library and suite.
My idea was to using a mix of AOT (for platform such as iPhone) and JIT for usual PC builds, altrough this idea might change.

A JIT would be implemented by using Luajit's DynAsm while AOT would be implemented by using LLVM, altrough I'm currently not sure if it would require a different llvm backend or it can be done without touching the source code, and the target for AOT/JIT would be for later on in the code.
Also, LLVM JIT is not supported on platform such as Android (having to rely fully on AOT which might not be the best idea) and maintainign two JIT code would be hard to bother with, perhaps JIT can be skipped entirely and just go with AOT or keep JIT without doing AOT at all.

A plan for integrating LLVM would work as follows:

1. Load danmaku scripts into memory (eg: by using SDL2 RWOps)
2. Use Antlr to execute a tokenizer and lexter
3. Use antlr tree navigation to build CG for LLVM (reference)
4. Generate object files
5. Link the generates object files with a special, stripped down version of the executor that will execute the generated files and skip everything else like game selection

HLSL shader support

BGFX works by having it's own shader compiler and it's GLSL rules. Every platform will need to compile it's own shader. This will be problematic with games that had external shaders by using HLSL (DirectX9), specifically, we would need to take ShaderConductor and build specific GLSL shaders to be built with bgfx shaderc (so some herustic would be required to udnerstand the differences between GLSL or BGFX GLSL) which would be very consuming.
Another idea would be having a fork of ShaderConduct specifically made to export BGFX compatible shaders, this would elimiate the need to analyze output shaders but might require more work as bgfx shaderc has to be integrated.

This pipeline would also be required to be made accessible during the game runtime, as Danmakufu has an HLSL string api to compile shaders on the fly.

Precompiled shaders should be putted into a shader cache container, where the game would try to directly access them without compiling them every single time, for shader passed by strings, a simple CRC of the string can work to avoid recompiling the same shader every time (as it would be time consuming)

Debug statistics conversion

Might be usefull to someone who wants to debug code with this engine (for example seeing if scripts are correct)

The original design relied on winapi windows, a simple approach would be exposing a Debug window with ImGui, which might also mean more thinking into imgui abstraction or something more decoupled/hardcoded.

Rendering support

The biggest block of Danmakufu. The best way would be having the game to force DirectX9 (Ex) and expose the DirectX9 device to the danmakufu code so we can proceed by doing simple conversions on top of the existing ones to avoid dealing with a massive piece of converted code at the same time. Static tests must also be written in order to see if conversion was correct.
In the future, the engine code should contain support for the view code, altrough at the moment all that code will remain to be hidden to avoid breaking GcLibDx api.

• Implement simple bgfx with callback
• Implement program/shader loading (vs,fs no need to install cs)
• Create a simple shader cache file format to contain precompiled default shaders
• Convert render objects and their own shaders
• Convert metasoquoia (MSQ) 3d mesh support
• Implement custom effects (like fog and texture effects) via a view shader
• Convert render targets/different frame buffers
• Extend options to include support such as rendering api, msaa and so on
• Topology conversions, this is required so we can keep the same script api for creating mesh, while in really it should simply convert index buffers or creating new ones (bgfx doesn't have options for other primitives rather than triangles)
• Something else?

Linux porting/build

After some time checking/testing what is known so far, I have decided to ressurect some planning and activity from this repo (in also hope to remove the trello one).

As this project is intended to be a porting of Danmakufu to Linux which stays truthfull to the API, games that were made with ph3 should be 1:1 compatible with this version of the engine, therefore my plan is to make short releases that contains some pieces of code converted or refactored to test, which can help spotlight issues before the biggest task (the rendering rework) happens.

Components to change:

• Font rendering (GDI to Freetype)
• 3D rendering (DirectX9 to BGFX)
• Windows handling (Winapi to SDL2)
• Input handling (DirectInput to SDL2)
• Audio handling (DirectSound to ?)
• Extra IO operations (Winapi to SDL2 or C++ features)
• Build system (MSBuild to CMake)
• DirectX9 shader (bgfx won't like it so it needs a conversion)

Android/touch screen builds

Another next step after the Linux build is done would be having the game work under Android. As I don't own a touchscreen PC the only way to test all of the mess that will be created would be by having Android. I also don't own any iDevice or any jailbroken one so I would leave iPhone/iPad support unless someone wants to do it.

NOTE: this issue will certanly receive updates later on.

• Should the player have a virtual joypad or touch the player character?
• Should touch be threated as it's own input in the script or should it gimmik the mouse?
• Should the reply record touch input as mouse inputs or touch inputs?
• How would Java code play a role in all of this? We will certanly need to have a new executor written in Java and the whole game exported as a dynamic library.
• What would be the safest minimum Android API to target?

SDL Input

The code was designed to support only mouse and keyboard, after the switch to a basic SDL window system, the input would eventually need to be converted.
An idea would be to use specifically scankeys and have the reply system report them accordingly, unfortunatelly, this would also mean breaking the reply files (and there might be no way to restore them).
As I would like to keep compatibility with as much things as possible, while adding support for controller devices as well (such as joystick and gamepads), here's what needs to be done:

• Convert DirectInput code with SDL2 input code (oldgen_lnxa/master can help with this)
• Implement an Action-like approach to associate a specific device with an action, this can help to have support for both keyboard and joystick at the same time.
• Add new API to cover with joystick and gamepad support
• Create a new reply format that will register action inputs and the device that comes with it (either a keyboard, joystick, etc...)
• Implement testing code for touch as well, this would be a good step for porting under Android/iOS or PC with touchscreens

Audio upgrade

Audio system currently uses DirectAudio (or something similar like XAudio2), currently the code could be converted to use either SDL2 Mixer / SDL2 Audio or use something more raw like Portaudio. More research should be done into selecting the best library for this job (considering this is a game).

Initial setup of the repository

This issue is intended as a cleanup for the repository, altrough the main code will be restarted from origin, it's good to keep track of what's done in order to start implementing code.

• Convert source code to UTF-8
• Setup cmake once again
• Enable a modern version of C++ and fix changes in case
• Change gstd pointers to std pointers
• Disable permissive flag to have the code more MSVC dependant free
• Check and fix all warnings reported by MSVC and GCC
• Fix compilation from mingw

Some of this fixes can be achieved by taking the original codes on multiple branch (in the start of removing them) which are oldgen_lnxa and master.

Font rendering

Danmakufu font rendering is done by using GDI, a proprietary Microsoft rendering system used in Windows, altrough libgdiplus was created to bring GDI support under Linux, it is generally not updated and doesn't have all the freedom that a complete API might have.
As Danmakufu should not have support for input, we don't have to care to implement things such as fribidi, and kerning can be done directly via freetype avoiding us to implement harfbuzz. There are multiple types of font supported and I would like to bring support for, at least, monochrome fonts and colored fonts such as emoji (so SVG fonts)

• Convert existing GDI code to use FreeType2 (branch ft2 can help but with libraqm support removed)
• Add support for emojis/colored fonts
• Add a way to load fonts from the system caches on Windows, Linux and so on
• Use Freetype cache to avoid loading fonts everytime (or use any cache provided inside danmakufu)
• Should vertifcal font render be added? What API should be given?