gothickit / zenkit Goto Github PK
View Code? Open in Web Editor NEWA re-implementation of file formats used by the early 2000's ZenGin
Home Page: http://zk.gothickit.dev/
License: MIT License
A re-implementation of file formats used by the early 2000's ZenGin
Home Page: http://zk.gothickit.dev/
License: MIT License
Followup to: #28
I've tested std::stringstream
performance today.
Change:
char name[128] = {};
char cls [128] = {};
sscanf_s(line.c_str(), "[%s %s %u %u]",
name, rsize_t(sizeof(name)),
cls, rsize_t(sizeof(cls)),
&obj.version, &obj.index);
obj.object_name = name;
obj.class_name = cls;
Benchmark (world = oldworld.zen)
auto buf = entry->open();
auto time = Tempest::Application::tickCount();
auto world = phoenix::world::parse(buf, version().game == 1 ? phoenix::game_version::gothic_1
: phoenix::game_version::gothic_2);
time = Tempest::Application::tickCount() - time;
Tempest::Log::d("parse time = ", time);
While change is rough one, and hopefully there is a better way than sscanf
, as prototype looking quite good:
// Before (time - milliseconds)
parse time = 2371
parse time = 2769
parse time = 2713
parse time = 2447
// After (roughly 40% improvement)
parse time = 1618
parse time = 1441
parse time = 1225
parse time = 1559
https://github.com/lmichaelis/phoenix/blob/0c29d03205a7bbf0ca5200a683a30be0157e43b8/include/phoenix/mesh.hh#L45
Last good revision: 32770f8
Blamed: 1957cbb
-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
...
[ 40%] Building CXX object lib/phoenix/CMakeFiles/phoenix.dir/source/animation.cc.o
In file included from /home/appveyor/projects/opengothic/lib/phoenix/include/phoenix/animation.hh:6,
from /home/appveyor/projects/opengothic/lib/phoenix/source/animation.cc:3:
/home/appveyor/projects/opengothic/lib/phoenix/include/phoenix/mesh.hh:45:49: error: ‘bool phoenix::polygon_flags::operator==(const phoenix::polygon_flags&) const’ cannot be defaulted
45 | bool operator==(const polygon_flags&) const = default;
| ^~~~~~~
make[3]: *** [lib/phoenix/CMakeFiles/phoenix.dir/build.make:76: lib/phoenix/CMakeFiles/phoenix.dir/source/animation.cc.o] Error 1
make[2]: *** [CMakeFiles/Makefile2:709: lib/phoenix/CMakeFiles/p
Also this operator appears to be unused, probably it would be good to remove this line.
Found this while debugging "Chronicles of Myrthana" mod.
Relevant piece of code: https://github.com/lmichaelis/tcom-decompilation/blob/c993234e0fd8520401491674c8ababe461859d09/Source/Raw/526.d
func int SPELL_TELEKINESIS_FOCUS_REMOVER() {
NPC = HLP_GETNPC(0x71b); // returns null. This is query for PC_HERE, but PC_HERE is not yet initialized
NPC.FOCUS_VOB = 0; // invalid access, causing explicit destructor call
STAGE = 0;
return FALSE;
}
Shortly after executing opcode::movi
VM crashes with memory error.
There are multiple "not fully parsed" error messages while running OpenGothic with the German version of Gothic 1.
To reproduce these, you don't have to do anything too difficult. Just open up OpenGothic with Gothic 1 and start a new game. No interaction with Diego is necessary.
Complete log:
OpenGothic v1.0 dev
no *.ini file in path - using default settings
GPU = NVIDIA GeForce RTX 2070
Depth format = Depth32F Shadow format = Depth16
[phoenix] world: parsing object [MeshAndBsp % 0 0]
[phoenix] bsp_tree: parsing chunk c000
[phoenix] bsp_tree: parsing chunk c010
[phoenix] bsp_tree: parsing chunk c040
[phoenix] bsp_tree: parsing chunk c045
[phoenix] bsp_tree: parsing chunk c050
[phoenix] bsp_tree: parsing chunk c0ff
[phoenix] world: parsing object [VobTree % 0 0]
[phoenix] world: parsing object [WayNet % 0 0]
[phoenix] world: parsing object [EndMarker % 0 0]
unable to load animation sequence: "PILLAR_7M-s_S1.MAN"
unable to load animation sequence: "PILLAR_7M-t_S1_2_S0.MAN"
unable to load sound fx: WOOD_NIGHT2
[phoenix] messages: oCMsgConversation("DIA_BaalTaran_Lehre_05_04") not fully parsed
[phoenix] messages: zCCSAtomicBlock("DIA_BaalTaran_Lehre_05_04") not fully parsed
[phoenix] messages: oCMsgConversation("DIA_Jarvis_Rest_08_01") not fully parsed
[phoenix] messages: zCCSAtomicBlock("DIA_Jarvis_Rest_08_01") not fully parsed
[phoenix] messages: oCMsgConversation("DIA_Mud_Nerve_14_07_00") not fully parsed
[phoenix] messages: zCCSAtomicBlock("DIA_Mud_Nerve_14_07_00") not fully parsed
[phoenix] messages: oCMsgConversation("DIA_Mud_Nerve_18_07_00") not fully parsed
[phoenix] messages: zCCSAtomicBlock("DIA_Mud_Nerve_18_07_00") not fully parsed
[phoenix] messages: oCMsgConversation("Info_Bloodwyn_PayDay_PayNoMore_08_01") not fully parsed
[phoenix] messages: zCCSAtomicBlock("Info_Bloodwyn_PayDay_PayNoMore_08_01") not fully parsed
[phoenix] messages: oCMsgConversation("Info_Diego_JoinAnalyze_Whistler_11_01") not fully parsed
[phoenix] messages: zCCSAtomicBlock("Info_Diego_JoinAnalyze_Whistler_11_01") not fully parsed
[phoenix] messages: oCMsgConversation("Info_GorNaToth_ARMOR_H_11_03") not fully parsed
[phoenix] messages: zCCSAtomicBlock("Info_GorNaToth_ARMOR_H_11_03") not fully parsed
[phoenix] messages: oCMsgConversation("Info_Jackal_Hello_WhatIf_07_02") not fully parsed
[phoenix] messages: zCCSAtomicBlock("Info_Jackal_Hello_WhatIf_07_02") not fully parsed
[phoenix] messages: oCMsgConversation("Info_Kirgo_Charge_Beer_15_02") not fully parsed
[phoenix] messages: zCCSAtomicBlock("Info_Kirgo_Charge_Beer_15_02") not fully parsed
[phoenix] messages: oCMsgConversation("Info_Vlk_1_EinerVonEuchWerden_01_01") not fully parsed
[phoenix] messages: zCCSAtomicBlock("Info_Vlk_1_EinerVonEuchWerden_01_01") not fully parsed
[phoenix] messages: oCMsgConversation("Info_Xardas_DANGER_14_03") not fully parsed
[phoenix] messages: zCCSAtomicBlock("Info_Xardas_DANGER_14_03") not fully parsed
[phoenix] messages: oCMsgConversation("Info_Xardas_LOADSWORD01_14_04") not fully parsed
[phoenix] messages: zCCSAtomicBlock("Info_Xardas_LOADSWORD01_14_04") not fully parsed
[phoenix] messages: oCMsgConversation("KDF_402_Corristo_KREIS4_Info_14_04") not fully parsed
[phoenix] messages: zCCSAtomicBlock("KDF_402_Corristo_KREIS4_Info_14_04") not fully parsed
[phoenix] messages: oCMsgConversation("KDF_402_Corristo_WANNBEKDF_Info_14_05") not fully parsed
[phoenix] messages: zCCSAtomicBlock("KDF_402_Corristo_WANNBEKDF_Info_14_05") not fully parsed
[phoenix] messages: oCMsgConversation("Org_819_Drax_Creatures_Fell_06_02") not fully parsed
[phoenix] messages: zCCSAtomicBlock("Org_819_Drax_Creatures_Fell_06_02") not fully parsed
[phoenix] messages: oCMsgConversation("Org_859_Aidan_Creatures_Fell_13_02") not fully parsed
[phoenix] messages: zCCSAtomicBlock("Org_859_Aidan_Creatures_Fell_13_02") not fully parsed
[phoenix] messages: oCMsgConversation("Sit_2_PSI_Yberion_BringFocus_WO_12_02") not fully parsed
[phoenix] messages: zCCSAtomicBlock("Sit_2_PSI_Yberion_BringFocus_WO_12_02") not fully parsed
comb not found: t_FIREPLACE_Stand_2_S0 -> c_FP_STAND_2_S0_(c_FP_STAND_2_S0_2)
comb not found: t_FIREPLACE_S0_2_Stand -> c_FP_STAND_2_S0_(c_FP_STAND_2_S0_2)
alias not found: t_Walk_2_WalkBL -> t_walkL_2_Walk
alias not found: t_SwimF_2_Dive -> t_swim_2_dive
alias not found: t_SwimF_2_Dive -> t_swim_2_dive
[phoenix] vm: accessing member "C_NPC.ATTRIBUTE" without an instance set
alias not found: r_Roam1 -> r_Scratch
alias not found: t_FallenB_2_Stand -> t_Fallen_2_Stand
inserNpc: invalid waypoint
inserNpc: invalid waypoint
alias not found: t_FistWalkBL_2_FistWalk -> t_FistWalk_2_FistWalkL
comb not found: t_FIREPLACE_Stand_2_S0 -> c_FP_STAND_2_S0_(c_FP_STAND_2_S0_2)
comb not found: t_FIREPLACE_S0_2_Stand -> c_FP_STAND_2_S0_(c_FP_STAND_2_S0_2)
alias not found: t_Walk_2_WalkBL -> t_walkL_2_Walk
comb not found: t_FIREPLACE_Stand_2_S0 -> c_FP_STAND_2_S0_(c_FP_STAND_2_S0_2)
comb not found: t_FIREPLACE_S0_2_Stand -> c_FP_STAND_2_S0_(c_FP_STAND_2_S0_2)
room not found: NLHU03
room not found: NLHU05
room not found: NLHU06
room not found: NLHU09
room not found: NLHU08
room not found: NLHU07
room not found: NLHU04
room not found: NLHU02
room not found: NLHU01
not implemented call [WLD_SETOBJECTROUTINE]
room not found: H�TTE5
room not found: H�TTE34
[phoenix] vm: accessing member "C_NPC.ID" without an instance set
[phoenix] vm: accessing member "C_NPC.NAME" without an instance set
not implemented call [WLD_EXCHANGEGUILDATTITUDES]
room not found: NLHU03
room not found: NLHU05
room not found: NLHU06
room not found: NLHU09
room not found: NLHU08
room not found: NLHU07
room not found: NLHU04
room not found: NLHU02
room not found: NLHU01
room not found: H�TTE5
room not found: H�TTE34
[phoenix] vm: accessing member "C_NPC.ID" without an instance set
[phoenix] vm: accessing member "C_NPC.NAME" without an instance set
[phoenix] world: parsing object [VobTree % 0 0]
[phoenix] world: parsing object [EndMarker % 0 0]
not implemented call [mdl_applyrandomani]
not implemented call [mdl_applyrandomanifreq]
not implemented call [NPC_HASREADIEDWEAPON]
not implemented call [AI_LOOKAT]
not implemented call [NPC_SETKNOWSPLAYER]
This would make sure that integers are always 32-bit (maybe some arcane, old system uses the LP32 model). This is especially important for the VM.
Run into this issue, when testing latest OpenGothic build on M1.
Apart from it, game runs fine, if detect_container_overflow
is disabled. Failure is quite consistent in trigger_list
parse code, yet I wasn't able to spot any suspicius code in relevant places.
Full log is below:
GPU = Apple M1
Depth format = 13 Shadow format = 13
[phoenix] world: parsing object [MeshAndBsp % 0 0]
[phoenix] bsp_tree: parsing chunk C000
[phoenix] bsp_tree: parsing chunk C010
[phoenix] bsp_tree: parsing chunk C040
[phoenix] bsp_tree: parsing chunk C045
[phoenix] bsp_tree: parsing chunk C050
[phoenix] bsp_tree: parsing chunk C0FF
[phoenix] mesh: 1 bytes remaining in section 0xB020
[phoenix] world: parsing object [VobTree % 0 0]
=================================================================
==89736==ERROR: AddressSanitizer: container-overflow on address 0x0002cdc006a0 at pc 0x0001051a1c60 bp 0x0002c142faa0 sp 0x0002c142f258
WRITE of size 24 at 0x0002cdc006a0 thread T8
#0 0x1051a1c5c in __asan_memcpy+0x240 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x3dc5c)
#1 0x1014a0cdc in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::basic_string(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&) string:1999
#2 0x1014a0c2c in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::basic_string(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&) string:2000
#3 0x1014a0a88 in phoenix::vobs::trigger_list::target::target(phoenix::vobs::trigger_list::target&&) trigger.hh:95
#4 0x1014a0994 in phoenix::vobs::trigger_list::target::target(phoenix::vobs::trigger_list::target&&) trigger.hh:95
#5 0x1014a0928 in phoenix::vobs::trigger_list::target* std::__1::construct_at<phoenix::vobs::trigger_list::target, phoenix::vobs::trigger_list::target, phoenix::vobs::trigger_list::target*>(phoenix::vobs::trigger_list::target*, phoenix::vobs::trigger_list::target&&) construct_at.h:37
#6 0x10149fe14 in void std::__1::allocator_traits<std::__1::allocator<phoenix::vobs::trigger_list::target> >::construct<phoenix::vobs::trigger_list::target, phoenix::vobs::trigger_list::target, void, void>(std::__1::allocator<phoenix::vobs::trigger_list::target>&, phoenix::vobs::trigger_list::target*, phoenix::vobs::trigger_list::target&&) allocator_traits.h:298
#7 0x10149f260 in void std::__1::vector<phoenix::vobs::trigger_list::target, std::__1::allocator<phoenix::vobs::trigger_list::target> >::__construct_one_at_end<phoenix::vobs::trigger_list::target>(phoenix::vobs::trigger_list::target&&) vector:948
#8 0x1014945c0 in phoenix::vobs::trigger_list::target& std::__1::vector<phoenix::vobs::trigger_list::target, std::__1::allocator<phoenix::vobs::trigger_list::target> >::emplace_back<phoenix::vobs::trigger_list::target>(phoenix::vobs::trigger_list::target&&) vector:1706
#9 0x101494218 in phoenix::vobs::trigger_list::parse(phoenix::vobs::trigger_list&, phoenix::archive_reader&, phoenix::game_version) trigger.cc:60
#10 0x1015c9f00 in phoenix::parse_vob_tree(phoenix::archive_reader&, phoenix::game_version) vob_tree.cc:180
#11 0x1015cc840 in phoenix::parse_vob_tree(phoenix::archive_reader&, phoenix::game_version) vob_tree.cc:251
#12 0x1015cc840 in phoenix::parse_vob_tree(phoenix::archive_reader&, phoenix::game_version) vob_tree.cc:251
#13 0x101566cec in phoenix::world::parse(phoenix::buffer&, phoenix::game_version) world.cc:88
#14 0x1006374b8 in World::World(GameSession&, std::__1::basic_string_view<char, std::__1::char_traits<char> >, bool, std::__1::function<void (int)>) world.cpp:75
#15 0x100638020 in World::World(GameSession&, std::__1::basic_string_view<char, std::__1::char_traits<char> >, bool, std::__1::function<void (int)>) world.cpp:64
#16 0x1002d4418 in GameSession::GameSession(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) gamesession.cpp:62
#17 0x1002d4ffc in GameSession::GameSession(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) gamesession.cpp:55
#18 0x1004e25d0 in MainWindow::startGame(std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_0::operator()(std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&) const mainwindow.cpp:866
#19 0x1004e251c in decltype(static_cast<MainWindow::startGame(std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_0&>(fp)(static_cast<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >>(fp0))) std::__1::__invoke<MainWindow::startGame(std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_0&, std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > >(MainWindow::startGame(std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_0&, std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&) type_traits:3918
#20 0x1004e24c4 in std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > std::__1::__invoke_void_return_wrapper<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >, false>::__call<MainWindow::startGame(std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_0&, std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > >(MainWindow::startGame(std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_0&, std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&) invoke.h:30
#21 0x1004e2464 in std::__1::__function::__alloc_func<MainWindow::startGame(std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_0, std::__1::allocator<MainWindow::startGame(std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_0>, std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>::operator()(std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&) function.h:178
#22 0x1004e0e88 in std::__1::__function::__func<MainWindow::startGame(std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_0, std::__1::allocator<MainWindow::startGame(std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_0>, std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>::operator()(std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&) function.h:352
#23 0x100338994 in std::__1::__function::__value_func<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>::operator()(std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&) const function.h:505
#24 0x1003386a8 in std::__1::function<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>::operator()(std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&) const function.h:1182
#25 0x100338308 in Gothic::implStartLoadSave(std::__1::basic_string_view<char, std::__1::char_traits<char> >, bool, std::__1::function<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>)::$_0::operator()() const gothic.cpp:417
#26 0x100338244 in decltype(static_cast<Gothic::implStartLoadSave(std::__1::basic_string_view<char, std::__1::char_traits<char> >, bool, std::__1::function<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>)::$_0>(fp)()) std::__1::__invoke<Gothic::implStartLoadSave(std::__1::basic_string_view<char, std::__1::char_traits<char> >, bool, std::__1::function<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>)::$_0>(Gothic::implStartLoadSave(std::__1::basic_string_view<char, std::__1::char_traits<char> >, bool, std::__1::function<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>)::$_0&&) type_traits:3918
#27 0x1003381e0 in void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, Gothic::implStartLoadSave(std::__1::basic_string_view<char, std::__1::char_traits<char> >, bool, std::__1::function<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>)::$_0>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, Gothic::implStartLoadSave(std::__1::basic_string_view<char, std::__1::char_traits<char> >, bool, std::__1::function<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>)::$_0>&, std::__1::__tuple_indices<>) thread:287
#28 0x1003378e0 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, Gothic::implStartLoadSave(std::__1::basic_string_view<char, std::__1::char_traits<char> >, bool, std::__1::function<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>)::$_0> >(void*) thread:298
#29 0x1861a94e8 in _pthread_start+0x90 (libsystem_pthread.dylib:arm64e+0x74e8)
#30 0x1861a42cc in thread_start+0x4 (libsystem_pthread.dylib:arm64e+0x22cc)
0x0002cdc006a0 is located 96 bytes inside of 128-byte region [0x0002cdc00640,0x0002cdc006c0)
allocated by thread T8 here:
#0 0x1051b0bd8 in wrap__Znwm+0x74 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x4cbd8)
#1 0x10149d8fc in void* std::__1::__libcpp_operator_new<unsigned long>(unsigned long) new:235
#2 0x10149d6ec in std::__1::__libcpp_allocate(unsigned long, unsigned long) new:261
#3 0x1014a2da4 in std::__1::allocator<phoenix::vobs::trigger_list::target>::allocate(unsigned long) allocator.h:108
#4 0x1014a2768 in std::__1::allocator_traits<std::__1::allocator<phoenix::vobs::trigger_list::target> >::allocate(std::__1::allocator<phoenix::vobs::trigger_list::target>&, unsigned long) allocator_traits.h:262
#5 0x1014a2378 in std::__1::__split_buffer<phoenix::vobs::trigger_list::target, std::__1::allocator<phoenix::vobs::trigger_list::target>&>::__split_buffer(unsigned long, unsigned long, std::__1::allocator<phoenix::vobs::trigger_list::target>&) __split_buffer:315
#6 0x1014a18f0 in std::__1::__split_buffer<phoenix::vobs::trigger_list::target, std::__1::allocator<phoenix::vobs::trigger_list::target>&>::__split_buffer(unsigned long, unsigned long, std::__1::allocator<phoenix::vobs::trigger_list::target>&) __split_buffer:314
#7 0x10149f71c in void std::__1::vector<phoenix::vobs::trigger_list::target, std::__1::allocator<phoenix::vobs::trigger_list::target> >::__emplace_back_slow_path<phoenix::vobs::trigger_list::target>(phoenix::vobs::trigger_list::target&&) vector:1687
#8 0x101494644 in phoenix::vobs::trigger_list::target& std::__1::vector<phoenix::vobs::trigger_list::target, std::__1::allocator<phoenix::vobs::trigger_list::target> >::emplace_back<phoenix::vobs::trigger_list::target>(phoenix::vobs::trigger_list::target&&) vector:1709
#9 0x101494218 in phoenix::vobs::trigger_list::parse(phoenix::vobs::trigger_list&, phoenix::archive_reader&, phoenix::game_version) trigger.cc:60
#10 0x1015c9f00 in phoenix::parse_vob_tree(phoenix::archive_reader&, phoenix::game_version) vob_tree.cc:180
#11 0x1015cc840 in phoenix::parse_vob_tree(phoenix::archive_reader&, phoenix::game_version) vob_tree.cc:251
#12 0x1015cc840 in phoenix::parse_vob_tree(phoenix::archive_reader&, phoenix::game_version) vob_tree.cc:251
#13 0x101566cec in phoenix::world::parse(phoenix::buffer&, phoenix::game_version) world.cc:88
#14 0x1006374b8 in World::World(GameSession&, std::__1::basic_string_view<char, std::__1::char_traits<char> >, bool, std::__1::function<void (int)>) world.cpp:75
#15 0x100638020 in World::World(GameSession&, std::__1::basic_string_view<char, std::__1::char_traits<char> >, bool, std::__1::function<void (int)>) world.cpp:64
#16 0x1002d4418 in GameSession::GameSession(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) gamesession.cpp:62
#17 0x1002d4ffc in GameSession::GameSession(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) gamesession.cpp:55
#18 0x1004e25d0 in MainWindow::startGame(std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_0::operator()(std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&) const mainwindow.cpp:866
#19 0x1004e251c in decltype(static_cast<MainWindow::startGame(std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_0&>(fp)(static_cast<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >>(fp0))) std::__1::__invoke<MainWindow::startGame(std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_0&, std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > >(MainWindow::startGame(std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_0&, std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&) type_traits:3918
#20 0x1004e24c4 in std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > std::__1::__invoke_void_return_wrapper<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >, false>::__call<MainWindow::startGame(std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_0&, std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > >(MainWindow::startGame(std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_0&, std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&) invoke.h:30
#21 0x1004e2464 in std::__1::__function::__alloc_func<MainWindow::startGame(std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_0, std::__1::allocator<MainWindow::startGame(std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_0>, std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>::operator()(std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&) function.h:178
#22 0x1004e0e88 in std::__1::__function::__func<MainWindow::startGame(std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_0, std::__1::allocator<MainWindow::startGame(std::__1::basic_string_view<char, std::__1::char_traits<char> >)::$_0>, std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>::operator()(std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&) function.h:352
#23 0x100338994 in std::__1::__function::__value_func<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>::operator()(std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&) const function.h:505
#24 0x1003386a8 in std::__1::function<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>::operator()(std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&) const function.h:1182
#25 0x100338308 in Gothic::implStartLoadSave(std::__1::basic_string_view<char, std::__1::char_traits<char> >, bool, std::__1::function<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>)::$_0::operator()() const gothic.cpp:417
#26 0x100338244 in decltype(static_cast<Gothic::implStartLoadSave(std::__1::basic_string_view<char, std::__1::char_traits<char> >, bool, std::__1::function<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>)::$_0>(fp)()) std::__1::__invoke<Gothic::implStartLoadSave(std::__1::basic_string_view<char, std::__1::char_traits<char> >, bool, std::__1::function<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>)::$_0>(Gothic::implStartLoadSave(std::__1::basic_string_view<char, std::__1::char_traits<char> >, bool, std::__1::function<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>)::$_0&&) type_traits:3918
#27 0x1003381e0 in void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, Gothic::implStartLoadSave(std::__1::basic_string_view<char, std::__1::char_traits<char> >, bool, std::__1::function<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>)::$_0>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, Gothic::implStartLoadSave(std::__1::basic_string_view<char, std::__1::char_traits<char> >, bool, std::__1::function<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>)::$_0>&, std::__1::__tuple_indices<>) thread:287
#28 0x1003378e0 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, Gothic::implStartLoadSave(std::__1::basic_string_view<char, std::__1::char_traits<char> >, bool, std::__1::function<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>)::$_0> >(void*) thread:298
#29 0x1861a94e8 in _pthread_start+0x90 (libsystem_pthread.dylib:arm64e+0x74e8)
Thread T8 created by T0 here:
#0 0x10519df58 in wrap_pthread_create+0x54 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x39f58)
#1 0x100337880 in std::__1::__libcpp_thread_create(_opaque_pthread_t**, void* (*)(void*), void*) __threading_support:421
#2 0x100337638 in std::__1::thread::thread<Gothic::implStartLoadSave(std::__1::basic_string_view<char, std::__1::char_traits<char> >, bool, std::__1::function<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>)::$_0, void>(Gothic::implStartLoadSave(std::__1::basic_string_view<char, std::__1::char_traits<char> >, bool, std::__1::function<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>)::$_0&&) thread:314
#3 0x100322e80 in std::__1::thread::thread<Gothic::implStartLoadSave(std::__1::basic_string_view<char, std::__1::char_traits<char> >, bool, std::__1::function<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>)::$_0, void>(Gothic::implStartLoadSave(std::__1::basic_string_view<char, std::__1::char_traits<char> >, bool, std::__1::function<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>)::$_0&&) thread:306
#4 0x100322c64 in Gothic::implStartLoadSave(std::__1::basic_string_view<char, std::__1::char_traits<char> >, bool, std::__1::function<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>) gothic.cpp:411
#5 0x100322dc0 in Gothic::startLoad(std::__1::basic_string_view<char, std::__1::char_traits<char> >, std::__1::function<std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> > (std::__1::unique_ptr<GameSession, std::__1::default_delete<GameSession> >&&)>) gothic.cpp:393
#6 0x1004d3888 in MainWindow::startGame(std::__1::basic_string_view<char, std::__1::char_traits<char> >) mainwindow.cpp:864
#7 0x1004d2ce8 in MainWindow::MainWindow(Tempest::Device&) mainwindow.cpp:75
#8 0x1004d43d8 in MainWindow::MainWindow(Tempest::Device&) mainwindow.cpp:35
#9 0x1004cc964 in main main.cpp:107
#10 0x104ca50f0 in start+0x204 (dyld:arm64e+0x50f0)
HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_container_overflow=0.
If you suspect a false positive see also: https://github.com/google/sanitizers/wiki/AddressSanitizerContainerOverflow.
SUMMARY: AddressSanitizer: container-overflow (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x3dc5c) in __asan_memcpy+0x240
Shadow bytes around the buggy address:
0x007059ba0080: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x007059ba0090: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x007059ba00a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x007059ba00b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x007059ba00c0: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
=>0x007059ba00d0: 00 00 00 00[fc]fc fc fc fa fa fa fa fa fa fa fa
0x007059ba00e0: 00 00 00 00 00 00 fc fc fc fc fc fc fc fc fc fa
0x007059ba00f0: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
0x007059ba0100: 00 00 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
0x007059ba0110: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x007059ba0120: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==89736==ABORTING
This is something which has been on my mind for a long time. phoenix should also support writing the files it parses so it can be used as a modern modding tool.
This requires merging the feature/flexible-buffers branch first and then implementing an archive_writer
. After that is done, dump
functions can be implemented for each file type.
Danger: especially for the VOb tree, it is critical that the correct object version numbers are saved!
The following has been done:
WriteArchive
VirtualObject
and descendantsAxisAlignedBoundingBox
OrientedBoundingBox
CutsceneLibrary
DaedalusScript
Date
Font
Material
Mesh
(partially done)Model
ModelAnimation
ModelHierarchy
ModelMesh
ModelScript
MorphMesh
MultiResolutionMesh
SaveGame
SoftSkinMesh
Texture
Vfs
World
WayNet
BspTree
OpenGothic CI: https://ci.appveyor.com/project/Try/opengothic/builds/45436817/job/vljogctckhdu2sh0
[ 41%] Building CXX object lib/phoenix/CMakeFiles/phoenix.dir/source/world/bsp_tree.cc.o
/home/appveyor/projects/opengothic/lib/phoenix/source/world/bsp_tree.cc:60:3: error: extra ‘;’ [-Werror=pedantic]
60 | };
| ^
cc1plus: all warnings being treated as errors
make[3]: ***
Currently, deprecated symbols are always included in the build but there should be a way of excluding them at compile time. This can be done using a pre-processor symbol and a set of #ifndef
s.
Instead of defining a parse
static member function for each entity, there should instead be a uniform
namespace phoenix {
template <typename T>
T parse(buffer&);
template <typename T>
T parse(buffer&&);
template <typename T>
T parse(archive_reader&);
}
This can only be implemented if a heuristic for detecting the Gothic version within worlds can be found. One idea would be to search through the archive's key-map to find object keys only present in Gothic 2 VObs, like visualAniMode
(probably unlikely to yield good results since it's in the base zCVob
which is most likely packed), canMove
or fadeOutSky
. It might still be a good idea, however, to keep the explicit version around somehow, so consumers of the library can prevent loading the wrong version anyways.
This alteration should also apply to dumping function (if implemented):
namespace phoenix {
template <typename T>
buffer dump(const T&);
}
At the moment, phoenix only reports some failures using custom exception classes. Ideally, any failure which needs to be reported to the user should be wrapped in a custom exception class.
There is also a discussion to be had about using exceptions at all. It might be better to report failures using std::optional
or a custom std::expected
structure than throwing exceptions since some codebases don't allow exceptions at all.
Since this is a parser library, failure can occur more often than usual thus performance is also a point to keep in mind when using exceptions to report parsing failures.
One idea would be to search through the archive's key-map to find object keys only present in Gothic 2 VObs, like visualAniMode
(probably unlikely to yield good results since it's in the base zCVob which is most likely packed), canMove
or fadeOutSky
. It might still be a good idea, however, to keep the explicit version around somehow, so consumers of the library can prevent loading the wrong version anyways.
Some model script files don't follow proper syntax which leads to the parser failing. The parser fails when trying to parse any statement which is missing a closing parenthesis like this:
// ...
aniEnum {
ani("aniName1" 111 "aniNext1" 4.2 0.5 MI "aniModel1" F 221 331 FPS:25 CVS:0.2 // <- missing parenthesis!
// ...
The ZenGin itself seems to be able to parse these files just fine, so the parser in phoenix has to be adjusted so that it treats closing round parenthesis (and possible curly braces too) as whitespace.
Self-explanatory. phoenix used to have CI over on GitLab but since moving the project I've not set it up again. CI should cover clang-format
and clang-tidy
compliance as well as running the tests1.
CI would ideally build on the latest Ubuntu release as well as Windows to ensure compatibility. Since MacOS is unix-like, phoenix should run on it without problems.
Some tests can currently not run in CI due to copyright issues. There might be a way to use different files, though. ↩
These VObs are empty though they seem to be related to zCCSCamera
keyframes. ZenLib calles them zReference
but does not use them anywhere. Reverse-engineering of the original game binary might be required.
Leads:
zCArchiverBinSafe::ReadObject@0x51fb50
contains a check against 0xA7
at 0x51fbde
0x51fc63
contains string "D: zArchiver(::ReadObject): objRef, illegal object index: "
Offended code:
std::string buffer::get_line(bool skip_whitespace) {
...
if (skip_whitespace) {
auto count = mismatch([](char chr) { return !std::isspace(chr); }); // <--
if (count == -1) {
position(limit()); // the end of the buffer has been reached
} else {
position(position() + count);
}
}
return tmp;
}
char
can be signed on some C++ implementations, and have negative values.
Issue was hit, when parsing world mesh in steam-eng version of gothic2
Keyframes consist of a position and a rotation stored as a quaternion. Currently the rotation is parsed as a glm::mat4
and then converted to a glm::quat
which is not correct.
It should be parsed as a quaternion directly. The component order is w, x, y, z
.
Source: Try/OpenGothic#362
Hi,
i tried building this with CMake 3.24.1 Visual Studio 2019 project generator.
Cmake Output:
Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.19042.
The C compiler identification is MSVC 19.29.30146.0
The CXX compiler identification is MSVC 19.29.30146.0
Detecting C compiler ABI info
Detecting C compiler ABI info - done
Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - skipped
Detecting C compile features
Detecting C compile features - done
Detecting CXX compiler ABI info
Detecting CXX compiler ABI info - done
Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - skipped
Detecting CXX compile features
Detecting CXX compile features - done
GLM: Version 0.9.9.9
Module support is disabled.
Version: 9.1.1
Build type:
CXX_STANDARD: 20
Required features: cxx_variadic_templates
CMake Warning (dev) at C:/Program Files/CMake/share/cmake-3.24/Modules/FetchContent.cmake:1264 (message):
The DOWNLOAD_EXTRACT_TIMESTAMP option was not given and policy CMP0135 is
not set. The policy's OLD behavior will be used. When using a URL
download, the timestamps of extracted files should preferably be that of
the time of extraction, otherwise code that depends on the extracted
contents might not be rebuilt if the URL changes. The OLD behavior
preserves the timestamps from the archive instead, but this is usually not
what you want. Update your project to the NEW behavior or specify the
DOWNLOAD_EXTRACT_TIMESTAMP option with a value of true to avoid this
robustness issue.
Call Stack (most recent call first):
vendor/CMakeLists.txt:12 (FetchContent_Declare)
This warning is for project developers. Use -Wno-dev to suppress it.
CMake Deprecation Warning at build/_deps/libsquish-src/CMakeLists.txt:14 (CMAKE_MINIMUM_REQUIRED):
Compatibility with CMake < 2.8.12 will be removed from a future version of
CMake.
Update the VERSION argument <min> value or use a ...<max> suffix to tell
CMake that the project does not need compatibility with older versions.
Configuring done
Generating done
CMake Error: Cannot open file for write: C:/Users/User/Eigene Repositories/Renderer/phoenix/build/tools/zdump/phoenix::tools::zinfo.sln.tmp77450
CMake Error: : System Error: Invalid argument
CMake Error: Cannot open file for write: C:/Users/User/Eigene Repositories/Renderer/phoenix/build/tools/zmodel/phoenix::tools::zmodel.sln.tmp07b13
CMake Error: : System Error: Invalid argument
CMake Error: Cannot open file for write: C:/Users/User/Eigene Repositories/Renderer/phoenix/build/tools/zscript/phoenix::tools::zscript.sln.tmp17120
CMake Error: : System Error: Invalid argument
CMake Error: Cannot open file for write: C:/Users/User/Eigene Repositories/Renderer/phoenix/build/tools/ztex/phoenix::tools::ztex.sln.tmp44674
CMake Error: : System Error: Invalid argument
CMake Error: Cannot open file for write: C:/Users/User/Eigene Repositories/Renderer/phoenix/build/tools/zvdfs/phoenix::tools::zvdfs.sln.tmpdf74c
CMake Error: : System Error: Invalid argument
No idea whats up with the CMake errors at the end there, i ignored them.
Then to fix project setup i needed to change:
Properties (all configurations) -> C++/Command Line/Additional Options
:
-Wno-unused-parameter
-Wextra
, -Werror
, -Wconversion
-Wno-conversion
-Wno-conversion
I think these flags do only apply to certain compilers and should not be set if MSVC is used.
After that i can build much more than before, however I still get a lot of compile errors (Debug x64):
Severity | Code | Description | Project | File | Line |
---|---|---|---|---|---|
Error | C2665 | 'std::from_chars': none of the 14 overloads could convert all the argument types | phoenix | C:\Users\User\Eigene Repositories\Renderer\phoenix\source\archive\archive_ascii.cc | 146 |
Error | C2039 | 'base': is not a member of 'std::_String_iterator<std::_String_val<std::_Simple_types<_Elem>>>' | phoenix | C:\Users\User\Eigene Repositories\Renderer\phoenix\source\archive\archive_ascii.cc | 140 |
Error | C3536 | 'beg_it': cannot be used before it is initialized | phoenix | C:\Users\User\Eigene Repositories\Renderer\phoenix\source\archive\archive_ascii.cc | 146 |
Error | C2665 | 'std::from_chars': none of the 14 overloads could convert all the argument types | phoenix | C:\Users\User\Eigene Repositories\Renderer\phoenix\source\archive\archive_ascii.cc | 147 |
Error | C2665 | 'std::from_chars': none of the 14 overloads could convert all the argument types | phoenix | C:\Users\User\Eigene Repositories\Renderer\phoenix\source\archive\archive_ascii.cc | 148 |
Error | C2665 | 'std::from_chars': none of the 14 overloads could convert all the argument types | phoenix | C:\Users\User\Eigene Repositories\Renderer\phoenix\source\archive\archive_ascii.cc | 149 |
Error | C2039 | 'base': is not a member of 'std::_String_iterator<std::_String_val<std::_Simple_types<_Elem>>>' | phoenix | C:\Users\User\Eigene Repositories\Renderer\phoenix\source\archive\archive_ascii.cc | 163 |
Error | C3536 | 'beg_it': cannot be used before it is initialized | phoenix | C:\Users\User\Eigene Repositories\Renderer\phoenix\source\archive\archive_ascii.cc | 166 |
Error | C2665 | 'std::from_chars': none of the 14 overloads could convert all the argument types | phoenix | C:\Users\User\Eigene Repositories\Renderer\phoenix\source\archive\archive_ascii.cc | 166 |
Error | C2440 | 'initializing': cannot convert from 'int' to 'phoenix::alpha_function' | phoenix | C:\Users\User\Eigene Repositories\Renderer\phoenix\include\phoenix\material.hh | 87 |
Error | C2440 | 'initializing': cannot convert from 'int' to 'phoenix::alpha_function' | phoenix | C:\Users\User\Eigene Repositories\Renderer\phoenix\include\phoenix\material.hh | 87 |
Error | LNK1104 | cannot open file '....\lib\Debug\phoenix.lib' | zscript | C:\Users\User\Eigene Repositories\Renderer\phoenix\build\tools\zscript\LINK | 1 |
Error | LNK1104 | cannot open file '....\lib\Debug\phoenix.lib' | zmodel | C:\Users\User\Eigene Repositories\Renderer\phoenix\build\tools\zmodel\LINK | 1 |
Error | LNK1104 | cannot open file '....\lib\Debug\phoenix.lib' | zdump | C:\Users\User\Eigene Repositories\Renderer\phoenix\build\tools\zdump\LINK | 1 |
Error | LNK1104 | cannot open file '....\lib\Debug\phoenix.lib' | zvdfs | C:\Users\User\Eigene Repositories\Renderer\phoenix\build\tools\zvdfs\LINK | 1 |
Error | LNK1104 | cannot open file '....\lib\Debug\phoenix.lib' | ztex | C:\Users\User\Eigene Repositories\Renderer\phoenix\build\tools\ztex\LINK | 1 |
Error | C2079 | 'array' uses undefined class 'std::arraystd::byte,2' | phoenix-tests | C:\Users\User\Eigene Repositories\Renderer\phoenix\tests\test_buffer.cc | 259 |
Error | C2666 | 'phoenix::buffer::get': 3 overloads have similar conversions | phoenix-tests | C:\Users\User\Eigene Repositories\Renderer\phoenix\tests\test_buffer.cc | 260 |
Error | C2672 | 'std::equal': no matching overloaded function found | phoenix-tests | C:\Users\User\Eigene Repositories\Renderer\phoenix\tests\test_buffer.cc | 264 |
Error | C2780 | 'bool std::equal(_ExPo &&,const _FwdIt1,const _FwdIt1,const _FwdIt2,const _FwdIt2) noexcept': expects 5 arguments - 2 provided | phoenix-tests | C:\Users\User\Eigene Repositories\Renderer\phoenix\tests\test_buffer.cc | 264 |
Error | C2780 | 'bool std::equal(const _InIt1,const _InIt1,const _InIt2,const _InIt2)': expects 4 arguments - 2 provided | phoenix-tests | C:\Users\User\Eigene Repositories\Renderer\phoenix\tests\test_buffer.cc | 264 |
Error | C2780 | 'bool std::equal(_ExPo &&,const _FwdIt1,const _FwdIt1,const _FwdIt2,const _FwdIt2,_Pr) noexcept': expects 6 arguments - 2 provided | phoenix-tests | C:\Users\User\Eigene Repositories\Renderer\phoenix\tests\test_buffer.cc | 264 |
Error | C2780 | 'bool std::equal(const _InIt1,const _InIt1,const _InIt2,const _InIt2,_Pr)': expects 5 arguments - 2 provided | phoenix-tests | C:\Users\User\Eigene Repositories\Renderer\phoenix\tests\test_buffer.cc | 264 |
Error | C2780 | 'bool std::equal(_ExPo &&,const _FwdIt1,const _FwdIt1,const _FwdIt2) noexcept': expects 4 arguments - 2 provided | phoenix-tests | C:\Users\User\Eigene Repositories\Renderer\phoenix\tests\test_buffer.cc | 264 |
Error | C2780 | 'bool std::equal(const _InIt1,const _InIt1,const _InIt2)': expects 3 arguments - 2 provided | phoenix-tests | C:\Users\User\Eigene Repositories\Renderer\phoenix\tests\test_buffer.cc | 264 |
Error | C2780 | 'bool std::equal(_ExPo &&,const _FwdIt1,const _FwdIt1,const _FwdIt2,_Pr) noexcept': expects 5 arguments - 2 provided | phoenix-tests | C:\Users\User\Eigene Repositories\Renderer\phoenix\tests\test_buffer.cc | 264 |
Error | C2780 | 'bool std::equal(const _InIt1,const _InIt1,const _InIt2,_Pr)': expects 4 arguments - 2 provided | phoenix-tests | C:\Users\User\Eigene Repositories\Renderer\phoenix\tests\test_buffer.cc | 264 |
Error | C2079 | 'array' uses undefined class 'std::arraystd::byte,2' | phoenix-tests | C:\Users\User\Eigene Repositories\Renderer\phoenix\tests\test_buffer.cc | 274 |
Error | C2664 | 'void phoenix::buffer::get(uint64_t,std::spanstd::byte,18446744073709551615) const': cannot convert argument 2 from 'int' to 'std::spanstd::byte,18446744073709551615' | phoenix-tests | C:\Users\User\Eigene Repositories\Renderer\phoenix\tests\test_buffer.cc | 275 |
Error | C2672 | 'std::equal': no matching overloaded function found | phoenix-tests | C:\Users\User\Eigene Repositories\Renderer\phoenix\tests\test_buffer.cc | 278 |
Error | C2780 | 'bool std::equal(_ExPo &&,const _FwdIt1,const _FwdIt1,const _FwdIt2,const _FwdIt2) noexcept': expects 5 arguments - 2 provided | phoenix-tests | C:\Users\User\Eigene Repositories\Renderer\phoenix\tests\test_buffer.cc | 278 |
Error | C2780 | 'bool std::equal(const _InIt1,const _InIt1,const _InIt2,const _InIt2)': expects 4 arguments - 2 provided | phoenix-tests | C:\Users\User\Eigene Repositories\Renderer\phoenix\tests\test_buffer.cc | 278 |
Error | C2780 | 'bool std::equal(_ExPo &&,const _FwdIt1,const _FwdIt1,const _FwdIt2,const _FwdIt2,_Pr) noexcept': expects 6 arguments - 2 provided | phoenix-tests | C:\Users\User\Eigene Repositories\Renderer\phoenix\tests\test_buffer.cc | 278 |
Error | C2780 | 'bool std::equal(const _InIt1,const _InIt1,const _InIt2,const _InIt2,_Pr)': expects 5 arguments - 2 provided | phoenix-tests | C:\Users\User\Eigene Repositories\Renderer\phoenix\tests\test_buffer.cc | 278 |
Error | C2780 | 'bool std::equal(_ExPo &&,const _FwdIt1,const _FwdIt1,const _FwdIt2) noexcept': expects 4 arguments - 2 provided | phoenix-tests | C:\Users\User\Eigene Repositories\Renderer\phoenix\tests\test_buffer.cc | 278 |
Error | C2780 | 'bool std::equal(const _InIt1,const _InIt1,const _InIt2)': expects 3 arguments - 2 provided | phoenix-tests | C:\Users\User\Eigene Repositories\Renderer\phoenix\tests\test_buffer.cc | 278 |
Error | C2780 | 'bool std::equal(_ExPo &&,const _FwdIt1,const _FwdIt1,const _FwdIt2,_Pr) noexcept': expects 5 arguments - 2 provided | phoenix-tests | C:\Users\User\Eigene Repositories\Renderer\phoenix\tests\test_buffer.cc | 278 |
Error | C2780 | 'bool std::equal(const _InIt1,const _InIt1,const _InIt2,_Pr)': expects 4 arguments - 2 provided | phoenix-tests | C:\Users\User\Eigene Repositories\Renderer\phoenix\tests\test_buffer.cc | 278 |
Error | C2664 | 'void phoenix::buffer::get(uint64_t,std::spanstd::byte,18446744073709551615) const': cannot convert argument 2 from 'int' to 'std::spanstd::byte,18446744073709551615' | phoenix-tests | C:\Users\User\Eigene Repositories\Renderer\phoenix\tests\test_buffer.cc | 281 |
Looks similar when building in release mode. The projects are configured with ISO C++20 Standard (/std:c++20)
.
I might try again with a later version of Visual Studio, but for Visual Studio 19 it seems to much work to get it running. Maybe not all used C++ 20 features are supported by MSVC, i don't know.
phoenix should support the original save-game format.
The command-line interface for the tool scripts can be a bit fiddly with regards to argument placement. Ideally, their CLI would be generified as well to reduce the number of surprises for the user. Most tools also need more options for specifying file output locations, file types and more.
In OpenGothic the size of p.animations
is smaller for some mds files than if using a pre-phoenix build. E.g. for HumanS.mds
only the first 279 of 864 entries are loaded resulting in stuck npc behavior.
This is only a issue for G1, in G2 there's no difference.
Animation::Animation(phoenix::model_script &p, std::string_view name, const bool ignoreErrChunks) {
ref = std::move(p.aliases);
for(auto& ani : p.animations) {
This allows for find
-ing keys of type std::string
using std::string_view
which makes everything faster.
A std::stack
does not have any relevant performance requirements, so it might be unsuitable for a semi high-performance application like the VM. Replacing it with a wrapper around a std::vector
, for example should improve performance.
In cases where the pointer is not used outside of the external function, this can improve performance by not having to increment the shared_ptr
's refcount.
Noticed at top level testing in OpenGothic, presumable those were not loaded due to potential error in model-script parsing.
Do not have much more info at this point, can do extra testing, if needed.
Version: Steam
Parser: text based (lexy)
Currently there are two functions model_script::parse
and model_script::parse_binary
. Since we can detect whether we're dealing with a binary file or not by reading the first two bytes and checking whether the resulting uint16_t
is in range of any of the chunk IDs, there should only be one parse
function.
Hi, I've found an issue with level parsing. Offended object:
This is invisible ore nugget, that triggers the big bridge, after fire-dragon. focus_override
is needed, so player can aim it with bow.
focus_override
value is wrongly false
, with current main branch.
When debugging locally, I've found that problem is in read_bool
routine:
bool archive_reader_binsafe::read_bool() {
ensure_entry_meta(bs_bool);
- return input.get_uint() != 1;
+ return input.get_uint() != 0;
}
Proposed change is to replace all read_bool
variants with !=0
- what is similar to how code would behave, if original game uses implicit case.
MinGW has no -fsanitize
support, should be disabled:
- else()
- elif (NOT WIN32)
After building OpenGothic, sub-module OpenGothic\lib\phoenix\vendor\libsquish
becomes modifyed:
HEAD detached at b28220d
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
(commit or discard the untracked or modified content in submodules)
modified: vendor/libsquish (modified content)
...
HEAD detached at ae7054a
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: squish_export.h
I'm not sure, if this header usefull at all. Also part of that: libsquish is only to implement texture::as_rgba8
, right? Can it be removed eventually?
Currently, when loading MRMs (or phoenix::mesh
es), a secondary parameter for only including certain polygons can be provided. This is used mostly by phoenix::world::parse
to exclude level-of-detail polygons from the world mesh.
This functionality should
parse
.Hi team,
I want to set up phoenix as linked dependency to my C++ project.
Unfortunately calling a phoenix function leads to runtime exception:
The procedure entry point _ZNKSt7codecvtlwc9_MbstatetE10do_unshiftERS0_PcS3_RS3_ could not be located in the dynamic link library [...]\build\test_lib.exe
I checked repositories phoenix-studio and OpenGothic for some references how to handle but without success.
What am I missing?
I'm posting my setup here to reproduce:
main.cpp
#include <iostream>
#include <phoenix/vdfs.hh>
int main(int argc, char** argv) {
auto vdf = phoenix::vdf_file::open("speech_babe_speech_engl.VDF");
return 0;
}
CMakeLists.txt
cmake_minimum_required(VERSION 3.10)
set(CMAKE_CXX_STANDARD 17)
project(test_lib VERSION 0.1.0)
add_subdirectory(phoenix)
add_executable(test_lib main.cpp)
target_link_libraries(test_lib phoenix)
build commands
(cmake version = 3.25.2)
cmake -G "MinGW Makefiles" -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build
In isolated build envionments it might not be possible to access the network while in the CMake configure step. Ideally, it would be possible to include phoenix' dependences as Git submodules instead, like it was before c30bba8. There is one dependency which is not available as a Git submodule, however: libsquish. It can be downloaded as a ZIP from SourceForge but it is to my knownlege not possible to add it as a Git submodule.
It would require making a fork of libsquish
here on GitHub, GitLab or another Git hosting provider. Ideally, that fork would automatically be updatd as well though that might not be possible on GitHub due to SourceForge using Subversion.
I have made a clone of libsquish
at https://github.com/lmichaelis/phoenix-libsquish.
Origin: Try/OpenGothic#352
The API for phoenix as a whole needs to be properly documented for the end-user. Additionally, examples should be added for each feature.
I'm looking into KoM startup routine, to see if it's possible to have adequate support for this mod.
Before phoenix, I was able to react stating location and play tutorial segment on a ship.
symbol::set_int
void symbol::set_int(std::int32_t value, std::size_t index, const std::shared_ptr<instance>& context) {
...
if (is_member()) {
...
} else {
// _m_value is null
std::get<std::unique_ptr<std::int32_t[]>>(_m_value)[index] = value;
}
}
Script call-stack:
"HOOKENGINEF"
"INIT_FEATURES_GAMESTART"
"INIT_GAMESTART"
"INIT_GLOBAL"
Apparently it's related to assignment to HOOKENGINEF.FUNCTION
(function override?)
opcode::div
and opcode::mod
If script divides by zero - whole application crashes
allocate_instance
, when called for "$INSTANCE_HELP"
auto* parent = find_symbol_by_index(sym->parent()); // parent ==nullptr here
while (parent->type() != datatype::class_) {
parent = find_symbol_by_index(parent->parent());
}
[phoenix] vm: internal exception: illegal mutable access of const symbol WIN_GETLASTERROR.CALL
[phoenix] vm: internal exception: illegal mutable access of const symbol KMLIB_INITIALIZEGAMESTART._KMLIB_INITIALIZEGAMESTART_PTR
[phoenix] vm: internal exception: illegal mutable access of const symbol CALLINT_RETVALISFLOAT
[phoenix] vm: internal exception: illegal mutable access of const symbol CALLINT_PUTRETVALTO
[phoenix] vm: internal exception: illegal mutable access of const symbol CALLINT_NUMPARAMS
Proposed fix: introduce vm_allow_const_cast_access
, flag similar to vm_allow_null_instance_access
// CATINV_NEXTNONACTIVEITEM.* should be local variables
var int CATINV_NEXTNONACTIVEITEM.I = 0;
instance CATINV_NEXTNONACTIVEITEM.L(ZCLISTSORT)
instance CATINV_NEXTNONACTIVEITEM.ITM(C_ITEM)
func int CATINV_LASTNONACTIVEITEM(var int LIST, var int MAX) {
...
}
vm.override_function("repeat", [this](phoenix::symbol* var, int count){ repeat(var,count); })
var
as a reference to loop-counter.Proposed solution: remove check for phoenix::symbol
in check_external_params
, rest seem to work as-is
repeat
. All together makes workflow:// script
repeat(i, 32)
...
end; // variable-token
// c++
void Ikarus::repeat(phoenix::symbol* var, int count) {
auto end = vm.find_symbol_by_name("end");
if(end==nullptr)
return;
auto& called_from = ...; // need to get parent context
for(uint32_t i=called_from.pc(); i<vm.size(); ) {
auto inst = vm.instruction_at(i);
if(inst.op==phoenix::opcode::pushv && inst.symbol==end->address()) {
// inject jump-back to have a loop
break;
}
i += inst.size;
}
}
PS:
Probably it's better to have one ticket instead of million small ones.
There are multiple warnings in command line:
[phoenix] vob_tree: encountered unknown VOb [% § 0 25125]
[phoenix] vob_tree: encountered unknown VOb [% § 0 25126]
[phoenix] vob_tree: encountered unknown VOb [% § 0 25127]
[phoenix] vob_tree: encountered unknown VOb [% § 0 25128]
[phoenix] vob_tree: encountered unknown VOb [% § 0 25129]
[phoenix] vob_tree: encountered unknown VOb [% § 0 25130]
AFAIR §
is a special way to refer keyframe in cameras
The API needs to be stabilized and made more intuitive by making all classes and features behave in a similar way. Right now, for example, one can create a daedalus::script
from a buffer
or by passing a std::string_view
which reads a file. Ignoring for now, that it should take a std::filesystem::path
instead, this API should either be available for all classes or be removed in favour of just always passing a buffer
.
buffer&
to basically all user-facing functions which require one. While it is useful to have this feature within phoenix, users would almost never care about the buffer
but they have to keep it around anyways. User-facing functions should always be able to take a buffer
or a buffer&&
for the user's convenience.More FYI, rather than actual issues. When OpenGothic load *.vdf from file system, game crashes with exception:
for(auto& i:archives)
inst->gothicAssets.merge(phoenix::vdf_file::open(i.name), false); // buffer_underflow here
Callstack:
0x00007ff67c54ed76: dbg::call_stack<64u>::collect(unsigned int) in C:\Users\mrsta\Downloads\opengothic_win\Gothic2Notr.exe
0x00007ff67c3cf1dc: CrashLog::dumpStack(char const*) in C:\Users\mrsta\Downloads\opengothic_win\Gothic2Notr.exe
0x00007ff67c3cf738: terminateHandler() in C:\Users\mrsta\Downloads\opengothic_win\Gothic2Notr.exe
0x00007fff739263f6: ZN10__cxxabiv111__terminateEPFvvE in C:\Users\mrsta\Downloads\opengothic_win\libstdc++-6.dll
0x00007fff73a22d43: ZSt9terminatev in C:\Users\mrsta\Downloads\opengothic_win\libstdc++-6.dll
0x00007fff73a2bf56: _cxa_throw in C:\Users\mrsta\Downloads\opengothic_win\libstdc++-6.dll
0x00007ff67c430819: phoenix::buffer::slice(unsigned long long, unsigned long long) const in C:\Users\mrsta\Downloads\opengothic_win\Gothic2Notr.exe
0x00007ff67c47a368: phoenix::vdf_entry::read(phoenix::buffer&, unsigned int) in C:\Users\mrsta\Downloads\opengothic_win\Gothic2Notr.exe
0x00007ff67c47aa5b: phoenix::vdf_file::open(phoenix::buffer&) in C:\Users\mrsta\Downloads\opengothic_win\Gothic2Notr.exe
0x00007ff67c47ac87: phoenix::vdf_file::open(std::filesystem::__cxx11::path const&) in C:\Users\mrsta\Downloads\opengothic_win\Gothic2Notr.exe
0x00007ff67c3b7c39: Resources::loadVdfs(std::vector<std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t> >, std::allocator<std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t> > > > const&) in C:\Users\mrsta\Downloads\opengothic_win\Gothic2Notr.exe
I've added simple try-catch handler to workaround issue, so here is the list of files that are failed to load:
unable to load archive: "D:/Games/Gothic II/Data/$Templates$/asmcl.dll"
unable to load archive: "D:/Games/Gothic II/Data/Union.vdf"
unable to load archive: "D:/Games/Gothic II/Data/$Templates$/ZUNIONUTILS.DLL"
unable to load archive: "D:/Games/Gothic II/Data/$Templates$/ZBINKFIX.DLL"
unable to load archive: "D:/Games/Gothic II/Data/$Templates$/ZMOUSEFIX.DLL"
Proposed change:
Can you add new dedicated type of exception, if *.vdf file signature does not match VDF_SIGNATURE_G*
?
Thanks!
Currently, focusOverride
is parsed as a bool
but that might not be correct. A look into the disassembly might be helpful in determining whether is is actually a bool
or might be an enum
instead.
Related: #44
There is probably a lot of performance left on the table at the moment since it is not really possible to profile most parts of phoenix. Especially the script VM is critical to performance in Gothic re-implementations like OpenGothic.
Writing proper benchmarks will enable easier profiling of phoenix.
Hit runtime exception, at loading "Chronicles of Myrtana" mod:
...\TheChroniclesOfMyrtana\_work\Data\Scripts\_compiled\Fight.dat
- is empty file, that causes mmap
to fail internally.
// _deps\mio-src\include\mio\detail\mmap.ipp
inline mmap_context memory_map(const file_handle_type file_handle, const int64_t offset,
const int64_t length, const access_mode mode, std::error_code& error)
{
...
char* mapping_start = static_cast<char*>(::MapViewOfFile(
file_mapping_handle,
mode == access_mode::read ? FILE_MAP_READ : FILE_MAP_WRITE,
win::int64_high(aligned_offset),
win::int64_low(aligned_offset),
length_to_map));
if(mapping_start == nullptr) // <-- file exists, but mapping is null
{
...
Originally posted by @Try in Try/OpenGothic#271 (comment)
The code around suggest resize
instead of reserve
.
And camera_lock_mode
should be renamed to sprite_alignment_mode
GameScript::loadDialogOU()
become noticeably slower with phenix backend.
I haven't done proper profiling, but issue seem to be rooted in large amount of small allocations, while population hash-map.
Lazy profiling(debug break a few times) shows, that this place can be an issue:
bool archive_reader_binsafe::read_object_begin(archive_object& obj) {
...
std::stringstream ss {line.substr(1, line.size() - 2)}; // substr creates a copy; and well stringstream known to be slow
ss >> obj.object_name >> obj.class_name >> obj.version >> obj.index;
return true;
}
Current syntax:
symbol* script::find_symbol_by_name(const std::string& name)
forces client to bake std::string
into interface.
for find function this is especially unnecessary, since it does string copy immediatly after:
symbol* script::find_symbol_by_name(const std::string& name) {
std::string up {name}; // <-- fine, but no need to query name as string
std::transform(up.begin(), up.end(), up.begin(), ::toupper);
if (auto it = _m_symbols_by_name.find(up); it != _m_symbols_by_name.end()) {
return find_symbol_by_index(it->second);
}
return nullptr;
}
In general it's probably would be good to pass read-only strings as string_view
in all cases. If implementation then uses this string as key to has-map or similar - solve this locally inside the function.
Other places:
R vm::call_function(const std::string& name, P... args)
vm::init_instance(const std::string& name)
vm::allocate_instance(const std::string& name)
void vm::push_string(const std::string& value)
void register_external(const std::string& name, const std::function<R(P...)>& callback)
void override_function(const std::string& name, const std::function<R(P...)>& callback)
const way_point* way_net::waypoint(const std::string& name) const
const message_block* messages::block_by_name(const std::string& name) const
This is most important in the opcode::be
handler where the return value of the external is not pushed if it is not registered. Other opcodes might benefit from such a system as well. The most practical implementation would probably be an RAII helper class.
Currently, the CMake build script is pretty bare-bones and does not allow for easy, clean distribution of the library code. This is due to being statically linked as well as pulling in 3rd-party libraries directly using target_link_libraries(... PUBLIC ...)
which causes CMake's install process to malfunction by including 3rd-party library code in the installation.
This is unacceptable for installing phoenix into systems directly since the extra files might pollute existing installations of those libraries or could cause the package manager to break.
This makes sense because the string is copied internally anyways. Additionally, defaulting to taking std::string_view
provides more opportunities for copy elision. This is needed as a consequence to #30.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.