Issue with loading Union related archives

[Try]

More FYI, rather than actual issues. When OpenGothic load *.vdf from file system, game crashes with exception:

 for(auto& i:archives)
    inst->gothicAssets.merge(phoenix::vdf_file::open(i.name), false); // buffer_underflow here

Callstack:

0x00007ff67c54ed76: dbg::call_stack<64u>::collect(unsigned int) in C:\Users\mrsta\Downloads\opengothic_win\Gothic2Notr.exe
0x00007ff67c3cf1dc: CrashLog::dumpStack(char const*) in C:\Users\mrsta\Downloads\opengothic_win\Gothic2Notr.exe
0x00007ff67c3cf738: terminateHandler() in C:\Users\mrsta\Downloads\opengothic_win\Gothic2Notr.exe
0x00007fff739263f6: ZN10__cxxabiv111__terminateEPFvvE in C:\Users\mrsta\Downloads\opengothic_win\libstdc++-6.dll
0x00007fff73a22d43: ZSt9terminatev in C:\Users\mrsta\Downloads\opengothic_win\libstdc++-6.dll
0x00007fff73a2bf56: _cxa_throw in C:\Users\mrsta\Downloads\opengothic_win\libstdc++-6.dll
0x00007ff67c430819: phoenix::buffer::slice(unsigned long long, unsigned long long) const in C:\Users\mrsta\Downloads\opengothic_win\Gothic2Notr.exe
0x00007ff67c47a368: phoenix::vdf_entry::read(phoenix::buffer&, unsigned int) in C:\Users\mrsta\Downloads\opengothic_win\Gothic2Notr.exe
0x00007ff67c47aa5b: phoenix::vdf_file::open(phoenix::buffer&) in C:\Users\mrsta\Downloads\opengothic_win\Gothic2Notr.exe
0x00007ff67c47ac87: phoenix::vdf_file::open(std::filesystem::__cxx11::path const&) in C:\Users\mrsta\Downloads\opengothic_win\Gothic2Notr.exe
0x00007ff67c3b7c39: Resources::loadVdfs(std::vector<std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t> >, std::allocator<std::__cxx11::basic_string<char16_t, std::char_traits<char16_t>, std::allocator<char16_t> > > > const&) in C:\Users\mrsta\Downloads\opengothic_win\Gothic2Notr.exe

I've added simple try-catch handler to workaround issue, so here is the list of files that are failed to load:

unable to load archive: "D:/Games/Gothic II/Data/$Templates$/asmcl.dll"
unable to load archive: "D:/Games/Gothic II/Data/Union.vdf"
unable to load archive: "D:/Games/Gothic II/Data/$Templates$/ZUNIONUTILS.DLL"
unable to load archive: "D:/Games/Gothic II/Data/$Templates$/ZBINKFIX.DLL"
unable to load archive: "D:/Games/Gothic II/Data/$Templates$/ZMOUSEFIX.DLL"

Proposed change:
Can you add new dedicated type of exception, if *.vdf file signature does not match VDF_SIGNATURE_G* ?
Thanks!

[lmichaelis]

Hi, thanks I'll look at it. I might just implement a proper parsing routine if I can find documentation :)

[Try]

proper parsing routine

Not for this: ZUNIONUTILS.DLL :)

[lmichaelis]

@Try I've added the exception as requested. You can now catch phoenix::vdfs_signature_error for this specific error. Could you send over the VDF file that caused the error so I can investigate?

[Try]

Could you send over the VDF file that caused the error so I can investigate?

Unfortunately can't. This bug was reported on worldofplayer.ru forum, so don't have the file on my side.
Lets assume for now that issue was fixed :)

[lmichaelis]

I see. I'd still like to implement it so I'll keep this issue open :>

[Try]

Bump: got the Union.vdf file.
Union.zip

[lmichaelis]

Thanks, that helps out a lot!

I managed to find out, that it's a single entry in the VDF which is causing the problem: SYSTEM/ZBINKFIXOLD.DLL.

The entire archive looks like this:

_WORK
_WORK/DATA
_WORK/DATA/ANIMS
_WORK/DATA/ANIMS/_COMPILED
_WORK/DATA/ANIMS/_COMPILED/HUMANS_SKELETON-T_SPAWN.MAN
_WORK/DATA/ANIMS/_COMPILED/INVALID_SOURCE_FILE.MDL
_WORK/DATA/ANIMS/_COMPILED/INVALID_SOURCE_FILE.MDM
_WORK/DATA/ANIMS/_COMPILED/INVALID_SOURCE_FILE.MMB
_WORK/DATA/MESHES
_WORK/DATA/MESHES/_COMPILED
_WORK/DATA/MESHES/_COMPILED/INVALID_SOURCE_FILE.MRM
_WORK/DATA/SOUND
_WORK/DATA/SOUND/INVALID_SOURCE_FILE.OGG
_WORK/DATA/SOUND/INVALID_SOURCE_FILE.WAV
_WORK/DATA/TEXTURES
_WORK/DATA/TEXTURES/_COMPILED
_WORK/DATA/TEXTURES/_COMPILED/INVALID_SOURCE_FILE-C.TEX
SYSTEM
SYSTEM/ASMCL.DLL
SYSTEM/AUTORUN
SYSTEM/AUTORUN/MENU
SYSTEM/AUTORUN/MENU/ZUNIONMENU.D
SYSTEM/AUTORUN/UTILS
SYSTEM/AUTORUN/UTILS/ZUNIONTRIGGER.D
SYSTEM/AUTORUN/UTILS/ZUNIONVOB.D
SYSTEM/AUTORUN/ZBINKFIX.DLL
SYSTEM/AUTORUN/ZMOUSEFIX.DLL
SYSTEM/AUTORUN/ZPARSEREXTENDER.DLL
SYSTEM/AUTORUN/ZUNIONUTILS.DLL
SYSTEM/M3D
SYSTEM/M3D/MSSA3D.M3D
SYSTEM/M3D/MSSDS3D.M3D
SYSTEM/M3D/MSSDS3DH.M3D
SYSTEM/M3D/MSSDX7.M3D
SYSTEM/M3D/MSSEAX.M3D
SYSTEM/M3D/MSSRSX.M3D
SYSTEM/M3D/MSSSOFT.M3D
SYSTEM/OSTEAMWORKS.DLL
SYSTEM/STEAM_API.DLL
SYSTEM/ZBINKFIXOLD.DLL

That being said, there seems to be a lot of entropy in that file. When extracted as-is, all content is just random bytes and no structure can be found. The file tool reports a plain "data" format for all files in the archive.

I assume its some sort of compression or encryption but if it's compression I can't tell the type. I'll have to look into the Vdfs32g.dll Union ships with. That could take some time :)

Btw, I apologize for my absence this past week, I've been ill.

[Try]

Btw, I apologize for my absence this past week, I've been ill.

Welcome back :D

When extracted as-is, all content is just random bytes and no structure can be found.

Yes, most likely it's compressed. Yet it's probably not worth it to chaise after compression algorithm - just as long as it's not crashing it's fine.

[lmichaelis]

Yeah that looks like a custom implementation of Vdfs32g.dll. Just skimming I can already tell trouble ahead:

There might be mods using these features too since the custom VDF loader is the default.

just as long as it's not crashing it's fine.

That I can do. It'll be ugly but I'll just generate a zero-length entry if I detect an out-of-bounds condition.

[lmichaelis]

Hey @Try, can you confirm this is done?

[Try]

Tested today on latest phoenix commit(36d8d5f) in master branch:

Load fine without Union.vdf, but with Union.vdf, game throws exception upon game loading:

loading error: buffer underflow at byte 6 while reading 225443840 additional bytes [context: slicing]
Stack-trace:
1   libstdc++-6!.cxa_throw                                                                                                                                                                               
2   phoenix::buffer::slice          buffer.cc          
3   phoenix::buffer::extract        buffer.hh          
4   phoenix::animation::parse       animation.cc       
5   Animation::Sequence::Sequence   animation.cpp      
6   std::construct_at<Animation...  stl_construct.h    
7   std::allocator_traits<std::...  alloc_traits.h     
8   std::vector<Animation::Sequ...  vector.tcc         
9   std::vector<Animation::Sequ...  vector.tcc         
10  Animation::loadMAN              animation.cpp      
11  Animation::Animation            animation.cpp      
12  Resources::implLoadAnimation    resources.cpp      
13  Resources::loadAnimation        resources.cpp      
14  Resources::implLoadMeshMain     resources.cpp      
15  Resources::implLoadMesh         resources.cpp      
16  Resources::loadMesh             resources.cpp      
17  Resources::loadSkeleton         resources.cpp      
18  AnimationSolver::load           animationsolver.cpp
19  MdlVisual::load                 mdlvisual.cpp      
20  Npc::load                       npc.cpp 

Offended file, that been requested from archive: humans_skeleton-T_SPAWN.MAN

[lmichaelis]

Right, that fails because the VDF is compressed and I don't handle that. Just loading and accessing the files is fine but trying to actually parse data from them (animations in this case) will fail unless the data is properly decompressed beforehand.

[Try]

Application side, unfortunately is not aware of where compression been used. Is it possible to pretend, that compressed files are not in archive at all?

[lmichaelis]

Maybe it would be possible but it would be unreliable. Afaik there is no way of telling if a file is stored as compressed or not so I would have to try loading the file to check whether it's compressed or not. Or I would have to try to detect the file type using binary chunk IDs (for example).

If you can't handle the error in a try-catch, I recommend just not loading the VDF at all.

[Try]

If you can't handle the error in a try-catch, I recommend just not loading the VDF at all.

Yes, that's good solution, that I'm happy with. We probably need an api to test if VDF has compressed files.

[lmichaelis]

If you need to figure out if the VDF is made for/with Union, you can try checking the version stored in the header (vdf.header.version). For original Gothic 1 & 2 files it's 0x50 while Union.vdf uses 0xa0 instead. That's probably not reliable though since any file made with Union tools is likely to have that version. It's also the only difference between original files and the Union ones as far as I can tell.

Implementing a check for compressed files is probably not in scope for phoenix unless it's a really simple check. But it looks like there is no easy way of checking that.

[dreimer1986]

Just my 2 cents to this matter. The modified dll talks about zipped files, so shouldn't it be possible to get a rather reliable variant by first checking the VDF version and then the file for signatures/magic words/patterns? Very likely a open algorithm was used here and ZIP means Deflate many cases.
I learned that the hard way when I extracted stuff from unknown binary data without any success and then was told that the first 4 Bytes of my file in question were the magic pattern for a LZ4 compression. Boom! Got my file. If the algorithm is fixed, the whole thing could be way easier to accomplish. This list is far from complete, but maybe... Just one of these like my LZ4 example are enough for a quiick and precise way to detect compressed files. https://en.wikipedia.org/wiki/List_of_file_signatures
Of course this is absolutely irrelevant to compatibility of the original games in any language or version that ever showed up, but if mods are in scope in the far future, this might show up again.

[lmichaelis]

I was thinking about that initially, but I noticed none of the files had an obvious header. I did some more investigating and I did find something the could be considered a header.

At offset 0x18, in compressed files there will be the byte sequence 00 00 78 da which corresponds with ZLIB Best Compression (no preset dictionary) according to the Wiki page you linked. I tried extracting that part of the file and I can say that it was a success.

So it actually seems to be possible to detect this compression, I just didn't see those four bytes being the same everywhere. There is some data before that too but I haven't figured out what it means yet. I'm attaching a ZIP containing the first 50 bytes of all files in Union.vdf for reference. I'll look into implementing a check for this soon-ish unless one of you would like to do that earlier :)

[Try]

For now added workaround for OpenGothic: checking vdf header: f22560e1
Assuming that union is always version 160 :)