Custom DLL: DLLMain Segfault

[taviso]

Custom DLL: DLLMain Segfault

[taviso]

What is happening at the crash site? Try these commands x/i $pc and i r. The call to GetModuleFileNameA() seems sane, it's possible it didn't like the result. Maybe it was trying to find path separators in there? I think by default it will just be fakename.exe.

It looks like you just a few instructions into the routine at loc_1000D552, can you show the disassembly?

e.g.: x/20i loc_1000D552

[RichardWithnell]

Thanks for the quick response.

I've increased the verbosity of the winapi calls and the segfault occurs after a LeaveCriticalSection.

GetModuleFileNameA(): (nil), 0xf7d8a62c, 260 => 12
EnterCriticalSection(): : 0xf7d8a5e8
LeaveCriticalSection(): : 0xf7d8a5e8

Program received signal SIGSEGV, Segmentation fault.
0xf7d8055a in loc_1000D552 () at {standard input}:3208
3208	{standard input}: No such file or directory.

The output of the requested commands:

(gdb) x/i $pc
=> 0xf7d8055a <loc_1000D552>:	mov    (%esi),%al

(gdb) i r
eax            0x4	4
ecx            0xffffcf1c	-12516
edx            0x0	0
ebx            0x0	0
esp            0xffffcf1c	0xffffcf1c <RtlUnwind+136806056>
ebp            0xffffcf48	0xffffcf48 <RtlUnwind+136806100>
esi            0x0	0
edi            0x0	0
eip            0xf7d8055a	0xf7d8055a <loc_1000D552>
eflags         0x10246	[ PF ZF IF RF ]
cs             0x23	35
ss             0x2b	43
ds             0x2b	43
es             0x2b	43
fs             0x73	115
gs             0x63	99

(gdb) x/20i loc_1000D552
=> 0xf7d8055a <loc_1000D552>:	mov    (%esi),%al
   0xf7d8055c <loc_1000D552+2>:	cmp    %bl,%al
   0xf7d8055e <loc_1000D552+4>:	je     0xf7d80572 <loc_1000D56A>
   0xf7d80560 <loc_1000D552+6>:	cmp    $0x3d,%al
   0xf7d80562 <loc_1000D552+8>:	je     0xf7d80565 <loc_1000D55D>
   0xf7d80564 <loc_1000D552+10>:	inc    %edi
   0xf7d80565 <loc_1000D55D>:	push   %esi
   0xf7d80566 <loc_1000D55D+1>:	call   0xf7d81828 <_strlen>
   0xf7d8056b <loc_1000D55D+6>:	pop    %ecx
   0xf7d8056c <loc_1000D55D+7>:	lea    0x1(%esi,%eax,1),%esi
   0xf7d80570 <loc_1000D55D+11>:	jmp    0xf7d8055a <loc_1000D552>
   0xf7d80572 <loc_1000D56A>:	lea    0x4(,%edi,4),%eax
   0xf7d80579 <loc_1000D56A+7>:	push   %eax
   0xf7d8057a <loc_1000D56A+8>:	call   0xf7d7c600 <_malloc>
   0xf7d8057f <loc_1000D56A+13>:	mov    %eax,%esi
   0xf7d80581 <loc_1000D56A+15>:	pop    %ecx
   0xf7d80582 <loc_1000D56A+16>:	cmp    %ebx,%esi
   0xf7d80584 <loc_1000D56A+18>:	mov    %esi,0xf7d8a57c
   0xf7d8058a <loc_1000D56A+24>:	jne    0xf7d80594 <loc_1000D58C>
   0xf7d8058c <loc_1000D56A+26>:	push   $0x9
(gdb) 

[taviso]

So it's a NULL dereference, it was expecting the value in %esi to be an ANSI string (because it then passes it to strlen()), but it's NULL. It's tough to guess what went wrong from this, but the critical section stuff is probably not related.

Because it's so close to a call to GetModuleFileNameA, and it was clearly expecting an ANSI string here...I think it must be related to that. A shot in the dark, try changing GetModuleFileNameA to return something like C:\foo\whatever.exe, and see if it works (remember to double slashes, like \\).

It's hard to guess any further without seeing what the dll is trying to do. If you also have a Windows machine, you could put a breakpoint there and see what the string is (da @esi in windbg commands), that might help you guess what went wrong.

[taviso]

Oh, but this line is interesting:

0xf7d80560 <loc_1000D552+6>: cmp $0x3d,%al

It looks like it was expecting an = character in there. Hmm. It would be surprising if that was a pathname, maybe it's trying to parse the environment - it would be normal for those to contain equals.

It does call GetEnvironmentStringsW earlier on, so maybe it was expecting an environment variable to be set and didn't find it. There's no way to guess which one variable it wanted from that debugging output, you will need to poke around in the code (or maybe you will just get really lucky and can guess from the strings output!).

[RichardWithnell]

Thanks for the pointers! I've taken a quick look at the API calls made in Windows.

The first call to WideCharToMultiByte takes in the environment string (just looking for where = is passed around), it also passes NULL for lpMultiByteStr, so your stub function returns 0 here, while it returns a positive value in Windows.

The documentation of the return value is a little confusing:

If successful, returns the number of bytes written to the buffer pointed to by lpMultiByteStr. If the function succeeds and cbMultiByte is 0, the return value is the required size, in bytes, for the buffer indicated by lpMultiByteStr. Also see dwFlags for info about how the WC_ERR_INVALID_CHARS flag affects the return value when invalid sequences are input.

I think the stub needs to account for the second condition, where cbMultiByte is 0 so the return value indicates how big lpMultiByteStr needs to be.

Simply patching your WideCharToMultiByte function to return cchWideChar instead of 0 eliminated the segfault.

I'll tidy up my changes and make a pull request.