Tutorial: How to attempt identifying a crashing condition?

[valinet]

Due to the nature of ExplorerPatcher, mainly the fact that it injects other processes (like explorer.exe) and assumes certain things about the injected applications, many being conclusions after looking on disassembled executable code, rather than official specifications, sometimes application crashes might happen due to errors in the code, oversights and so on.

After all, ExplorerPatcher is written by humans and humans make mistakes. Identifying these problems and fixing them is paramount to continuously improving this application. You can often do much more than merely report a crash in a generic way. Also, it's always much more useful to provide a solution to a crash rather than provide or ask for a workaround.

With these being said, this tutorial will show a method for gathering clues and trying to identify the cause of a particular crash, without any particular setup beforehand. Of course, it's always better to diagnose problems using full crash dumps, but those have to be enabled in advance, while the method described here requires no special setup. Full crash dumps will be described at the end of this tutorial.

Get basic information about the crash

Windows always stores basic information regarding a crash in the Event Viewer. To open Event Viewer, open Run (Win+R) and type eventvwr.msc and hit Enter. In there, go to "Event Viewer (Local)" - "Windows Logs" - "Application". In there, look for entries of type "Error":

To identify the correct event, look in the output of the event for:

• Faulting application name: Explorer.EXE, version: 10.0.22000.527, time stamp: 0x59dc9944 - This line specifies that the faulting process is explorer.exe; ExplorerPatcher runs in-process in explorer.exe. explorer.exe also hosts the entire Windows shell, besides displaying folder windows. A crashing explorer.exe event log might mean the crash happened in ExplorerPatcher. Further on:
• Faulting module path: C:\Windows\dxgi.dll - This line indicates that the module from explorer.exe that crashed is C:\Windows\dxgi.dll. That's not Windows' DXGI (DirectX Graphics Infrastructure Library), but ExplorerPatcher. Windows' DXGI is located in C:\Windows\System32. The reason EP is called dxgi.dll and placed there is architectural (to get injected in explorer.exe, it has to be placed in C:\Windows and named like that in order to exploit the library search order in Windows).

Other paths where ExplorerPatcher is placed and masqueraded as 'dxgi.dll' are:

• C:\Windows
• C:\Windows\SystemApps\Microsoft.Windows.StartMenuExperienceHost_cw5n1h2txyewy (EP injects the Start menu)
• C:\Windows\SystemApps\ShellExperienceHost_cw5n1h2txyewy (EP injects the host process for UWP/modern UI elements of the shell - action center, various flyouts etc)

ExplorerPatcher is not only active under explorer.exe. Modules of it might get injected, depending on your configuration, in these particular executables as well:

• explorer.exe
• StartMenuExperienceHost.exe - This process hosts the Start menu.
• ShellExperienceHost.exe - This process hosts UWP/modern UI elements of the shell.
• dllhost.exe - This process ("COM Surrogate") hosts COM components that ship as libraries but request to run out of process; regarding ExplorerPatcher, that's the case for the Windows 10 taskbar weather widget.
• ep_dwm.exe - This process runs as a service under the SYSTEM user account and hosts the application that manipulates the memory of the Desktop Window Manager in order to enable not rounded corners for windows, if the corresponding option is checked in “Properties”

After you identify the correct event, you can look at the whole output, which looks something like this:

Faulting application name: Explorer.EXE, version: 10.0.22000.527, time stamp: 0x59dc9944
Faulting module name: dxgi.dll, version: 22000.469.41.16, time stamp: 0x61f204a2
Exception code: 0xc0000005
Fault offset: 0x00000000000130a4
Faulting process id: 0x3d7c
Faulting application start time: 0x01d823cf7e6477dd
Faulting application path: C:\Windows\Explorer.EXE
Faulting module path: C:\Windows\dxgi.dll
Report Id: 2025b7b1-25c0-40b5-9bcf-ccc354c6b04f
Faulting package full name: 
Faulting package-relative application ID: 

Of very big interest here is now the Fault offset: 0x00000000000130a4. This indicates the address in the assembly of the EP module of the instruction that generated the fault (crash). Take note of that. We'll use that in the next part.

Also, have a look at the Exception code: 0xc0000005. That indicates access violation (Google the code). What this means is that EP tried to access things at a wrong memory address. We don't know what that address was, but at least we know that it was not valid. We'll keep this mind for the next part, where we analyze the disassembly.

Obtain the whole build archive for the ExplorerPatcher version that you are using

By looking at the event log above, you can determine the version of ExplorerPatcher that you are using by looking at this line: Faulting module name: dxgi.dll, version: 22000.469.41.16, time stamp: 0x61f204a2. Thus, in the case of the crash reported above, we are talking about version 22000.469.41.16.

To obtain the artifacts for this build, head to ExplorerPatcher GitHub Releases. In the list, identify the build that you are using. Under the description, there is a line that reads: "An archive containing all the files generated during the build process (including dxgi.dll and symbol files) is available here."

Click the "here" link to go to the artifacts page. In there, near the bottom of the page, there will be a link to an archive. Download that (you need a GitHub account to be able to download from there, registration is free).

After downloading and unpacking the archive, you'll get a folder containing a whole bunch of files:

These are all necessary in order to correctly and easily identify the place where the crash happened.

Disassemble ExplorerPatcher

Moving on, the easiest way is to disassemble the ExplorerPatcher module binary. In our case, since the faulting module is dxgi.dll, we'll disassemble that.

As a disassembler of choice, I will use IDA Freeware. You can also use Ghidra or whatever disassembler of your choice, the process is similar.

Open IDA, click "New" and locate dxgi.dll on your disk:

In the window that appears, click "OK":

Next, IDA will ask whether it is okay to use the symbol files for ExplorerPatcher to help in aiding the disassembly. It is VERY important that you allow this, since we have the symbols for ExplorerPatcher just fine and these help A LOT in identifying where the problem is exactly. You can read more about what symbol files are here.

Since you probably haven't extracted the files at the path suggested previously, IDA will ask whether you'd like to manually locate the symbol files on the disk. Again, agree to that.

Since our module of interest is dxgi.dll, which is just ExplorerPatcher.amd64.dll with another name, use the dialog to browse to the folder where you extracted the files. In there, identify `ExplorerPatcher.amd64.pdb" and choose that.

Next, IDA may show some warning, it's not important and you can safely dismiss it.

Wait for the initial autoanalysis to finish. This should be pretty quick. You can tell whether it is done by looking for this text: "The initial autoanalysis has been finished" in the log of IDA at the bottom of the window:

Next, remember from the last section that we had that Fault offset: 0x00000000000130a4. We are interested in this number: 0x130a4. This indicates the offset from the beginning of the executable where the fault has occurred.

To determine where the line of interest is, you need to add that number to the module image base address. That can be found by going to "Edit" - "Segments" - "Rebase program...". Copy the number in there, in this case 0x180000000 and add to that the previous 0x130a4. We obtain 0x1800130A4 (type 0x180000000+0x130a4 on Google). That's where the fault occurred. To jump to that address, press "G" or go to "Jump" - "Jump to address...". Type that number in there and press OK.

.text:00000001800130A1                 mov     rbx, rax
.text:00000001800130A4                 mov     rdx, [rax]
.text:00000001800130A7                 call    qword ptr [rdx+30h]

The offending disassembly will be shown. This is very helpful, as we can determine what function this is in. For that, scroll up until you see the first "SUBROUTINE" label:

.text:0000000180013020 ; void __fastcall Explorer_RefreshUI(int unused)
.text:0000000180013020 Explorer_RefreshUI proc near            ; CODE XREF: LoadSettings+F06↑p
.text:0000000180013020                                         ; DATA XREF: Inject+25D↓o

The problematic function is Explorer_RefreshUI. You can identify that in the source code (this one's in dllmain.c) and from the disassembly identify which exact instructions might cause this.

Also, you can also use the IDA decompiler to decompile the disassembly, generating C-like code that is more readable and might help you identify the offending line on the actual source code even easier. For that, with the cursor placed on the line that generated the fault (0x1800130A4), press F5. In the generated pseudocode, the line roughly corresponding to the disassembly will be highlighted:

void __fastcall Explorer_RefreshUI(int unused)
{
  HWND Window; // rax
  HWND v2; // rax
  HWND v3; // rax
  HWND v4; // rdi
  _BYTE *WindowLongPtrW; // rbx
  HWND v6; // rdi
  HWND v7; // rax
  HWND v8; // rax
  HWND v9; // rsi
  _BYTE *v10; // rbx

  SendNotifyMessageW(HWND_BROADCAST, 0x1Au, 0i64, (LPARAM)L"TraySettings");
  Window = FindWindowExW(0i64, 0i64, L"Shell_TrayWnd", 0i64);
  if ( Window )
  {
    v2 = FindWindowExW(Window, 0i64, L"TrayNotifyWnd", 0i64);
    if ( v2 )
    {
      v3 = FindWindowExW(v2, 0i64, L"TrayClockWClass", 0i64);
      v4 = v3;
      if ( v3 )
      {
        WindowLongPtrW = (_BYTE *)GetWindowLongPtrW(v3, 0);
        (*(void (__fastcall **)(_BYTE *))(*(_QWORD *)WindowLongPtrW + 48i64))(WindowLongPtrW); <-- this is the problem
        WindowLongPtrW[547] = 1;
        InvalidateRect(v4, 0i64, 1);
      }
    }
  }

  v6 = 0i64;
  while ( 1 )
  {
    v7 = FindWindowExW(0i64, v6, L"Shell_SecondaryTrayWnd", 0i64);
    v6 = v7;
    if ( !v7 )
      break;

    v8 = FindWindowExW(v7, 0i64, L"ClockButton", 0i64);
    v9 = v8;
    if ( v8 )
    {
      v10 = (_BYTE *)GetWindowLongPtrW(v8, 0);
      (*(void (__fastcall **)(_BYTE *))(*(_QWORD *)v10 + 48i64))(v10);
      v10[547] = 1;
      InvalidateRect(v9, 0i64, 1);
    }
  }
}

With this information at hand, let's look on the real source code. First, if you look on the source code, you'll see Explorer_RefreshUI only contains 2 lines of code:

void WINAPI Explorer_RefreshUI(int unused)
{
    SendNotifyMessageW(HWND_BROADCAST, WM_WININICHANGE, 0, (LPARAM)L"TraySettings");
    Explorer_RefreshClock(0);
}

What happened here is that, because Explorer_RefreshClock is only used once in the entire program, the compiler decided to inline the Explorer_RefreshClock in the Explorer_RefreshUI function (it copied the contents of Explorer_RefreshClock into Explorer_RefreshUI and skipped storing Explorer_RefreshClock separately). Look in Explorer_RefreshClock:

void Explorer_RefreshClock(int unused)
{
    HWND hShellTray_Wnd = FindWindowExW(NULL, NULL, L"Shell_TrayWnd", NULL);
    if (hShellTray_Wnd)
    {
        HWND hTrayNotifyWnd = FindWindowExW(hShellTray_Wnd, NULL, L"TrayNotifyWnd", NULL);
        if (hTrayNotifyWnd)
        {
            HWND hClockButton = FindWindowExW(hTrayNotifyWnd, NULL, L"TrayClockWClass", NULL);
            if (hClockButton)
            {
                Explorer_RefreshClockHelper(hClockButton);
            }
        }
    }

    HWND hWnd = NULL;
    do
    {
        hWnd = FindWindowExW(
            NULL,
            hWnd,
            L"Shell_SecondaryTrayWnd",
            NULL
        );
        if (hWnd)
        {
            HWND hClockButton = FindWindowExW(hWnd, NULL, L"ClockButton", NULL);
            if (hClockButton)
            {
                Explorer_RefreshClockHelper(hClockButton);
            }
        }
    } while (hWnd);
}

This looks more like the disassembly from IDA, yet still, the same inlining happened for the Explorer_RefreshClockHelper. The crash is on that, so let's finally check that:

void Explorer_RefreshClockHelper(HWND hClockButton)
{
    INT64* ClockButtonInstance = (BYTE*)(GetWindowLongPtrW(hClockButton, 0)); // -> ClockButton
    // we call v_Initialize because all it does is to query the
    // registry and update the internal state to display seconds or not
    // to get the offset, simply inspect the vtable of ClockButton
    ((void(*)(void*))(*(INT64*)((*(INT64*)ClockButtonInstance) + 6 * sizeof(uintptr_t))))(ClockButtonInstance); // v_Initialize
    // we need to refresh the button; for the text to actually change, we need to set this:
    // inspect ClockButton::v_OnTimer
    *((BYTE*)ClockButtonInstance + 547) = 1;
    // then, we simply invalidate the area
    InvalidateRect(hClockButton, NULL, TRUE);
}

This line from the source code ((void(*)(void*))(*(INT64*)((*(INT64*)ClockButtonInstance) + 6 * sizeof(uintptr_t))))(ClockButtonInstance); // v_Initialize seems to correspond to this line from IDA's decompilation: (*(void (__fastcall **)(_BYTE *))(*(_QWORD *)WindowLongPtrW + 48i64))(WindowLongPtrW);.

So, now that we identified the offending instruction, what's the problem with it? Well, as it seems to me, if ClockButtonInstance is NULL, then dereferencing it is not a valid operation and will yield to a crash. That can happen on a short window a time - once the window is created, there is an opportunity when window information at index 0 is not yet populated with the pointer to the instance of the class the window belongs to. A solution, in this case at least, is to check the value against NULL and not act on it if that's the case.

Speaking of this bug, the offending behavior has been corrected in the latest commits to ExplorerPatcher.

Using full dumps

The alternative to the above, which requires prior setup, is to investigate full crash dumps. The slight inconvenience with these is the fact that they have to be enabled on the system beforehand. You cannot obtain a memory dump for a crash that already happened which was not recorded.

To enable full crash dumps on your system, follow these instructions.

Afterwards, every time a program crashes, a dump will be saved in your folder of choice. You can open the dump file with any capable program. I personally use WinDbg Preview.

When you open the dump file, the program will show the exception details directly. Full crash dumps offer much more information than event logs; of particular interest is the stack trace, which shows which functions were being called when the crash happened.

Conclusion

I hope this brings a bit more light on the procedures to follow when trying to investigate problems with this application, or with any other application for that matter.

Thanks