Monday, March 23, 2015

Debugging BSOD when trying to delete malformed registry key on Windows 8.1

The last days my computer started to crash suddenly, with bug check 0x51 (REGISTRY_ERROR). It was totally random, so i decided to analyze the crash. I decided to write an article about this because, in spite of the fact that it doesn’t seem a security problem, i learnt some interesting things about windows registry.

Before reading this article, it’s good to have some knowledge about internals of windows registry. Here is a very interesting article about it.

The machine where the BSOD occured is x64 running windows 8.1. First, we activate full memory dumps and we wait for the crash.

When the crash occurs, we get the memory dump. We open it with windbg, load symbols for kernel and user mode, and we analyze -v.

3: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

REGISTRY_ERROR (51)
Something has gone badly wrong with the registry. If a kernel debugger
is available, get a stack trace. It can also indicate that the registry got
an I/O error while trying to read one of its files, so it can be caused by
hardware problems or filesystem corruption.
It may occur due to a failure in a refresh operation, which is used only
in by the security system, and then only when resource limits are encountered.
Arguments:
Arg1: 0000000000000001, (reserved)
Arg2: ffffc000a7277000, (reserved)
Arg3: 00000000ccbe055b, depends on where Windows bugchecked, may be pointer to hive
Arg4: 00000000000001ea, depends on where Windows bugchecked, may be return code of
 HvCheckHive if the hive is corrupt.

Debugging Details:
------------------


DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT

BUGCHECK_STR: 0x51

PROCESS_NAME: svchost.exe

CURRENT_IRQL: 0

ANALYSIS_VERSION: 6.3.9600.16384 (debuggers(dbg).130821-1623) amd64fre

LAST_CONTROL_TRANSFER: from fffff8011914ce46 to fffff80118d529a0

STACK_TEXT: 
ffffd000`8b3137b8 fffff801`1914ce46 : 00000000`00000051 00000000`00000001 ffffc000`a7277000 00000000`ccbe055b : nt!KeBugCheckEx
ffffd000`8b3137c0 fffff801`18fe3463 : ffffc000`b211da78 00000000`04223a78 00000000`00000000 ffffc000`a7277000 : nt! ?? ::NNGAKEGL::`string'+0x1db76
ffffd000`8b313820 fffff801`18fe434a : ffffc000`b2375e64 00000000`03d01a50 ffffc000`37643535 ffffc000`ccbe055b : nt!HvFreeCell+0xf7
ffffd000`8b3138a0 fffff801`18fe2836 : ffffc000`a7277000 ffffc000`af1574b4 ffffc000`a7277000 00000000`03d01a90 : nt!CmpFreeValueData+0x6a
ffffd000`8b3138d0 fffff801`18fe10e5 : 00000000`00000000 ffffe000`28d8b4c0 ffffc000`a7277000 00000000`00000000 : nt!CmpFreeValue+0x26
ffffd000`8b313900 fffff801`19034075 : 01d063e1`4dd2cd24 ffffd000`8b3139a9 ffffc000`af1574b4 ffffc000`a995b948 : nt!CmpFreeKeyByCell+0xf9
ffffd000`8b313940 fffff801`19033d26 : ffffc000`a8e89af0 ffffd000`8b313a01 00000000`00000101 00000000`00000000 : nt!CmDeleteKey+0x1fd
ffffd000`8b313a10 fffff801`18d5e1b3 : 00000000`00000008 ffffe000`28d8b080 00000000`000001b0 00000000`00000000 : nt!NtDeleteKey+0x25e
ffffd000`8b313b00 00007ff8`15ea19ba : 00007ff8`1308e7eb 00000096`9953ccb0 00000000`00000000 00000000`0000ff00 : nt!KiSystemServiceCopyEnd+0x13
00000096`9953cbd8 00007ff8`1308e7eb : 00000096`9953ccb0 00000000`00000000 00000000`0000ff00 00000000`00010000 : ntdll!NtDeleteKey+0xa
00000096`9953cbe0 00007ff8`1308e6c5 : 00000000`000001a8 00000000`000001b0 00000000`00000000 00000000`00000000 : KERNELBASE!LocalBaseRegDeleteKeyEx+0xf7
00000096`9953cc70 00007ff8`1305efa1 : 00000000`000001a8 00000000`00000000 00000000`00000000 00000000`00000000 : KERNELBASE!RegDeleteKeyExInternalW+0xa1
00000096`9953ccd0 00007fff`fc90195c : 00000000`000001a8 00000096`98b2de20 00007ff8`00000101 00000000`000001a4 : KERNELBASE!RegDeleteTreeW+0x14d
00000096`9953cd50 00007fff`fc8fabc5 : 00000000`00000001 00000000`00000004 00000000`00000004 00000000`00000000 : defragsvc!SxRegDeleteKey+0x60
00000096`9953cdc0 00007fff`fc8ee0ec : 00007fff`fc8eb860 00007fff`fc8eb860 00000096`991965b0 00000096`98b2d710 : defragsvc!CDefragAsyncWorker::_LoadStatisticsFromRegistry+0x6f5
00000096`9953d360 00007fff`fc8f2956 : 00000000`00000000 00000096`9953d9e0 00000096`98b2e060 00000096`98b2e060 : defragsvc!CDefragAsyncWorker::_Initialize+0x19c
00000096`9953d8d0 00007fff`fc8ef347 : 00000000`00000000 00000096`9953dbf0 00000000`00000000 00000000`00000000 : defragsvc!CDefragAsyncWorker::CreateInstance+0xe6
00000096`9953d990 00007fff`fc8edca8 : 00000000`00000000 00000000`00000000 00000000`00000000 00007ff8`00000001 : defragsvc!CDefragEngine::_CreateAsyncForVolume+0x1f5
00000096`9953dab0 00007fff`fc8f3428 : 00000000`00000000 00000000`00000198 00000000`00000198 00000096`99196440 : defragsvc!CDefragEngine::_RefreshVolumeList+0x51b
00000096`9953e060 00007fff`fc8f32a5 : 00000000`00000000 00000096`991964f0 00000096`99196530 00000000`00000158 : defragsvc!CDefragEngine::FinalConstruct+0x12c
00000096`9953e0f0 00007fff`fc8ea51b : 00000000`00000000 00000000`00000001 00000096`991962f0 00000096`991962a0 : defragsvc!CSxComObjectCached<CDefragEngine>::CreateInstance+0xa1
00000096`9953e120 00007ff8`143e7cb4 : 00000096`991962a0 00007fff`fc8ea4a0 00000000`00000000 00000000`00000000 : defragsvc!CSxComClassFactorySingleton<CDefragEngine>::CreateInstance+0x7b
00000096`9953e170 00007ff8`143e7ad0 : 00000000`00000000 00007ff8`143cd330 00000096`9953e240 00000000`00000000 : combase!GetInstanceHelperMulti+0x4c [d:\9147\com\combase\objact\objact.cxx @ 2821]
00000096`9953e1c0 00007ff8`14012053 : 00000000`00000000 00000096`9953e840 00007ff8`143e7550 00000096`98b1eb40 : combase!CObjServer::CreateInstance+0x57e [d:\9147\com\combase\objact\defcxact.cxx @ 662]
00000096`9953e3a0 00007ff8`1401c70d : 00000096`9953e820 00007ff8`1444a722 00000096`98b26418 00000000`00000000 : RPCRT4!Invoke+0x73
00000096`9953e400 00007ff8`14029cb3 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : RPCRT4!NdrStubCall2+0x35e
00000096`9953ea70 00007ff8`145287e3 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : RPCRT4!NdrStubCall3+0x156
00000096`9953ed90 00007ff8`145297bd : 00000000`00000001 00007ff8`14528770 00000000`00000000 00000000`00000000 : combase!CStdStubBuffer_Invoke+0x6b [d:\9147\com\combase\ndr\ndrole\stub.cxx @ 1586]
00000096`9953edd0 00007ff8`143b4f5a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : combase!SyncStubInvoke+0x21d [d:\9147\com\combase\dcomrem\channelb.cxx @ 1664]
00000096`9953ef70 00007ff8`1452951f : 00000000`00000000 00000096`98b11f70 00000096`9953f240 00000000`00000000 : combase!CCtxComChnl::ContextInvoke+0x27a [d:\9147\com\combase\dcomrem\ctxchnl.cxx @ 1377]
00000096`9953f180 00007ff8`14528fb0 : 00000096`98b25f30 00000096`98b25f30 00000096`98b12600 00007ff8`14528dc9 : combase!AppInvoke+0x1af [d:\9147\com\combase\dcomrem\channelb.cxx @ 1481]
00000096`9953f270 00007ff8`14529b35 : 00000000`00000000 00000000`00070005 00000096`98b157d0 00000000`00000000 : combase!ComInvokeWithLockAndIPID+0x676 [d:\9147\com\combase\dcomrem\channelb.cxx @ 2314]
00000096`9953f4b0 00007ff8`140123c7 : 00000000`00000158 00007ff8`14529cf0 00000000`00000000 00000096`98b23eb0 : combase!ThreadInvoke+0x48a [d:\9147\com\combase\dcomrem\channelb.cxx @ 5488]
00000096`9953f580 00007ff8`14012220 : 00000096`98b1f360 00000000`00000000 00000096`9953f740 00000000`000618c8 : RPCRT4!DispatchToStubInCNoAvrf+0x33
00000096`9953f5d0 00007ff8`14029a78 : 00000096`98b1f360 00000000`00000000 00000096`00000000 00000096`98b25c90 : RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0x190
00000096`9953f6d0 00007ff8`14012c86 : 00000000`000618c8 00000096`98b25c90 00000000`00000000 00000096`98b25c90 : RPCRT4!LRPC_SCALL::DispatchRequest+0x4c9
00000096`9953f7e0 00007ff8`14012ad8 : 00000096`00000000 00000096`98b189f0 00000096`98b25c90 00000096`99140000 : RPCRT4!LRPC_SCALL::HandleRequest+0x291
00000096`9953f890 00007ff8`140118bd : 00000096`98b13020 00000096`00000000 00007ff8`1406edbc 00000096`98b13128 : RPCRT4!LRPC_SASSOCIATION::HandleRequest+0x238
00000096`9953f920 00007ff8`140116be : 00000096`98b01088 00000096`98b13128 00000000`00000000 00007ff8`1406edbc : RPCRT4!LRPC_ADDRESS::ProcessIO+0x444
00000096`9953fa60 00007ff8`15e46b30 : 00000096`98af4010 00000096`98b01040 00007ff8`14011600 00000096`9953fb18 : RPCRT4!LrpcIoComplete+0x144
00000096`9953fb00 00007ff8`15e44e68 : 00000000`00000000 00007ff8`15e46900 00000000`00000000 00000000`00000000 : ntdll!TppAlpcpExecuteCallback+0x210
00000096`9953fb70 00007ff8`13c213d2 : 00000000`00000000 00007ff8`15e445e0 00000096`98af4010 00000000`00000000 : ntdll!TppWorkerThread+0x888
00000096`9953ff50 00007ff8`15e7e954 : 00007ff8`13c213b0 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0x22
00000096`9953ff80 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x34


STACK_COMMAND: kb

FOLLOWUP_IP: 
nt! ?? ::NNGAKEGL::`string'+1db76
fffff801`1914ce46 cc int 3

SYMBOL_STACK_INDEX: 1

SYMBOL_NAME: nt! ?? ::NNGAKEGL::`string'+1db76

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: nt

IMAGE_NAME: ntkrnlmp.exe

DEBUG_FLR_IMAGE_TIMESTAMP: 54c832b2

BUCKET_ID_FUNC_OFFSET: 1db76

FAILURE_BUCKET_ID: 0x51_nt!_??_::NNGAKEGL::_string_

BUCKET_ID: 0x51_nt!_??_::NNGAKEGL::_string_

ANALYSIS_SOURCE: KM

FAILURE_ID_HASH_STRING: km:0x51_nt!_??_::nngakegl::_string_

FAILURE_ID_HASH: {e5bde1d4-9584-25e7-f283-f0bca450261c}
We can see the stack:

nt!KeBugCheckEx 
nt! ?? ::NNGAKEGL::`string'+0x1db76 
nt!HvFreeCell+0xf7 
nt!CmpFreeValueData+0x6a 
nt!CmpFreeValue+0x26 
nt!CmpFreeKeyByCell+0xf9 
nt!CmDeleteKey+0x1fd 
nt!NtDeleteKey+0x25e 
nt!KiSystemServiceCopyEnd+0x13 
ntdll!NtDeleteKey+0xa 
KERNELBASE!LocalBaseRegDeleteKeyEx+0xf7 
KERNELBASE!RegDeleteKeyExInternalW+0xa1 
KERNELBASE!RegDeleteTreeW+0x14d 
defragsvc!SxRegDeleteKey+0x60 
defragsvc!CDefragAsyncWorker::_LoadStatisticsFromRegistry+0x6f5 
defragsvc!CDefragAsyncWorker::_Initialize+0x19c 
defragsvc!CDefragAsyncWorker::CreateInstance+0xe6 
defragsvc!CDefragEngine::_CreateAsyncForVolume+0x1f5 
defragsvc!CDefragEngine::_RefreshVolumeList+0x51b 
defragsvc!CDefragEngine::FinalConstruct+0x12c 
defragsvc!CSxComObjectCached<CDefragEngine>::CreateInstance+0xa1 
defragsvc!CSxComClassFactorySingleton<CDefragEngine>::CreateInstance+0x7b 
combase!GetInstanceHelperMulti+0x4c [d:\9147\com\combase\objact\objact.cxx @ 2821] 
combase!CObjServer::CreateInstance+0x57e [d:\9147\com\combase\objact\defcxact.cxx @ 662] 
RPCRT4!Invoke+0x73 
RPCRT4!NdrStubCall2+0x35e 
RPCRT4!NdrStubCall3+0x156 
combase!CStdStubBuffer_Invoke+0x6b [d:\9147\com\combase\ndr\ndrole\stub.cxx @ 1586] 
combase!SyncStubInvoke+0x21d [d:\9147\com\combase\dcomrem\channelb.cxx @ 1664] 
combase!CCtxComChnl::ContextInvoke+0x27a [d:\9147\com\combase\dcomrem\ctxchnl.cxx @ 1377] 
combase!AppInvoke+0x1af [d:\9147\com\combase\dcomrem\channelb.cxx @ 1481] 
combase!ComInvokeWithLockAndIPID+0x676 [d:\9147\com\combase\dcomrem\channelb.cxx @ 2314] 
combase!ThreadInvoke+0x48a [d:\9147\com\combase\dcomrem\channelb.cxx @ 5488] 
RPCRT4!DispatchToStubInCNoAvrf+0x33 
RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0x190 
RPCRT4!LRPC_SCALL::DispatchRequest+0x4c9 
RPCRT4!LRPC_SCALL::HandleRequest+0x291 
RPCRT4!LRPC_SASSOCIATION::HandleRequest+0x238 
RPCRT4!LRPC_ADDRESS::ProcessIO+0x444 
RPCRT4!LrpcIoComplete+0x144 
ntdll!TppAlpcpExecuteCallback+0x210 
ntdll!TppWorkerThread+0x888 
KERNEL32!BaseThreadInitThunk+0x22 
ntdll!RtlUserThreadStart+0x34
As we can see the problem occurs when the service defragsvc tries to delete a key. If we see the stack, at fffff801`18fe3463 is the ret addr of the call to the function that calls KeBugCheck, HvpRemoveFreeCellHint:

fffff800`dd7f9451 8b943498000000 mov edx,dword ptr [rsp+rsi+98h]
fffff800`dd7f9458 458bcc mov r9d,r12d
fffff800`dd7f945b 498bcf mov rcx,r15
fffff800`dd7f945e e895010000 call nt!HvpRemoveFreeCellHint (fffff800`dd7f95f8)
fffff800`dd7f9463 4883c604 add rsi,4

We disassemble the function with IDA and we see the exact point where KeBugCheck is called:



In the image we can see the exact point where KeBugCheckEx is called (from HvpRemoveFreeCellHint function).

HvpRemoveFreeCellHint has these parameters:

VOID
HvpRemoveFreeCellHint(
    PHHIVE          Hive,
    HCELL_INDEX     Cell,
    ULONG           Index,
    HSTORAGE_TYPE   Type
    )
In the stack we see the pointer to the HHIVE is ffffc000`a7277000, and we can explore it:
3: kd> dt _HHIVE ffffc000`a7277000
nt!_HHIVE
 +0x000 Signature : 0xbee0bee0
 +0x008 GetCellRoutine : 0xfffff801`18fb7490 _CELL_DATA* nt!HvpGetCellPaged+0
 +0x010 Allocate : 0xfffff801`18fe1d08 void* nt!CmpAllocate+0
 +0x018 Free : 0xfffff801`18fe2a44 void nt!CmpFree+0
 +0x020 FileWrite : 0xfffff801`190146a4 unsigned char nt!CmpFileWrite+0
 +0x028 FileRead : 0xfffff801`19083128 unsigned char nt!CmpFileRead+0
 +0x030 HiveLoadFailure : (null) 
 +0x038 BaseBlock : 0xffffc000`a7279000 _HBASE_BLOCK
 +0x040 DirtyVector : _RTL_BITMAP
 +0x050 DirtyCount : 0x520
 +0x054 DirtyAlloc : 0x52d8
 +0x058 UnreconciledVector : _RTL_BITMAP
 +0x068 UnreconciledCount : 0
 +0x06c BaseBlockAlloc : 0x1000
 +0x070 Cluster : 1
 +0x074 Flat : 0y0
 +0x074 ReadOnly : 0y0
 +0x074 Reserved : 0y000000 (0)
 +0x075 DirtyFlag : 0x1 ''
 +0x078 HvBinHeadersUse : 0x97000
 +0x07c HvFreeCellsUse : 0x542620
 +0x080 HvUsedCellsUse : 0x4cfe9e0
 +0x084 CmUsedCellsUse : 0
 +0x088 HiveFlags : 0x10200
 +0x08c CurrentLog : 5
 +0x090 CurrentLogSequence : 0xb53
 +0x094 CurrentLogOffset : 0
 +0x098 MinimumLogSequence : 0xb53
 +0x09c LogFileSizeCap : 0x2000000
 +0x0a0 LogDataPresent : [2] ""
 +0x0a2 PrimaryFileValid : 0x1 ''
 +0x0a3 BaseBlockDirty : 0 ''
 +0x0a4 FirstLogFile : 0y101
 +0x0a4 SecondLogFile : 0y000
 +0x0a4 HeaderRecovered : 0y0
 +0x0a4 LegacyRecoveryIndicated : 0y0
 +0x0a4 RecoveryInformationReserved : 0y00000000 (0)
 +0x0a4 RecoveryInformation : 5
 +0x0a6 LogEntriesRecovered : [2] "."
 +0x0a8 RefreshCount : 0
 +0x0ac StorageTypeCount : 2
 +0x0b0 Version : 5
 +0x0b8 Storage : [2] _DUAL
BaseBlock field of the HHIVE points to the regf (the start of the hive). FirstLogFile field is the page of the first hbin block. So BaseBlock + FirstLogFile*0x1000 points to the first hbin block (read the article about registry internals to know about regf and hbin blocks).

We can see in the call to RegDeleteTreeW a pointer as parameter, 00000096`98b2de20, pointing to:

V.o.l.u.m.e.{.9.c.2.f.1.1.c.0.-.1.3.f.6.-.4.2.4.b.-.8.a.1.2.-.8.d.7.0.1.1.e.a.7.6.1.5.}

Now we will search this string from the start of first hbin, and we find it at:

3: kd> s -a ffffc000`a737a000 L10000000 "Volume{9c2f11c0-13f6-424b-8a12-8d7011ea7615}"

ffffc000`a9164590 56 6f 6c 75 6d 65 7b 39-63 32 66 31 31 63 30 2d Volume{9c2f11c0-
ffffc000`af157500 56 6f 6c 75 6d 65 7b 39-63 32 66 31 31 63 30 2d Volume{9c2f11c0-
 
Some bytes before of the second one string found, we can see the nk cell, exactly at ffffc000`af1574b4 (we see this pointer in the stack near the call to CmpFreeKeyByCell too):

ffffc000`af1574b4 nk .l.onXa.......8".....................`. 
ffffc000`af1574de G.(...............8...........,...Volume{9 
ffffc000`af157508 c2f11c0-13f6-424b-8a12-8d7011ea7615}...0..

It seems when this cell is going to be deleted, BSOD occurs.
We can see this key with regedit:


This is the key that causes the BSOD when we try to delete it. If we try to delete it with regedit.exe, i get the same KeBugCheck. Here is the stack:

nt!KeBugCheckEx 
nt! ?? ::NNGAKEGL::`string'+0x1db76 
nt!HvFreeCell+0xf7 
nt!CmpFreeValueData+0x6a 
nt!CmpFreeValue+0x26 
nt!CmpFreeKeyByCell+0xf9 
nt!CmDeleteKey+0x1fd 
nt!NtDeleteKey+0x25e 
nt!KiSystemServiceCopyEnd+0x13 
ntdll!NtDeleteKey+0xa 
ADVAPI32!LocalBaseRegDeleteKeyEx+0xf2 
ADVAPI32!RegDeleteKeyW+0x7f 
regedit!RegDeleteKeyRecursive+0x13a 
regedit!RegDeleteKeyRecursive+0x108 
regedit!RegDeleteKeyRecursive+0x108 
regedit!RegEdit_OnKeyTreeDelete+0xcb 
regedit!RegEdit_OnKeyTreeCommand+0x115 
regedit!RegEdit_OnCommand+0x47 
regedit!RegEditWndProc+0x298 
USER32!UserCallWinProcCheckWow+0x149 
USER32!DispatchClientMessage+0xa2 
USER32!_fnDWORD+0x3e 
ntdll!KiUserCallbackDispatcherContinue 
USER32!NtUserTranslateAccelerator+0xa 
regedit!WinMain+0x265 
regedit!__mainCRTStartup+0x1ab 
KERNEL32!BaseThreadInitThunk+0x22 
ntdll!RtlUserThreadStart+0x34
The registry key seems to be malformed. You can see in the capture some duplicated values: BytesPerCluster, LastRunFullDefragTime, … I guess the registry key became malformed with some sudden reboot (due to power failure). Since the registry became malformed, i had been getting BSOD randomly (between 10 to 30 minutes, each time that defragsvc started to defrag).

We can get the software hive in disk: %systemroot%\system32\config\SOFTWARE (the file is locked but you can copy it with fget.exe tool, of Hbgary) and see the malformed entry there:

I
I used the SOFTWARE file that we extracted, and i tried to load this hive (with regedit you can do it) under HKEY_LOCAL_MACHINE or HKEY_USERS, on XP, Win7 and Win8 operating systems. But, when you try to delete the malformed key from the new loaded hive, the system doesn’t crash.
I decided to publish the analysis because i found some interesting points, and, maybe, this problem should be fixed because, in spite of the fact this is not a security problem, a system could be totally disabled because of a registry malformation like this or similar.

No comments:

Post a Comment