When does Windows write registry changes to disk?

TL;DR:

I've noticed that if I make a registry change and then hard-shutdown my Windows 10 system, the registry change does not appear after reboot.

I've also noticed that the deletion of a hibernation file can impact the ability for a Linux recovery tool to make changes to the Windows registry in an offline state. The tool seems unable to make persistent changes after the deletion of a hibernation file. I will list specific examples below.

Example 1:

I add a key called "1111" into "HKLM\SOFTWARE". I then hard power down my box by holding the power button for 5 seconds.
When I pull the registry back up, that key and its values are gone:
(These images were edited for formatting purposes)

Example 2:

Make a change in the registry (using regedit)
Hibernate the system
Boot into a Linux recovery tool
Delete the hibernation file in order to mount the disk.
Read the windows registry (from the recovery tool)
The registry change is gone.

This seems equivalent to a hard shutdown on the box.

Example 3:

I see stranger behavior when I:

Hibernate the box
Boot to a Linux recovery tool and delete the hibernation file
Make changes to the registry (from the recovery tool)
Reboot the box

Those changes are not reflected in the registry either.

So what's going on here?

When does Windows 10 write the registry changes to the disk?
Why does the deletion of a hibernation file (in Example 3) prevent registry changes made from the recovery tool from being reflected on the next boot?

Hoping to get some clarification!

Shrout1

Posted 2018-06-14T17:58:31.327

Reputation: 929

2"he registry change does not appear after reboot." - The registry change does not go into effect until the reboot, and typically, only File Explorer has to be restarted. It all depends on what behavior the registry change is intended to do if a reboot is actually required. The actual key is modified, when it's modified, which answers your question. – Ramhound – 2018-06-14T18:09:46.093

8What do you mean by a hard shutdown? Holding the power button until it shuts down? That process bypasses writing out any pending disk cache writes. – DavidPostill – 2018-06-14T18:10:49.877

2@Ramhound I get that it doesn't go into effect, but why hasn't it been written to the disk? I make the edit, hard power down and then change isn't there any more. Whether or not it went into effect in memory doesn't matter at that point – Shrout1 – 2018-06-14T18:10:58.043

1Does your Linux registry tool support reading from the transaction log? – user1686 – 2018-06-14T18:11:15.013

@DavidPostill So the registry edit is a "pending change"? – Shrout1 – 2018-06-14T18:11:20.197

@grawity I'm using chntpw but I'm not intimately familiar with it. – Shrout1 – 2018-06-14T18:11:39.910

@Shrout1 Anything that is written to disk is a pending change until it is actually written. – DavidPostill – 2018-06-14T18:12:52.007

@DavidPostill Ok - so when does it commit the change to disk? At logoff? Shutdown? Seems like I can circumvent the change by interrupting the normal reboot cycle. – Shrout1 – 2018-06-14T18:13:55.810

@Shrout1 Read https://www.google.com/search?q=when+does+windows+flush+disk+caches&ie=utf-8&oe=utf-8&client=firefox-b-ab

– DavidPostill – 2018-06-14T18:14:18.970

2@Shrout1 - Why are you forcing the machine to turn off instead of safely shutting the machine done? – Ramhound – 2018-06-14T18:14:59.290

6@Ramhound I'm running tests to see what happens if the system doesn't shut down gracefully. Random, I know, but I need to know exactly how this is committed to memory so that we don't lose anything on our systems if it isn't handled correctly. – Shrout1 – 2018-06-14T18:15:47.850

@DavidPostill Hey I disabled Disk Write Caching per this tenforums post but it didn't seem to make a difference. Let me know if I'm missing something else.

– Shrout1 – 2018-06-14T18:43:01.770

@Shrout1: Do we know that changing something in regedit.exe actually changes the registry immediately? My impression was that regedit.exe changes only got written to registry when regedit.exe was closed. An easy way to check would be to open a second copy of regedit.exe after adding 1111 but before closing the original window and see if the key appeared in the new copy too. – Jacob Manaker – 2018-06-17T04:50:56.227

@Jacob Manaker I closed regedit.exe after adding the key, before the hard power down. Regedit adds to a cache that Windows flushes to the disk at some unknown interval. Google regflushkey and lazy flush for more info :) – Shrout1 – 2018-06-17T05:11:36.517

@Shrout1: Ah, OK; great. The question was ambiguous enough that I got worried you weren't testing what you thought you were. – Jacob Manaker – 2018-06-18T14:15:59.213

Answers

TL;DR: Shut your system down properly.

Hibernation is nothing to do with shutting down, it is closely related to suspend-to-RAM (sleep) except with the contents of RAM pushed to disk in order for it to be read back in and operation resumed from exactly where the system left off.

If you want the changes to persist then you need to disable hibernation and Windows Fastboot (which is a subset of hibernation). Or you can actually reboot rather than hibernate and restart.

The reason changes are not persisted is because they are not written to disk yet except in the hibernation file. Which you are deleting, meaning that the filesystem may well have to repair itself and go back to a "last known good" state.

While the system is hibernated there are going to be several key filesystem structures that may not have been written out to disk and are instead in RAM. The system will, upon resuming from hibernation, expect the disk to be in a very particular state and it is possible that disk caches and important system files get saved to the hibernation file rather than to the actual disk.

If you do a proper shutdown then Windows will properly flush working memory to disk, and then unmount the disk cleanly before powering down.

To force a proper shutdown open a command prompt and type

shutdown /s /f /t 0

/s is "shutdown", /f to force, and /t 0 to mean "now" (time = 0 seconds)

Or you can just disable fastboot and hibernation.

Related to you doing a hard shutdown the problem there is that Windows is not guaranteed to write any changes out to disk the same millisecond (or even minute) that you make the change. It will almost certainly be written within a few minutes but the probablility of it having actually been written will increase over time. It's unlikely to be written immediately, then near to the time you make the change the probability increases sharply, and will almost certainly have been written within an hour.

The thing is though, that by forcing a hard shutdown you are not giving the system a chance to safely write changes to disk.

Most modern filesystems are written to make changes in the safest way possible. In the past they have been referred to as "atomic", as in the change has either happened or it has not.

Today we know them as Journalled file systems because they keep a log of operations that will happen that can be either reverted or rolled forwards in the event of system failure and reboot. Upon startup from a power failure the system checks the journal and for each transaction it checks if the actual file data is written to disk and is "good". If it is then the transation rolls forwards and is completed, if not then it rolls back to the old data.

By using this order the disk is almost always in an easy to repair state.

But by forcing your system to power down unexpectedly you cannot guarantee whether the transaction has progressed far enough to roll forwards upon repair, and chances are that an operating system such as Linux will not care as much as Windows about the transaction history and is more likely than not to just make changes that roll everything backwards rather than forwards.

If you rebooted into Windows it might try or be able to repair the disk properly as it has a more intimate knowledge of the filesystem.

Mokubai

Posted 2018-06-14T17:58:31.327

Reputation: 64 434

Thank you! I appreciate your response :) I turned off hibernation, rebooted and re-ran the same test. Example 1 has the same result, no hibernation file involved. It appears that registry changes are only written at some point during logoff / shutdown... I also agree that the system might be looking for a "last good" state when waking from a broken hibernation - it tends to run integrity checks. – Shrout1 – 2018-06-14T18:29:02.213

@Shrout1 added more information, particularly about the fact that the filesystem will also be doing some self-healing and "last known good" book keeping that Linux might not be handling. – Mokubai – 2018-06-14T18:43:29.110

Thanks for elaborating! I guess I'm just surprised by the fact that the registry doesn't actually write the change to disk when it's made, but instead waits until a specific trigger (logoff or reboot etc.) I believe that it's done that way, it's just news to me! – Shrout1 – 2018-06-14T18:45:14.677

@Shrout1 It's not even necessarily a specific trigger. There is going to be a window of time when caches are likely to have been flushed and changes actually appear on the physical disk in a consistent fashion, but you cannot guarantee exactly when that time is. It could be a minute, an hour, or even a day. It's unlikely to be that long, but there is an outside chance that maybe it will. Shutdown forces it to clear caches and write everything to disk. Hibernation, sleep, and other not-quite shutdown type things won't have that guarantee. – Mokubai – 2018-06-14T18:49:48.027

1Hmmm... So I downloaded sysinternals "sync" which forces the write cache to disk and that didn't do anything (still lost the keys / values on hard shut off). I also ran process explorer and looked at the Disk I/O in the properties for Regedit. It had read I/O but no write I/O. This makes me think that it might be waiting for shutdown... Do you know of any exceedingly dry M$ documentation that might lay this out plainly? Thanks again for all your help!! – Shrout1 – 2018-06-14T19:24:17.993

The NTFS journal makes no guarantees about file data. It's merely to keep the NTFS metadata consistent, so you don't end up with a broken filesystem. Individual files can still have partial changes recorded to them, breaking the file. There is Transactional NTFS but that's not recommended and very rarely used. This is also why database engines keep their own journal for data consistency. See also https://blogs.msdn.microsoft.com/oldnewthing/20130101-00/?p=5673

– Bob – 2018-06-15T03:40:45.203

@Bob Wouldn’t running the systernals “sync” utility flush the write cache to the disk? Even after doing this my changes weren’t reflected. – Shrout1 – 2018-06-15T03:54:32.187

@Shrout1 That's assuming the registry changes are pending in the write-back cache. It's entirely possible that they are instead managed separately and that this has nothing to do with the filesystem, caching or otherwise. For example, Last Known Good Configuration is only updated on successful startup. (I don't know enough about registry internals to be certain when changes are saved. And then TxR might be involved.)

– Bob – 2018-06-15T03:58:19.373

Unlikely to be the case, but the hard drive usually has its own cache which might not be flushed on improper-shutdown. I think that cache is usually flushed more frequently than the Windows cache, but it probably varies – reffu – 2018-06-15T12:41:00.047

1Is there a reason you're forcing shutdown? AFAIK that only forces running applications to close, which doesn't do much other than kill applications that you can't otherwise close and makes it more likely that you'd lose some unsaved changes somewhere if you don't know what you're doing or that you'd put some application in an unrecoverable state - I wouldn't really call that a "proper" shutdown. – NotThatGuy – 2018-06-16T17:15:58.460

As documented on the MSDN page for RegFlushKey:

Calling RegFlushKey is an expensive operation that significantly affects system-wide performance as it consumes disk bandwidth and blocks modifications to all keys by all processes in the registry hive that is being flushed until the flush operation completes. RegFlushKey should only be called explicitly when an application must guarantee that registry changes are persisted to disk immediately after modification. All modifications made to keys are visible to other processes without the need to flush them to disk.

Alternatively, the registry has a 'lazy flush' mechanism that flushes registry modifications to disk at regular intervals of time. In addition to this regular flush operation, registry changes are also flushed to disk at system shutdown. Allowing the 'lazy flush' to flush registry changes is the most efficient way to manage registry writes to the registry store on disk.

This suggests that apart from flushing a specific key to disk immediately (which locks everyone else out of the registry until the flush is complete), the registry is automatically flushed periodically: a time is not given, but presumably it is at least more than the time you waited between writing the key and hard shutdown. In addition, it is flushed at shutdown, as you had already figured out.

You can use the RegFlushKey function in the software that manipulates said key, or create an additional tool with it to force writing a registry key to disk immediately, if this is crucial to your usage case.

user914877

Posted 2018-06-14T17:58:31.327

Reputation: 381

Hey! I'll give you this one if you throw a link in to the following MSDN article: Saving application registry changes on Windows 8 or Windows Server 2012 and add a brief blockquote. Your answer led me down the rabbit hole to that article and it basically says the same thing you did. Thanks a whole lot!

– Shrout1 – 2018-06-15T14:16:54.440

But does anybody know how a user can call RegFlushKey? – Scott – 2018-06-15T18:19:50.160

@Scott It appears that it has to be done in code... The MSDN article listed in the post has a C++ example and I've seen it implemented in C# elsewhere. Not sure it can be done from the command line. – Shrout1 – 2018-06-15T19:53:15.440

I’m really disappointed that we (as a community) haven’t been able to answer your question any better than we have so far. Mokubai says “It will almost certainly be written within a few minutes … and will almost certainly have been written within an hour.”, but nobody has identified a way to determine that it has happened, or to ensure that it happens (other than shutting down). I’m shocked that sync doesn’t do it. Then again, we’re accustomed to this sort of unclarity and non-functionality when it comes to Microsoft Windows. – Scott – 2018-06-15T20:06:31.020

According to Windows Internals Sixth Edition by Mark Russinovich, the lazy writer starts five seconds after the Configuration Manager makes a sync request. There is also a logging mechanism to ensure the Registry is left in a consistent state after a system crash. This applies to Windows 7. Windows 10 is covered in the seventh edition but the volume which includes the Registry has yet to be published. – David Marshall – 2018-06-15T21:45:44.470

3@Scott: I would be shocked if sync did flush the registry to the disk. It just wouldn't make any sense. Why should flushing the file system cache (which is used by the configuration manager, not the other way around) suddenly flush the configuration manager's own cache? It doesn't even know what the structure of the higher-level cache is, if one even exists at all. – user541686 – 2018-06-16T04:21:39.793

@Scott You're assuming there's a concrete answer. Most software nowadays will use a heuristic. How much is the registry being updated? How full is the cache? Is the disk under load? If I were implementing this, I'd want to take all those factors - and many more - into account, and time my writes to minimise system disruption. – Basic – 2018-06-16T18:32:45.167

@Basic: Yes, it’s foolish of me to assume that Microsoft Windows would provide or even tolerate something so user-friendly. Of course I understand the use of a write cache for normal operation, but there should be a way for the user to tell the OS to flush the cache now. Unix has had the sync command for over 40 years; this post talks about why this is useful. Even Windows provides the “Safely Remove Hardware” functionality — … (Cont’d)

– Scott – 2018-06-16T20:15:18.160

(Cont’d) … is it so unreasonable that it would provide a similar function for the system drive and other non-removable media? Based on Mokubai’s answer, changes that the user thinks he has made might not really have been committed to secondary storage (e.g., disk) an hour later, and there’s no way for the user to expedite the process or even know when it has happened on its own. As I said in the first comment, we’re accustomed to this sort of behavior from Windows. But it feels too much like the user saying “Do this” and the computer saying “I don’t care what you want me to do”; that’s bad. – Scott – 2018-06-16T20:15:20.607

1@Scott There is a way. The API has already been linked. There are tools out there that give you a GUI to call arbitrary Win32 functions. Thing is... It's an implementation detail. If you expose it, people rely on it and then you can't change it. Also worth noting that the system disk is a totally different beast to removable media. The system disk is used for many things that require a handle to be always open (eg page file). Windows doesn't support unmounting the system partition, so has no need for this "feature". Plus... try to keep the comments neutral. You hate Windows, we get it. – Basic – 2018-06-17T00:57:16.760

@Scott: FYI, in my experience, the "might not really have been committed an hour later" thing is incorrect for Windows. In fact it's the kind of thing I would say is more accurate for Linux. In my experience, the only way something can remain unflushed for a long time is if the disk simply hasn't had sufficient time/bandwidth to flush the previously buffered data yet. Otherwise, Windows really does flush the file system cache periodically. In particular, on my current system I measure a pretty darn consistent ~16-second flush interval. – user541686 – 2018-06-18T09:40:48.040

TL;DR: It is very careful to do this at the right moment.

The documentation on FSCTL_MARK_AS_SYSTEM_HIVE has this to say:

The FSCTL_MARK_AS_SYSTEM_HIVE control code informs the file system that the specified file contains the registry's system hive. The file system must flush system hive data to disk at just the right moment to avoid deadlocks and to ensure data integrity.

I don't believe there is any more detail available publicly than this.

Bear in mind that flushing the file system does not imply flushing the registry, because the registry can perform caching on top of the file system. To flush the registry first, you need to somehow cause NtFlushKey or ZwFlushKey to be called on your key of interest.

user541686

Posted 2018-06-14T17:58:31.327

Reputation: 21 330

Oh good catch! That's my new favorite MSDN-ism: just the right moment. My previous favorite was "Use caution when specifying the TOP clause in a query that contains a UNION, UNION ALL, EXCEPT, or INTERSECT operator. It is possible to write a query that returns unexpected results" which was an euphemism for "this feature is badly broken and doesn't do what any reasonable person would expect and instead of fixing it we're going to document it".

– davidbak – 2018-06-17T02:13:13.073

@davidbak: lol, well in MSDN's defense, I don't really see what more detail they could provide that a user should be relying on. – user541686 – 2018-06-17T02:17:58.477

Oh you're right; clearly this was just something documented as a fallout of the long-ago EU settlement where Microsoft was required to document all APIs - no matter how internal they are. This page you linked even says "Only kernel-level components can use this filesystem control code" which is a strong hint to keep your hands off. Still, "just the right moment" - one day I'll document some API of mine with "call this method only at just the right moment" and see what happens.... – davidbak – 2018-06-17T02:20:58.470

@davidbak: I think you're a little... excited. :-P My guess is that this was documented in order to let file system drivers know what file contains the SYSTEM registry hive, which may be important in that they wouldn't attempt to write anything to the registry at the same time that that file is being flushed to disk. I have no proof of this though; it's just one reason I find conceivable. One doubt I have about it, though, is why a disk driver wouldn't need to know this too. – user541686 – 2018-06-17T02:25:26.490