Non-hidden directory doesn't show in Explorer, but the program that made it can access it

26

9

I'm using IMAPSize to backup my mailboxes. The process just dumps the contents of your mailbox into .eml files on your disk. Anyway, the first mailbox I backed up shows up as I expect. However, the next one doesn't show in Explorer and, if I enter it in the address bar, it tells me that it doesn't exist.

However, IMAPSize is insisting that the files are there. (A lot of disk writing was being performed during the operation.) Moreover, the directory does appear in the Windows directory dialog and - as I have GNU tools installed - it shows in ls (but not in the native dir). I can't cd into the directory, but ls does seem to be able to access it (and, indeed, the directories and files I expect appear to be there).

command prompt window showing the listings

Also note that the timestamp and filesize of backup.db are different in the dir and ls -l output.

In short: What is going on here and how do I fix it!?

Xophmeister

Posted 2016-04-01T23:56:31.390

Reputation: 433

1The only conspicuous difference I see is that ls -l is showing a peculiar number of hardlinks to the "hidden" directories and files... I don't know why that is, nor how a hardlink is even interpreted in NTFS-land. – Xophmeister – 2016-04-01T23:59:37.140

...Just to confirm: I can access the data in the "hidden" files, from the command line, using cat (another GNU coreutil). Again, however, native Windows tools (e.g., type) can't even find the file. – Xophmeister – 2016-04-02T00:08:36.507

Answers

63

Let me guess: the program that created the file, and also the GNU utilities, are not running as administrator.

First, some history. In the days of Windows XP, lots of programs assumed they would always be run as admin, and would write to places like C:\Windows and C:\Program Files (x86) with wild abandon. With Vista, Microsoft tried to make fewer people admins, but standard users can't write to those places. They needed those dubious programs to keep working (or else people wouldn't upgrade). So, they introduced a magical feature called UAC virtualization.

Programs running as standard users might think their writes to important locations succeeded, but in reality, Windows squirreled the data away in a per-user location. When those programs look for files in a directory, Windows checks to see whether there are any files in that place's virtual store, and if so, it adds them to the directory listing. (There is equivalent functionality for the Registry.)

It looks like your mail program tried to write to a place under Program Files (x86) while running as a normal user. The write was redirected, so it didn't actually go to that place. The program can still see it, because Windows is keeping up the illusion for it. Explorer doesn't see it because it announces to the operating system that it's well-behaved and so doesn't need redirection. The command prompt's dir command isn't a program (it's just a feature of cmd.exe), so it is also considered "in the know" and so is not shown the compatibility files. ls is a program that is, evidently, not in the know, so it gets to see the compatibility files.

You'll find your file here:

%LOCALAPPDATA%\VirtualStore\Program Files (x86)\IMAPSize\backup

While poking around in VirtualStore, you might be surprised at what programs are not well-behaved and need the virtualization safety net.

If you want to stop the redirection, run the program as administrator, or save your backups in a location that you can actually write to without admin privileges.

Ben N

Posted 2016-04-01T23:56:31.390

Reputation: 32 973

16And here I thought I knew Windows. Either this is a well-done april's-fool-joke, or I learned somethign significant today. thanks, @Ben N! – Aganju – 2016-04-02T01:34:08.963

3@Aganju A few months ago I had to help a Windows user create a keypair for WinSCP. And those became invisible in pretty much the same way as described in the question. At the time I hypothesized an explanation pretty much the same as given in this answer. I didn't bother to verify my hypothesis, because the problem was solved by putting the file under the users home directory instead. But at least I can confirm that this answer does not look at all like a joke to me. – kasperd – 2016-04-02T07:02:22.147

3Well I'll be damned! There they all are. That is weird behaviour, even if it is intended. I thought I had a filesystem error!... This would also explain why my first mailbox backup does show up: The program was first run from the installer ("Do you want to run such-and-such now?"), which would have elevated it to having Administrator privileges; whereas the "hidden" run was as the local user. – Xophmeister – 2016-04-02T10:31:37.527

1@Aganju I'm guessing you never used Vista much. It had a feature to show you virtualized files in Explorer; because the issue was sufficiently common for a few years. By the time Win7 came around, most widely used software stopped assuming it could write to subfolders of its install directory; and the feature's been made significantly less visible or possibly outright removed in the most recent versions of Windows. – Dan is Fiddling by Firelight – 2016-04-02T17:21:20.327

4@Xophmeister It's a trade-off. As noted in the answer, software writing to its own application directory was scarily common before Vista. This is a massive security hole, so it had to be solved. The alternatives were 1) prevent the write and break all the applications, 2) use virtualization to save the files in a safe location. 1) is obviously untenable and very much against Microsoft's SOP, and in the end it would only lead to everyone running everything under administrator as before again. It can be confusing, but it's still probably the best option. Don't write to dangerous directories. – Luaan – 2016-04-02T18:07:05.290

1Similar redirection to offline files, dir /s is your friend. – mckenzm – 2016-04-02T21:33:26.173

There should actually be a Compatibility Files button in the toolbar. See also: http://superuser.com/a/401979/117590 (screenshot)

– Bob – 2016-04-03T10:50:50.937

@Bob Yes, in Vista there is, but it seems to have vanished later. (It never appears for me in Windows 8.) – Ben N – 2016-04-03T15:36:49.650

@BenN I see it in Win7, though perhaps only in those folders that are actually redirected. IIRC it was removed in Win8 (not too sure though). In any case, the OP here is clearly using Win7 so that's at least useful info here. – Bob – 2016-04-03T16:09:22.417

Asked: 2016-04-01T23:56:31.390

Viewed: 2 270 times

Active: 2016-04-02T04:24:22.440