5

A team member informed me that one of our Windows Server 2008 (not 2008 R2) based MSSQL Servers had begun to generate CAPI2 Event ID 513 Errors in the Application Event Log:

Application\CAPI2

Cryptographic Services failed while processing the OnIdentity() call in the System Writer Object.

Details:
AddCoreCsiFiles: BeginFileEnumeration() failed.

System Error: Access is denied.



A little PowerShell reveals that the issue started back on 08/06/14 and primarily seems to occur after 22:00 on a daily basis:

PS C:\Users\Administrator> Get-EventLog -LogName Application | ? { $_.EventID -like "513" -and $_.Source -like "Microsoft-Windows-CAPI2" } | Select -Property TimeGenerated

TimeGenerated
-------------
8/18/2014 10:41:32 AM
8/18/2014 10:25:17 AM
8/18/2014 10:15:20 AM
8/17/2014 10:55:41 PM
8/17/2014 10:55:27 PM
8/17/2014 10:55:26 PM
8/16/2014 10:49:44 PM
8/16/2014 10:49:28 PM
8/16/2014 10:49:28 PM
8/15/2014 10:52:11 PM
8/15/2014 10:51:58 PM
8/15/2014 10:51:57 PM
8/15/2014 1:03:06 AM
8/15/2014 1:02:45 AM
8/15/2014 1:02:45 AM
8/13/2014 10:58:49 PM
8/13/2014 10:58:32 PM
8/13/2014 10:58:31 PM
8/12/2014 10:57:09 PM
8/12/2014 10:56:56 PM
8/12/2014 10:56:56 PM
8/11/2014 10:56:13 PM
8/11/2014 10:55:56 PM
8/11/2014 10:55:55 PM
8/10/2014 10:50:15 PM
8/10/2014 10:50:04 PM
8/10/2014 10:50:03 PM
8/10/2014 7:12:09 AM
8/10/2014 7:11:52 AM
8/10/2014 7:11:51 AM
8/8/2014 10:57:00 PM
8/8/2014 10:56:44 PM
8/8/2014 10:56:43 PM
8/6/2014 9:47:26 PM
8/6/2014 9:47:03 PM
8/6/2014 9:47:02 PM
8/6/2014 10:48:33 AM



Curious no? I wonder what the System Writer Object is used for? Shadow Copy! Oh duh! I started doing VSS-based Application Aware backups of this virtual machine using Veeam this month. Naturally the backup process starts at 22:00 which explains the repeated frequency instead of the the error just occurring at "random" times.

Interestingly enough Veeam did not register this as a failed backup attempt which makes me wonder if the Restore Points are actually transaction consistent. Regardless, I did a quick search through the Veeam Backup Logs and did not find anything obviously wrong but it is probably worth reviewing them closer and confirming that a recovery from those Restore Points is transaction consistent.

The Event ID 513 TechNet reference recommended Resolution indicates a NTFS permission issue could be at fault, however the C:\Windows\Registration COM+ registration folder has the appropriate permissions.

Ideas?

Mathias R. Jessen
  • 24,907
  • 4
  • 62
  • 95

1 Answers1

6

Mathias R. Jessen got me pointed in the right direction, the ever famous WinSxS folder. However I did not see any of the VSS errors in the Event Log which made me a little hesitant to just nuke all the NTFS permissions lest I break something else.

Lesson 1: Read

I went back and read Event ID 513 TechNet reference again and noted that under the Verify section it was recommended that I check to see the System Writer was available as VSS writer using vssadmin list writers and sure enough it was NOT. Lesson Learned #1: Read the whole KB/TechNet/Blog

Lesson 2: Reproduce

Doing a bit more researching I came across Missing System Writer Case Explained which seemed to indicate the issue was originating with Cryptographic Services. I found that I could reproduce by CAPI2 error at will by stopping and starting the CryptSVC service. Lesson Learned #2: Try to figure out a way to reproduce your error at will.

Using ProcMon

At this point, I pretty much followed the post's instructions. I located the PID of which instance of svchost was wrapping CryptSVC using Task Manager. You could alternatively force CryptSVC to run as its own process using sc config if you can reboot the server in question. Depending on how deep you get into ProcMon it is worth isolating services under a single PID just to cut down the amount of events you have to sort through.

From here it is back to good old ProcMon. Setup a filter to exclude all PIDs that are not the one used by the svchost process that is wrapping CryptSVC:

ProcMon Exclude Filter

Lesson 3: Love ProcMon

I applied my trusty first pass filter which is to exclude all events that have the results of SUCCESS. This reduced the events from 31,118 to a much more manageable 139 and at the bottom I found the ACCESS DENIED event I was looking for, not surprisingly in the WinSxS folder (C:\Windows\winsxs\FileMaps\$$.cdf-ms). Lesson Learned #3: Learn to use and love ProcMon

ProcMon ACCESS DENIED on C:\Windows\winsxs\FileMaps\$$.cdf-ms

Lesson 4: Verify

Now what? KB2009272 that Mathias linked has the solution but now I know why. Lesson Learned #4: Don't guess, know!

Lesson 5: Start small

The resolution is exactly as explained in KB2009272. Take ownership and reset the permissions of the FileMaps folder and then restart CryptSVC:

Takeown /f %windir%\winsxs\filemaps\* /a
icacls %windir%\winsxs\filemaps\*.* /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\filemaps\*.* /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\filemaps\*.* /grant BUILTIN\Users:(RX)

net stop cryptsvc
net start cryptsvc



and... we have lift off! The System Writer is now available as a VSS writer. No need to change the permissions for the PendingRename folder. Lesson Learned #5: Start with the smallest changes and work your way out to changes that effect more things.

C:\Users\administrator>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {98c52075-429a-4487-8b77-e42b18767458}
   State: [1] Stable
   Last error: No error


Restarting CryptSVC at will no longer produced the CAPI2 error and after a day or two of monitoring it looks resolved.

Community
  • 1