Event ID: 36870, Schannel error

This was a very nasty error that I found in the System Event logs of my Windows 2000 webserver while upgrading a Digital ID for Secure Email certificate.

Event Type:
ErrorEvent Source: Schannel
Event Category: None
Event ID: 36870
Date: 7/11/2007
Time: 1:50:10 PM
User: N/A
Computer:
Description: A fatal error occurred when attempting to access the SSL client credential private key. The error code returned from the cryptographic module is 0xffffffff.

Strange thing was that it happened only on a few of the Windows 2000 servers in our web farm.

To explain, we use a browser certificate to encrypt a small subset of transactions on our website. Verisign calls this a "Digital ID for Secure Email." During our yearly update of the certificate, we encountered the Schannel error shown above. Customers on our website would then a failure when they hit a webserver showing evidence of the problem. Again, not all webservers showed the problem, only a subset.

After four hours of troubleshooting and googling, I stumbled upon a post that suggested to look at the permissions on the following directory:
C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys

Somehow, when the certificate got updated earlier that morning, the administrator and Everyone user had lost ALL their privileges to Read, Write or Modify files in that directory. Because this was a Severity One condition for our web application, I decided to take the easy road and give Administrator and Everyone Read/Read&Execute/List/Write permissions on that directory.

This solved the problem and allowed the customers to complete the transaction; however, it didn't tell us the cause of why installing the new certificate changed the permissions on the MachineKeys directory. I am still researching this. If I find out why this happened, I will update this post.