Showing posts with label ssl. Show all posts
Showing posts with label ssl. Show all posts

Saturday, January 17, 2009

VMware clone and IIS 6.0 SSL problem

Yesterday, we cloned our first virtual machine running Win2K3 that uses IIS 6.0 and runs multiple websites via separate IP addresses. The clone process went smoothly: the new VM was assigned a new IP address and hostname and the IWAM and IUSR accounts and permissions migrated to that new machine name. However, during testing, we encountered a problem with the SecureBindings in the IIS metabase.

Because this VM had multiple websites in IIS configured, the SecureBindings setting in the IIS metabase (c:\windows\system32\inetsrv\metabase.xml) kept the old IP address information that was configured in the source VM. I'm not sure if this is caused by VMware or IIS, but the solution is to manually edit SecureBindings in IIS to point the website's SSL to the correct IP.

MS Article on Editing the Metabase

TAG

Thursday, May 15, 2008

Verisign Class 1 CA Individual Subscriber cert expires

Yesterday, 5/14/2008, we were rudely awakened to errors in our SSL checkout process. Basically, the dialog box that informs a person browsing the website that a problem has occurred showed up. Investigating our IIS 5.0 web servers, we found that none of the SSL certificates had expired. Digging deeper, we then looked at the certificate store on the machine. In the certificate store, we found that the Intermediate Certificate called Verisign Class 1 CA Individual Subscriber certificate had expired. Yuck!


For Windows 2000
We called Verisign and confirmed that the expired certificate was affecting our site. Verisign was able to give us a link to download the latest certificate:
https://knowledge.verisign.com/support/digital-id-support/index?page=content&id=SO6052



We followed the install and verfication instructions to install the cert on each server in the farm. After making those changes, I verified in Firefox that the SSL dialog box no longer appeared. Thank God!

For Windows 2003
Win2K3 was slightly different. We actually needed to make the certificate appear in our Intermediate Certificate list:


This was accomplished by downloading the latest intermediate cert that works with your chosen level of SSL certification from here:
https://knowledge.verisign.com/support/ssl-certificates-support/index?page=content&id=AR657

For us, we use Non-EV Premium SSL certificates. So we used that link from the main page. Once we installed them by following the directions on the linked page,
http://www.verisign.com/support/verisign-intermediate-ca/secure-site-pro-intermediate/index.html, we saw the intermediate cert appear (date of 10/24/2011) in the proper certificate store:


Finally, a reset of IIS was needed. Yuk!

Afterword
The one thing I do not have clarity on is exactly what certificates does a typical web server running ASP and ASP.net need to run correctly. I know that there is a hierarchy of certs:
1) Root certificates
2) Root cert signs Intermediate
3) Intermediate signs Personal (www.mysite.com)

We had a similar problem back a couple of years ago and I had been too busy to research the question then. I will try to research it and give an update to this page.

Good luck, folks!
'sodo

Monday, August 13, 2007

nCipher hardserver process memory leak

We are using nCipher's 500 F2 SSL acceleration cards in a small farm of four Windows 2000 production web servers. About a month and a half ago, we installed these cards into the servers. Last week, one of the servers went down, gave us the lovely blue screen of death and became unbootable. The only way our server admin could bring the server back up was to remove the nCipher card.

Two days after the card was removed, I started seeing these errors in the System logs:
Event ID: 2019
The server was unable to allocate from the system nonpaged pool because the pool was empty.

That error lead me to a Microsoft Q Article that stated that this error could be associated with an application that was using up too much memory; ie, an application that had a memory leak:
/technet/prodtechnol/windows2000serv/reskit/w2000msgs/4746.mspx
Therefore, I started hunting for a rogue application. I used Performance Monitor to map all the running processes' "Pool Nonpaged Bytes", like so:


I simply looked at each running process and found the ones using the most Pool Nonpaged Byte memory. I set my refresh to every three seconds, so that I could see the increase in Pool Nonpaged Byte memory as I searched for the rogue. As there were seventy or so processes running, it took a bit of time to identify "hardserver" as the rogue process. I simply looked for the process consuming the most memory! Also, I saw that the memory usage was increasing as I watched the chart.

But what is "hardserver?" Apparently, hardserver is part of nCipher's SSL card driver install:
http://www.ncipher.com/resources/97/sa14_presence_of_flaws_in_firmware_security

Once I charted hardserver's Pool Nonpaged Byte memory use against the Total Pool Nonpaged Byte memory used, it was easy to see how the hardserver process was driving up Nonpaged memory utilization, as in the screen cap below:


The side effect of this memory leak brought down the ASPNet worker process on the webserver:
"aspnet_wp.exe could not be started. The error code for the failure is 800705AA. This error can be caused when the worker process account has insufficient rights to read the .NET Framework files. Please ensure that the .NET Framework is correctly installed and that the ACLs on the installation directory allow access to the configured account."

This, in turn, showed the ugly error "Server Application Unavailable" to the end user:


So, either the card itself is bad or nCipher's software has a memory leak only when the cards' driver software is installed, but the card is not available.

We put the card in a test server in the lab and it showed the same bad behaviour as in the production server: blue screen of death. Speaking to an nCipher engineer, I found out that the blue indicator light on the back of the card will tell us whether the card is functioning properly. As expected, instead of a constant blink at the 3 or 4 second mark, the indicator light on the card flashed randomly. This tells us the card is not working correctly for some reason.

The nCipher engineers and customer service folks were very helpful and we soon had an RMA number to return the card. Thanks nCipher!

10/23/07 Update
Here's a concise set of instructions about finding processes that are triggering memory leaks. From Microsoft, no less:
http://support.microsoft.com/kb/130926

'sodo

Friday, June 22, 2007

from IIS, convert an SSL cert to PEM format

As we were moving our SSL termination from the web servers to an F5 3400 load balancing switch, I needed to convert our SSL certificate to PEM format. This is a subject that has been the source of much frustration and I thought I'd clarify the process for individuals suffering the same fate as I using step by step instructions.

Update 2010/04/06
On Windows 2003, you can only export the private key via the Certificate Manager Snap-In, not the IIS MMC.
*** end update ***

Requirements
In order to do the conversion, you'll need access to your IIS box with an SSL certificate installed and OpenSSL. I run OpenSSL using cygwin (http://www.cygwin.com/), a set of Unix tools for NT.

Overview
1. Export the certificate from IIS
2. Convert the PKCS#12 certificate
3. Verify the certificate
4. Move the sections in the certificate into the correct order
5. Final test

Detailed Steps

Export the certificate from IIS
1. Open the Internet Information Services (IIS) Manager administration tool.
2. Expand the Web Sites node and locate the SSL-enabled Web site.
3. Right-click this Web site and click Properties.
4. Click the Directory Security tab and in the Secure Communications section of the window, select the View Certificate box.
5. Click the Details tab and click Copy to File


6. Click Next on the Welcome to the Certificate Export Wizard page.


7. Select Yes, export the private key and click Next:


Note: The private key MUST be exported for SSL to work. If the option to export the private key is unavailable, see Microsoft article 232154 – IIS: Export Private Key Option is Grayed When Exporting a Server Certificate.

8. Ensure that the Personal Information Exchange –PKCS #12 radio button is selected and select ONLY the Include all certificates in the certification path if possible check box. Click Next:


9. Enter a password and click Next:


10. Enter a file name and location and click Next. Give the file an extension of .PFX:


11. Click Finish:




Convert the PKCS#12 certificate
1. Move the exported .PFX certificate file to a location from where it may be converted with OpenSSL
2. Convert it with openssl
# openssl.exe pkcs12 -in exportPKCS12.pfx -out cert.pem -nodes

bash-3.00$ openssl.exe pkcs12 -in exportPKCS12.pfx -out cert.pem -nodes
Enter Import Password:
MAC verified OK


Verify the certificate
1. Ensure that the converted certificate is in correct x509 format. Do this by verifying that the following command produces no errors:
# openssl x509 -in cert.pem -text

bash-3.00$ openssl x509 -in cert.pem -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
74:e0:f4:04:1d:af:04:43:aa:89:27:69:ab:74:f9:55
Signature Algorithm: sha1WithRSAEncryption
Issuer: O=VeriSign Trust Network, OU=VeriSign, Inc., OU=VeriSign International Server CA - Class 3, OU=www.veris
ign.com/CPS Incorp.by Ref. LIABILITY LTD.(c)97 VeriSign
Validity
Not Before: Aug 16 00:00:00 2005 GMT
Not After : Aug 26 23:59:59 2007 GMT
Subject: C=US, ST=New York, L=New York, O=Test, OU=IT eServicesTerms of use at www.verisign.com/rpa (c
)00, OU=Terms of use at www.verisign.com/rpa (c)00, CN=www.test.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public Key: (1024 bit)
Modulus (1024 bit):
00:cb:70:5f:ce:ee:d6:8c:5f:13:7a:06:df:51:56:
7d:d9:50:11:dc:69:d7:5d:d8:90:f7:9f:47:df:8d:
cc:56:ae:af:e0:c5:b5:5e:40:a8:8c:5d:03:a8:5e:
49:1c:13:42:2e:bd:d5:29:5e:16:49:18:9b:a3:84:
32:d8:f9:8e:84:2d:14:27:a6:80:b5:19:03:93:39:
44:9f:4f:b8:c5:49:42:9a:ef:0c:9a:cf:05:9e:6f:
02:d7:fb:0b:04:d5:1d:f8:fa:3b:29:d8:ed:bb:fd:
9b:ae:fc:27:1d:d3:ed:d6:86:8a:fb:f0:a3:61:85:
13:a0:84:1c:8a:12:64:0e:17
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Key Usage:
Digital Signature, Key Encipherment
X509v3 CRL Distribution Points:
URI:http://crl.verisign.com/Class3InternationalServer.crl

X509v3 Certificate Policies:
Policy: 2.16.840.1.113733.1.7.23.3
CPS: https://www.verisign.com/rpa

X509v3 Extended Key Usage:
Netscape Server Gated Crypto, Microsoft Server Gated Crypto, TLS Web Server Authentication, TLS Web Clie
nt Authentication
Authority Information Access:
OCSP - URI:http://ocsp.verisign.com

1.3.6.1.5.5.7.1.12:
0_.].[0Y0W0U..image/gif0!0.0...+..............k...j.H.,{..0%.#http://logo.verisign.com/vslogo.gif
Signature Algorithm: sha1WithRSAEncryption
88:ff:10:98:c4:90:62:c2:77:61:33:5e:ba:fa:5f:ca:7e:2d:
7c:1e:b7:64:7c:b8:df:33:4b:92:8a:0a:29:85:b8:9c:c5:93:
db:b2:f0:fc:77:b4:50:db:ec:db:df:c2:36:b9:b6:9e:7c:12:
c4:8d:83:02:43:f1:fc:6a:94:f9:d9:75:e5:66:70:04:74:04:
89:f9:15:8f:1f:45:36:51:b7:a9:7f:7d:94:a9:10:e3:51:c5:
b0:1c:b6:d4:23:cb:66:1d:c0:b0:0d:72:69:dd:49:9d:a8:e4:
33:12:70:8a:b7:84:85:a5:3a:5c:7f:c9:d1:e7:e2:b4:31:57:
ff:e4
-----BEGIN CERTIFICATE-----
MIIErTCCBBagAwIBAgIQdOD0BB2vBEOqiSdpq3T5VTANBgkqhkiG9w0BAQUFADCB
ujEfMB0GA1UEChMwVmVyaVNpZ24gVHJ1c3QgTmV0d29yazEXMBUGA1UECxMOVmVy
aVNpZ24sIEluYy4xfzAxBgNVBAsTKlZlcmlTaWduIEludGVybmF0aW9uYWwgU2Vy
dmVyIENBIC0gQ2xhcsMgMzFJMEcGA1UECxNAd3d3LnZlcmlzaWduLmNvbS9DUFMg
SW5jb3JwLmJ5IFJlZi4gTElBQklMSVRZIExURC4oYyk5NyBWZXJpU2lnbjAeFw0w
NTA4MTYwMDAwMDBaFw0wNzA4MjYyMzU5NTlaMIHlMQswCQYDVQQGEwJVUzETMBEG
A1UECBMKTmV3IEplcnNleTEOMAwGA1UEBxQFVW5pb24xGDAWBgNVBAoUD0JlZCBC
YXRoIEJleW9uZDE/MD0GA1UECxQ2SVQgZVNlcnZpY2VzVGVybXMgb2YgdXNlIGF0
IHd3dy52ZXJpc2lnbi5jb20vcnBhIChjKTAwMTMwMQYDVQQLFCpUZXJtcyBvZiB1
c2UgYXQgd3d3LnZlcmlzaWduLmNvbS9ycGEgKGMpMDAxITAfBgNVBAMUGHd3dy5i
ZWRiYXRoYW5kYmV5b25kLmNvbTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEA
y3Bfzu7WjF8TegbfUVZ92VAR3GnXXdiQ959H343MVq6v4MW1XkCojF0DqF5JHBNC
Lr3VKV4WSRibo4Qy2PmOhC0UJ6aAtRkDkzlEn0+4xUlCmu8Mms8Fnm8C1/sLBNUd
+Po7Kdjtu/2brvwnHdPt1oaK+/CjYYUToIQcihJkDhcCAwEAAaOCAYUwggGBMAkG
A1UdEwQCMAAwCwYDVR0PBAQDAgWgMEYGA1UdHwQ/MD0wO6A5oDeGNWh0dHA6Ly9j
cmwudmVyaXNpZ24uY29tL0NsYXNzM0ludGVybmF0aW9uYWxTZXJ2ZXIuY3JsMEQG
A1UdIAQ9MDswOQYLYIZIAYb4RQEHFwMwKjAoBggrBgEFBQcCARYcaHR0cHM6Ly93
d3cudmVyaXNpZ24uY29tL3JwYTA0BgNVHSUELTArBglghkgBhvhCBAEGCisGAQQB
gjcKAwMGCCsGAQUFBwMBBggrBgEFBQcDAjA0BggrBgEFBQcBAQQoMCYwJAYIKwYB
BQUHMAGGGGh0dHA6Ly9vY3NwLnZlcmlzaWduLmNvbTBtBggrBgEFBQcBDARhMF+h
XaBbMFkwVzBVFglpbWFnZS9naWYwITAfxAcGBSsOAwIaBBSP5dMahqyNjmvDz4Bq
1EgYLHsZLjAlFiNodHRwOi8vbG9nby52ZXJpc2lnbi5jb20vdnNsb2dvLmdpZjAN
BgkqhkiG9w0BAQUFAAOBgQCI/xzYxJBiwndhM166+l/Kfi18HrdkfLjfM0uSigop
hbicxZPbsvD8d7RQ2+zb38I2ubaefBLEjYMCQ/H8apT52XXlZnAEdASJ+RWPH0U2
Ubepf32UqRDjUcWwHLbUI8tmHcCwDXJp3UmdqOQzEnCKt4SFpTpcf8nR5+K0MVf/
5A==
-----END CERTIFICATE-----


2. Now that we've verified that the certificate is in correct x509 format, check that the certificate file contains a private key:
# cat cert.pem

Look at the output (example shown below) and check for a section that looks similar to this:
-----BEGIN RSA PRIVATE KEY-----
XaBbMFkwVzBVFglpbWFnZS9naWYwITAfxAcGBSsOAwIaBBSP5dMahqyNjmvDz4Bq
1EgYLHsZLjAlFiNodHRwOi8vbG9nby52ZXJpc2lnbi5jb20vdnNsb2dvLmdpZjAN
BgkqhkiG9w0BAQUFAAOBgQCI/xzYxJBiwndhM166+l/Kfi18HrdkfLjfM0uSigop
hbicxZPbsvD8d7RQ2+zb38I2ubaefBLEjYMCQ/H8apT52XXlZnAEdASJ+RWPH0U2
-----END RSA PRIVATE KEY-----

Move the sections in the certificate into the correct order

For some reason, OpenSSL converts the PFX to PEM where the sections of the certificate are in the wrong order. The key sections (shown below) must be in the following order:
a. Private key
b. Intermediate certificate
c. Server certificate

As the PEM is a text file, use your favorite editor (vi or emacs or any editor that does not add or remove line feeds) to reorder the sections as noted above.

Key sections within the certificate

The following is the RSA PRIVATE KEY section:
Bag Attributes
1.3.6.1.4.1.311.17.2:
localKeyID: 01 00 00 00
Microsoft CSP Name: Microsoft RSA SChannel Cryptographic Provider
friendlyName: 4b9cef4cc8c9b849ff5c662fd3e0ef7e_76267e3e-6183-4d45-886e-6e067297b38f
Key Attributes
X509v3 Key Usage: 10
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: QWS-EDE3-CCC,43E7ACA5F4423968
pZJ2SfsSVqMbRRf6ug37Clua5gY0Wld4frPIxFXyJquUHr31dilW5ta3hbIaQ+Rg
... (random characters)
v8dMugeRplkaH2Uwt/mWBk4t71Yv7GeHmcmjafK8H8iW80ooPO3D/ENV8X4U/tlh
5eU6ky3WYZ1BTy6thxxLlwAullynVXZEflNLxq1oX+ZYl6djgjE3qg==
-----END RSA PRIVATE KEY-----

The following is the SERVER CERTIFICATE section:

Bag Attributes
localKeyID: 01 00 00 00
friendlyName: Test Certificate
subject=/C=AU/ST=NSW/L=test/O=Furnishing/OU=Support/CN=test.retail.home
issuer=/DC=ranch/DC=retail/CN=home
-----BEGIN CERTIFICATE-----
MIIFiTCCBHGgAwIBAgIKCGryDgAHzANBgkqhkiG9w0BAQUFADA8MRMwEQYK
... (random characters)
5pLDWYVHhLkA1pSxvFjNJHRSIydWHc5ltGyKqIUcBezVaXyel94pNSUYx07NpPV/
MY2ovQyQZM8gGe3+lGFum0VHbv/y/gB9HhFesog=
-----END CERTIFICATE-----

The following is the INTERMEDIATE CA CERTIFICATE section:

Bag Attributes:
subject=/DC=ranch/DC=retail/CN=home
issuer=/DC=ranch/DC=retail/CN=home
-----BEGIN CERTIFICATE-----
MIIESDCCAzCgAwIBAgIQah20fCRYTY9LRXYMIRaKGjANBgkqhkiG9w0BAQUFADA8
... (random characters)
Nt0nksawDnbKo86rQcNnY5xUs7c7pj2zxj/IOsgNHUp5W6dDI9pQoqFFaDk=
-----END CERTIFICATE-----


Further Intermediate CA certificates may follow depending on the certification path of the exported certificate.

Example of key sections of PEM in wrong order:


Example of key sections of PEM in correct order:


Final test procedure
Use OpenSSL’s verify command to verify that the certificate is properly formed:
# openssl verify cert.pem

bash-3.00$ openssl verify cert.pem
cert2.pem: OK


That's it folks!

Troubleshooting
When I did not move the server certificate portion of the PEM file to the bottom of the PEM file, I received the error:
error 20 at 0 depth lookup:unable to get local issuer certificate

As WordPad does not add control characters to the document, I used WordPad to move the server certificate portion of the PEM file to the bottom. After doing this, I then received the success message:

bash-2.02$ openssl verify certorder.pem
certorder.pem: OK
Feel free to drop me a line or ask me a question.