HTTPS, Authentication, and Certificates

Computer Security Lecture, Dr. Lawlor

The original web server-to-client communication protocol HTTP (hypertext transfer protocol) includes absolutely no privacy or authentication--the request URL is sent in cleartext, and a response is accepted without question. This means any attacker that controls the network can read or write your traffic.

By contrast, HTTPS (HTTP Secure) is designed to ensure two things:

Secrecy: only the client and server can understand the network traffic. This is done using Diffie-Hellman key exchange to set up a secret key, and then the rest of the traffic is encrypted using an ordinary symmetric cipher such as AES.
Authentication: when a client connects to Amazon, they're really talking to Amazon, not a fake site designed to collect credit card numbers from people who think they're shopping at Amazon. This is a lot harder task, but it's done using a hierarchical system where the browsers trust a list of certificate authorities (CAs), and the CAs sign certificates using RSA. The browser then checks the signature against its hardcoded public key, the trust root. Normally the root CA signs a shorter-lived certificate for an intermediate CA, who then signs your server's certificate.

The certificates are the part that provides the authentication.

The format of certificates is X.509 (see design rationale presentation). Starting from the outside and working inward, we have:

PEM (Privacy Enhanced Mail format): ASCII wrapper around a DER certificate. It's just Base64 encoded binary DER data. You see the same files with extension ".crt" (for certificate), but usually a ".crt" only contains one certificate, while a ".pem" can contain a whole list of certificates concatenated together.
DER (Distinguished Encoding Rules): a hierarchical binary format used to assemble the pieces of the certificate into one big binary glob. DER is one of the many forms of ASN.1, but none of them are as popular as XML or JSON.
An RSA-signed hash of the DER certificate so far.

Verify a Certificate
OpenSSL is a neat little command line tool for generating and verifying certificates.

In your browser, open the debug console, Security, View Certificate, Details, Export, and save the whole https certificate chain to a Base64 .pem file. You can dump this to the screen with:

openssl x509 -in somewhere.pem -noout -text

You can verify the certificate against your system's disturbingly long trusted certificate lists (/usr/share/ca-certificates/mozilla/ or /etc/ssl/certs/ on my machine) using:

openssl verify googleIA.pem
googleIA.pem: OK

PEM

PEM stands for "Privacy Enhanced Mail", since certificates were first used for authenticating email. You see the same data in a ".crt" certificate file, or ".csr" key signing request, or ".pem" extension.

Google's .pem file looks like this:

With openssl, you can dump a PEM or .crt file's contents using:

   openssl x509 -in google.pem -noout -text

This produces the following text version of the certificate:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            14:85:0d:9e:00:00:00:00:7d:40
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, O=Google Inc, CN=Google Internet Authority
        Validity
            Not Before: Feb 20 13:34:56 2013 GMT
            Not After : Jun  7 19:43:27 2013 GMT
        Subject: C=US, ST=California, L=Mountain View, O=Google Inc, CN=www.google.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (1024 bit)
                Modulus:
                    00:e0:f5:15:b3:32:1b:42:13:f0:93:a2:d8:49:7a:
                    45:57:22:04:9a:78:10:d7:d3:92:92:76:be:7f:c7:
                    52:af:a0:8e:9a:e6:50:dc:4b:4a:f7:0a:f8:3f:2f:
                    06:99:2a:f0:de:35:42:ff:bc:d8:fc:93:b9:92:01:
                    d2:98:36:08:97:ec:d3:b0:b9:f9:90:10:9f:09:b4:
                    d4:38:90:8c:cf:e3:da:32:9b:e4:67:a0:38:14:58:
                    9e:06:d1:90:03:d2:18:2b:9f:cc:9c:be:40:15:9b:
                    77:58:6d:a9:c7:88:44:41:08:fc:91:fb:a5:41:a0:
                    83:dd:17:51:08:01:75:d0:5b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Key Identifier: 
                08:19:9A:02:9E:C8:AF:0F:5C:81:8E:01:4F:BF:E6:BF:F1:37:2E:61
            X509v3 Authority Key Identifier: 
                keyid:BF:C0:30:EB:F5:43:11:3E:67:BA:9E:91:FB:FC:6A:DA:E3:6B:12:24

            X509v3 CRL Distribution Points: 

                Full Name:
                  URI:http://www.gstatic.com/GoogleInternetAuthority/GoogleInternetAuthority.crl

            Authority Information Access: 
                CA Issuers - URI:http://www.gstatic.com/GoogleInternetAuthority/GoogleInternetAuthority.crt

            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Alternative Name: 
                DNS:www.google.com
    Signature Algorithm: sha1WithRSAEncryption
         9b:e8:97:20:cc:16:89:e5:6c:cc:e5:30:49:e5:75:c1:66:4d:
         1a:c2:76:82:e5:df:ca:40:da:00:12:40:be:45:63:80:b0:30:
         8c:dd:c3:ca:1d:bd:af:72:3c:f3:62:2f:b1:b0:76:5d:94:fd:
         77:71:79:63:e3:73:fe:f1:d7:d8:32:7d:96:af:56:78:38:28:
         61:97:7e:a2:99:6b:1c:3b:b1:c1:6a:4a:ed:29:ff:f9:78:12:
         6a:fa:43:47:9c:71:45:8a:1a:57:22:cd:4b:16:15:d2:d5:cc:
         b9:e1:4f:86:6f:14:b4:db:2e:39:4e:1c:d2:9d:bb:17:b3:c3:
         89:88

DER

You can extract the underlying binary DER glob from a PEM with

   grep -v "[-]----" google.pem | base64 -d > google.der

Here's the raw binary DER data in the above PEM file. This is actually a pretty compact format--2500 bytes of certificate text is 1200 bytes of base64, and only 900 bytes of DER.

0<82>^C<80>0<82>^B<E9><A0>^C^B^A^B^B
^T<85>^M<9E>^@^@^@^@}@0^M^F     *<86>H<86><F7>^M^A^A^E^E^@0F1^K0        ^F^CU^D
^F^S^BUS1^S0^Q^F^CU^D
^S
Google Inc1"0 ^F^CU^D^C^S^YGoogle Internet Authority0^^^W^M130220133456Z^W^M130607194327Z0h1^K0 ^F^CU^D^F^S^BUS1^S0^Q^F^CU^D^H^S
California1^V0^T^F^CU^D^G^S^MMountain View1^S0^Q^F^CU^D
^S
Google Inc1^W0^U^F^CU^D^C^S^Nwww.google.com0<81><9F>0^M^F       *<86>H<86><F7>^M^M^A^A^A^E^@^C<81>
<8D>^@0<81><89>^B<81><81>^@<E0><F5>^U<B3>2ESCB^S<F0><93><A2>
<D8>IzEW"^D<9A>x^P<D7><D3><92><92>v<BE>^?<C7>R<AF><A0><8E><9A><E6>P<DC>KJ<F7>
<F8>?/^F<99>*<F0><DE>5B<FF><BC><D8><FC><93><B9><92>^A<D2><98><97><EC><D3><B0>
<B9><F9><90>^P<9F> <B4><D4>8<90><8C><CF><E3><DA>2<9B><E4>g<A0>8^TX<9E>^F<D1><90>^C<D2>^X+<9F><CC>
<9C><BE>@^U<9B>wXm<A9><C7><88>D<FC><91><FB><A5>A<A0><83><DD>^W
^Au<D0>[^B^C^A^@^A<A3><82>^AQ0<82>^AM0^]^F^CU^]%^D^V0^T^F^H+^F^A^E^E^G^C^A^F^H+
^F^A^E^E^G^C^B0^]^F^CU^]^N^D^V^D^T^H^Y<9A>^B<9E><C8><AF>^O\<81><8E>^AO<BF><E6>
<BF><F1>7.a0^_^F^CU^]#^D^X0^V<80>^T<BF><C0>0<EB><F5>C^Q>g<BA><9E><91><FB><FC>j
<DA><E3>k^R$0[^F^CU^]^_^DT0R0P<A0>N<A0>L
<86>Jhttp://www.gstatic.com/GoogleInternetAuthority/GoogleInternetAuthority.crl0f^F^H+^F^A^E^E^G^A^A^DZ0X0V^F^H+^F^A^E^E^G0^B
<86>Jhttp://www.gstatic.com/GoogleInternetAuthority/GoogleInternetAuthority.crt0^L^F^CU^]^S^A^A<FF>^D^B0^@0^Y^F^CU^]^Q^D^R0^P
<82>^Nwww.google.com0^M^F
      *<86>H<86><F7>^M^A^A^E^E^@^C<81><81>^@<9B><E8><97> <CC>^V<89><E5>l<CC><E5>0I<E5>u<C1>fM^Z<C2>v<82><E5><DF><CA>@<DA>^@^R@
<BE>Ec<80><B0>0<8C><DD><C3><CA>^]
<BD><AF>r<<F3>b/<B1><B0>v]<94><FD>wqyc<E3>s<FE><F1><D7><D8>2}<96><AF>Vx8(a<97>~
<A2><99>k^\;<B1><C1>jJ<ED>)<FF><F9>x^Rj<FA>CG<9C>qE<8A>^ZW"<CD>K^V^U<D2><D5><CC><B9><E1>O<86>o^T<B4><DB>.9N^\<D2><9D><BB>^W<B3><C3><89><88>

The format is a little weird here (it's from "less"):

<8D> means the single binary byte 0x8D. Note that the very end of the DER is the binary bytes from the signatureAlgorithm above.
^M means "control-M", which is ASCII 'M' (0x4D) minus 0x40, or 0x0D.
Everything else is direct byte-for-byte data.

Note that the strings are length-counted, then appear directly.

See the Wikipedia DER encoding example for how this works. Class tags you see in certificates include:

0x00: END OF CONTENT (EOC) tag. This indicates the end of a sequence or message.
0x04: OCTET tag. This is followed by a length byte, then raw binary data. It's used for things like hashes or encryption keys.
0x13: PRINTABLE-STRING tag (^S here). This is followed by a length byte, then ASCII text.
0x30: SEQUENCE tag. This is followed by a sequence length field, then an EOC-terminated list of subtags. This is how hierarchical data is represented in the DER.
0x86: context-specific OBJECT IDENTIFIER. Seems to be a string.
0xA0: CONSTRUCTED object.

There's some fairly self-contained DER certificate decoding code as part of PolarSSL, but it's still pretty long and confusing. I/O code is often long and confusing!

Generate Server Certificate

OpenSSL's "req" command can generate "self-signed" certificates. These provide no protection against man-in-the-middle attacks (anybody could just sign their own certificate to impersonate the server), but are secure against eavesdroppers who don't modify your traffic.

openssl req \
  -x509 -nodes -days 9999 \
  -subj '/O=Assembly Extremists Society/OU=AES/CN=www.aes.edu/C=US/ST=Alaska/L=Fairbanks' \
  -newkey rsa:2048 -keyout mycert.key -out mycert.crt

The /C and /OU markers are X.509 certificate subject fields.

You can dump the info to the screen as before:

openssl x509 -in mycert.crt -noout -text

You can even start a simple demo HTTPS server using that certificate:

openssl s_server -cert mycert.crt -key mycert.key -www

Now you can visit https://localhost:4433 and talk to your server. You'll get a warning about the self-signed certificate. It doesn't have any content, just info about the ciphers used, but it is HTTPS!

The same certificate could be used with a real web server, like Apache, using mod_ssl (after "a2enmod ssl"). The relevant config lines are:

<VirtualHost *:443>
    ... lists of permissions and directories ...

    SSLEngine On

    # My public key, as self-signed or signed by my CA:
    SSLCertificateFile /etc/apache2/ssl/crt/mycert.crt

    # Certificate chain, from root to me: AddTrust -> InCommon -> me
    #  See below for how to get this file:
    SSLCertificateChainFile /etc/apache2/ssl/crt/mycert_chain.crt

    # My private key, which never leaves my machine:
    SSLCertificateKeyFile /etc/apache2/ssl/key/mycert.key

    <Location />
        SSLRequireSSL On
        SSLVerifyClient optional
        SSLVerifyDepth 1
        SSLOptions +StdEnvVars +StrictRequire
    </Location>

</VirtualHost>

Generate Key Signing Request (KSR)

You can generate a Key Signing Request or .csr similar to generating a self-signed key, but you leave out the -days and -x509 flags:

openssl req -new -newkey rsa:2048 -nodes -keyout mycert.key -out mycert.csr -subj  '/O=Assembly Extremists Society/OU=AES/CN=www.aes.edu/C=US/ST=Alaska/L=Fairbanks'

Again, the /C and /OU markers are X.509 certificate subject fields.

You now upload the .csr to your Certificate Authority. UAF uses InCommon. You can get a free 90-day trial cert from Comodo, if you can receive email at the admin or webmaster account at the controlled name.

Debug the files you get back from your CA by dumping the first PEM in a file with:

openssl x509 -in mycert.crt -noout -text

Some CAs concatenate all the PEM files together, so you need to split them manually (!?) and reorder them so web clients are happy with them.

Debug your server setup using s_client with showcerts:

echo | openssl s_client -showcerts -CApath /etc/ssl/certs/  -connect localhost:443

(Byte count is useful to determine setup speed.)

Doublecheck with gnutls, which is both name and order-sensitive (and used by git, for example):

echo | gnutls-cli --x509cafile /etc/ssl/certs/ca-certificates.crt -p https lawlor.cs.uaf.edu

OK, you are now using HTTPS!

Let's Encrypt: DNS-based Authentication

One of the hardest things about being a certificate authority is validating the requests: how do you know the person sending you this email is *really* working for Ebay?

There's a simple technical fix for authentication, based on doing DNS lookups:

You: I'm ebay.com, and I need to update my certificate.
Them: OK, just post "123456" to the file "http://ebay.com/789012.txt"
You: OK, done.
Them: OK, I did a DNS lookup and I see that file. Here's your signed cert.
You: Thanks!

It's all scripted with "Let's Encrypt". In theory it can work with your existing http server, but I've had better luck just asking the script to open its own server (with --standalone):

service apache2 stop
./certbot-auto certonly --standalone -d lawlor.cs.uaf.edu -d netrun.cs.uaf.edu
service apache2 start

One downside: if an attacker grabs control of your server or your IP address, even briefly, Let's Encrypt will sign their cert too. This lets them masquerade as you (or successfully man-in-the-middle your https traffic) until the cert expires, so Let's Encrypt certs expire in just 3 months.

How Hard is Authentication?

It's really hard to verify the person you're talking to is really who they claim they are.

"Rafael Espinoza" was hired as an Anchorage Police Officer in 2005 and was quite successful for over 5 years before the federal government noticed he was actually Rafael Mora-Lopez, when the real Rafael Espinoza applied for a passport at the same time as Mora-Lopez.
"Jose Morales" was hired as a Milwaukee Police Officer in 2004 and was by all accounts successful, until an anonymous tip led to the discovery he was actually Oscar Ayala-Cornejo, an illegal immigrant and the cousin of the real Jose Morales, who died as a child.

Cheat sheets: