HTTPS, Authentication, and Certificates

Computer Security Lecture, Dr. Lawlor

The original web server-to-client communication protocol HTTP (hypertext transfer protocol) includes absolutely no privacy or authentication--the request URL is sent in cleartext, and a response is accepted without question.  This means any attacker that controls the network can read or write your traffic.

By contrast, HTTPS (HTTP Secure) is designed to ensure two things:
The certificates are the part that provides the authentication. 

The format of certificates is X.509 (see design rationale presentation). Starting from the outside and working inward, we have:

Verify a Certificate
OpenSSL is a neat little command line tool for generating and verifying certificates.

In your browser, open the debug console, Security, View Certificate, Details, Export, and save the whole https certificate chain to a Base64 .pem file.  You can dump this to the screen with:
openssl x509 -in somewhere.pem -noout -text
You can verify the certificate against your system's disturbingly long trusted certificate lists (/usr/share/ca-certificates/mozilla/ or /etc/ssl/certs/ on my machine) using:
openssl verify googleIA.pem
googleIA.pem: OK

PEM

PEM stands for "Privacy Enhanced Mail", since certificates were first used for authenticating email.  You see the same data in a ".crt" certificate file, or ".csr" key signing request, or ".pem" extension.

Google's .pem file looks like this:
-----BEGIN CERTIFICATE-----
MIIDgDCCAumgAwIBAgIKFIUNngAAAAB9QDANBgkqhkiG9w0BAQUFADBGMQswCQYD
VQQGEwJVUzETMBEGA1UEChMKR29vZ2xlIEluYzEiMCAGA1UEAxMZR29vZ2xlIElu
dGVybmV0IEF1dGhvcml0eTAeFw0xMzAyMjAxMzM0NTZaFw0xMzA2MDcxOTQzMjda
MGgxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQHEw1N
b3VudGFpbiBWaWV3MRMwEQYDVQQKEwpHb29nbGUgSW5jMRcwFQYDVQQDEw53d3cu
Z29vZ2xlLmNvbTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEA4PUVszIbQhPw
k6LYSXpFVyIEmngQ19OSkna+f8dSr6COmuZQ3EtK9wr4Py8GmSrw3jVC/7zY/JO5
kgHSmDYIl+zTsLn5kBCfCbTUOJCMz+PaMpvkZ6A4FFieBtGQA9IYK5/MnL5AFZt3
WG2px4hEQQj8kfulQaCD3RdRCAF10FsCAwEAAaOCAVEwggFNMB0GA1UdJQQWMBQG
CCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQUCBmaAp7Irw9cgY4BT7/mv/E3
LmEwHwYDVR0jBBgwFoAUv8Aw6/VDET5nup6R+/xq2uNrEiQwWwYDVR0fBFQwUjBQ
oE6gTIZKaHR0cDovL3d3dy5nc3RhdGljLmNvbS9Hb29nbGVJbnRlcm5ldEF1dGhv
cml0eS9Hb29nbGVJbnRlcm5ldEF1dGhvcml0eS5jcmwwZgYIKwYBBQUHAQEEWjBY
MFYGCCsGAQUFBzAChkpodHRwOi8vd3d3LmdzdGF0aWMuY29tL0dvb2dsZUludGVy
bmV0QXV0aG9yaXR5L0dvb2dsZUludGVybmV0QXV0aG9yaXR5LmNydDAMBgNVHRMB
Af8EAjAAMBkGA1UdEQQSMBCCDnd3dy5nb29nbGUuY29tMA0GCSqGSIb3DQEBBQUA
A4GBAJvolyDMFonlbMzlMEnldcFmTRrCdoLl38pA2gASQL5FY4CwMIzdw8odva9y
PPNiL7Gwdl2U/XdxeWPjc/7x19gyfZavVng4KGGXfqKZaxw7scFqSu0p//l4Emr6
Q0eccUWKGlcizUsWFdLVzLnhT4ZvFLTbLjlOHNKduxezw4mI
-----END CERTIFICATE-----
With openssl, you can dump a PEM or .crt file's contents using:
   openssl x509 -in google.pem -noout -text 
This produces the following text version of the certificate:
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
14:85:0d:9e:00:00:00:00:7d:40
Signature Algorithm: sha1WithRSAEncryption
Issuer: C=US, O=Google Inc, CN=Google Internet Authority
Validity
Not Before: Feb 20 13:34:56 2013 GMT
Not After : Jun 7 19:43:27 2013 GMT
Subject: C=US, ST=California, L=Mountain View, O=Google Inc, CN=www.google.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (1024 bit)
Modulus:
00:e0:f5:15:b3:32:1b:42:13:f0:93:a2:d8:49:7a:
45:57:22:04:9a:78:10:d7:d3:92:92:76:be:7f:c7:
52:af:a0:8e:9a:e6:50:dc:4b:4a:f7:0a:f8:3f:2f:
06:99:2a:f0:de:35:42:ff:bc:d8:fc:93:b9:92:01:
d2:98:36:08:97:ec:d3:b0:b9:f9:90:10:9f:09:b4:
d4:38:90:8c:cf:e3:da:32:9b:e4:67:a0:38:14:58:
9e:06:d1:90:03:d2:18:2b:9f:cc:9c:be:40:15:9b:
77:58:6d:a9:c7:88:44:41:08:fc:91:fb:a5:41:a0:
83:dd:17:51:08:01:75:d0:5b
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Key Identifier:
08:19:9A:02:9E:C8:AF:0F:5C:81:8E:01:4F:BF:E6:BF:F1:37:2E:61
X509v3 Authority Key Identifier:
keyid:BF:C0:30:EB:F5:43:11:3E:67:BA:9E:91:FB:FC:6A:DA:E3:6B:12:24

X509v3 CRL Distribution Points:

Full Name:
URI:http://www.gstatic.com/GoogleInternetAuthority/GoogleInternetAuthority.crl

Authority Information Access:
CA Issuers - URI:http://www.gstatic.com/GoogleInternetAuthority/GoogleInternetAuthority.crt

X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Alternative Name:
DNS:www.google.com
Signature Algorithm: sha1WithRSAEncryption
9b:e8:97:20:cc:16:89:e5:6c:cc:e5:30:49:e5:75:c1:66:4d:
1a:c2:76:82:e5:df:ca:40:da:00:12:40:be:45:63:80:b0:30:
8c:dd:c3:ca:1d:bd:af:72:3c:f3:62:2f:b1:b0:76:5d:94:fd:
77:71:79:63:e3:73:fe:f1:d7:d8:32:7d:96:af:56:78:38:28:
61:97:7e:a2:99:6b:1c:3b:b1:c1:6a:4a:ed:29:ff:f9:78:12:
6a:fa:43:47:9c:71:45:8a:1a:57:22:cd:4b:16:15:d2:d5:cc:
b9:e1:4f:86:6f:14:b4:db:2e:39:4e:1c:d2:9d:bb:17:b3:c3:
89:88

DER

You can extract the underlying binary DER glob from a PEM with
   grep -v "[-]----" google.pem | base64 -d > google.der
Here's the raw binary DER data in the above PEM file.  This is actually a pretty compact format--2500 bytes of certificate text is 1200 bytes of base64, and only 900 bytes of DER.
0<82>^C<80>0<82>^B<E9><A0>^C^B^A^B^B
^T<85>^M<9E>^@^@^@^@}@0^M^F *<86>H<86><F7>^M^A^A^E^E^@0F1^K0 ^F^CU^D
^F^S^BUS1^S0^Q^F^CU^D
^S
Google Inc1"0 ^F^CU^D^C^S^YGoogle Internet Authority0^^^W^M130220133456Z^W^M130607194327Z0h1^K0 ^F^CU^D^F^S^BUS1^S0^Q^F^CU^D^H^S
California1^V0^T^F^CU^D^G^S^MMountain View1^S0^Q^F^CU^D
^S
Google Inc1^W0^U^F^CU^D^C^S^Nwww.google.com0<81><9F>0^M^F *<86>H<86><F7>^M^M^A^A^A^E^@^C<81>
<8D>^@0<81><89>^B<81><81>^@<E0><F5>^U<B3>2ESCB^S<F0><93><A2>
<D8>IzEW"^D<9A>x^P<D7><D3><92><92>v<BE>^?<C7>R<AF><A0><8E><9A><E6>P<DC>KJ<F7>
<F8>?/^F<99>*<F0><DE>5B<FF><BC><D8><FC><93><B9><92>^A<D2><98><97><EC><D3><B0>
<B9><F9><90>^P<9F> <B4><D4>8<90><8C><CF><E3><DA>2<9B><E4>g<A0>8^TX<9E>^F<D1><90>^C<D2>^X+<9F><CC>
<9C><BE>@^U<9B>wXm<A9><C7><88>D<FC><91><FB><A5>A<A0><83><DD>^W
^Au<D0>[^B^C^A^@^A<A3><82>^AQ0<82>^AM0^]^F^CU^]%^D^V0^T^F^H+^F^A^E^E^G^C^A^F^H+
^F^A^E^E^G^C^B0^]^F^CU^]^N^D^V^D^T^H^Y<9A>^B<9E><C8><AF>^O\<81><8E>^AO<BF><E6>
<BF><F1>7.a0^_^F^CU^]#^D^X0^V<80>^T<BF><C0>0<EB><F5>C^Q>g<BA><9E><91><FB><FC>j
<DA><E3>k^R$0[^F^CU^]^_^DT0R0P<A0>N<A0>L
<86>Jhttp://www.gstatic.com/GoogleInternetAuthority/GoogleInternetAuthority.crl0f^F^H+^F^A^E^E^G^A^A^DZ0X0V^F^H+^F^A^E^E^G0^B
<86>Jhttp://www.gstatic.com/GoogleInternetAuthority/GoogleInternetAuthority.crt0^L^F^CU^]^S^A^A<FF>^D^B0^@0^Y^F^CU^]^Q^D^R0^P
<82>^Nwww.google.com0^M^F
*<86>H<86><F7>^M^A^A^E^E^@^C<81><81>^@<9B><E8><97> <CC>^V<89><E5>l<CC><E5>0I<E5>u<C1>fM^Z<C2>v<82><E5><DF><CA>@<DA>^@^R@
<BE>Ec<80><B0>0<8C><DD><C3><CA>^]
<BD><AF>r<<F3>b/<B1><B0>v]<94><FD>wqyc<E3>s<FE><F1><D7><D8>2}<96><AF>Vx8(a<97>~
<A2><99>k^\;<B1><C1>jJ<ED>)<FF><F9>x^Rj<FA>CG<9C>qE<8A>^ZW"<CD>K^V^U<D2><D5><CC><B9><E1>O<86>o^T<B4><DB>.9N^\<D2><9D><BB>^W<B3><C3><89><88>
The format is a little weird here (it's from "less"):
Note that the strings are length-counted, then appear directly.  

See the Wikipedia DER encoding example for how this works.  Class tags you see in certificates include:
There's some fairly self-contained DER certificate decoding code as part of PolarSSL, but it's still pretty long and confusing.  I/O code is often long and confusing!

Generate Server Certificate

OpenSSL's "req" command can generate "self-signed" certificates.  These provide no protection against man-in-the-middle attacks (anybody could just sign their own certificate to impersonate the server), but are secure against eavesdroppers who don't modify your traffic.
openssl req \
-x509 -nodes -days 9999 \
-subj '/O=Assembly Extremists Society/OU=AES/CN=www.aes.edu/C=US/ST=Alaska/L=Fairbanks' \
-newkey rsa:2048 -keyout mycert.key -out mycert.crt
The /C and /OU markers are X.509 certificate subject fields.

You can dump the info to the screen as before:

openssl x509 -in mycert.crt -noout -text
You can even start a simple demo HTTPS server using that certificate:
openssl s_server -cert mycert.crt -key mycert.key -www
Now you can visit https://localhost:4433 and talk to your server.  You'll get a warning about the self-signed certificate.  It doesn't have any content, just info about the ciphers used, but it is HTTPS!

The same certificate could be used with a real web server, like Apache, using mod_ssl  (after "a2enmod ssl").  The relevant config lines are:
<VirtualHost *:443>
... lists of permissions and directories ...

SSLEngine On

# My public key, as self-signed or signed by my CA:
SSLCertificateFile /etc/apache2/ssl/crt/mycert.crt

# Certificate chain, from root to me: AddTrust -> InCommon -> me
# See below for how to get this file:
SSLCertificateChainFile /etc/apache2/ssl/crt/mycert_chain.crt

# My private key, which never leaves my machine:
SSLCertificateKeyFile /etc/apache2/ssl/key/mycert.key

<Location />
SSLRequireSSL On
SSLVerifyClient optional
SSLVerifyDepth 1
SSLOptions +StdEnvVars +StrictRequire
</Location>

</VirtualHost>

Generate Key Signing Request (KSR)

You can generate a Key Signing Request or .csr similar to generating a self-signed key, but you leave out the -days and -x509 flags:
openssl req -new -newkey rsa:2048 -nodes -keyout mycert.key -out mycert.csr -subj  '/O=Assembly Extremists Society/OU=AES/CN=www.aes.edu/C=US/ST=Alaska/L=Fairbanks'
Again, the /C and /OU markers are X.509 certificate subject fields.

You now upload the .csr to your Certificate Authority.  UAF uses InCommon.  You can get a free 90-day trial cert from Comodo, if you can receive email at the admin or webmaster account at the controlled name.

Debug the files you get back from your CA by dumping the first PEM in a file with:
openssl x509 -in mycert.crt -noout -text
Some CAs concatenate all the PEM files together, so you need to split them manually (!?) and reorder them so web clients are happy with them.

Debug your server setup using s_client with showcerts:
echo | openssl s_client -showcerts -CApath /etc/ssl/certs/  -connect localhost:443
(Byte count is useful to determine setup speed.)

Doublecheck with gnutls, which is both name and order-sensitive (and used by git, for example):
echo | gnutls-cli --x509cafile /etc/ssl/certs/ca-certificates.crt -p https lawlor.cs.uaf.edu
OK, you are now using HTTPS!

Let's Encrypt: DNS-based Authentication

One of the hardest things about being a certificate authority is validating the requests: how do you know the person sending you this email is *really* working for Ebay?

There's a simple technical fix for authentication, based on doing DNS lookups:
It's all scripted with "Let's Encrypt".  In theory it can work with your existing http server, but I've had better luck just asking the script to open its own server (with --standalone):
service apache2 stop
./certbot-auto certonly --standalone -d lawlor.cs.uaf.edu -d netrun.cs.uaf.edu
service apache2 start
One downside: if an attacker grabs control of your server or your IP address, even briefly, Let's Encrypt will sign their cert too.  This lets them masquerade as you (or successfully man-in-the-middle your https traffic) until the cert expires, so Let's Encrypt certs expire in just 3 months.

How Hard is Authentication?

It's really hard to verify the person you're talking to is really who they claim they are.
Cheat sheets: