The (Leaning) Tower of Data Storage Abstractions

CS 493 Lecture, Dr. Lawlor

General Principles


Anthropomorphic wolf hacks computer using physical access.


Level of System
Attack
Countermeasures
Front door website
Intruder gains access to a legitimate user's account.
Require strong passwords.
Expire old accounts.
Two+ factor authentication (e.g., crypto key).
Audit user access logs regularly.
Database
SQL injection or other direct database manipulation allows intruder to read & write data.
Allow only safe characters instead of trying to find and exclude all possible unsafe characters.
Use PDO prepare/execute to automatically quote queries.
Restrict database user account permissions to limit damage (role based access control).
Audit system logs regularly.
Operating System
Local user attaches debugger to vulnerable process.
Run servers as separate user accounts. 
Keep OS patched. 
Audit system regularly.
Local user directly reads vulnerable files.
Lock down file permissions.
Consider using virtual machines, or containers, for better isolation.
Block Level
Attacker reads storage device at raw byte level, below the filesystem. 
Full disk encryption.
Physical Machine
Ninjas sneak into building and physically access hardware.
Full disk encryption.
Set a BIOS password to prevent external boot devices.
Fill external USB ports with epoxy. 
Desolder all unused ports from system motherboard.
Lock computer in an Intrusion Prevention System (IPS) safe.
Video surveillance system.
Armed security guards on site.
Adversary finds old backup tapes.
Encrypted backups.
Don't erase, shred.
Adversary buys old server disks at surplus auction for data remanence attack.
Melt your old data storage devices, don't sell them.


Specific Example: Extracting Database Passwords from Files

If we have OS level access to a machine running a mysql database, there are files storing everything in the database. 

By default in Linux, they're at /var/lib/mysql; you can find this directory in several ways:
root@minbox:~# ps aux | grep mysql
mysql 1051 0.1 13.2 544580 135268 ? Ssl 20:58 0:00 /usr/sbin/mysqld
root 1412 0.0 0.0 5104 848 pts/0 S+ 21:04 0:00 grep --color=auto mysql
root@minbox:~# cd /proc/1051/
root@minbox:/proc/1051# ls -al
total 0
dr-xr-xr-x 9 mysql mysql 0 Sep 4 20:58 .
dr-xr-xr-x 115 root root 0 Sep 4 20:58 ..
dr-xr-xr-x 2 mysql mysql 0 Sep 4 21:03 attr
-rw-r--r-- 1 mysql mysql 0 Sep 4 21:03 autogroup
-r-------- 1 mysql mysql 0 Sep 4 21:03 auxv
-r--r--r-- 1 mysql mysql 0 Sep 4 21:03 cgroup
--w------- 1 mysql mysql 0 Sep 4 21:03 clear_refs
-r--r--r-- 1 mysql mysql 0 Sep 4 20:59 cmdline
-rw-r--r-- 1 mysql mysql 0 Sep 4 21:03 comm
-rw-r--r-- 1 mysql mysql 0 Sep 4 21:03 coredump_filter
-r--r--r-- 1 mysql mysql 0 Sep 4 21:03 cpuset
lrwxrwxrwx 1 mysql mysql 0 Sep 4 21:03 cwd -> /var/lib/mysql
-r-------- 1 mysql mysql 0 Sep 4 21:03 environ
lrwxrwxrwx 1 mysql mysql 0 Sep 4 20:58 exe -> /usr/sbin/mysqld
dr-x------ 2 mysql mysql 0 Sep 4 20:58 fd
dr-x------ 2 mysql mysql 0 Sep 4 21:03 fdinfo
-rw-r--r-- 1 mysql mysql 0 Sep 4 21:03 gid_map
-r-------- 1 mysql mysql 0 Sep 4 21:03 io
-r--r--r-- 1 mysql mysql 0 Sep 4 21:03 limits
-rw-r--r-- 1 mysql mysql 0 Sep 4 21:03 loginuid
dr-x------ 2 mysql mysql 0 Sep 4 21:03 map_files
-r--r--r-- 1 mysql mysql 0 Sep 4 21:03 maps
-rw------- 1 mysql mysql 0 Sep 4 21:03 mem
-r--r--r-- 1 mysql mysql 0 Sep 4 21:03 mountinfo
-r--r--r-- 1 mysql mysql 0 Sep 4 21:03 mounts
-r-------- 1 mysql mysql 0 Sep 4 21:03 mountstats
dr-xr-xr-x 5 mysql mysql 0 Sep 4 21:03 net
dr-x--x--x 2 mysql mysql 0 Sep 4 21:03 ns
-rw-r--r-- 1 mysql mysql 0 Sep 4 21:03 oom_adj
-r--r--r-- 1 mysql mysql 0 Sep 4 21:03 oom_score
-rw-r--r-- 1 mysql mysql 0 Sep 4 21:03 oom_score_adj
-r-------- 1 mysql mysql 0 Sep 4 21:03 pagemap
-r-------- 1 mysql mysql 0 Sep 4 21:03 personality
-rw-r--r-- 1 mysql mysql 0 Sep 4 21:03 projid_map
lrwxrwxrwx 1 mysql mysql 0 Sep 4 21:03 root -> /
-rw-r--r-- 1 mysql mysql 0 Sep 4 21:03 sched
-r--r--r-- 1 mysql mysql 0 Sep 4 21:03 schedstat
-r--r--r-- 1 mysql mysql 0 Sep 4 21:03 sessionid
-rw-r--r-- 1 mysql mysql 0 Sep 4 21:03 setgroups
-r--r--r-- 1 mysql mysql 0 Sep 4 21:03 smaps
-r-------- 1 mysql mysql 0 Sep 4 21:03 stack
-r--r--r-- 1 mysql mysql 0 Sep 4 20:58 stat
-r--r--r-- 1 mysql mysql 0 Sep 4 21:03 statm
-r--r--r-- 1 mysql mysql 0 Sep 4 20:58 status
-r-------- 1 mysql mysql 0 Sep 4 21:03 syscall
dr-xr-xr-x 30 mysql mysql 0 Sep 4 21:03 task
-r--r--r-- 1 mysql mysql 0 Sep 4 21:03 timers
-rw-r--r-- 1 mysql mysql 0 Sep 4 21:03 uid_map
-r--r--r-- 1 mysql mysql 0 Sep 4 21:03 wchan
root@minbox:/proc/1051# ls -al fd
total 0
dr-x------ 2 mysql mysql 0 Sep 4 20:58 .
dr-xr-xr-x 9 mysql mysql 0 Sep 4 20:58 ..
lr-x------ 1 mysql mysql 64 Sep 4 20:58 0 -> /dev/null
l-wx------ 1 mysql mysql 64 Sep 4 20:58 1 -> /var/log/mysql/error.log
lrwx------ 1 mysql mysql 64 Sep 4 20:58 10 -> /var/lib/mysql/ibtmp1
lrwx------ 1 mysql mysql 64 Sep 4 20:58 11 -> /tmp/ibrkMbTv (deleted)
lrwx------ 1 mysql mysql 64 Sep 4 20:58 12 -> /var/lib/mysql/mysql/innodb_table_stats.ibd
lrwx------ 1 mysql mysql 64 Sep 4 20:58 13 -> /var/lib/mysql/mysql/plugin.ibd
lrwx------ 1 mysql mysql 64 Sep 4 20:58 14 -> /var/lib/mysql/mysql/innodb_index_stats.ibd
lrwx------ 1 mysql mysql 64 Sep 4 20:58 15 -> /var/lib/mysql/mysql/gtid_executed.ibd
lrwx------ 1 mysql mysql 64 Sep 4 20:58 16 -> /var/lib/mysql/mysql/server_cost.ibd
lrwx------ 1 mysql mysql 64 Sep 4 20:58 17 -> /var/lib/mysql/webstore/orders.ibd
lrwx------ 1 mysql mysql 64 Sep 4 20:58 18 -> /var/lib/mysql/mysql/engine_cost.ibd
lrwx------ 1 mysql mysql 64 Sep 4 20:58 19 -> /var/lib/mysql/webstore/customers.ibd
l-wx------ 1 mysql mysql 64 Sep 4 20:58 2 -> /var/log/mysql/error.log
lrwx------ 1 mysql mysql 64 Sep 4 20:58 20 -> socket:[16805]
lrwx------ 1 mysql mysql 64 Sep 4 20:58 21 -> socket:[16806]
lrwx------ 1 mysql mysql 64 Sep 4 20:58 22 -> /var/lib/mysql/mysql/help_category.ibd
lrwx------ 1 mysql mysql 64 Sep 4 20:58 23 -> /var/lib/mysql/mysql/db.MYI
lrwx------ 1 mysql mysql 64 Sep 4 20:58 24 -> /var/lib/mysql/mysql/help_relation.ibd
lrwx------ 1 mysql mysql 64 Sep 4 20:58 25 -> /var/lib/mysql/mysql/db.MYD
lrwx------ 1 mysql mysql 64 Sep 4 20:58 26 -> /var/lib/mysql/mysql/help_keyword.ibd
lrwx------ 1 mysql mysql 64 Sep 4 20:58 27 -> /var/lib/mysql/mysql/user.MYI
lrwx------ 1 mysql mysql 64 Sep 4 20:58 28 -> /var/lib/mysql/mysql/user.MYD
lrwx------ 1 mysql mysql 64 Sep 4 20:58 29 -> /var/lib/mysql/mysql/event.MYI
lrwx------ 1 mysql mysql 64 Sep 4 20:58 3 -> /var/lib/mysql/ib_logfile0
lrwx------ 1 mysql mysql 64 Sep 4 20:58 30 -> /var/lib/mysql/mysql/servers.ibd
lrwx------ 1 mysql mysql 64 Sep 4 20:58 31 -> /var/lib/mysql/mysql/event.MYD
lrwx------ 1 mysql mysql 64 Sep 4 20:58 32 -> /var/lib/mysql/mysql/time_zone.ibd
lrwx------ 1 mysql mysql 64 Sep 4 20:58 33 -> /var/lib/mysql/mysql/time_zone_leap_second.ibd
lrwx------ 1 mysql mysql 64 Sep 4 20:58 34 -> /var/lib/mysql/mysql/time_zone_name.ibd
lrwx------ 1 mysql mysql 64 Sep 4 20:58 35 -> /var/lib/mysql/mysql/time_zone_transition.ibd
lrwx------ 1 mysql mysql 64 Sep 4 20:58 36 -> /var/lib/mysql/mysql/time_zone_transition_type.ibd
lrwx------ 1 mysql mysql 64 Sep 4 20:58 4 -> /tmp/ibhkbgQS (deleted)
lrwx------ 1 mysql mysql 64 Sep 4 20:58 5 -> /tmp/ibY0mtIc (deleted)
lrwx------ 1 mysql mysql 64 Sep 4 20:58 6 -> /tmp/ibOzUGAw (deleted)
lrwx------ 1 mysql mysql 64 Sep 4 20:58 7 -> /tmp/ibrDPXqa (deleted)
lrwx------ 1 mysql mysql 64 Sep 4 20:58 8 -> /var/lib/mysql/ib_logfile1
lrwx------ 1 mysql mysql 64 Sep 4 20:58 9 -> /var/lib/mysql/ibdata1

So if we're looking for data about the database users, those mysql/user files look promising.  There's also a third file called .frm that didn't happen to be open when we looked:
root@minbox:~# cd /var/lib/mysql/mysql
root@minbox:/var/lib/mysql/mysql# ls -al user*
-rw-r----- 1 mysql mysql 10816 Aug 30 15:55 user.frm
-rw-r----- 1 mysql mysql 652 Aug 30 15:55 user.MYD
-rw-r----- 1 mysql mysql 4096 Aug 30 15:55 user.MYI
There is documentation available for the .frm and .myd database file formats, but in general a tool like grep is useful for this sort of thing:
root@minbox:/var/lib/mysql/mysql# grep -a -i storemaster user.*
user.MYD:w�� localhostrootmysql_native_password)*9383B6232A1FCC15D9B8690D7E031D916F59764EY�X|�� localhost mysql.sysmysql_native_password)*THISISNOTAVALIDPASSWORDTHATCANBEUSEDHEREY�Pp��� localhostdebian-sys-maintmysql_native_password)*27EA63E48B6EF2E052FA0550F653A6F378ADE789Y�X~�� localhost
storemastermysql_native_password)*mysql.sessionmysql_native_password)*THISISNOTAVALIDPASSWORDTHATCANBEUSEDHEREY�Pp
mysql.session�mysql.sys|�root�debian-sys-maint��
storemaster�
The data comes out in a little easier to understand format from "strings":
root@minbox:/var/lib/mysql/mysql# strings user.MYD
localhost
root
mysql_native_password)
*9383B6232A1FCC15D9B8690D7E031D916F59764E
localhost mysql.sys
mysql_native_password)
*THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE
localhost
debian-sys-maint
mysql_native_password)
*27EA63E48B6EF2E052FA0550F653A6F378ADE789
localhost
storemaster
mysql_native_password)
*ED5F95C9F5236B7D788C86BB58A92308B04EB5CF
localhost
mysql.session
mysql_native_password)
*THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE
The database user passwords aren't in plaintext (for this database!), but there are two promising strings of hex values, both 40 chars long.  I'll refer to these as '93' (root password) and 'ED' (storemaster password) for short.
root@minbox:/var/lib/mysql/mysql# echo -n '9383B6232A1FCC15D9B8690D7E031D916F59764E' | wc
0 1 40
root@minbox:/var/lib/mysql/mysql# echo -n 'ED5F95C9F5236B7D788C86BB58A92308B04EB5CF' | wc
0 1 40
40 chars is the length of a SHA-1 hash, so the database is protecting its passwords using a Password-Based Key Derivation Function based on SHA-1, probably PBKDF2:
root@minbox:/var/lib/mysql/mysql# echo -n 'foo' | sha1sum
0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33 -
root@minbox:/var/lib/mysql/mysql# echo -n '0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33' | wc
0 1 40

Recovering Passwords from Hashes with Hashcat

A tool like hashcat can recover passwords given only their hashes.  I used a raw SHA-1 hash for 'foo' above, so I use -m 100 (raw SHA1 hash) and -a 3 (bruteforce, lowercase only by default):
~/class/cs493/hashcat/hashcat-3.6.0$ ./hashcat -a 3 -m 100 0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33
0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33:foo
It's a little tricker to bruteforce hashes that include mixed case or numbers; hashcat has a counterintuitive 'charset' interface for this:
olawlor@omlet:/me/olawlor/class/cs493/hashcat/hashcat-3.6.0$ echo -n "h3rpDe" | sha1sum
4778e55f074157af15ce67aff6b4554802e21dfc -
olawlor@omlet:/me/olawlor/class/cs493/hashcat/hashcat-3.6.0$ time ./hashcat -a 3 -1 '?l?u?d' -m 100 4778e55f074157af15ce67aff6b4554802e21dfc '?1?1?1?1?1?1?1?1?1?1' -i
hashcat (v3.6.0) starting...

4778e55f074157af15ce67aff6b4554802e21dfc:h3rpDe

Session..........: hashcat
Status...........: Cracked
Hash.Type........: SHA1
Hash.Target......: 4778e55f074157af15ce67aff6b4554802e21dfc
Time.Started.....: Mon Sep 4 22:41:01 2017 (1 sec)
Time.Estimated...: Mon Sep 4 22:41:02 2017 (0 secs)
Guess.Mask.......: ?1?1?1?1?1?1 [6]
Guess.Charset....: -1 ?l?u?d, -2 Undefined, -3 Undefined, -4 Undefined
Guess.Queue......: 6/10 (60.00%)
Speed.Dev.#1.....: 3634.1 MH/s (13.73ms)
Recovered........: 1/1 (100.00%) Digests, 1/1 (100.00%) Salts
Progress.........: 6851395584/56800235584 (12.06%)
Rejected.........: 0/6851395584 (0.00%)
Restore.Point....: 1572864/14776336 (10.64%)
Candidates.#1....: GSBoL1 -> H4rxGi
HWMon.Dev.#1.....: Temp: 57c Util: 97% Core:1354MHz Mem:3802MHz Bus:8
Recovering that 6 character alphanumeric password only took 6 seconds on my GPU (a 1070), but the time grows exponentially with each additional character.

The hash we extracted for storemaster, ED5F95C9F5236B7D788C86BB58A92308B04EB5CF, is a MySQL password (which is just SHA1(UNHEX(SHA1(password)))).  We can verify the MySQL password formatting by testing it against known plaintext:
olawlor@omlet:/me/olawlor/class/cs493/hashcat/hashcat-3.6.0$ echo -n LMAO | sha1sum | xxd -r -p | sha1sum
ed5f95c9f5236b7d788c86bb58a92308b04eb5cf -

In hashcat, MySQL 4/5 passwords are explicitly supported using the -m 300 argument, but otherwise we make exactly the same call as above:
~/class/cs493/hashcat/hashcat-3.6.0$ time ./hashcat -a 3 -1 '?l?u?d' -m 300 ED5F95C9F5236B7D788C86BB58A92308B04EB5CF '?1?1?1?1?1?1?1?1?1?1' -i 
hashcat (v3.6.0) starting...

ed5f95c9f5236b7d788c86bb58a92308b04eb5cf:LMAO

Session..........: hashcat
Status...........: Cracked
Hash.Type........: MySQL4.1/MySQL5
Hash.Target......: ed5f95c9f5236b7d788c86bb58a92308b04eb5cf
Time.Started.....: Mon Sep 4 22:48:12 2017 (0 secs)
Time.Estimated...: Mon Sep 4 22:48:12 2017 (0 secs)
Guess.Mask.......: ?1?1?1?1 [4]
Guess.Charset....: -1 ?l?u?d, -2 Undefined, -3 Undefined, -4 Undefined
Guess.Queue......: 4/10 (40.00%)
Speed.Dev.#1.....: 31993.6 kH/s (0.35ms)
Recovered........: 1/1 (100.00%) Digests, 1/1 (100.00%) Salts
Progress.........: 10009776/14776336 (67.74%)
Rejected.........: 0/10009776 (0.00%)
Restore.Point....: 0/238328 (0.00%)
Candidates.#1....: Lari -> TXvZ
HWMon.Dev.#1.....: Temp: 57c Util:100% Core:1455MHz Mem:3802MHz Bus:8

That only took 4 seconds, because the password is so short.  (If you use the wrong password format, it either won't start, or it never finds a match.)

The MySQL root password hash we extracted above is 9383B6232A1FCC15D9B8690D7E031D916F59764E.  The underlying password is actually 9 characters long, and alphanumeric, so this command will recover it ... eventually:
./hashcat -a 3 -1 '?l?u?d' -m 300 9383B6232A1FCC15D9B8690D7E031D916F59764E '?1?1?1?1?1?1?1?1?1?1' -i 
The downside is the time it takes to find long passwords this way.   Here's hashcat's estimated times to brute force different length MySQL passwords on my laptop GPU, a decent NVIDIA GTX 1070 that can do about 2 billion (!) mysql hashes per second:
Password Length
Brute Force Time
5 alphanumeric
3 seconds
6 alphanumeric
30 seconds
7 alphanumeric
30 minutes
8 alphanumeric
1 day
9 alphanumeric
80 days
10 alphanumeric
13 years

Clearly, longer passwords are better!  Since bruteforce rapidly becomes infeasible, it soon becomes more useful to feed relevant sources of possible passwords, e.g., the dictionary, the phonebook, then the entire company facebook presence into hashcat.

If you don't have OpenCL, John the Ripper will run on the CPU, and is still fairly quick for short passwords.  You'll need the "jumbo" version to crack raw hashes; to build this you'll need to download the git repo (git clone https://github.com/magnumripper/JohnTheRipper), install OpenSSL-dev (sudo apt install libssl-dev), and then configure and make.  The program comes out in JohnTheRipper/run/john, but fails with missing AVX2 support unless you first "export CPUID_DISABLE=1".  Then, finally, you can run:
run/john --encoding=ascii --format=Raw-SHA1 --min-length=1 --max-length=4  hash_in_a_file.txt
You actually display the cracked passwords with:
run/john hash_in_a_file.txt --show