Direct Operating System Access via Syscalls

CS 301: Assembly Language Programming Lecture, Dr. Lawlor

Normally, to interact with the outside world (files, network, etc) from assembly language, you just call an existing function, usually the exact same function you'd call from C or C++.  But sometimes, such as when you're implementing a C library, when there is no C library call to access the functionality you need, or when your code needs to operate behind enemy lines, you want to talk to the OS kernel directly. 

In fact, to get anything dangerous done, like talking directly to the screen or keyboard, you *must* talk to the OS kernel, because the hardware itself has been instructed to not let you access those things.  On our x86 machines, the IOPL (I/O Privilege Level) flag values range from 0 (kernel mode, anything is allowed) to 3 (user mode, no hardware access is allowed).  The hardware lets you give up IOPL rights easily, but there's only one way to gain IOPL rights: make a system call.  This makes system calls the primary gateway to kernel functionality. 

Linux 64-bit System Calls

On a 64-bit Linux machine, there's a special instruction "syscall" to make system calls: a request to the kernel to do something.

You identify which system call you'd like to make by loading a syscall number into register rax.  A full list of syscall numbers is here, or on  /usr/include/asm/unistd_64.h.

Here we're calling the magic operating system call "fork();", which creates a new process.

mov rax,57 ; the system call number of "fork".
syscall ; Issue the system call
ret ; return rax directly

(Try this in NetRun now!)

Syscall parameters are passed in registers rdi, rsi, rdx, r10, r8, r9, which you'll notice is *somewhat* like a function call but with slightly different registers!  The return value, normally an integer error code, is returned in rax.

Here's a direct syscall for: write(1,"Yo\n",3);
; "syscall" instruction call numbers are listed in "asm/unistd_64.h"
mov rax,1 ; the (syscall) system call number of "write".
mov rdi,1 ; first parameter: 1, the stdout file descriptor
mov rsi,myStr ; data to write
mov rdx,3 ; bytes to write
syscall ; Issue the system call
; leave syscall's return value in rax

section .data
db "Yo",0xa

section .text

(Try this in NetRun now!)

Linux 32-bit Syscalls

The older 32-bit syscall interface uses a software interrupt, "int 0x80".  As with the 64-bit interface, register rax describes what to do (open a file, write data, etc).  The registers ebx, ecx, edx, esi, and edi have the parameters describing how to do it.  This register-based parameter passing is similar to how we call functions in 64-bit x86, but using different register numbers and smaller 32-bit registers, and the Linux kernel allows the use of this convention both in 32 and 64 bit mode.  

"int 0x80" uses a special x86 instruction to do this called "int", short for interrupt.  More generally, an "interrupt" is a hardware feature where the CPU saves what it was doing and does something else for a while.  For example, every time a packet arrives from the network, the network card will interrupt the CPU, so some low-level operating system code can look at the packet and decide if it should keep running the current program, or switch to some new program (such as the web browser, or a network server).  Handling interrupts is the central responsibility of the operating system.

The operating system allows you to perform a wide variety of almost magical features.  For example, Linux syscall number 2 is "fork", which creates a complete duplicate of your process.

; System call numbers are listed in "asm/unistd.h"
mov rax,2 ; the system call number of "fork"
int 0x80 ; Issue "fork" system call (Linux 32-bit interface)


(Try this in NetRun now!)

The return value comes back in rax.  If it's negative, that indicates an error, which are listed in errno.h.

Konstantin Boldyshev has a good writeup and examples of Linux, BSD, and BeOS x86 syscalls, and a list of common Linux syscalls.  The full list of Linux syscalls is in /usr/include/asm/unistd_32.h.  Here's a netrun-friendly version of his Linux example:

push rbx  ; <- we'll be using ebx below

; System calls are listed in "asm/unistd.h"
mov rax,4 ; the system call number of "write".
mov rbx,1 ; first parameter: 1, the stdout file descriptor
mov rcx,myStr ; data to write
mov rdx,3 ; bytes to write
int 0x80 ; Issue the system call

pop rbx ; <- restore ebx to its old value

section .data
db "Yo",0xa

(Try this in NetRun now!)

This same "int 0x80" approach works on both 32-bit or 64-bit Linux systems, *BUT* since the above is a 32-bit interface, any parameters you pass need to fit in 32 bits.  In particular, if you have a pointer to some data on the stack, the pointer (0x7fffffffbcde or something) is too big to fit, so you get "bad address" errors (EFAULT, error -14).  

Syscalls on non-Linux systems

Other operating systems such as BSD UNIX store syscall parameters on the stack, like the 32-bit x86 function call interface.

In ancient 16-bit DOS mode, you access DOS functionality via INT 0x21, with the equivalent of a system call number in register AH.

In a PC BIOS boot block, you can access BIOS functionality via several interrupts, including INT 0x10 for screen access, or INT 0x13 for disk access.

On modern 32-bit Windows, you use the sysenter instruction, with the system call number in eax (table of Windows system call numbers).  Older pre-XP windows used interrupt 0x2E.