Encoding instructions

CS 301 / Prof. Chris Hartman

 

The parts of an instruction:

First, you need to have some idea of the parts of the instruction. An instruction is formed of an opcode and from zero to two operands. Abbreviations for the types of operands are found in Table 1.

 

When encoding/decoding an instruction, the first byte (the opcode) determines the instruction (almost — as we will see soon, some forms of some instructions like (INC and DEC) or (ROR and ROL and RCR and RCL) share their first byte).

 

If necessary, the second byte is the ModR/M byte, which encodes effective addresses (that is, indirect memory references). This byte can also include an opcode extension, that is, the part that lets us differentiate between INC and DEC (for instance) when their encodings share the first byte. In some cases this byte is also used to specify register operands.

 

For complicated effective addresses (for instance, MOV EAX, [2*ESI+EDI+FE20]) the ModR/M byte needs to be extended by the SIB byte (scale/index/base), which encodes the index register (ESI in the example), the scale (2), and the base register (EDI).

 

Instructions will also often need a memory address or offset (such as the example in the previous paragraph) — these are easy to deal with as they are encoded directly into the instruction stream. You do need to be careful to keep your bytes in order. Remember, the least significant byte goes in the lowest memory location.

 

16-bit and 32-bit processor modes:

You also need to know a little about processor modes. The processor can run in 16-bit mode or 32-bit mode, but changing modes is not efficient (and sometimes not possible) so there is a mechanism to cause single instructions to run in the other mode. (For instance, to “DEC AX” the processor needs to be in 16-bit mode.) The mechanism is to prefix the instruction with the byte 66h (the operand size prefix) to tell the CPU that this instruction should treat registers as if it were in the other mode (that is, if it is in 32-bit mode it will use 16-bit registers and vice versa.) You can also prefix an instruction with the byte 67h (the address size prefix) to tell the CPU to treat registers used as addresses as if it were in the other mode. (This is almost always for the “string” instructions such as MOVSx or LODSx or any PUSH/POP type instructions. They use si/esi, di/edi or sp/esp as addresses). Since we will be assuming the processor is in 32-bit mode, you can just remember that you probably need to do something special if you want to use a 16-bit register.

 

Understanding the NASM instruction set reference

So, you look up an instruction in appendix A of the NASM manual and it tells you what the opcode is. But what are all of these other codes? We’ll treat them one at a time:

 

HHh +r — There is a range of opcodes for this instruction. It can operate on any general purpose register. Add the code for the register (found in Table 2) to the hex number HH to get the opcode.

 

HHh +cc — Branching or jumping instructions have a range of opcodes depending on the condition code. Add the value for the condition code (from Table 3) to the hex number HH to get the opcode.

 

/# — A slash followed by a digit (from 0 to 7) means this instruction will be encoded with an MIB byte. The binary value of the digit goes in the spare/register field (bits 5-3) of the MIB byte. See below for MIB encoding.

 

/r — One of the operands is r/m# or mem#, the other is reg#. The instruction should be encoded with an MIB byte to describe the r/m# or mem# operand. The code for the reg# operand (from Table 2) is encoded in the spare/register field (bits 5-3).

 

ib/iw/id — One of the operands is imm#. It should be encoded directly into the instruction stream (little endian — that is, least significant byte first).

 

rb/rw/rd — One of the operands is imm#. The difference between this value and the address of the end of the instruction should be encoded directly into the instruction stream (little endian — that is, least significant byte first). (In other words, this is a loop or jump instruction and the address is an offset of how many bytes to jump.)

 

o32/o16 — This instruction needs the operand size prefix if the processor is not in the specified mode.

 

a32/a16 — This instruction needs the address size prefix if the processor is not in the specified mode.

 

 

 

 

 

 

 

 

 

mod

spare/register

r/m

 The ModR/M byte

 

 

The ModR/M byte has of three fields: mod (bits 7-6), spare/register (bits 5-3), and r/m (bits 2-0). It is used to encode an effective address or a direct register reference.

 

This is how the ModR/M byte works when the processor is in 32-bit mode. In 16-bit mode the rules are entirely different.

 

The mod field tells you whether this is encoding a direct register reference (11) or an effective address (00,01, or 10). For effective addresses, the length of the displacement (memory offset) field is shown: (00 – 0 bytes of displacement, 01 – 1 byte of displacement, 10 – 4 bytes of displacement).

 

The spare/register field contains either an opcode extension or the register code for another operand (see /r above).

 

The r/m field encodes which register is used. For a direct register reference (mod=11) look up the register code in Table 2. (The size of the register will already be known from the opcode.) For an effective address of the form [reg + displacement], do the same thing, unless the register is ESP. For more complicated effective addresses (such as [ESP+2*ESI+FACEh]) put 100 (which would normally encode [ESP + displacement]) and include an SIB byte.

 

There are two special case exceptions to the above rules. If you look carefully, there is no way to encode the effective addresses [displacement] or [displacement + scale*index]. These will be handled after discussion of the SIB byte, but in the meantime, be aware that encoding [EBP] and [EBP + scale*indexreg] will not follow the above rules.

 

 

 

 

 

 

 

 

 

 

scale

index

base

The SIB byte

 

 

The SIB byte has three fields: scale (bits 7-6), index (bits 5-3) and base (bits 2-0).  It is necessary when encoding effective addresses of the form [scale*index + base + displacement]. It’s presence is indicated by the ModR/M byte showing an effective address (mod=00,01, or 10) and a reg field of 100.

 

The scale field is a multiplier for the register encoded in the index field. The multiplier is 2bb where bb are the digits in the field. In other words 00 means times 1, 01 means times 2, 10 means times 4, and 11 means times 8.

 

The index field describes the index register encoded as in Table 2.

 

The base field describes the base register encoded as in Table 2.

 

Exceptions to the above rules

When the above rules would indicate [ebp] (mod=00 implying no displacement and r/m=5 implying ebp) the address is instead [disp32] and a four byte displacement field is present. (Thus to indicate [ebp] you need to encode [epb+0] where 0 is a one byte displacement.)

 

When the above rules would indicate [ebp + scale*indexreg] (mod=00 implying no displacement, r/m=4 implying an SIB byte, base=4 implying ebp) the address is instead [disp32+scale*index] and there is a four byte displacement field present. (Thus, to indicate [ebp+scale*indexreg] you need to encode [EBP + scale*indexreg + 0] where 0 is a one byte displacement. Alternatively, if scale is 00 meaning times one, you could encode [indexreg + EBP].)

 

Table 1 – abbs. for operand types

reg8

8-bit register

reg16

16-bit register

reg32

32-bit register

imm

immediate operand

imm8

8-bit immediate operand

imm16

16-bit immediate operand

imm32

32-bit immediate operand

mem

generic memory reference

mem16

16-bit memory reference

mem32

32-bit memory reference

r/m8

reg8 or mem8

r/m16

reg16 or mem16

r/m32

reg32 or mem32

Table 2 – Register codes

code

Operand Type

reg8

reg16

reg32

0 or 000

al

ax

eax

1 or 001

cl

cx

ecx

2 or 010

dl

dx

edx

3 or 011

bl

bx

ebx

4 or 100

ah

sp

esp

5 or 101

ch

bp

ebp

6 or 110

dh

si

esi

7 or 111

bh

di

edi

 

Table 3 – Condition codes

code

Meaning

x=0

x=1

 0 or  1   (000x)

O

NO

 2 or  3   (001x)

B, C, NAE

NB, NC, AE

 4 or  5   (010x)

E, Z

NE, NZ

 6 or  7   (011x)

BE, NA

NBE, A

 8 or  9   (100x)

S

NS

10 or 11 (101x)

P, PE

NP, PO

12 or 13 (110x)

L, NGE

NL, GE

14 or 15 (111x)

LE, NG

NLE, G