I've been trying to get my head around the x86 instruction format lately, and have been looking at various disassemblers to understand how it fits together (as well as writing my own as an exercise).

I've noticed that the implementation in LLVM seems to disagree with other assemblers/docs in this detail.

Consider the following instruction bytes (compiled for x86-64)
43 80 04 05 78 56 34 12 00

One online disassembler (http://onlinedisassembler.com/odaweb/) gives the following:
add    BYTE PTR [r8*1+0x12345678],0x0

which after much consideration I agree with.


The crux here is that the sib byte (05) corresponds to the special case where the base register is ignored and a 4 byte displacement follows the SIB byte.  The docs (both Intel and AMD) are rather unclear on this, but my understanding is that this special case should apply regardless of whether the REX prefix is present; as does the special case for the MODR/M byte that indicates the SIB byte is required.  My understanding is that the REX prefix shouldn't affect the length of the instruction.

However, the code in LLVM (from x86DissasemblerDecoder.c, reproduced here) takes the REX.B prefix into account before performing the test, and so does not read the displacement:


   base = baseFromSIB(insn->sib) | (bFromREX(insn->rexPrefix) << 3);

   switch (base) {
   case 0x5:
      ...


I think, much like the case in the readModRM function the switch should also allow 0xd as so:


   base = baseFromSIB(insn->sib) | (bFromREX(insn->rexPrefix) << 3);

   switch (base) {
   case 0x5:
   case 0xd:   /* in case REXW.b is set */
      ...