Background

While working on a method to guard pointers in memory from being accessed by unwanted code, I thought of an idea to utilize exceptions and an exception handler to guard a function from being reverse engineered. This would work at both dynamic and static analysis.

The Idea 💡

Create a map of opcodes, and junk bytes to correlate to each opcode. Then, write over the opcodes in the function with the junk bytes. Once the application is running, an ILLEGAL_INSTRUCTION exception should occur. Once this happens, we catch the exception, fix the opcode on that instruction, setup a SINGLE_STEP exception, then continue execution. This way, the instruction after will raise another exception, allowing us to patch out the fixed opcode with the junk byte from before.

Why is this good? Because it is almost impossible to guess the instruction without the opcode, especially if we threw in some XOR cipher on the remaining bytes. Furthermore, because the attacker does not know the real opcode, they will not be able to determine the control flow of the function. On top of that, it should cause little performance overhead due to the type of exception being caused.

While I did think of this concept by myself, please note that I am not claiming to be the first person who has thought of this. After showing my idea to others, one of them mentioned that this concept was used in an old packer called "Armadillo".

The Problem

When testing began, I realized pretty quick that just creating a map for all opcodes would not work. This is because occasionally the junk byte would cause a different exception, like an ACCESS_VIOLATION. While we could just account for this in our exception handler, I decided ultimately against it for whatever reason. I presume the reason this happens is because by chance the junk byte causes a jmp, mov, etc.

The Solution

I created some code that would swap out the opcode with a random byte, then log the type of exception caused. There is definitely a better way of going about this, but after a solid 10 minuets of research, I could not find any documentation (it definitely exists somewhere). Once I gathered all junk bytes that cause an ILLEGAL_INSTRUCTION exception for each opcode, I created a map with that information.

Proof of Concept

I have provided my exception handler below. You should be able to guess exactly how this works from a quick glance.

LONG WINAPI exception_handler(PEXCEPTION_POINTERS pExceptionInfo)
{
    static DWORD64 old_instruction = 0x0;
    static DWORD64 new_memory = 0x0;
    static DWORD64 next_instruction = 0x0;

    DWORD exceptionCode = pExceptionInfo->ExceptionRecord->ExceptionCode;
    auto XIP = pExceptionInfo->ContextRecord->Rip; // instruction pointer for x64
    
    if (exceptionCode == EXCEPTION_ILLEGAL_INSTRUCTION)
    { 
        // repair bytes
        DWORD oldProtect;
        VirtualProtect((LPVOID)XIP, 0x15, PAGE_EXECUTE_READWRITE, &oldProtect);
        for (size_t i = 0; i < 0x15; i++)
        {
            BYTE a1 = opcode_map[*(BYTE*)(XIP + i)];
            if (a1)
                *(BYTE*)(XIP + i) = opcode_map[*(BYTE*)(XIP + i)];
        }
        VirtualProtect((LPVOID)XIP, 0x15, oldProtect, &oldProtect);
  
        pExceptionInfo->ContextRecord->EFlags |= 0x100; // cause single step

        old_instruction = XIP;

        return EXCEPTION_CONTINUE_EXECUTION;
    }
    else if (pExceptionInfo->ExceptionRecord->ExceptionCode == STATUS_SINGLE_STEP)
    {
        // re corrupt
        DWORD oldProtect;
        VirtualProtect((LPVOID)old_instruction, 0x15, PAGE_EXECUTE_READWRITE, &oldProtect);
        for (size_t i = 0; i < 0x15; i++)
        {
            BYTE corrupted_byte = fetch_corrupted_byte(*(BYTE*)(old_instruction + i));
            if (corrupted_byte)
                *(BYTE*)(old_instruction + i) = corrupted_byte;
        }
        VirtualProtect((LPVOID)old_instruction, 0x15, oldProtect, &oldProtect);

        return EXCEPTION_CONTINUE_EXECUTION;
    }

    return EXCEPTION_CONTINUE_SEARCH;
}

Defeating

One way to defeat this is to have another exception handler set up to catch SINGLE_STEP first, then dump the instruction prior. Then recreate the map, allowing you to clean up the binary on disk with the correct instructions. After asking around, someone suggested to move the exception handler to kernel with callbacks, or hooking lower down before KiUserExceptionDispatcher even gets the exception, giving you a better chance of catching the exception before an attacker. While this is not a 100% guarantee, it will definitely up the skill level needed to reconstruct the map.

Use Case

My proof of concept was to use this method on every function that was called rarely or only once. I wanted to utilize this in DRM software, ensuring the application could not start without verifying the device is allowed access to the application.

Combining this idea with not-byfron, I believe it would provide a great way to discourage cheat development and cracking for games.