Malware Development Essentials

Portable Executable (PE) File Format

The Portable Executable (PE) format is a file format used for executables, object code, DLLs, FON font files, and other file types in 32-bit and 64-bit versions of Windows operating systems. The PE format is a data structure that encapsulates the information required by the Windows OS loader to manage the executable code. This includes dynamic library references for linking, API export and import tables, resource management data, and thread-local storage (TLS) data. In NT-based operating systems, the PE format is used for EXE, DLL, SYS (device driver), and other file types.

Key PE Sections

  • .text: Contains executable code (useful for malware developers).

  • .rdata: Contains read-only data.

  • .data: Contains the application’s global variables (useful for malware developers).

  • .pdata: Contains exception handling information.

  • .rsrc: Contains resources such as objects, pictures, icons, manifest files, embedded EXEs, and DLLs (useful for malware developers).

  • .reloc: Contains relocation information, allowing Windows loaders to properly load DLLs or EXEs into memory.

PE Bear

A Portable Executable reversing tool with a user-friendly GUI.

PE Bear Software

EXE vs. DLL

  • EXE (Executable)

    • Programs that can be loaded into memory as independent processes.

    • Require a main function.

  • DLL (Dynamic Link Library)

    • PE modules loaded into an existing process.

    • Provide functionalities required by the process.

    • Require a DllMain function.

The only difference between an EXE and a DLL is how they are invoked in a program.

Payload Storage

Payload Storage in PE Files

1. Dropper

  • A dropper is a specialized program or file (e.g., PDF, DOCX) used to deliver a payload to a target machine.

  • Its primary function is to bypass detection and execute the payload on the victim's system.

2. Payload

  • A sophisticated program designed to perform malicious actions, such as:

    • Deploying malware.

    • Establishing a reverse shell.

    • Disabling antivirus (AV).

    • Downloading/executing additional programs.

    • Escalating privileges.

3. Storing Payload in PE Sections

Payloads can be embedded in different sections of a Portable Executable (PE) file:

Section

Storage Method

Use Case

.text

Stored as local variables inside functions.

Executable code region; useful for hiding small payloads.

.data

Stored as global variables.

Stores writable data; payload remains accessible throughout execution.

.rsrc

Embedded as resources (e.g., icons, binaries).

Evades detection by storing payload as non-executable data.

Generate payload using MSFVENOM

Store Payload .text section

File: compile.bat

Payload is defined as a local variable inside main function.

Store Payload .data section

File: compile.bat

Payload is defined as a global variable.

Store Payload .rsrc section

Payload is embedded as a resource (e.g., fake icon file).

File: resources.rc

File: resources.h

File: compile.bat

Payload Encoding and Encryption

The purpose is to evade AV/EDR and make it difficult to detect

Difference between encoding and encryption

Encode: The purpose is to transform data using a public schema to and make it easy to process by software. (To decode needs to know the decode type)

  • Base64

Encryption: The purpose is to transform data using an encryption algorithm in order to keep it secret from others (To decrypt needs Encryption type and key)

  • XOR

  • AES

Base64 encoding

File: compile.bat

XOR encryption

File: xor_encrypt.py

File: compile.bat

AES Encryption

File: aes_encrypt.py

File: compile.bat

Function Call Obfuscation

Function call obfuscation is a technique used in malware development to hide calls to external functions (usually from Windows DLLs) in order to evade detection by antivirus (AV) software.

How does a normal executable work?

  • EXE or DLL files use external functions from system DLLs like kernel32.dll, user32.dll, etc.

  • These external functions are imported normally, and their names appear clearly in the Import Address Table (IAT).

  • AV engines analyze these imports to detect suspicious or malicious behavior.

How does AV detect malware?

AV software inspects imported DLLs and their functions. If the executable uses functions often seen in malware—such as:

  • VirtualAlloc

  • WriteProcessMemory

  • CreateRemoteThread

...then the AV might flag the binary as suspicious or malicious.

How Does Function Call Obfuscation Help?

To evade AV detection, malware developers avoid importing sensitive functions directly. Instead, they:

  • Resolve function addresses at runtime using Windows API functions:

    • GetModuleHandle("DLL_Name"): Returns a handle to a loaded DLL.

    • GetProcAddress(handle, "FunctionName"): Returns the address of a function in the specified DLL.

Listing imports using dumpbin

This tells AVs that VirtualProtect is used—which can be suspicious.

How to search for function (E.x. VirtualProtect)

  1. Example: msdn virtualprotect

  2. Copy the function structure

E.x. VirtualProtect Structure

  1. First, define a global function pointer:

Then resolve it at runtime:

The problem here is the VirtualProtect into the code and it can be detectable, to fix it use XOR encryption.

Obfuscating Strings using XOR encryption and CyberChef

Site: CyberChef

Key: RANDOMKEY

Input: FUNCTION_NAME

OLD CODE WITHOUT CALL OBFUSCATION

This code calls VirtualProtect directly, exposing its name in the binary/

NEW CODE USING FUNCTION CALL OBFUSCATION

This version hides the call to VirtualProtect using XOR and dynamic resolution.

Using Function Call Obfuscation:

  • Prevents AV from easily detecting sensitive API calls.

  • Allows dynamic resolution of Windows API functions at runtime.

  • Can be extended to any function: CreateRemoteThread, LoadLibrary, etc.

  • Increases stealth of malware or offensive tools.

Code Injection

Payload injection is a technique used in malware development where a malicious code payload is inserted into a legitimate process or system.

Most Common Combination

  • VirtualAllocEx: Allocates memory in the remote process.

  • WriteProcessMemory: Writes the shellcode into the allocated memory.

  • CreateRemoteThread: Starts a new thread in the remote process to execute the shellcode.

DLL Injection

  • LoadLibrary: The address of this function is obtained in dropper.exe using GetProcAddress.

  • VirtualAllocEx: Called in the target process (e.g., outlook.exe) to allocate space for the DLL path.

  • WriteProcessMemory: Writes the DLL path into the allocated memory.

  • CreateRemoteThread: Executes LoadLibrary with the path to the DLL as its argument, effectively loading the malicious DLL into the target process.

File: implantDLL.cpp

File: injectDLL.cpp

File: compileDLL.bat

File: CompileInjector.bat

Hiding console

Using FreeConsole(); but the problem is the console will pop up a milliseconds.

Use a WinMain instead of main method and compile using

Right Way to hide console

Will pop up the console even using FreeConsole

And should compile the program as GUI program using /SUBSYSTEM:WINDOWS

Final project

  1. Create a dropper where the shellcode is loading from .rsrc

  2. Encrypt/Decrypt using XOR encryption

  3. Inject the shellcode into explorer.exe or notepad.exe

  4. Get rid of console windows (pop up)

Payload:

File: compile.bat

File: resources.rc

File: resources.h

File: xorencrypt.py

File: implant.cpp

Bypassing AV (Detectable)

Implement function call obfuscation to VirtualAllocEx, WriteProcessMemory, CreateRemoteThread .

Binary Analysis

Last updated