Preprocessing and AST Generation, and Object Code

Intuition

The C++ compilation pipeline transforms human-readable source code into machine-executable binary through a series of stages. The preprocessor is a text manipulation engine that handles file inclusion and macros before the compiler understands C++ syntax. The frontend then parses tokens into an abstract syntax tree, validates semantics, and generates intermediate representation. Understanding these stages helps you debug cryptic compiler errors, optimise build times, and appreciate why modules represent such a significant improvement over the textual inclusion model.

Understanding the translation stages are critical for debugging compilation errors, optimizing build Times, and understanding. The pipeline processes one Translation Unit (TU) at a time. A TU is the output of the Preprocessor (a single source file plus all included headers).

Stage 1: The Preprocessor

The preprocessor is a legacy text-manipulation engine inherited from C. It operates on source text before the compiler understands C++ syntax. It does not know what a class or function is; it Only knows tokens.

Mechanics

1. File Inclusion (#include): The preprocessor locates the target file and recursively copy-pastes its entire contents into the current file. This explains why C++ compilation is slow: a single .cpp file often expands to tens of thousands of lines of code after headers are merged.
2. Macro Expansion (#define): Token replacement. This happens blindly, disregarding scope or namespaces.
3. Conditional Compilation (#ifdef): Text is stripped or kept based on conditions.

Inspection

To debug macro expansion or header inclusion issues, you can stop the pipeline after preprocessing.

## -E: Run preprocessor only
## -P: Disable linemarker generation (cleaner output)
clang++ -E -P main.cpp -o main.i

:: /P: Preprocess to a file (.i)
cl /P main.cpp

The output (main.i) is the exact text the compiler frontend receives. If a syntax error occurs “inside a macro,” inspecting this file reveals the generated code causing the issue.

Stage 2: The Frontend (AST Generation)

Once the preprocessor generates the TU, the compiler frontend (Clang/GCC) takes over. This stage Validates the code against the C++ Language Standard.

1. Lexical Analysis (Tokenization)

The source stream is converted into a sequence of tokens (kw_int``identifier_main``l_paren r_paren).

2. Syntax and Semantic Analysis

The tokens are assembled into an Abstract Syntax Tree (AST).

• Syntax: Does the code follow the grammar? (e.g., matching braces, semicolons).
• Semantics: Does the code make sense? (e.g., type checking, overload resolution, access control).

Template Note: At this stage, templates are parsed for basic syntax, but they are not fully Instantiated until they are used.

Inspection (Clang)

Clang allows direct visualization of the AST. This is invaluable for understanding how the compiler Interprets complex templates or precedence rules.

# -Xclang -ast-dump: Dump the AST to stdout
# -fsyntax-only: Do not generate code
clang++ -Xclang -ast-dump -fsyntax-only main.cpp

Example Output:

FunctionDecl 0x... <line:3:1, line:5:1> line:3:5 main 'int ()'
`-CompoundStmt 0x... <col:12, line:5:1>
  `-ReturnStmt 0x... <line:4:5, col:12>
    `-IntegerLiteral 0x... <col:12> 'int' 0

Stage 3: Intermediate Representation (IR)

The AST is translated into an Intermediate Representation. This is an architecture-agnostic Assembly language. This is where the majority of optimizations (inlining, loop unrolling, Vectorization) occur.

• GCC: Uses GIMPLE and RTL.
• LLVM (Clang): Uses LLVM IR (.ll).

LLVM IR is a Static Single Assignment (SSA) based language. It models an infinite-register machine.

Inspection

# -emit-llvm: Generate LLVM bitcode/text
# -S: Output as text (assembly/IR) rather than binary
# -O3: Apply optimizations to the IR
clang++ -S -emit-llvm -O3 main.cpp -o main.ll

# -fdump-tree-all: Dumps internal tree representations
g++ -fdump-tree-all main.cpp

Reading the optimized IR confirms whether high-level C++ abstractions (like std::vector or Lambdas) were successfully compiled away into efficient scalar operations (“Zero-Cost Abstractions”).

Stage 4: The Backend (Code Generation)

The Backend maps the generic IR to the specific Instruction Set Architecture (ISA) of the target Machine (x86_64, ARM64, WASM).

1. Instruction Selection: Mapping IR operations to machine opcodes.
2. Register Allocation: Mapping infinite IR registers to the limited physical CPU registers (RAX, RDI, etc.).
3. Instruction Scheduling: Reordering instructions to optimize pipeline throughput.

Output: Assembly

To see the final assembly before it is encoded into binary:

# -S: Compile only; do not assemble or link
# -masm=intel: Use Intel syntax (dest, source) instead of AT&T
g++ -S -masm=intel -O3 main.cpp -o main.s

Stage 5: The Object File

The Assembler converts the textual assembly (.s) into a relocatable object file (.o or .obj). This file contains machine code, but it is not executable. It contains placeholders for memory Addresses (symbols) that are outside the current TU.

Binary Formats

• ELF (Executable and Linkable Format): Linux, BSD, Android.
• PE/COFF (Portable Executable): Windows.
• Mach-O: macOS, iOS.

Anatomy of an Object File

An object file is divided into Sections. Understanding these is essential for systems Programming and embedded work.

Inspection Tools

Analyze the contents of object files to debug linking errors or binary size bloat.

# Headers and Sections
readelf -h -S main.o

# Symbol Table (Defined vs Undefined)
nm -C main.o
# 'T' = Text (Defined), 'U' = Undefined (Needs Linking)

# Disassembly
objdump -d -C -M intel main.o

# Sections
otool -l main.o

# Symbols
nm -m main.o

# Headers and Sections
dumpbin /HEADERS main.obj

# Symbols
dumpbin /SYMBOLS main.obj

The compiler produces a collection of Object Files. These files contain optimized machine code But rely on “Undefined Symbols” (like std::cout or function calls to other files). They cannot run Yet. Resolving these symbols is the job of the Linker.

Translation Phases Defined by the Standard

The C++ standard [N4950 §5.1.1] defines the translation of a source file into an executable as a Sequence of translation phases (also called “phases of translation”). These are numbered 1 Through 9:

The Preprocessor in Depth

Translation Units and Header Units

A Translation Unit (TU) [N4950 §5.1.1] is the output of phases 1-7: a single preprocessed source File. The compiler processes one TU at a time, which is why the C++ model is called “separate Compilation.” The boundary between TUs is at the .cpp file level. When you compile a.cpp and b.cpp Separately, each is an independent TU. The compiler knows nothing about b.cpp when compiling a.cpp. This is why declarations (in headers) must be consistent across TUs — the linker cannot Detect type mismatches.

Macro Pitfalls

Macros are textual substitution with no understanding of C++ scope, types, or namespaces. This Causes a category of bugs that no other C++ feature can produce. Problem 1: Argument evaluation side effects.

#include <cstdio>
#define MAX(a, b) ((a) > (b) ? (a) : (b))
int main() \{
    int x = 1;
    int y = MAX(x++, x++);  // UB: x++ is evaluated twice
    std::printf("y = %d, x = %d\n", y, x);
    // Possible output: y = 3, x = 4 (x incremented twice in one expression)
\}

Problem 2: Operator precedence.

#include <cstdio>
#define SQUARE(x) x * x
int main() \{
    int result = SQUARE(1 + 2);  // Expands to: 1 + 2 * 1 + 2 = 5, not 9
    std::printf("result = %d\n", result);  // Prints 5
\}

Fixing with extra parentheses (#define SQUARE(x) ((x) * (x))) helps but doesn’t solve the Side-effect problem. The modern C++ solution is to use constexpr functions or inline functions Instead:

constexpr int square(int x) \{ return x * x; \}  // Safe: evaluates argument once

Problem 3: Name shadowing.

#include <vector>
#define VALUE 42
int main() \{
    // If any header or code defines a variable or macro named VALUE,
    // the preprocessor silently replaces it everywhere
    int VALUE = 10;  // Expands to: int 42 = 10; — compile error
\}

Predefined Macros

Compilers define a set of macros that identify the platform, compiler version, and C++ standard. These are essential for writing portable code.

#include <iostream>
int main() \{
    std::cout << "Compiler: "
#ifdef __clang__
              << "Clang " << __clang_major__ << "." << __clang_minor__
#elif defined(__GNUC__)
              << "GCC " << __GNUC__ << "." << __GNUC_MINOR__
#elif defined(_MSC_VER)
              << "MSVC " << _MSC_VER
#endif
              << "\n";
    std::cout << "C++ Standard: " << __cplusplus << "\n";
    // C++11: 201103L, C++14: 201402L, C++17: 201703L, C++20: 202002L, C++23: 202302L
    std::cout << "Platform: "
#ifdef _WIN32
              << "Windows"
#elif defined(__linux__)
              << "Linux"
#elif defined(__APPLE__)
              << "macOS"
#endif
              << "\n";
\}

Important: Never use __cplusplus alone to detect compiler features. Use feature-test macros Defined by the standard (e.g., __cpp_constexpr``__cpp_concepts):

#if __cpp_concepts >= 201907L
    // Concepts are available
    template<typename T>
    concept Addable = requires(T a, T b) \{ a + b; \};
#endif

Include Guards vs #pragma once

Traditional include guards use preprocessor conditionals to prevent multiple inclusion:

// my_header.h
#ifndef MY_HEADER_H
#define MY_HEADER_H
// ... header contents ...
#endif // MY_HEADER_H

#pragma once is a non-standard but universally supported directive that achieves the same result More concisely:

// my_header.h
#pragma once
// ... header contents ...

Both approaches prevent the same header from being included multiple times in a TU. #pragma once Is faster (the preprocessor can short-circuit based on the file’s inode rather than tokenizing to Find the guard), but it is not part of the C++ standard. Every major compiler supports it, and C++20 Modules will eventually eliminate the need for both approaches.

The AST in Depth

How the Compiler Represents C++ Constructs

The AST is a tree where each node represents a language construct. Understanding the AST structure Helps debug surprising compiler behavior.

// input.cpp
const int* p = nullptr;

clang++ -Xclang -ast-dump -fsyntax-only input.cpp

The AST for this declaration includes:

VarDecl 0x... <line:1:1, col:20> col:15 p 'const int *' cinit
|-ValueStmt 0x...
| `-ImplicitCastExpr 0x... 'const int *' <NullToPointer>
|   `-CXXNullPtrLiteralExpr 0x... 'std::nullptr_t'

Note: the type is const int * (pointer to const int), not const int* (const pointer to int). The AST makes the actual type binding unambiguous, even though the human-readable syntax can be Confusing.

Name Mangling

The compiler encodes function signatures, namespaces, and template arguments into a single string Called the mangled name. This is what appears in the object file’s symbol table. Name mangling Is necessary because the linker only sees flat strings, not C++ scope.

// code.cpp
namespace math \{
    int add(int a, int b) \{ return a + b; \}
    double add(double a, double b) \{ return a + b; \}
\}

nm -C code.o
# _ZN4math3addEdd  →  math::add(double, double)
# _ZN4math3addEii  →  math::add(int, int)

The mangling scheme is compiler-specific (Itanium ABI for GCC/Clang, MSVC ABI for MSVC). Use nm -C (demangle) or c++filt to decode mangled names. The Itanium ABI mangling encodes:

• _Z prefix
• N for nested names
• Length-prefixed identifiers (4math = “math”, 3add = “add”)
• Type encodings (i = int, d = double, St = std::)

c++filt _ZN4math3addEii
# Output: math::add(int, int)

Intermediate Representation in Depth

LLVM IR Structure

LLVM IR is a typed, SSA-based assembly language. Every value is computed exactly once and assigned To a register (%0``%1Etc.). This makes dataflow analysis trivial for the optimizer.

// sum.cpp
int sum(int n) \{
    int total = 0;
    for (int i = 0; i < n; ++i)
        total += i;
    return total;
\}

clang++ -S -emit-llvm -O2 sum.cpp -o sum.ll

Key IR concepts visible in the output:

define i32 @sum(i32 %n) \{
entry:
  %cmp = icmp sgt i32 %n, 0
  %nminus1 = select i1 %cmp, i32 %n, i32 0
  %mul = mul i32 %nminus1, %n
  %div = lshr i32 %mul, 1
  ret i32 %div
\}

The loop has been replaced with n*(n-1)/2 — a textbook example of loop optimization. This is why Reading the optimized IR is critical: it confirms that your high-level abstractions compiled away to Efficient code.

GCC GIMPLE

GCC uses a different IR called GIMPLE. It is a simplified form of C where every statement is an Assignment, a function call, or a branch — no complex expressions.

g++ -fdump-tree-optimized sum.cpp

The output shows the same optimization (loop elimination) in GCC’s representation.

Object Files in Depth

Relocation Entries

When the compiler generates an object file, it cannot know the final addresses of symbols defined in Other translation units. It emits relocation entries — instructions to the linker saying “patch This offset with the final address of symbol X.”

# View relocations in an object file
readelf -r main.o

Each relocation entry specifies:

• Offset: Where in the section to patch.
• Type: How to apply the patch (absolute, PC-relative, GOT-relative, etc.).
• Symbol: Which symbol’s address to use.
• Addend: An optional constant offset. The linker resolves these relocations by computing the final virtual address of each symbol and Writing it into the appropriate location in the machine code.

Common Object File Pitfalls

• Debug builds without -g produce useless backtraces. The compiler emits DWARF debug info (.debug_info``.debug_line) only when -g is specified. Without it, gdb and addr2line cannot map addresses to source lines.
• Not using -fPIC for shared library code. Position-Independent Code generates PC-relative addressing instead of absolute addressing. Without it, the loader must perform text relocations (patching code sections at load time), which is slow and prevents code segment sharing across processes.
• Stripping before creating a debug link. Always run objcopy --only-keep-debug before strip and use objcopy --add-gnu-debuglink to associate the debug file with the stripped binary. Otherwise, you lose the ability to debug crashes in production.

Common Pitfalls

1. Memorising content without understanding the underlying principles. This leads to poor application in unfamiliar contexts.
2. Focusing only on content knowledge without developing exam technique and question-answering skills.
3. Ignoring feedback from marked work and failing to address recurring weaknesses.
4. Not making connections between different topics within the subject to build a coherent understanding.

Summary

The key principles covered in this topic are linked in the sub-pages above. Focus on understanding the definitions, applying the formulas or frameworks, and evaluating strengths and limitations of each approach.

Worked Examples

Worked examples demonstrating the application of key concepts are covered in the detailed sub-pages linked above.

Cross-References

• Site Home: Main landing page for Programming notes.
• Types: Type system fundamentals.
• Resource Management: RAII and smart pointers.
• Templates: Generic programming and metaprogramming.