The Linux kernel is written in C, but it also leverages extensionsprovided by GCC. In 2022, it moved from GCC/Clang-std=gnu89 to -std=gnu11. This articleexplores my notes on how these GNU extensions are utilized within thekernel.

Statement expressions

Statementexpressions are commonly used in macros.

Locallabels

Some macros use this extension to restart a for loop in a macro'sreplacement list.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

// include/linux/wait.h
#define ___wait_event(wq_head, condition, state, exclusive, ret, cmd)           \
({                                                                              \
        __label__ __out;                                                        \
        struct wait_queue_entry __wq_entry;                                     \
        long __ret = ret;       /* explicit shadow */                           \
                                                                                \
        init_wait_entry(&__wq_entry, exclusive ? WQ_FLAG_EXCLUSIVE : 0);        \
        for (;;) {                                                              \
                long __int = prepare_to_wait_event(&wq_head, &__wq_entry, state);\
                                                                                \
                if (condition)                                                  \
                        break;                                                  \
                                                                                \
                if (___wait_is_interruptible(state) && __int) {                 \
                        __ret = __int;                                          \
                        goto __out;                                             \
                }                                                               \
                                                                                \
                cmd;                                                            \
        }                                                                       \
        finish_wait(&wq_head, &__wq_entry);                                     \
__out:  __ret;                                                                  \
})

include/linux/instruction_pointer.h utilizes the featureto obtain the current instruction pointer:

1

#define _THIS_IP_  ({ __label__ __here; __here: (unsigned long)&&__here; })

Labelsas values and computed goto statements

Bytecode interpreters often leverage this extension. BPF, forinstance, utilizes this extension. We can also see this extension indrm_exec_retry_on_contention for the Direct RenderingManager.

When it comes to Position-Dependent Code (PDC), a switch statementwith a default label marked as __builtin_unreachable can bejust as efficient as using labels as values. However, forPosition-Independent Code (PIC), absolute addresses offer a slightperformance edge, although at the cost of requiring more dynamicrelocations.

1
2
3
4
5
6
7
8

static void *tbl[] = {&&do_ret, &&do_inc, &&do_dec};
#define NEXT goto *tbl[*pc++]
NEXT;
for(;;) {
do_ret: return val;
do_inc: val++; NEXT;
do_dec: val--; NEXT;
}

typeof and__auto_type

typeof is used in numerous code.__auto_type has a few occurrences.

C23 standardizes typeof and auto.

Conditionalswith omitted operands

x ?: y is used in numerous code.

Emptystructures

The C standard (at least C11 and C23) specifies that:

If the member declaration list does not contain any named members,either directly or via an anonymous structure or anonymous union, thebehavior is undefined.

The empty structure extension enables a dummy structure when aconfiguration option disables the functionality.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

#ifdef CONFIG_GENERIC_ENTRY

struct syscall_user_dispatch {
        char __user     *selector;
        unsigned long   offset;
        unsigned long   len;
        bool            on_dispatch;
};

#else

struct syscall_user_dispatch {};

#endif

Caseranges

This extension (case low ... high:), frequently used inthe kernel, allows us to specify a range of consecutive values in asingle case label within a switch statement.

Objectsize checking

glibc 2.3.4 introduced _FORTIFY_SOURCE in 2004 to catchsecurity errors due to misuse of some C library functions (primarilybuffer overflow). The implementation leverages inline functions and__builtin_object_size.

Linux kernel introduced CONFIG_FORTIFY_SOURCE in 2017-07.Like the userspace, the implementation relies on optimizations likeinlining and constant folding.

For example, include/linux/string.h combines thisfeature with BUILD_BUG_ON(__attribute__((error(...)))) to provide compile-timeerrors:

1
2
3
4
5
6
7
8

#define strtomem(dest, src)     do {                                    \
        const size_t _dest_len = __builtin_object_size(dest, 1);        \
        const size_t _src_len = __builtin_object_size(src, 1);          \
                                                                        \
        BUILD_BUG_ON(!__builtin_constant_p(_dest_len) ||                \
                     _dest_len == (size_t)-1);                          \
        memcpy(dest, src, strnlen(src, min(_src_len, _dest_len)));      \
} while (0)

The Clang implementation utilizes__attribute__((pass_object_size(type))) and__attribute__((overloadable(...))).

Clang introduced__builtin_dynamic_object_size in 2019 to possiblyevaluate the size dynamically. The feature was ported to GCC in 2021 andis used by the kernel.

Pragmas

#pragma GCC diagnosticis occasionally used to disable a local diagnostic.

include/linux/hidden.h uses #pragma GCC visibility("hidden")to force direct access (instead of GOT indirection) for external symbolsfor -fpie/-fpic.

bpf uses #pragma GCC poisonto forbid some undesired identifiers.

Structure-layoutpragmas are commonly used.

Inline assembly

The Linux kernel uses inline assembly for a few reasons

• Hardware interaction: Certain operations require direct interactionwith the underlying hardware capabilities, which might not beexpressible in standard C.
• Performance optimization: In critical code paths, inline assemblycan be used to squeeze out the last bit of performance by utilizingprocessor-specific instructions or bypassing certain C languageconstructs that might introduce overhead.
• Special facility: Some features, such as static keys, alternativesruntime patching, cannot be implemented with standard C.

Built-in functions

Special machine codeinstructions

GCC provides built-in functions to generate machine code instructionsdesigned for certain tasks like popcount/clz/ctz. The compiler has moreinformation about the function's purpose and leverages internaloptimizations tailored for these tasks.

__builtin_choose_expr

This is analogous to ? : but the condition is a constantexpression and the return type is not altered by promotion rules. InGCC, __builtin_choose_expr is not available in C++.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

// include/linux/math.h
#define abs(x)  __abs_choose_expr(x, long long,                         \
                __abs_choose_expr(x, long,                              \
                __abs_choose_expr(x, int,                               \
                __abs_choose_expr(x, short,                             \
                __abs_choose_expr(x, char,                              \
                __builtin_choose_expr(                                  \
                        __builtin_types_compatible_p(typeof(x), char),  \
                        (char)({ signed char __x = (x); __x<0?-__x:__x; }), \
                        ((void)0)))))))

#define __abs_choose_expr(x, type, other) __builtin_choose_expr(        \
        __builtin_types_compatible_p(typeof(x),   signed type) ||       \
        __builtin_types_compatible_p(typeof(x), unsigned type),         \
        ({ signed type __x = (x); __x < 0 ? -__x : __x; }), other)

__builtin_constant_p

__builtin_constant_p identifies whether an expressioncan be evaluated to a constant. This capability unlocks several codepatterns.

Conditional static assertions:

1

BUILD_BUG_ON(__builtin_constant_p(nr) && nr != 1);

__builtin_constant_p decides whetherMAYBE_BUILD_BUG_ON expands to a compile-time static assertmechanism (__attribute__((error(...)))) or a runtimemechanism.

1
2
3
4
5
6
7

#define MAYBE_BUILD_BUG_ON(cond)                        \
        do {                                            \
                if (__builtin_constant_p((cond)))       \
                        BUILD_BUG_ON(cond);             \
                else                                    \
                        BUG_ON(cond);                   \
        } while (0)

Alternative code paths:

Sometimes, the constant expression case might enable a more efficientcode path.

1

#define __trace_if_var(cond) (__builtin_constant_p(cond) ? (cond) : __trace_if_value(cond))

Constant Folding Optimizations:

By knowing expressions are constant, the compiler can performconstant folding optimizations. This means pre-calculating theexpression's value during compilation and replacing it with the actualconstant in the code. This leads to potentially smaller and faster codeas the calculation doesn't need to be done at runtime. However, I noticethat __builtin_constant_p is often abused. This is one ofthe source why the kernel cannot be built with -O0.

__builtin_expect

Used by likely and unlikely macros to givethe compiler hints for optimization.

1
2
3

// include/linux/compiler.h
# define likely(x)   __builtin_expect(!!(x), 1)
# define unlikely(x) __builtin_expect(!!(x), 0)

__builtin_frame_address

Used by stack trace utilities.

__builtin_offsetof

Used to implement offsetof.

__builtin_*_overflow

Used by include/linux/overflow.h for overflowchecking.

__builtin_prefetch

Used as the default implementation of prefetch andprefetchw.

__builtin_unreachable

Used by include/linux/compiler.h:unreachable to informunreachability for optimization or diagnostics suppressing purposes.

This is often used to emulate C11 static_assert

1
2
3
4
5
6
7
8

#define __same_type(a, b) __builtin_types_compatible_p(typeof(a), typeof(b))

// include/linux/highmem-internal.h
#define kunmap_atomic(__addr)                                   \
do {                                                            \
        BUILD_BUG_ON(__same_type((__addr), struct page *));     \
        __kunmap_atomic(__addr);                                \
} while (0)

or implement C++ function overloading, e.g.

1
2
3
4
5
6
7
8
9

// fs/bcachefs/util.h
#define strtoi_h(cp, res)                                               \
        ( type_is(*res, int)            ? bch2_strtoint_h(cp, (void *) res)\
        : type_is(*res, long)           ? bch2_strtol_h(cp, (void *) res)\
        : type_is(*res, long long)      ? bch2_strtoll_h(cp, (void *) res)\
        : type_is(*res, unsigned)       ? bch2_strtouint_h(cp, (void *) res)\
        : type_is(*res, unsigned long)  ? bch2_strtoul_h(cp, (void *) res)\
        : type_is(*res, unsigned long long) ? bch2_strtoull_h(cp, (void *) res)\
        : -EINVAL)

include/linux/jump_label.h combines__builtin_types_compatible_p and dead code elimination toemulate C++17 if constexpr.____wrong_branch_error is not defined. If the unexpectedcode path is taken, there will be a linker error.

1
2
3
4
5
6
7
8
9
10

#define static_branch_likely(x)                                                 \
({                                                                              \
        bool branch;                                                            \
        if (__builtin_types_compatible_p(typeof(*x), struct static_key_true))   \
                branch = !arch_static_branch(&(x)->key, true);                  \
        else if (__builtin_types_compatible_p(typeof(*x), struct static_key_false)) \
                branch = !arch_static_branch_jump(&(x)->key, true);             \
        else                                                                    \
                branch = ____wrong_branch_error();                              \
        likely_notrace(branch);                                                         \                                                                                 })

BTF built-in functions

Clangthread safety analysis

See perfmutex: Add thread safety annotations.

Function attributes

tools/include/linux/compiler.h defines macros for manycommonly used attributes. Many of them are commonly used by otherprojects and probably not worth describing.

tools/testing/selftests has a few__attribute__((constructor)) for initialization.

__attribute__((error(...))) is used with inlining anddead code elimination to provide static assertion.

1
2
3
4
5
6
7
8
9
10
11
12

// include/linux/compiler_types.h
#ifdef __OPTIMIZE__
# define __compiletime_assert(condition, msg, prefix, suffix)           \
        do {                                                            \
                __noreturn extern void prefix ## suffix(void)           \
                        __compiletime_error(msg);                       \
                if (!(condition))                                       \
                        prefix ## suffix();                             \
        } while (0)
#else
# define __compiletime_assert(condition, msg, prefix, suffix) do { } while (0)
#endif

__attribute__((naked)) (__naked) is used byarm and bpf code.

__attribute__((section(...))) is used to place functionsin the specified section, e.g. .init.text.

__attribute__((weak)) is used to allow a genericfunction to be replaced by an arch-specific implementation. The weakreference feature is used by some__start_<sectionname>/__stop_<sectionname>encapsulation symbols.

#define __noendbr __attribute__((nocf_check)) is usedto disable Intel CET for some special functions.

Variable attributes

__attribute__((cleanup(...))) runs a function when thevariable goes out of scope. include/linux/cleanup.h definesDEFINE_FREE and __free based on thisfeature.

1
2
3
4
5

// include/linux/slab.h
DEFINE_FREE(kfree, void *, if (!IS_ERR_OR_NULL(_T)) kfree(_T))
// kernel/irq/irq_sim.c
        struct irq_sim_work_ctx *work_ctx __free(kfree) =
                                kmalloc(sizeof(*work_ctx), GFP_KERNEL);

Type attributes

__attribute__((aligned(...))) and__attribute__((packed)) are often to control structurelayouts.

Language dialects

Assume that programs can safely dereference null pointers, and codeor data element may reside at address zero.

-fno-strict-aliasing

The C11 standard (section 6.5) defines strict aliasing rules.

An object shall have its stored value accessed only by an lvalueexpression that has one of the following types:

— a type compatible with the effective type of the object, — aqualified version of a type compatible with the effective type of theobject, — a type that is the signed or unsigned type corresponding tothe effective type of the object, — a type that is the signed orunsigned type corresponding to a qualified version of the effective typeof the object, — an aggregate or union type that includes one of theaforementioned types among its members (including, recursively, a memberof a subaggregate or contained union), or — a character type.

Compilers leverage these aliasing rules for optimizations, which canbe disabled with -fno-strict-aliasing.

Linus Torvalds has expressed reservations about strict aliasing. Hereis a discussionfrom 2018.

-fno-strict-overflow

In GCC, this option is identical to-fwrapv -fwrapv-pointer: make signed integer overflowdefined using wraparound semantics. While avoiding undefined behaviors,unexpected arithmetic overflow bugs might be lurking.

[RFC]Mitigating unexpected arithmetic overflow is a 2024-05 thread aboutdetecting and mitigating these issues.