x86_64 Microarchitecture Levels

2024-07-29

Newer versions of the x86_64 processors support instructions that did not exist in the older versions. These newer instructions can perform tasks that would previously have taken multiple instructions using only what was available in the original x86_64 baseline instruction set. These instructions are generally useful for "hardcore" math, graphics, or encryption operations.

If a program tries to run one of these new instructions on a CPU which doesn't support them, you'll get an "Illegal instruction" error and the program will crash.

The x86 CPUs have a set of "flags" which programs can query if they plan to use any of the newer instructions, so they can either fail gracefully (with a better error message), or use different code to do the same job using the older, slower instructions.

ℹ️ On a Linux machine, you can see these flags in the /proc/cpuinfo file.

$ grep ^flags /proc/cpuinfo
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov ...

Each of these codes corresponds to a capability which may or may not exist on an x86 CPU.

Having to query for a whole list of instructions can be rather tedious, so Intel, AMD, Red Hat, and SUSE created a set of "levels" to encapsulate different sets of instructions that were added over time.

• Original suggestion on the llvm-dev mailing list
• Wikipedia has a good description of how the levels work and which CPU instructions are included in each level
• Medium article also explains it fairly clearly
• Red Hat blog post explains how and why Red Hat built RHEL 9 to require and use the x86-64-v2 capabilities

AlmaLinux (and RHEL) 9

I first ran into this when I tried to run AlmaLinux 9 in a Docker container, under colima on an Apple M2 machine. (I'm currently in the process of migrating my own servers from CentOS 7 to AlmaLinux 9.) AlmaLinux 9 and RHEL 9 require a minimum of x86-64-v2, and the virtual CPU created by QEMU (used by colima to run the x86_64 Linux VM where Docker is actually running) by default only supports x86-64 without any extensions.

At first, when I tried to start an AlmaLinux 9 (or UBI 9) container, I got this error:

$ docker run -it --rm almalinux:9.4
Fatal glibc error: CPU does not support x86-64-v2

This was the problem that made me dig into this issue. I found that adding --cpu-type max when creating the colima VM made the virtual CPU suport the same capabilities as the hardware CPU, which for the Apple M2 includes everything in the x86-64-v3 list.

Detect

If you need to detect which "level" a script is running on, here are a few suggestions:

awk script

⇒ Source

#!/usr/bin/awk -f

BEGIN {
    while (!/flags/) if (getline < "/proc/cpuinfo" != 1) exit 1
    if (/lm/&&/cmov/&&/cx8/&&/fpu/&&/fxsr/&&/mmx/&&/syscall/&&/sse2/) level = 1
    if (level == 1 && /cx16/&&/lahf/&&/popcnt/&&/sse4_1/&&/sse4_2/&&/ssse3/) level = 2
    if (level == 2 && /avx/&&/avx2/&&/bmi1/&&/bmi2/&&/f16c/&&/fma/&&/abm/&&/movbe/&&/xsave/) level = 3
    if (level == 3 && /avx512f/&&/avx512bw/&&/avx512cd/&&/avx512dq/&&/avx512vl/) level = 4
    if (level > 0) { print "CPU supports x86-64-v" level; exit level + 1 }
    exit 1
}

bash script

⇒ Source

Too long to incude here, but it looks like it's doing the same thing as the awk script above.

Changelog

2024-07-29 jms1

• initial version