Performance Framework Autobench3's CPU Monitor – MySQL Benchmarking

Solving the Backlog Problem in MySQL Benchmarking

In database scalability testing, Autobench3 (AB3) steps through thread counts to evaluate performance under increasing load—typically using values like 4, 8, 16, 32, 64, 128, 256...4096 and beyond.

To improve stability and reproducibility, each thread count is executed multiple times (usually 3 iterations, but up to 7 if needed). Final results are averaged across iterations.

The Backlog Problem

One challenge with thread stepping is that MySQL may still be processing work after the client stops sending traffic. This backlog is especially common in write-heavy workloads, but even read tests can leave MySQL active briefly after disconnect.

Initially, AB3 tried to address this by inserting sleep intervals between phases—during MySQLD startup, database load, and between iterations. But sleep was a guessing game. As thread counts increased, so did the time needed to clear the backlog, and fixed sleep durations weren’t reliable.

Enter CpuMonitor

CpuMonitor was created to solve this problem by detecting when MySQL has truly reached a “rest” state before proceeding to the next iteration or thread count. It attaches to the MySQLD process using PID and monitors CPU usage, waiting for sustained low activity before continuing.

CpuMonitor Configuration

• CPU_REST_VALUE: CPU usage threshold considered “rest” (typically ≤ 10%)
• MAX_COUNT: Number of consecutive rest samples required to confirm rest
• CPU_REST_RESET_VALUE: CPU usage threshold that resets the rest counter (typically ≥ 70%)
• INTERVAL: How often CPU usage is checked (in seconds)

Impact on Benchmarking

• Read tests: Required minimal rest time, reducing overall waits between iterations
• Write tests: Showed longer rest times at higher thread counts, especially for I/O-bound workloads
• Low concurrency: Needed less rest time, but high concurrency demanded more patience

Early versions lacked reset logic, which led to false positives—brief rest followed by CPU spikes (e.g., redo/undo logs flushing). Adding reset logic ensured true rest before moving on (e.g., requiring 10 consecutive rest samples).

Why It Matters

CpuMonitor ensures that no residual work from the previous iteration contaminates the next iteration. This improves stability, reproducibility, and confidence in performance results.

Once stabilized, AB3 began tracking rest duration as a metric—adding another lens for detecting performance changes across versions and workloads.

Rewriting the Tool

The original CpuMonitor was a Java-based utility I built for the Autobench3 framework, owned by Oracle. It relied on the Sigar library for cross-platform CPU monitoring but became increasingly cumbersome due to heavy dependencies and Sigar’s eventual obsolescence.

Over the last few weeks, having some extra time on my hands, I wrote a new Python version from the ground up—not as a direct port, but as a lean, standalone tool. It was a chance to dive deeper into Python’s ecosystem, share something useful, and simplify what had become a bulky Java solution.

https://github.com/JonathanBMiller/process_cpu_monitor

Thanks for following along.

Tags: #MySQL #Benchmarking #Autobench3 #PerformanceTesting #CpuMonitor #AutomationFramework #DatabasePerformance #Scalability #TechLeadership #MariaDB