Under Construction: Building the MariaDB Benchmarking Test Automation Framework (TAF)

Building a New Test Automation Framework for MariaDB

MariaDB has given me the chance to pursue a lifelong dream: creating a new Test Automation Framework (TAF) — an improved, expanded evolution of the Autobench3 framework I originally built for MySQL.

Autobench3 was never just a benchmark API. It was a framework that wrapped benchmark APIs to provide a consistent platform for configuration and abstraction. Instead of requiring deep knowledge of each benchmark tool, Autobench3 allowed developers to work with simple command‑line options and property files. The framework made clear what suite was being run, what test case was in play, which database software was targeted, and what overrides were applied.

When running, the framework would load the chosen test suite into itself — becoming Sysbench, becoming DBT2 — and then drive the workload through a consistent set of lifecycle stages.

Autobench3: Still in Service, But Limited

Autobench3 continues to serve MySQL well in worklog development, commit monitoring, and release performance testing. Yet it has important limitations:

• Hardwired to MySQL only — no MariaDB, PostgreSQL, or other database support.
• XML results only — output tied directly to the Automated Test Results (ATR) parser, database, and web stack.
• Single‑install limitation — only one MySQL install could be active at a time, with no concept of an active install marker.
• Large driver size — even with helper libraries, the driver was nearly 9,000 lines of code.

When Oracle ended my role, I was crushed. I had just finished coding developer compare functionality through Gerrit/Jenkins/AB3 and was dreaming of a lights‑out solution: automated detection of performance regressions, commit hunting, compare runs, profiling, and automatic outreach to developers.

A New Beginning with MariaDB

Then came an unbelievable opportunity — from the very family that started MySQL and MariaDB. The CEO reached out and asked if I wanted to rebuild a framework, perform benchmarking, and blog about improving MariaDB’s performance.

My answer was immediate: Absolutely.

Since then, I’ve been heads‑down building. November was spent creating the driver and the first beta test suites:

• Sysbench‑Lua with BMK abilities (BMK)
• HammerDB for TPROC-C and TPROC-H (HammerDB)

With the driver running and suites loading correctly, I put them to use.

Why Profile Guided Optimization (PGO) Matters

One thing many don’t realize: Oracle’s Enterprise Editions of MySQL are PGO (Profile Guided Optimized) builds. That optimization is locked behind a paywall. The biggest gains from PGO show up in cached database workloads — point‑selects and read‑heavy cases in particular.

When I built MariaDB 11.8.3 both with and without PGO, the difference was undeniable: throughput improvements of up to +30% in many test cases, with nearly every run showing measurable gains. And here’s the key point — MariaDB can deliver those optimizations for free.

Just as thread pool was once a feature you had to pay Oracle for in MySQL but is freely available in MariaDB, my hope is that PGO builds will become another example of MariaDB giving the community what Oracle keeps gated.

Framework Progress

Today, the framework is under heavy construction:

• Core driver and test suites (basic beta form, still need more testing)

This is not a finished product. The current pieces are proof‑of‑concept only, showing that the driver can load and run suites correctly. Much more work remains before it’s ready for general use. The next stages include:

• Expanding database plugin logic (MariaDB, MySQL, and a generic layout for PostgreSQL, MSSQL, etc.)
• Adding support for multiple unpacked installs with easy switching
• Continuing to refine plugin‑based reporters

Call to Action

My hope is that when the framework is released — as a free product under the MariaDB Foundation, just as the Foundation always does — others will download it, use it, and contribute. Not only by improving the framework itself, but also by reporting any performance issues they encounter with MariaDB while using it.

Together, we can make MariaDB better for everyone.

When beta is released it will just be the beginning. There’s plenty of growing up to do, and I welcome feedback — good, bad, even brutal. I won’t take it personally; I’ll take it as proof that you care.

Stay tuned!

#MariaDB #MariaDBFoundation #MySQL #PerformanceTesting