Leopard: A General Test Suite for Isolation Level Verification

Isolation levels (ILs) play as the correctness contract between applications and a database management system (DBMS). They are implemented to ensure the correct database status w.r.t. complex business logic under concurrent transaction processing. However, verifying the IL implementations of a DBMS has always been a challenging task due to the 1) inaccessibility of the codebase; 2) non-deterministic execution orders among concurrent transactions; 3) various IL definitions and implementations and 4) requirement of highly efficient IL verification. To expose potential IL anomalies in DBMSs, we propose to design and implement a general IL test suite Leopard. Leopard can be configured to verify various IL implementations in different DBMSs in a black-box way by analyzing client-side workload traces. With the help of Leopard, we have successfully discovered 24 bugs (14 fixed, 16 confirmed and 8 open reported) in several commercial DBMSs.