Efficient auto-increment keys generation for distributed log-structured storage systems

Recent years, writing-intensive workloads on big data make log-structured style storage popular in distributed data storage systems, which provides both large-volume storage capacity and high-performance data updates. Rapidly generating valid keys for append records can significantly improve the data write performance of log-structured storage systems. In distributed and high concurrency environment, however, both the huge disk IO and the interaction overhead of a traditional lock manager limit the transactional throughput for generating auto-increment keys. In this paper, we design an efficient auto-increment keys generation manager (AKGM), a memory management structure that cannot only avoid disk IO but also eliminate the interaction overhead of traditional lock manager for transactions of generating auto-increment keys. We also propose a protocol called adaptive batch processing (ABP), which enables systems implementing AKGM to achieve high transactional throughput even under high contention workloads. We implement these protocols in an open-source database based on log-structured storage, and our experimental results show the superior performance of AKGM and ABP.