LLMonCAR: A Benchmark for Exploring Large Language Models on Cryptographic Algorithm Recognition

The Cryptographic Algorithm Recognition (CAR) task is a critical problem in cryptography, with significant implications for the security of cryptographic algorithm design. While Large Language Models (LLMs) demonstrate promising potential in addressing this task, evaluating their performance remains a challenge due to the absence of aligned input-output specifications and standardized evaluation metrics for it in LLMs. In this paper, we construct an evaluation dataset and the corresponding metrics to analyze the performance and factors that influence effectiveness in CAR. The evaluation includes seven different cryptographic algorithms, along with performance of five main LLMs in this dataset. Experimental results indicate that LLMs exhibit limitations in algorithm identification, achieving an average accuracy of 63.9%. The performance is significantly influenced by the cryptographic algorithm and the fundamental capabilities of LLMs. Surprisingly, a mainstream cryptographic algorithm called Keccak can be relatively recognized by LLMs, which it shouldn’t be, unlike other modern algorithms. Furthermore, we introduce six different prompt engineering methods and find that most do not significantly enhance LLM performance in CAR. However, the prompting approach of snapshot-based exemplar reference effectively improves performance of CAR, resulting in an average increase of 7.7%, with varying degrees of improvement under different conditions.