RVLLM-Bench: A Comprehensive Benchmark for Large Language Model Inference with RISC-V Vector Extension

Large language models (LLMs) are increasingly being deployed on edge devices, where their inference is constrained by computational resources. This makes hardware acceleration for LLM inference necessary and challenging. The RISC-V vector (RVV) extension, with variable vector length, configurable element width, and vector register group multiplier, offers a pathway to hardware acceleration on resource-constrained edge devices. However, an extensive benchmark for RVV-based LLM inference performance remains lacking. In this paper, we propose RVLLM-Bench, a benchmark suite to evaluate the effectiveness and cross-platform portability of RVV for LLM inference. It incorporates both the pre-filling (prompt processing) and decoding (token generation) phases across different workload patterns on typical RISC-V platforms with two C/C++ engines and multiple model scales. The benchmark results show the significant gains of RVV in both phases in various configurations. In summary, we provide a comprehensive and reproducible baseline for RVV-based acceleration of LLM inference. Our code and data are publicly available at https://github.com/JocelynPanPan/rvllm-bench.