跳到主要导航 跳到搜索 跳到主要内容

Accelerating Framework of Transformer by Hardware Design and Model Compression Co-Optimization

  • East China Normal University
  • University of Connecticut
  • Northeastern University

科研成果: 书/报告/会议事项章节 › 会议稿件 › 同行评审

摘要

State-of-the-art Transformer-based models, with gigantic parameters, are difficult to be accommodated on resource constrained embedded devices. Moreover, with the development of technology, more and more embedded devices are available to run a Transformer model. For a Transformer model with different constraints (tight or loose), it can be deployed onto devices with different computing power. However, in previous work, designers did not choose the best device among multiple devices. Instead, they just used an existing device to deploy model, which was not necessarily the best fit and may lead to underutilization of resources. To address the deployment challenge of Transformer and the problem to select the best device, we propose an algorithm hardware closed-loop acceleration framework. Given a dataset, a model, latency constraint LC and accuracy constraint AC, our framework can provide a best device satisfying both constraints. In order to generate a compressed model with high sparsity ratio, we propose a novel pruning technique, hierarchical pruning (HP). We optimize the sparse matrix storage format for HP matrix to further reduce memory usage for FPGA implementation. We design a accelerator that takes advantage of HP to solve the problem of concurrent random access. Experiments on Transformer and TinyBert model show that our framework can find different devices for various LC and AC, covering from low-end devices to high-end devices. Our HP can achieve higher sparsity ratio and is more flexible than other sparsity pattern. Our framework can achieve 37×, 1.9×, 1.7× speedup compared to CPU,GPU and FPGA,respectively.

源语言英语
主期刊名2021 40th IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2021 - Proceedings
出版商Institute of Electrical and Electronics Engineers Inc.
ISBN(电子版)9781665445078
DOI
出版状态已出版 - 2021
活动40th IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2021 - Virtual, Online, 德国
期限: 1 11月 2021 → 4 11月 2021

丛书

姓名IEEE/ACM International Conference on Computer-Aided Design, Digest of Technical Papers, ICCAD
卷2021-November
ISSN(印刷版)1092-3152

会议

会议40th IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2021
国家/地区德国
市Virtual, Online
时期1/11/21 → 4/11/21

学术指纹

探究 'Accelerating Framework of Transformer by Hardware Design and Model Compression Co-Optimization' 的科研主题。它们共同构成独一无二的学术指纹。

引用此