Stable Quantization-Aware Training with Adaptive Gradient Clipping

Jihoon Park; Seunghyun Lee; Byung Cheol Song

doi:10.1109/ICEIC57457.2023.10049939

Stable Quantization-Aware Training with Adaptive Gradient Clipping

Jihoon Park, Seunghyun Lee, Byung Cheol Song

Inha University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Quantization-Aware Training (QAT) uses batch normalization (BN) folding during fine-tuning, so it may not use the normalization effects of the BN layer. HAWQ, which achieved SOTA with QAT, has significant accuracy degradation as the training becomes longer. In this paper, we apply Adaptive Gradient Clipping (AGC) to stable quantization-aware training and improve accuracy by adding Dropout. Moreover, we have ablation studies about AGC.

Original language	English
Title of host publication	2023 International Conference on Electronics, Information, and Communication, ICEIC 2023
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9798350320213
DOIs	https://doi.org/10.1109/ICEIC57457.2023.10049939
State	Published - 2023
Event	2023 International Conference on Electronics, Information, and Communication, ICEIC 2023 - Singapore, Singapore Duration: 5 Feb 2023 → 8 Feb 2023

Publication series

Name	2023 International Conference on Electronics, Information, and Communication, ICEIC 2023

Conference

Conference	2023 International Conference on Electronics, Information, and Communication, ICEIC 2023
Country/Territory	Singapore
City	Singapore
Period	5/02/23 → 8/02/23

Bibliographical note

Publisher Copyright:
© 2023 IEEE.

Access to Document

10.1109/ICEIC57457.2023.10049939

Cite this

Park, J., Lee, S., & Song, B. C. (2023). Stable Quantization-Aware Training with Adaptive Gradient Clipping. In 2023 International Conference on Electronics, Information, and Communication, ICEIC 2023 (2023 International Conference on Electronics, Information, and Communication, ICEIC 2023). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICEIC57457.2023.10049939

@inproceedings{ad600feb6a6f4339b13eba02f0774251,

title = "Stable Quantization-Aware Training with Adaptive Gradient Clipping",

abstract = "Quantization-Aware Training (QAT) uses batch normalization (BN) folding during fine-tuning, so it may not use the normalization effects of the BN layer. HAWQ, which achieved SOTA with QAT, has significant accuracy degradation as the training becomes longer. In this paper, we apply Adaptive Gradient Clipping (AGC) to stable quantization-aware training and improve accuracy by adding Dropout. Moreover, we have ablation studies about AGC.",

author = "Jihoon Park and Seunghyun Lee and Song, {Byung Cheol}",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 2023 International Conference on Electronics, Information, and Communication, ICEIC 2023 ; Conference date: 05-02-2023 Through 08-02-2023",

year = "2023",

doi = "10.1109/ICEIC57457.2023.10049939",

language = "English",

series = "2023 International Conference on Electronics, Information, and Communication, ICEIC 2023",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2023 International Conference on Electronics, Information, and Communication, ICEIC 2023",

address = "United States",

}

Park, J, Lee, S & Song, BC 2023, Stable Quantization-Aware Training with Adaptive Gradient Clipping. in 2023 International Conference on Electronics, Information, and Communication, ICEIC 2023. 2023 International Conference on Electronics, Information, and Communication, ICEIC 2023, Institute of Electrical and Electronics Engineers Inc., 2023 International Conference on Electronics, Information, and Communication, ICEIC 2023, Singapore, Singapore, 5/02/23. https://doi.org/10.1109/ICEIC57457.2023.10049939

Stable Quantization-Aware Training with Adaptive Gradient Clipping. / Park, Jihoon; Lee, Seunghyun; Song, Byung Cheol.
2023 International Conference on Electronics, Information, and Communication, ICEIC 2023. Institute of Electrical and Electronics Engineers Inc., 2023. (2023 International Conference on Electronics, Information, and Communication, ICEIC 2023).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Stable Quantization-Aware Training with Adaptive Gradient Clipping

AU - Park, Jihoon

AU - Lee, Seunghyun

AU - Song, Byung Cheol

PY - 2023

Y1 - 2023

N2 - Quantization-Aware Training (QAT) uses batch normalization (BN) folding during fine-tuning, so it may not use the normalization effects of the BN layer. HAWQ, which achieved SOTA with QAT, has significant accuracy degradation as the training becomes longer. In this paper, we apply Adaptive Gradient Clipping (AGC) to stable quantization-aware training and improve accuracy by adding Dropout. Moreover, we have ablation studies about AGC.

AB - Quantization-Aware Training (QAT) uses batch normalization (BN) folding during fine-tuning, so it may not use the normalization effects of the BN layer. HAWQ, which achieved SOTA with QAT, has significant accuracy degradation as the training becomes longer. In this paper, we apply Adaptive Gradient Clipping (AGC) to stable quantization-aware training and improve accuracy by adding Dropout. Moreover, we have ablation studies about AGC.

UR - http://www.scopus.com/inward/record.url?scp=85150450476&partnerID=8YFLogxK

U2 - 10.1109/ICEIC57457.2023.10049939

DO - 10.1109/ICEIC57457.2023.10049939

M3 - Conference contribution

AN - SCOPUS:85150450476

T3 - 2023 International Conference on Electronics, Information, and Communication, ICEIC 2023

BT - 2023 International Conference on Electronics, Information, and Communication, ICEIC 2023

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2023 International Conference on Electronics, Information, and Communication, ICEIC 2023

Y2 - 5 February 2023 through 8 February 2023

ER -

Park J, Lee S, Song BC. Stable Quantization-Aware Training with Adaptive Gradient Clipping. In 2023 International Conference on Electronics, Information, and Communication, ICEIC 2023. Institute of Electrical and Electronics Engineers Inc. 2023. (2023 International Conference on Electronics, Information, and Communication, ICEIC 2023). doi: 10.1109/ICEIC57457.2023.10049939

Stable Quantization-Aware Training with Adaptive Gradient Clipping

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this