Akira Sakai, Yuma Ichikawa
The paper identifies that weight sign persistence is a bottleneck in sub-bit model compression and proposes methods to reduce sign flips while maintaining performance.
In machine learning, compressing models to use less storage space is important for efficiency. When trying to compress models to use less than one bit per weight, the sign of the weight becomes a significant obstacle. The researchers found that the signs of weights tend to stay the same as their initial values, which limits further compression. They propose a new approach to initializing weights and a regularization technique to reduce unnecessary changes in sign, which helps maintain model performance while achieving better compression.
