Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via lp Regularization

arXiv:2609.28998v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) has become a popular parameter-efficient fine-tuning method for large language models. A key challenge in LoRA is how to determine the rank of each adaptation matrix, as rank directly controls its capacity and efficiency. Existing adaptive-rank methods typically allocate ranks according to manually designed importance scores, which are not directly derived from an optimization objective. In this work, we propose $\ell_p$-LoRA, a principled rank-allocation method based on $\ell_p$ regularization with $0 <1$, which is a classical sparsity-inducing technique in signal pro