Federated Parameter-Efficient Fine-Tuning

FraQ: Efficient Coordinate-Space Recompression for Federated Low-Rank Adaptation

Shenghui Li¹ and Thiemo Voigt^1,2

¹Uppsala University, Uppsala, Sweden ²Research Institutes of Sweden, Stockholm, Sweden

{shenghui.li,thiemo.voigt}@angstrom.uu.se

shenghui.li@angstrom.uu.se
thiemo.voigt@angstrom.uu.se

Paper PDF

arXiv

BibTeX

Abstract

Federated fine-tuning with Low-Rank Adaptation (LoRA) enables collaborative and efficient adaptation of Large Language Models (LLMs) without centralizing private data. However, its two-factor parameterization introduces an aggregation mismatch across clients, as naive factor-wise averaging does not recover the average of the induced updates. Forming the exact aggregate in the full weight space and recompressing it avoids this mismatch, but decomposing the resulting full-size matrix is computationally expensive and memory-intensive. We propose FraQ, an efficient coordinate-space recompression method for federated LoRA. Starting from stacked factors that represent the exact aggregate, FraQ expresses the aggregate as the product of an orthonormal basis and a compact coordinate matrix. It then recovers the singular spectrum from a small Gram matrix of this coordinate representation and selects the smallest rank satisfying a prescribed energy threshold. The selected coordinate subspace is mapped back via the basis to construct the global adapter. Experiments across text classification and commonsense reasoning benchmarks demonstrate that FraQ achieves accuracy close to that of uncompressed baselines while substantially reducing downlink communication and incurring low server-side recompression overhead.

Overview

FraQ aggregates heterogeneous client-side LoRA adapters in a compact coordinate space. The server orients the weighted factor stacks along the smaller weight dimension, computes a single reduced Householder QR factorization, and recovers the spectrum from a small Gram matrix rather than reconstructing the full dense update.

Highlights

Exact aggregation before truncation. FraQ preserves the weighted aggregate represented by stacked LoRA factors before applying rank-adaptive recompression.
One-sided, direction-adaptive factorization. A single reduced Householder QR factorization is applied along the smaller weight dimension; the larger dimension appears only in matrix multiplications involving compact factors.
Communication-aware rank selection. An energy threshold controls the downlink size of the compact global adapter while maintaining strong accuracy.

Method

In each communication round, client $k$ trains a local LoRA adapter $\Delta W_k = s_k B_k A_k$ with rank $r_k$ and scale $s_k$ . The server first constructs weighted stacked factors by concatenating the client adapters:

B = [\sqrt{\alpha_1} B_1,\sqrt{\alpha_2} B_2,\cdots,\sqrt{\alpha_K} B_K], \quad A = [\sqrt{\alpha_1}s_1 A_1;\sqrt{\alpha_2}s_2 A_2;\cdots;\sqrt{\alpha_K}s_K A_K],

where $\alpha_k$ is the aggregation weight for client $k$ . This gives an exact factorized representation of the weighted average update:

\overline{\Delta W} = BA.

FraQ then recompresses this aggregate without forming the dense matrix. It orients the factors along the smaller weight dimension:

(X,Y)= \begin{cases} (B,A), & m\le n,\\ (A^\top,B^\top), & m>n. \end{cases}

A reduced Householder QR factorization of $X$ obtains $R$ and an implicit representation of the orthonormal basis $Q$ , with $X=QR$ . FraQ forms the coordinate matrix $H=RY$ and the compact Gram matrix

G = HH^\top.

Because the oriented aggregate satisfies $M=QH$ and $Q$ is orthonormal, the eigenvalues of $G$ equal the squared singular values of the exact aggregate. FraQ selects the smallest rank $p$ whose retained spectral energy exceeds $\tau$ and computes

H_p=U_pU_p^\top H, \qquad QH_p=(QU_p)(U_p^\top H).

The stored Householder reflectors are applied directly to $U_p$ , so the full basis $Q$ need not be materialized. The resulting factors are mapped back to the original orientation and broadcast as the compact global adapter $(B_g,A_g)$ .

BibTeX

@misc{li2026fraqefficientcoordinatespacerecompression,
  title={FraQ: Efficient Coordinate-Space Recompression for Federated Low-Rank Adaptation},
  author={Shenghui Li and Thiemo Voigt},
  year={2026},
  eprint={2608.03605},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2608.03605},
}