Federated Parameter-Efficient Fine-Tuning

FraQ: Efficient Core-Space Recompression for Federated Low-Rank Adaptation

Shenghui Li¹ and Thiemo Voigt^1,2

¹Uppsala University, Uppsala, Sweden ²Research Institutes of Sweden, Stockholm, Sweden

{shenghui.li,thiemo.voigt}@angstrom.uu.se

shenghui.li@angstrom.uu.se
thiemo.voigt@angstrom.uu.se

Paper PDF

BibTeX

Abstract

Federated Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning of large language models without sharing local data, but aggregating LoRA adapters is fundamentally different from aggregating full model updates. Directly averaging LoRA factors introduces cross-client approximation errors, while recovering the exact averaged update and recompressing it with truncated SVD incurs substantial server-side cost. We propose FraQ, an efficient core-space recompression method for federated LoRA aggregation. Instead of materializing the dense weight update, FraQ represents the exact aggregate through weighted stacked LoRA factors, computes a compact orthonormal basis via thin QR factorization, and recovers the singular spectrum from a small Gram matrix. An energy threshold then selects a compact merge patch, allowing the server to control the trade-off between accuracy and downlink communication.

Overview

FraQ aggregates heterogeneous client-side LoRA adapters by working in a compact core space. The server stacks weighted local factors, computes a thin QR basis for the shared column space, and performs eigendecomposition on a small Gram matrix rather than reconstructing the full dense update.

Highlights

Exact aggregation before truncation. FraQ preserves the weighted aggregate represented by stacked LoRA factors before applying rank-adaptive recompression.
Avoids full-update SVD. The server never materializes the dense update $\overline{\Delta W}$ . It replaces full-matrix SVD with thin QR factorization and eigendecomposition of a small $q \times q$ Gram matrix.
Communication-aware rank selection. An energy threshold controls the downlink size of the global LoRA patch while maintaining strong accuracy.

Method

In each communication round, client $k$ trains a local LoRA adapter $\Delta W_k = B_k A_k$ with rank $r_k$ . The server first constructs weighted stacked factors by concatenating the client adapters:

B = [\sqrt{\alpha_1} B_1,\sqrt{\alpha_2} B_2,\cdots,\sqrt{\alpha_K} B_K], \quad A = [\sqrt{\alpha_1} A_1;\sqrt{\alpha_2} A_2;\cdots;\sqrt{\alpha_K} A_K],

where $\alpha_k$ is the aggregation weight for client $k$ . This gives an exact factorized representation of the weighted average update:

\overline{\Delta W} = BA.

FraQ then recompresses this aggregate without forming the dense matrix $\overline{\Delta W}$ . It computes a thin QR factorization $B = QR$ , forms the compact core matrix $H = RA$ , and obtains the singular spectrum from the small Gram matrix

G = HH^\top.

Because $\overline{\Delta W} = QH$ and $Q$ is orthonormal, decomposing this core space recovers the same nonzero singular values as the aggregated update while keeping the expensive computation in the stacked-rank dimension. FraQ then selects the smallest rank whose retained singular energy exceeds a threshold $\tau$ , constructs the corresponding global LoRA patch $(B_g, A_g)$ , and sends that compact patch to all clients.

BibTeX

@misc{li2026fraq,
  title  = {FraQ: Efficient Core-Space Recompression for Federated Low-Rank Adaptation},
  author = {Li, Shenghui and Voigt, Thiemo},
  year   = {2026},
  note   = {Preprint},
}