CAPT: Cross-modal Anchored Prompt Tuning for Universal Cross-Domain Retrieval

Both CLIP encoders are frozen during training. We update only DSIP, a lightweight output-refinement layer (a residual MLP, about 0.26M parameters), and a small text context for image–text supervision, about 1.35M trainable parameters in total. At inference time, retrieval uses only the final image embeddings.

Deep Self-Instance Prompting (DSIP)

The anchor bank $A\in\mathbb R^{C\times d_t}$ encodes “a photo of a [class]” for the $C$ seen classes with the frozen CLIP text encoder, so it uses no unseen-class names. At layer $\ell$, the anchors and the current patch tokens are projected to a bottleneck of width $r$, $\tilde A=AW_a$ and $\bar H^{\ell}=[h^{\ell}_1,\dots,h^{\ell}_N]W_h$, and the anchors are conditioned on the image by a gated attention with a learnable scalar gate $\sigma(\gamma)$:

$$\hat A^{\ell}=\tilde A+\sigma(\gamma)\,\mathrm{softmax}\!\Big(\frac{\tilde A(\bar H^{\ell})^{\top}}{\sqrt r}\Big)\bar H^{\ell}.$$

Static learnable queries $Q_0\in\mathbb R^{P_q\times r}$ then read out the visual prompt from the conditioned anchors. $Q_0$ is shared across images, but $P^{\ell}$ is image-specific because $\hat A^{\ell}$ is conditioned on the current image:

$$P^{\ell}=\mathrm{MHA}_{\ell}\big(Q_0,\hat A^{\ell},\hat A^{\ell}\big)\,W_u^{\ell}\in\mathbb R^{P_q\times d}.$$

The prompt is appended to the token sequence at every transformer layer, $H^{\ell+1}=\mathrm{CLIPBlock}^{\ell}([H^{\ell};P^{\ell}])$. No domain prompt or domain label is used, so the same generator applies directly to unseen domains.

Per-sample Alignment via Cross-domain Gap Estimation (PACE)

PACE keeps a FIFO feature bank with class and domain labels. Domain labels are used only at training time; inference is label-free. CDCM treats same-class samples from other domains as positives, $\mathcal P(i)=\{a: y_a=y_i,\ e_a\neq e_i\}$, with the $K=256$ hardest negatives from the bank in the denominator. CDPA pulls each sample toward its flat class prototype, $\mathcal L^{i}_{\mathrm{CDPA}}=1-z_i^{\top}\mu_{y_i}$. CDCM spreads, CDPA compacts, so a fixed balance is not optimal for every sample. PACE uses the prototype confidence $s_i=z_i^{\top}\mu_{y_i}$ and the residual domain gap

$$g_i=z_i^{\top}\nu_{y_i,e_i}-z_i^{\top}\mu_{y_i},$$

where $\nu_{y_i,e_i}$ is the same-domain prototype of class $y_i$ in domain $e_i$. Both signals are standardized in the mini-batch, and

$$w_i=\mathrm{sg}\big[\sigma(\tilde s_i-\tilde g_i)\big],\qquad \mathcal L^{i}_{\mathrm{PACE}}=(1-w_i)\,\mathcal L^{i}_{\mathrm{CDCM}}+w_i\,\mathcal L^{i}_{\mathrm{CDPA}}.$$

A larger gap increases the CDCM contribution, while a confident and already aligned sample receives more CDPA. The full objective is $\mathcal L=\mathcal L_{\mathrm{ITC}}+\lambda\,\frac{1}{|S|}\sum_i\mathcal L^{i}_{\mathrm{PACE}}$ with $\lambda=0.5$.