Sun Yat-sen Researchers Beat Black-Box Prompt Tuning Across 7 GLUE Tasks

Researchers at Sun Yat-sen University have proposed a projection-free method for optimizing continuous prompts on frozen language models. The approach searches the full embedding space, adapts its mutation step size using the space’s estimated intrinsic dimension, and adds a confidence regularizer to limit few-shot overfitting. On seven GLUE tasks, it improved on common black-box baselines while maintaining a reported evaluation time of about 0.15 seconds.

The API contract comes first

Continuous prompt optimization prepends trainable embedding vectors to each input while leaving the model weights frozen. The optimizer receives losses or output scores from repeated model calls, without gradients or parameter access. A 50-token prompt for RoBERTa-Large , whose embeddings contain 1,024 values per token, creates a 51,200-dimensional search problem.

The method requires an endpoint that accepts continuous prompt embeddings and returns token-level logits or probabilities. Its confidence regularizer also needs scores across the vocabulary. Text-only chat APIs that accept strings and return sampled text cannot provide the required interface.

Black-Box Tuning, or BBT, reduces the search space by mapping a small parameter vector into the full prompt through a fixed random projection. CMA-ES then optimizes within that affine subspace. This reduces the optimizer’s working dimension, but it permanently excludes directions outside the sampled basis.

Random projections miss useful directions

The researchers sampled 5,000 random prompts, calculated a loss gradient at each point, and applied a maximum-likelihood intrinsic-dimension estimator to the resulting gradient manifold. Those gradients supported the diagnostic analysis only; the proposed optimizer does not use them. Across tasks and prompt lengths, the estimated intrinsic dimension remained at or below roughly 250.