Multi-Agent LLMs for Adaptive Acquisition in Bayesian Optimization

Bayesian optimization (BO) accelerates search over expensive black-box objectives by training a surrogate and selecting new queries via an acquisition function. Classical acquisitions tightly couple behavior to surrogate statistics and fixed hyperparameters, which can yield brittle exploration–exploitation trade-offs, especially in higher-dimensional or irregular design spaces. We propose an agentic large language model (LLM) framework that makes acquisition design itself adaptive. Instead of fixing a selector, one or more LLM agents (i) recommend task-appropriate metrics and weights and (ii) compose a problem-specific acquisition that balances four desiderata—exploration, exploitation, representativeness, and diversity—explicitly surfaced for transparent trade-off reasoning. We implement two protocols: a single-agent variant that selects metrics, constructs the acquisition, and proposes the next point; and a multi-agent variant that separates metric selection from acquisition/candidate generation for modularity and auditability.

We evaluate on three representative settings: (1) the Rosenbrock function (smooth but narrow valley), (2) hyperparameter tuning for machine-learning models (costly evaluations), and (3) robot pushing (discontinuous dynamics). Across tasks, the agent-designed acquisitions improve early-budget search quality, produce more balanced sampling patterns, and maintain competitive objective values while offering human-interpretable rationale (selected metrics and weights). The framework integrates into standard BO loops with minimal code changes and naturally supports decision-support workflows: practitioners can inspect, adjust, or constrain the metric set before execution. This work advances intelligent systems for modeling and optimization by making acquisition design adaptive, explainable, and user-steerable, enabling reliable application of BO in engineering and other human-centered, high-cost domains.