Swarm Optimization for LLMs (PSO, ACO)

Problem

You have a prompt that scores 62% on your task eval. You want to improve it. The naive move is gradient-free manual tweaking, which scales badly. Reinforcement learning needs reward signals and enough rollouts to train. Backprop through prompts is not really possible — the prompt is a discrete string, not a differentiable parameter.

Classical bio-inspired optimization — PSO for continuous search spaces, ACO for path selection — was designed exactly for this regime: gradient-free, population-based, cheap per evaluation. Pair them with LLMs for the gradient-free search step, and you get a surprisingly practical optimizer.

The same patterns apply to agent routing in multi-agent systems. An ACO-style pheromone trail records which agent worked best on which task-type, lets the router exploit the trail, and decays pheromones so routes can be rediscovered.

Concept

PSO refresher (Kennedy & Eberhart 1995)

Particle Swarm Optimization: population of particles in a continuous search space. Each particle has position x_i and velocity v_i. Each iteration:

Copyv_i <- w * v_i + c1 * r1 * (p_best_i - x_i) + c2 * r2 * (g_best - x_i)
x_i <- x_i + v_i
evaluate fitness(x_i)
update p_best_i if improved
update g_best if global best

Where p_best is particle's own best, g_best is swarm's best, w, c1, c2 are inertia + cognitive + social weights, r1, r2 are random factors.

PSO on LLM outputs — LMPSO

arXiv:2504.09247 adapts PSO for LLM-generated structured outputs (math expressions, programs). Each particle is a candidate output. Velocity is a prompt that describes how to modify the current output toward the personal/global best. The LLM generates the new output from the velocity prompt. The "inertia" of the velocity is a prompt like "make small incremental changes."

This works well when:

• The output is structured (parseable, evaluable).
• Fitness is automatic (test runs, arithmetic evaluation).
• Population is small (~10-30 particles) so total LLM calls stay manageable.

It does not work well when fitness needs human review — the per-iteration cost becomes prohibitive.

Model Swarms

arXiv:2410.11163 takes PSO off the output layer and into the model layer. Each "particle" is an expert LLM (parameters). The swarm moves the parameters toward the collective best via a gradient-free update. Reported: 13.3% average gain over 12 baselines on 9 datasets, with just 200 instances per iteration.

The key insight is that LLM expert models are already nearby in a shared parameter manifold (adapter weights, LoRA deltas). PSO on this low-dimensional subspace is cheap and effective.

ACO refresher (Dorigo 1992)

Ant Colony Optimization: ants traverse a graph; each path has a pheromone trail. Ant move probabilities weight by pheromone strength. Ants that complete the task deposit pheromone proportional to solution quality. Pheromone decays over time.

AMRO-S — ACO for agent routing

arXiv:2603.12933 uses ACO for multi-agent routing. Each task-type is a "destination"; each agent is a possible route. Pheromones strengthen routes that produce good outputs. Key contributions:

• Interpretable routing evidence. Pheromone strength is a human-readable signal.
• Quality-gated asynchronous update. Pheromones update only after quality checks pass, decoupling inference from learning.
• 4.7x speedup on the multi-agent routing benchmark.

The quality gate matters: without it, fast-but-wrong agents accrue pheromone, and the system locks in on bad routes.

When to use PSO / ACO for LLMs

Use PSO when:

• Search space is continuous or maps to continuous parameters (prompt embeddings, LoRA weights, numeric generation parameters).
• Fitness is cheap and automatic.
• Population can be small (10-30).

Use ACO when:

• You have a routing or path-selection problem.
• Decisions reinforce over time (the same task types come back).
• You need interpretable evidence for routing decisions.

Do not use either when:

• Fitness requires human review (too expensive per iteration).
• The search space is discrete and combinatorial in a way that PSO does not cover (use genetic algorithms instead).
• Real-time decisions need strict latency (PSO/ACO converge slowly relative to single-pass heuristics).

Why bio-inspired still wins

Gradient-based methods need differentiable signals. LLM outputs and routing decisions are not trivially differentiable. Pseudo-gradient methods (reinforcement-learned routers, DPO-style prompt tuners) work but need expensive training.

PSO and ACO need only an evaluator function. If you can score a candidate output or a routing decision, you can optimize over the space. That makes the bar for applicability much lower.

Practical limits

• Population budget. N particles × T iterations × per-eval cost. For LLM evals at ~$0.02 / call, a 20-particle PSO running 50 iterations costs ~$20. Plan accordingly.
• Exploration vs exploitation. Pheromone decay rate and PSO inertia trade off; too fast decay → forget solutions; too slow → stuck on early local optima.
• Catastrophic drift. Both algorithms can converge and then diverge if fitness landscape shifts (new data distribution). Monitor best-fitness stability.

Build It

code/main.py implements:

• LMPSO — PSO over numeric prompt parameters (temperature, top_k weights). Each particle's "LLM generation" is simulated as a scripted fitness function. Runs the algorithm for 30 iterations and shows g_best convergence.
• AMRO_S — ACO-style routing. 3 agents, 4 task types, pheromone matrix, 100 routed tasks. Prints (task_type → agent choices) distribution over time to show trail formation.
• Comparison: random routing vs ACO routing on the same task stream. Measures quality and latency.

Run:

Copypython3 code/main.py

Expected output:

• LMPSO: g_best fitness improves from random to near-optimal over 30 iterations.
• AMRO-S: pheromone table stabilizes on the right agent per task-type; ACO routing beats random by ~30-40% on quality and also reduces latency (fewer retries).

Use It

outputs/skill-swarm-optimizer.md helps choose between PSO, ACO, genetic algorithms, and gradient-based optimizers for LLM / agent optimization problems.

Ship It

• Start small. 10-20 particles, 20-50 iterations. Scale up only if the convergence curve shows clear gain.
• Log pheromones or g_best per iteration. Debugging swarm optimizers without a trail is painful.
• Quality-gate updates. Especially for ACO routing: fast-and-wrong agents must not accrue pheromone.
• Reset decay on distribution shift. When your eval distribution changes, aged pheromones are stale; reset or double the decay rate temporarily.
• Cap the per-iteration cost. Emit a cost-per-iteration metric. PSO that costs $500 / iteration and gains 0.5% is not shippable.

Exercises

1. Run code/main.py. Observe LMPSO convergence. Vary population size 5, 10, 20, 50. At what size does time-to-converge saturate?
2. Implement a "catastrophic drift" experiment: after iteration 30, change the fitness function. How fast does PSO adapt? Does resetting p_best help?
3. Add a quality gate to AMRO-S: pheromone deposit only on runs with eval score > 0.7. How does this change convergence vs the un-gated version?
4. Read LMPSO (arXiv:2504.09247). Map the paper's "velocity as a prompt" back to your numeric velocity. What is lost in the simulation and what is preserved?
5. Read AMRO-S (arXiv:2603.12933). Implement the decoupled "inference fast-path" with asynchronous pheromone update. How does this change system latency under sustained load?

Key Terms

Further Reading

• Kennedy & Eberhart — Particle Swarm Optimization — the 1995 PSO paper
• Dorigo — Ant Colony Optimization — 1992 ACO foundations
• LMPSO — Language Model Particle Swarm Optimization — PSO for structured LLM outputs
• Model Swarms — gradient-free LLM expert optimization — PSO on model-weight subspace
• AMRO-S — ant-colony multi-agent routing — pheromone-driven routing with quality gate

Diagrams1 figure