SCoUT: Scalable Communication via Utility-Guided Temporal Grouping in Multi-Agent Reinforcement Learning

Abstract

Communication can improve coordination in partially observed multi-agent reinforcement learning (MARL), but learning when and who to communicate with requires choosing among many possible sender-recipient pairs, and the effect of any single message on future reward is hard to isolate. We introduce SCoUT (Scalable Communication via Utility-guided Temporal grouping), which addresses both these challenges via temporal and agent abstraction within traditional MARL. During training, SCoUT resamples soft agent groups every K environment steps (macro-steps) via Gumbel-Softmax; these groups are latent clusters that induce an affinity used as a differentiable prior over recipients. Using the same assignments, a group-aware critic predicts values for each agent group and maps them to per-agent baselines through the same soft assignments, reducing critic complexity and variance. Each agent is trained with a three-headed policy: environment action, send decision, and recipient selection. To obtain precise communication learning signals, we derive counterfactual communication advantages by analytically removing each sender's contribution from the recipient's aggregated messages. This counterfactual computation enables precise credit assignment for both send and recipient-selection decisions. At execution time, all centralized training components are discarded and only the per-agent policy is run, preserving decentralized execution. Experiments on large-scale benchmarks show that SCoUT learns targeted communication and remains effective in large scenarios, while prior methods degrade as the population grows. Finally, ablations confirm that temporal grouping and counterfactual communication credit are both critical for scalability.

Methodology

SCoUT learns both whom to talk to (via a soft grouping policy over agent descriptors) and what to send (via a shared descriptor and message pool), using a centralized PPO with group-aware value baselines and counterfactual advantages for communication actions.

Results

SCoUT maintains high win rates and elimination rates across Battle scales and strong catch rates across Pursuit scales; ablations show that both temporal grouping and counterfactual communication credit are critical.

Battle: training curves across 20v20, 64v64, 81v81, 100v100.

Pursuit: catch rate and completion rate across scales.

MAgent Battle

Win rate, elimination rate, milestone reach rates R₅₀/R₇₅, and decisiveness via TT₅₀/TT₇₅ (20 evaluation seeds).

Pursuit (PettingZoo SISL)

Catch%, Done%, and milestone times TT₅₀/TT₇₅ (20 evaluation seeds; horizon=500). For milestone rows, each entry is TT_k (R_k), where R_k is the % of episodes reaching the k% capture milestone.

SCoUT in action

SCoUT policies across environment scales.

MAgent Battle

20v20

64v64

81v81

100v100

PettingZoo Pursuit

20p-8e

40p-16e

60p-24e

80p-32e

100p-40e

BibTeX

To be added upon publication.