Multi-Agent Debate and Collaboration Systems with AutoGen
A practical use of multi-agent systems is structured debate and collaboration, where multiple AI agents with different perspectives discuss a problem to arrive at better solutions than any single agent could produce alone. AutoGen, developed by Microsoft Research, provides a solid framework for this kind of multi-agent interaction.
AutoGen's group chat functionality allows you to create teams of agents that take turns speaking, challenge each other's reasoning, and collectively work through complex problems. I will walk through building debate and collaboration systems using AutoGen 0.2: group chat orchestration, speaker selection strategies, and patterns for getting good outputs from agent teams.
Why Multi-Agent Debate?
Research has shown that LLMs produce higher-quality outputs when they engage in self-reflection and critique. Multi-agent debate takes this further by externalizing the reflection process across separate agents, each with distinct instructions and perspectives. A "devil's advocate" agent challenges assumptions. A "fact checker" agent verifies claims. A "synthesizer" agent combines the best elements into a final answer.
This approach is particularly effective for:
- Complex reasoning tasks where a single pass often misses edge cases
- Creative problem-solving that benefits from diverse viewpoints
- Code review and debugging where multiple perspectives catch different issues
- Decision analysis where trade-offs need explicit examination
Installation and Setup
pip install pyautogen
import os
os.environ["OPENAI_API_KEY"] = "your-api-key-here"
llm_config = {
"config_list": [
{
"model": "gpt-4o",
"api_key": os.environ["OPENAI_API_KEY"],
}
],
"temperature": 0.7,
"cache_seed": None, # Disable caching for varied responses
}
Building a Two-Agent Debate
Let us start with the simplest debate pattern: two agents with opposing roles discussing a topic.
import autogen
# The proposer argues in favor
proposer = autogen.AssistantAgent(
name="Proposer",
system_message=(
"You are a technology strategist who advocates for adopting new "
"technologies. When discussing technical decisions, you emphasize "
"benefits, innovation potential, and competitive advantages. Support "
"your arguments with specific examples and data points. Be persuasive "
"but honest about trade-offs when directly challenged."
),
llm_config=llm_config,
)
# The critic argues against or raises concerns
critic = autogen.AssistantAgent(
name="Critic",
system_message=(
"You are a senior technical architect focused on risk management and "
"pragmatism. When discussing technical decisions, you identify potential "
"pitfalls, hidden costs, operational complexity, and adoption risks. "
"You are not opposed to new technology but insist on rigorous evaluation. "
"Ask probing questions and demand evidence for claims."
),
llm_config=llm_config,
)
# Human proxy to initiate the conversation
user_proxy = autogen.UserProxyAgent(
name="Moderator",
human_input_mode="NEVER",
max_consecutive_auto_reply=0,
code_execution_config=False,
)
Initiate the debate:
user_proxy.initiate_chat(
proposer,
message=(
"Should our engineering team migrate from a monolithic architecture "
"to microservices? Please present your case, then the Critic will "
"respond with concerns."
),
max_turns=6,
)
Group Chat: Multi-Agent Collaboration
AutoGen gets more interesting with group chats, where multiple agents interact in a managed conversation:
import autogen
researcher = autogen.AssistantAgent(
name="Researcher",
system_message=(
"You are a thorough technical researcher. When given a topic, you "
"provide detailed factual information, cite relevant studies or "
"documentation, and present data objectively. You do not advocate "
"for any position — you present facts for others to analyze."
),
llm_config=llm_config,
)
architect = autogen.AssistantAgent(
name="Architect",
system_message=(
"You are a systems architect who focuses on design patterns, "
"scalability, and maintainability. You evaluate technical approaches "
"based on architectural merit and long-term sustainability. You draw "
"diagrams in text form when helpful and always consider non-functional "
"requirements."
),
llm_config=llm_config,
)
security_expert = autogen.AssistantAgent(
name="SecurityExpert",
system_message=(
"You are a cybersecurity specialist who evaluates every technical "
"decision through a security lens. You identify threat vectors, "
"compliance requirements, and data protection concerns. You suggest "
"specific mitigations for every risk you identify."
),
llm_config=llm_config,
)
synthesizer = autogen.AssistantAgent(
name="Synthesizer",
system_message=(
"You are a technical lead who synthesizes discussions into actionable "
"recommendations. You listen to all perspectives, identify areas of "
"agreement and disagreement, and produce a clear summary with concrete "
"next steps. You speak last and your summary should be the definitive "
"output of the discussion."
),
llm_config=llm_config,
)
user_proxy = autogen.UserProxyAgent(
name="Admin",
human_input_mode="NEVER",
max_consecutive_auto_reply=0,
code_execution_config=False,
)
Configuring the Group Chat
group_chat = autogen.GroupChat(
agents=[user_proxy, researcher, architect, security_expert, synthesizer],
messages=[],
max_round=12,
speaker_selection_method="auto",
allow_repeat_speaker=False,
)
manager = autogen.GroupChatManager(
groupchat=group_chat,
llm_config=llm_config,
)
user_proxy.initiate_chat(
manager,
message=(
"We need to design a real-time notification system for our platform "
"that handles 50,000 concurrent users. The system must support push "
"notifications, email, and SMS channels. Please discuss the best "
"approach, covering architecture, technology choices, and security "
"considerations. The Synthesizer should provide a final recommendation."
),
)
The speaker_selection_method="auto" setting lets the LLM decide which agent should speak next based on the conversation context. This produces natural discussion flow. Alternatively, you can use "round_robin" for strict turn-taking or provide a custom function.
Custom Speaker Selection
For more control over the conversation flow, implement a custom speaker selection function:
def custom_speaker_selection(last_speaker, group_chat):
"""Custom logic to determine who speaks next."""
messages = group_chat.messages
# Researcher always goes first after the admin
if len(messages) <= 2:
return researcher
# After 8 messages, let the synthesizer wrap up
if len(messages) >= 8:
return synthesizer
# Alternate between architect and security expert
if last_speaker == researcher or last_speaker == security_expert:
return architect
elif last_speaker == architect:
return security_expert
return None # Fall back to auto selection
group_chat = autogen.GroupChat(
agents=[user_proxy, researcher, architect, security_expert, synthesizer],
messages=[],
max_round=10,
speaker_selection_method=custom_speaker_selection,
)
Adding Code Execution Capabilities
AutoGen's code execution feature lets agents write and run code during their discussion, which is powerful for technical debates where claims can be verified programmatically:
code_executor = autogen.UserProxyAgent(
name="CodeExecutor",
human_input_mode="NEVER",
max_consecutive_auto_reply=3,
code_execution_config={
"work_dir": "coding_workspace",
"use_docker": False, # Set True in production for safety
},
)
coder = autogen.AssistantAgent(
name="Coder",
system_message=(
"You are a Python developer who writes code to verify claims, "
"run benchmarks, and prototype solutions during technical discussions. "
"When someone makes a performance claim or proposes an algorithm, "
"write code to test it. Always include clear output messages."
),
llm_config=llm_config,
)
In a debate about algorithm performance, the coder agent might write a benchmark script, the code executor runs it, and the results feed back into the discussion with real data rather than speculation.
Structured Debate Pattern
Here is a complete pattern for a structured technical debate with defined phases:
def run_structured_debate(topic: str, num_rounds: int = 3):
"""Run a structured multi-phase debate on a technical topic."""
advocate = autogen.AssistantAgent(
name="Advocate",
system_message=(
"You advocate for the proposed approach. Present strong arguments "
"with evidence. Respond directly to criticisms raised by the Skeptic."
),
llm_config=llm_config,
)
skeptic = autogen.AssistantAgent(
name="Skeptic",
system_message=(
"You critically examine the proposed approach. Identify weaknesses, "
"risks, and alternatives. Challenge assumptions with specific "
"counter-examples."
),
llm_config=llm_config,
)
judge = autogen.AssistantAgent(
name="Judge",
system_message=(
"You are an impartial judge. After hearing both sides, evaluate "
"the strength of each argument. Identify which points were well-"
"supported and which were weak. Deliver a final verdict with a "
"clear recommendation and conditions for success."
),
llm_config=llm_config,
)
moderator = autogen.UserProxyAgent(
name="Moderator",
human_input_mode="NEVER",
max_consecutive_auto_reply=0,
code_execution_config=False,
)
group_chat = autogen.GroupChat(
agents=[moderator, advocate, skeptic, judge],
messages=[],
max_round=(num_rounds * 2) + 3,
speaker_selection_method="round_robin",
)
manager = autogen.GroupChatManager(
groupchat=group_chat,
llm_config=llm_config,
)
moderator.initiate_chat(
manager,
message=(
f"DEBATE TOPIC: {topic}\n\n"
"Format: The Advocate will present the case, the Skeptic will "
f"challenge it, and they will go back and forth for {num_rounds} "
"rounds. The Judge will then deliver a final verdict.\n\n"
"Advocate, please begin with your opening argument."
),
)
return group_chat.messages
# Run the debate
messages = run_structured_debate(
"Should we adopt GraphQL to replace our REST API layer?",
num_rounds=3,
)
Extracting Structured Results
After a debate, you often want to extract the key findings in a structured format:
import json
def extract_debate_summary(messages: list, llm_config: dict) -> dict:
"""Extract a structured summary from debate messages."""
summarizer = autogen.AssistantAgent(
name="Summarizer",
system_message=(
"You extract structured summaries from debates. Given a conversation, "
"output a JSON object with these fields: topic, arguments_for (list), "
"arguments_against (list), verdict, confidence (high/medium/low), "
"and next_steps (list). Output only valid JSON, no other text."
),
llm_config=llm_config,
)
proxy = autogen.UserProxyAgent(
name="Proxy",
human_input_mode="NEVER",
max_consecutive_auto_reply=0,
code_execution_config=False,
)
conversation_text = "\n".join(
f"{msg['name']}: {msg['content']}" for msg in messages
)
proxy.initiate_chat(
summarizer,
message=f"Summarize this debate as JSON:\n\n{conversation_text}",
max_turns=1,
)
last_message = summarizer.last_message()["content"]
return json.loads(last_message)
Best Practices
After building several multi-agent debate systems with AutoGen, here are the patterns that consistently produce the best results:
- Distinct personas: Each agent needs a clearly differentiated role and perspective. Vague system messages lead to agents that all say the same thing.
- Constrained rounds: Set explicit limits on debate rounds. Without them, agents tend to become repetitive after 4-5 exchanges.
- Structured output: Always include a synthesizer or judge agent that produces a structured final output. Open-ended debates without a conclusion are less useful.
- Temperature tuning: Use slightly higher temperatures (0.7-0.9) for debate agents to encourage diverse perspectives, but lower temperatures (0.1-0.3) for synthesizer agents that need to be precise.
- Disable caching: Set
cache_seed=Noneso that repeated runs produce different debates rather than returning cached results.
Conclusion
AutoGen's group chat framework makes it straightforward to build multi-agent debate and collaboration systems. In practice, multiple agents that challenge and refine each other's reasoning often beat a single, more powerful model. The debate pattern consistently produces more thorough results than single-agent approaches. Start with a simple two-agent debate, observe how the agents interact, and add more specialized agents as your use cases demand.