3,089
Failure records
from 8 public repositories
43.2%
Fail at a component interface
compatibility + parsing + tool
24.8%
Largest single category: Functionality / reasoning defect
95% CI 23.3–26.4%
31.8%
Escape runtime detection
reach the user as wrong output
97.0%
Taxonomy coverage
92 records unclassified
What this study found
Four results, each with the evidence that supports it.
Most failures are interface failures. The three categories that describe a break between components — compatibility and API contracts, output parsing, and tool integration — account for 43.2% of records. The two categories unique to multi-agent architectures, communication and context, account for 8.4%. Orchestration failures are real and measurable, but they are not the plurality.
Cause is associated with type, but the causal evidence is thin. Type and stated cause are strongly associated (Cramér's V = 0.374, χ² = 3881, p < 1e-12), yet only 374 records (12.1%) name a checkable mechanism. The rest are graded supported or hypothesised, and no causal claim in this study rests on them.
Frameworks fail differently, not worse. Composition differs (V = 0.262) and no two frameworks share a dominant category. Blocking-severity rate ranges from 36.5% to 96.0% — a property of how loudly a framework fails, not of its quality.
About a third of failures are silent. 31.8% were not detected by the framework runtime and reached the user as incorrect output rather than as an error. This is the strongest argument in the study for output validation over exception monitoring.
Distribution of failure types
Denominator includes the 92 records coded unknown; intervals are Wilson 95% score intervals.
| Code | Category | n | Share | 95% CI |
|---|---|---|---|---|
| A1 | Functionality / reasoning defect | 766 | 24.8% | 23.3–26.4% |
| A3 | Compatibility / API-contract defect | 541 | 17.5% | 16.2–18.9% |
| A6 | Data structure / output-parsing defect | 420 | 13.6% | 12.4–14.8% |
| A2 | Tool / integration defect | 374 | 12.1% | 11.0–13.3% |
| A7 | Configuration / environment defect | 357 | 11.6% | 10.5–12.7% |
| A5 | Context / memory / state defect | 255 | 8.3% | 7.3–9.3% |
| A8 | Performance / resource defect | 170 | 5.5% | 4.8–6.4% |
| A10 | Insufficient evidence | 92 | 3.0% | 2.4–3.6% |
| A4 | Agent communication / orchestration defect | 91 | 2.9% | 2.4–3.6% |
| A9 | Safety / guardrail defect | 23 | 0.7% | 0.5–1.1% |
Where the failures are
Explore the corpus
Click a record to see its evidence span, the rules that produced its label, and a link to the original report.
Framework comparison
Composition, not quality. A blocking rate is a property of the issue and of the framework's error handling.
| Framework | Records | Share | Blocking | Dominant category | Share |
|---|---|---|---|---|---|
| AutoGPT | 598 | 19.4% | 50.0% | Functionality / reasoning defect | 37.3% |
| LangGraph | 539 | 17.4% | 60.7% | Context / memory / state defect | 33.6% |
| CrewAI | 533 | 17.3% | 58.5% | Functionality / reasoning defect | 23.1% |
| LlamaIndex | 436 | 14.1% | 36.5% | Functionality / reasoning defect | 35.8% |
| OpenAI-Agents | 337 | 10.9% | 36.5% | Tool / integration defect | 30.6% |
| AutoGen | 300 | 9.7% | 96.0% | Compatibility / API-contract defect | 28.7% |
| MetaGPT | 238 | 7.7% | 88.2% | Data structure / output-parsing defect | 29.0% |
| SWE-agent | 108 | 3.5% | 86.1% | Configuration / environment defect | 32.4% |
No two frameworks share a dominant category. That spread is
the strongest descriptive evidence in this study that orchestration design
shapes which part of the system breaks.
Severity and silent failure
Escape rate is the share of records not detected by the framework runtime.
| Failure type | n | Runtime-caught | Escape rate |
|---|---|---|---|
| Safety / guardrail defect | 23 | 9 | 60.9% |
| Functionality / reasoning defect | 766 | 358 | 53.3% |
| Tool / integration defect | 374 | 218 | 41.7% |
| Context / memory / state defect | 255 | 149 | 41.6% |
| Data structure / output-parsing defect | 420 | 291 | 30.7% |
| Agent communication / orchestration defect | 91 | 67 | 26.4% |
| Performance / resource defect | 170 | 129 | 24.1% |
| Compatibility / API-contract defect | 541 | 436 | 19.4% |
| Configuration / environment defect | 357 | 357 | 0.0% |
Declared complexity
Share of orchestration and context failures by complexity tier. Complexity is declared in the report, not instrumented; read this as a stratification, not a causal estimate.
| Tier | Records | Orchestration / context | Share | 95% CI |
|---|---|---|---|---|
| Tier 0 | 699 | 18 | 2.6% | 1.6–4.0% |
| Tier 1 | 1258 | 68 | 5.4% | 4.3–6.8% |
| Tier 2 | 921 | 189 | 20.5% | 18.0–23.2% |
| Tier 3 | 211 | 71 | 33.7% | 27.6–40.3% |
The share rises monotonically from
2.6% to
33.7% (Spearman ρ =
1.00). Complexity relocates the failure
surface toward coordination; it does not demonstrably shrink it.
Stated causes
Cause class across the corpus. The final column counts records where the report names a checkable mechanism.
| Cause class | n | Share | Demonstrated |
|---|---|---|---|
| Expectation mismatch | 655 | 21.2% | 0 |
| Unknown | 487 | 15.8% | 0 |
| Configuration error | 471 | 15.2% | 88 |
| Specification / documentation ambiguity | 469 | 15.2% | 117 |
| API / version drift | 267 | 8.6% | 34 |
| Framework defect | 260 | 8.4% | 88 |
| Context / state mismanagement | 194 | 6.3% | 30 |
| Tool or environment unavailability | 110 | 3.6% | 6 |
| Model capability limitation | 109 | 3.5% | 7 |
| Application logic defect | 67 | 2.2% | 4 |
The codebook
Five dimensions, coded independently. Every category below has a definition, an inclusion list and an exclusion list, so that two coders can reach the same label from the same text. Click a category to expand it.
The dimensions are orthogonal on purpose: a compatibility failure can be caused by a framework defect, a version bump or a user misconfiguration, and those have different remediations. Collapsing them is why "agent errors" reports rarely generalise.
A. Failure type
What broke, structurally — 10 categories
Structural signals (tool invocation, inter-agent handoff, output parsing, environment setup) outrank generic complaint language. When a report contains a traceback, the raising component decides the label. Generic phrases such as "does not work" never override a structural signal.
A1_functionality — Functionality / reasoning defect
The agent architecture ran as designed but produced an incorrect result: wrong answer, wrong plan, wrong code, wrong decision, or an incomplete task outcome.
Includes:
- Output is wrong but syntactically well formed
- Plan or delegation choice is logically incorrect
- Agent silently skips part of the requested task
Excludes:
- A specific component raised an exception (code that category)
- The reported problem is purely about installation or configuration
A2_tool_integration — Tool / integration defect
Failure in the interface between the agent and an external capability: tool registration, tool selection, argument construction, MCP server connection, or tool result handling.
Includes:
- Tool not found / not registered / not exposed to the model
- Arguments built wrongly for a tool call
- MCP server, plugin or connector fails to load
- Tool returns a value the agent cannot consume
Excludes:
- The tool works but the agent chose the wrong tool (code A1)
A3_compatibility — Compatibility / API-contract defect
A mismatch between component versions, or between the framework's expected interface and the installed/upstream interface. Includes breaking changes, removed parameters, renamed symbols and dependency conflicts.
Includes:
- TypeError on an unexpected or missing keyword argument
- ImportError / ModuleNotFoundError for a moved or renamed symbol
- Version pin conflict between framework and dependency
- Deprecated parameter removed upstream
Excludes:
- A user-supplied value of the wrong type with correct API (code A6)
A4_communication — Agent communication / orchestration defect
Failure originating in the exchange between agents: message routing, handoff, delegation, speaker selection, turn taking, termination, or shared-conversation state.
Includes:
- Handoff or delegation never occurs, or targets the wrong agent
- Conversation loops, never terminates, or terminates early
- Manager/supervisor misroutes a task
- Agents disagree and no resolution mechanism applies
Excludes:
- A single agent fails alone with no inter-agent exchange (code A1/A2)
A5_context_memory — Context / memory / state defect
Failure caused by what the agent could see or remember: context truncation, lost history, memory persistence errors, or corrupted run state.
Includes:
- Earlier instructions dropped from the context window
- Memory store returns stale or cross-session data
- Checkpoint/resume restores an inconsistent state
Excludes:
- The context was present but the model ignored it (code A1)
A6_data_structure — Data structure / output-parsing defect
The agent produced output whose structure was unusable: schema validation failure, JSON parsing failure, type mismatch, or serialisation error. Includes the supervisor's "data structure failures".
Includes:
- Pydantic/JSON schema validation error on agent output
- Output parser cannot extract the expected block
- KeyError / AttributeError on a malformed payload
Excludes:
- Parsing failed because the tool was never called (code A2)
A7_configuration — Configuration / environment defect
Failure in setup rather than in agent behaviour: missing environment variables, credentials, settings, install steps, container setup, or unsupported runtime.
Includes:
- Missing API key or environment variable
- Installation or dependency resolution fails before any agent runs
- Configuration option ignored or has no effect
Excludes:
- Version incompatibility between packages (code A3)
A8_performance — Performance / resource defect
Failure of completion due to resource or timing limits: timeouts, rate limits, memory exhaustion, excessive token consumption, or prohibitive latency.
Includes:
- Request times out or is killed by a deadline
- Provider rate limit reached
- Run exhausts memory or token budget
Excludes:
- Slow but correct completion with no failure (not a record)
A9_safety — Safety / guardrail defect
Failure of control rather than capability: unsafe action executed, permission boundary crossed, prompt injection accepted, or guardrail bypassed.
Includes:
- Agent executes a destructive action it should not
- Injected instruction from tool output is followed
- Sandbox or permission restriction is not enforced
Excludes:
- Policy refusal that is merely inconvenient (code A1)
A10_unknown — Insufficient evidence
The report does not contain enough information to assign a category. Retained in the dataset and reported explicitly; never silently dropped.
Includes:
- Title-only report with no described symptom
- Report describes a question or feature request rather than a failure
B. Cause class
The mechanism the report names — 10 categories
B1_framework_defect — Framework defect
A bug in the orchestration framework's own code.
B2_api_version_drift — API / version drift
A change in an upstream dependency or framework version broke an existing application.
B3_configuration_error — Configuration error
Incorrect or missing configuration by the application developer.
B4_specification_ambiguity — Specification / documentation ambiguity
The documented behaviour is unclear, absent, or contradicted the actual behaviour.
B5_model_limitation — Model capability limitation
The underlying model could not perform the required reasoning or instruction following.
B6_context_management — Context / state mismanagement
Information needed for the task was not preserved, routed, or restored.
B7_tool_environment — Tool or environment unavailability
An external dependency was unavailable, misbehaving, or unconfigured.
B8_application_logic — Application logic defect
The defect is in the developer's own agent code, not the framework or model.
B9_expectation_mismatch — Expectation mismatch
The behaviour is by design; the reporter expected something else.
B10_unknown — Unknown
No cause recoverable from the report.
C. Lifecycle stage
Where in the run it broke — 8 categories
C1_setup — Installation, dependency resolution, environment or credentials
C2_construction — Building agents, teams, graphs, or tools before execution
C3_prompt_assembly — Assembling task description, roles, or context for a call
C4_inference — The model call itself
C5_tool_execution — Invoking or receiving from a tool or external service
C6_handoff — Passing control or state between agents
C7_output_parsing — Interpreting or validating the model's structured output
C8_runtime_other — Other runtime behaviour after a successful call
D. Severity
How bad the consequence was — 4 categories
D1_blocking — Execution cannot continue; crash, exception, or hard stop
D2_incorrect_output — Run completes but the deliverable is wrong or silently corrupted
D3_degraded — Run completes with loss of quality, workaround, or partial output
D4_minor — Cosmetic, noisy logging, or non-blocking annoyance
E. Detection point
Who noticed, and when — 5 categories
E1_framework_runtime — Raised by the framework or runtime as an exception
E2_external_test — Caught by a test, evaluation harness, or assertion
E3_human_observation — Noticed by a human inspecting output or behaviour
E4_production_monitoring — Detected by logging, metrics, or alerting in operation
E5_configuration_validation — Caught during setup or configuration validation
Coding rules
The precedence order applied when more than one category appears to fit.
R1 - Exception decides
If the report contains a Python exception, the component that raised it decides dimension A. TypeError on a call signature is A3; ValidationError is A6; a tool-registry error is A2.
R2 - Tool beats compatibility
When a failure concerns a tool call whose arguments fail validation, code A2, not A6, because the agent-tool interface is the failing mechanism.
R3 - Communication requires two agents
A4 applies only when the report describes an exchange between agents. A single agent looping internally is A1.
R4 - Never force a label
If the evidence supports no category with at least moderate confidence, code A10/B10 and record the reason. Unknown records stay in every denominator unless explicitly excluded.
R5 - Cause separate from type
Dimension B must not restate dimension A. A3 (type) can be caused by B2 (version drift) or B1 (framework defect); record which.
Research questions and agent assignment
A question is assigned to a single agent only when one pass over one table answers it. Questions that require deriving a variable, testing an association and then checking the claim against the source records are assigned to a chain, because the verification step is what prevents unsupported numbers from reaching a report.
RQ1. What is the distribution of failure types across agentic applications?agent chain
Frequency table with Wilson 95% CIs; denominator = retained failure records. · TaggerAgent, StatisticianAgent
24.8% of records are functionality / reasoning defect.
RQ2. Which failure types dominate, and how precisely is dominance estimated?agent chain
Ranking with CIs, plus a sensitivity analysis restricted to high-confidence labels. · TaggerAgent, StatisticianAgent, CriticAgent
Restricting to confident labels leaves the ranking unchanged.
RQ3. Which causes are associated with which failure types?agent chain
Contingency analysis, chi-square with Cramer's V, BH-corrected p-values, and lift per cell. · TaggerAgent, CausalAgent, CriticAgent
V = 0.374; 374 causes are demonstrated.
RQ4. Do frameworks differ in their mix of failure types?agent chain
Chi-square across frameworks; per-framework composition reported descriptively. · TaggerAgent, ComparatorAgent, StatisticianAgent
No two frameworks share a dominant category (V = 0.262).
RQ5. Does framework choice affect the rate of blocking-severity failures?agent chain
Blocking-rate per framework with Wilson CIs; pairwise Fisher tests; BH correction. · TaggerAgent, ComparatorAgent, StatisticianAgent, CriticAgent
Blocking rate spans 36.5% to 96.0%.
RQ6. Does declared agent complexity increase the likelihood of orchestration or context failures?agent chain
Complexity index (0-3) derived from reported architecture; trend test across tiers. · ComplexityAgent, StatisticianAgent, CriticAgent
Orchestration/context share rises from 2.6% to 33.7% across tiers.
RQ7. Are some failure types more likely to escape to users rather than be caught at runtime?agent chain
Escape rate = share not detected by framework runtime, tested across types. · TaggerAgent, StatisticianAgent
31.8% of failures escape runtime detection.
RQ8. How much of the corpus cannot be classified, and does that vary by framework?single agent
Unknown share with CIs per framework; lexical-coverage discussion. · StatisticianAgent
Coverage is 97.0%; unknowns stay in every denominator.
RQ9. Has the failure mix changed as the frameworks matured?agent chain
Failure composition by year with a chi-square trend; requires adequate per-year n. · TaggerAgent, ComparatorAgent, StatisticianAgent
Composition changed over time (V = 0.111), confounded by collection.
RQ10. Which failure types co-occur with tool and orchestration infrastructure?agent chain
Stratified composition: tool-using vs non-tool records; orchestrated vs single-agent records. · ComplexityAgent, CausalAgent, CriticAgent
Tool-heavy stacks move the failure surface to tool contracts (V = 0.338).
How the application is organised
Eight agents on a shared blackboard. Any agent can be re-run in isolation and its contribution inspected.
| Agent | Responsibility |
|---|---|
| Planner | Maps each question to the variables, the test and the output shape it requires. |
| Tagger | Resolves labels and confidence; exposes the evidence span for each. |
| Statistician | Distributions, intervals, association tests, corrected p-values. |
| Causal | Cause-to-type contingency analysis with lift per cell, graded by evidence. |
| Comparator | Framework and time contrasts with multiple-testing control. |
| Complexity | Declared capability index and trend across tiers. |
| Critic | Recomputes every headline metric on an independent code path; blocks a mismatch. |
| Reporter | Assembles verified results into tables, figures and prose. |
In the run shown here, all ten questions passed
verification with no rejections.
What the instrument can and cannot support
Five experiments test the labelling instrument rather than the findings. One is still pending, and it is stated as pending rather than approximated.
| ID | Experiment | Design | Result | Reading |
|---|---|---|---|---|
| E1 | Reference agreement | n = 100 | 52.0% exact · macro-F1 0.545 · κ = 0.45 | Moderate — residual error on two boundaries |
| E2 | Perturbation retention | n = 150 | 80.0% (95% CI 72.9–85.6%) | Labels survive rewording and anonymisation |
| E3 | Exception-rule ablation | n = 3089 | 2.1% of labels change | Not dependent on pasted tracebacks |
| E4 | Confidence calibration | n = 100 | 39% → 57% across bins | Informative for triage, not calibrated |
| E5 | Blind second coder | 40 records | pending | Requires a coder independent of the author |
Agreement of κ =
0.45 with a first-pass reference
annotation is moderate, not high. The residual error concentrates on two
codebook boundaries — compatibility versus configuration, and functionality
versus output parsing. Those are codebook problems, and they are the agenda
for v1.1.
Confusion against reference annotation
Downloads
The corpus, the codebook and the result tables, in the same folder as this page.
- Corpus (CSV)
- 1683.9 kB
- Corpus (JSONL)
- 3069.8 kB
- Excluded records
- 1.3 kB
- Codebook (YAML)
- 15.3 kB
- Research questions (YAML)
- 5.4 kB
- RQ1 table
- 0.9 kB
- RQ3 table
- 8.2 kB
- RQ4 table
- 5.2 kB
- Validation summary
- 20.7 kB
Every record shown here is traceable to a public bug report,
and the original report is linked from it. Users sometimes paste their own
API keys into those reports; any credential-shaped string is therefore
masked as [redacted: credential] on this page. The
stored corpus is left unmodified so the text still matches the source it
cites, and it is not published.
Record schema
One JSON object per line in the corpus file; the CSV carries the same fields.
| Field | Meaning |
|---|---|
| record_id | Stable hash; the join key across all artefacts. |
| repo, issue_number, url | Provenance — the original report. |
| created_at, year, state, labels | Tracker context. |
| A_failure_type, A_confidence, A_evidence | Primary dimension, confidence, and the text span that produced the label. |
| B_cause_class, B_evidence_grade | Stated cause, graded demonstrated / supported / hypothesised. |
| C_lifecycle_stage, D_severity, E_detection_point | Remaining three dimensions. |
| matched_rules, taxonomy_version | Audit trail: which rules fired, against which codebook. |
Reproducing the analysis
From the project root, with PYTHONPATH=src set:
python -m afs.cli build # normalise + label python -m afs.cli analyze # multi-agent pipeline with critic verification python -m afs.cli validate # validation experiments python -m afs.cli export # Word report python -m afs.cli paper # journal-style manuscript python -m afs.cli app # this page
Regenerating the corpus is deterministic: the labeller is
rule-based, so the same text yields the same label and the same evidence
span. Collection is rate-limited by the public API and takes about six
minutes; every later stage is offline.