AI Prompt to Triage Flaky CI Test Failures Using GitHub's MCP Server

0 views Updated Oct 14, 2026

This AI prompt helps engineers and tech leads figure out which failing CI tests are genuine bugs and which are flaky tests that fail intermittently for unrelated reasons, using GitHub's MCP Server to pull the actual workflow run history, logs, and commit diffs for a repository instead of relying on a single failed run. A model connected through MCP can query recent runs of the same test across different commits and spot whether a failure correlates with a specific code change or shows up randomly regardless of what changed.

The prompt asks the model to compare the failing test's recent run history, check whether the failure message is identical across runs or varies (a sign of a race condition or timing issue), and cross-reference the commits that landed around the first failure. This saves engineers from re-running a flaky suite repeatedly by hand or quarantining a genuinely broken test as 'just flaky' and shipping a real regression.

Because CI logs and run histories can run long and get expensive to pass through a model repeatedly during an active incident, teams pulling large volumes of log output alongside this prompt can use Token Optimizer to trim verbose logs down before each triage pass.

Prompt template

prompt-template

345 tokens

Role: You are a CI reliability engineer helping triage test failures. Context: - Repository: [REPO_NAME] - Failing test(s): [TEST_NAME_OR_FILE] - Workflow or CI pipeline: [WORKFLOW_NAME] - Number of recent runs to review: [NUMBER_OF_RUNS, e.g. 15] - Approximate time the failures started: [DATE_OR_COMMIT_RANGE] Task: 1. Retrieve the run history for the specified test across the last [NUMBER_OF_RUNS] workflow runs. 2. For each run, note whether the test passed or failed, and if it failed, record the exact failure message and stack trace line. 3. Determine whether the failure message and location are consistent across failing runs or vary from run to run. 4. Identify the commits merged in the window right before the failures began and flag any that touch the same files, dependencies, or test fixtures as the failing test. 5. Classify the failure as either "Likely flaky" (intermittent, inconsistent messages, no related code change) or "Likely real regression" (consistent failure tied to a specific commit). Constraints: - Do not classify a test as flaky without at least 3 failed runs to support it. - Always name the specific commit(s) you suspect if classifying as a real regression. - State your confidence level (high, medium, low) for the classification. Output format: Test: [name] Classification: [Likely flaky / Likely real regression] Confidence: [high/medium/low] Evidence: [run-by-run summary] Suspect commit(s), if any: [commit hash and message] Recommended next step: [one sentence]

Want it sharper? Optimize this prompt with Prompt Optimizer, check it with the Prompt Debugger or shorten it with the Token Optimizer.

Example input

example-input

74 tokens

Repository: acme-corp/checkout-service Failing test: tests/payments/test_retry_logic.py::test_retry_on_timeout Workflow: ci.yml (test suite) Number of recent runs to review: 15 Approximate time failures started: last 4 days, around commit range a1b2c3d..f9e8d7c

When to use it

A CI pipeline has intermittent test failures and the team needs to know which ones are flaky versus real regressions before merging.
You're deciding whether to quarantine a test or investigate it, and want evidence from run history instead of a gut call.
A release is blocked by failing tests and you need a fast triage of which failures are safe to re-run and which need a fix first.
You want a repeatable process for distinguishing timing-related test flakiness from genuine logic bugs introduced by a recent commit.

Best practices

Pull the test's run history across at least the last 10-15 runs through MCP rather than judging from one failed run, so you can see whether the failure is consistent or intermittent.
Ask the model to check whether the failure message, stack trace, or line number changes between runs, since identical failures point to a real bug and varying ones point to flakiness.
Have it cross-reference the commits merged around the first failure, not just the most recent one, since flaky tests sometimes start failing a few commits after the actual cause landed.
When log output from multiple runs is too long to review efficiently, trim it with Token Optimizer before feeding it into the triage prompt.

Common mistakes

Labeling a test as flaky after only one or two failures instead of checking its run history for a real pattern.
Quarantining a test without checking whether a genuine regression landed around the same time the failures started.
Ignoring differences in failure messages between runs, which is often the clearest signal separating a timing issue from a logic bug.
Triaging the symptom (the failing test) without looking at what changed in the surrounding code or infrastructure right before failures began.

FAQs

How do I tell if a failing CI test is flaky or a real bug?

Check the test's run history instead of a single failure: if the failure message and stack trace are identical across multiple runs and line up with a specific commit, it's likely a real regression; if the failures are inconsistent and uncorrelated with any code change, it's more likely flaky.

How many failed runs should I look at before calling a test flaky?

At least 3 to 5 failed runs is a reasonable minimum before quarantining a test as flaky, since one or two failures can easily be a real but intermittent bug rather than true flakiness.

Can an AI model actually pull my GitHub Actions run history automatically?

Yes, if it's connected through GitHub's MCP server it can query workflow run history, logs, and the commits associated with each run directly, rather than you needing to copy and paste logs manually.

What should I do with a test I've confirmed is genuinely flaky?

Quarantine it so it doesn't block merges, but track it separately and revisit it, since a test that's flaky due to a real race condition can mask a production bug if ignored indefinitely.

What tool can help me check this prompt's quality before using it on a live incident?

Prompt Debugger — scans the prompt for vague constraints or missed edge cases, which matters here since a weak classification rule could mislabel a real regression as flaky.

Found this prompt useful? Share it.

More in Engineering

Engineering

AI Prompt to Generate Release Notes From Jira Tickets Using Atlassian's MCP Server

This Jira MCP prompt turns a list of completed tickets into a readable set of release notes , built for engineering managers, release manage…

ROLE: You are a release notes writer working from live Jira data retrieved through an MCP connection.

CONTEXT:
- Jira project key: [PROJECT_KEY]
- Fix version or sprint to summarize: [FIX_VERSION_OR_SPRINT]
- Audience for these notes: [AUDIENCE, e.g. end users, internal team, enterprise customers]
- Tone: [TONE, e.g. plain and professional, upbeat and casual]

TASK:
1. Retrieve all tickets in [PROJECT_KEY] tagged with fix version or sprint [FIX_VERSION_OR_SPRINT] via the connected Jira MCP server.
2. Group the tickets into exactly these sections: "New Features", "Improvements", "Bug Fixes", and "Internal Changes (not customer-facing)".
3. For each ticket, write one short sentence in plain language describing what changed and why it matters to [AUDIENCE] — do not copy the raw ticket title verbatim.
4. Exclude the "Internal Changes" section from the customer-facing version if [AUDIENCE] is external; include it only in the internal version.
5. Flag any ticket labeled "security", "breaking-change", or with no description under a separate "Needs Manual Review" heading instead of guessing at its content.

CONSTRAINTS:
- Do not invent functionality that isn't described in the ticket data.
- Do not include Jira ticket keys in the customer-facing version; keep them in the internal version for traceability.
- Keep each bullet to one sentence unless the change is a breaking change, which gets two sentences: what changed and what action is required.

OUTPUT FORMAT:
Return the release notes as Markdown with the four section headings above (omitting empty sections), followed by a "Needs Manual Review" list if applicable.

Engineering

AI Prompt to Triage and Prioritize Engineering Issues Using Linear's MCP Server

This Linear MCP prompt helps engineering leads, team leads, and project managers use an AI assistant connected to Linear's Model Context Pro…

ROLE
You are an engineering triage assistant with access to a Linear workspace through its MCP server integration.

CONTEXT
Team or project: [TEAM_OR_PROJECT_NAME]
Time window: issues updated or opened in the last [TIME_WINDOW]
Priority scale in use: [PRIORITY_SCALE_DEFINITION]
Current sprint or cycle goal: [SPRINT_GOAL]

TASK
1. Pull all open issues matching the team, project, and time window above through the Linear MCP connection.
2. Group the issues into: blocked, in progress, and not started.
3. For each issue, assess severity using this rubric: [SEVERITY_RUBRIC], customer or revenue impact, how long it has been open, and whether it blocks other issues.
4. Flag any issue missing a priority, assignee, or label, and list it separately as a hygiene issue rather than ranking it.
5. Produce a prioritized list of the top [NUMBER] issues the team should work on next, with a one-line justification for each ranking.
6. Note any issues that appear to duplicate each other or contradict the stated sprint goal.

CONSTRAINTS
- Do not invent issue data that is not present in the Linear workspace; if a field is missing, say so explicitly.
- Keep each justification to one sentence.
- Do not take any write actions (status changes, reassignments) unless I explicitly confirm each one.

OUTPUT FORMAT
1. Hygiene issues (missing priority, assignee, or label) - bullet list
2. Blocked issues - bullet list with blocker noted
3. Prioritized list of top [NUMBER] issues - numbered, with one-line justification each
4. Possible duplicates or conflicts with sprint goal - bullet list

Engineering

AI Prompt to Write a Technical Design Document (RFC) for a New Feature

This technical design document prompt , sometimes called an RFC prompt, is for engineers and tech leads who need to turn a feature idea into…

ROLE: You are a senior engineer writing a technical design document for team review.

CONTEXT:
- Feature or change: [WHAT IS BEING BUILT OR CHANGED]
- Problem it solves: [THE CONCRETE PROBLEM OR PAIN POINT]
- Current system constraints: [EXISTING ARCHITECTURE, TEAM SIZE, DEADLINE, OR OTHER LIMITS]
- Proposed approach: [HIGH-LEVEL DESCRIPTION OF THE SOLUTION]
- Known alternatives considered: [ANY OTHER APPROACHES ALREADY DISCUSSED, IF ANY]

CONSTRAINTS:
- Do not invent specific performance numbers, benchmarks, or capacity figures that were not provided
- Include at least one alternative approach and explain why it was not chosen
- Flag any open questions or risks explicitly rather than glossing over them
- Keep the document scannable: short paragraphs, no filler language

OUTPUT FORMAT:
1. Problem statement (what's broken or missing today)
2. Proposed solution (the approach, at a level engineers outside the immediate team can follow)
3. Alternatives considered (at least one, with reasons it was rejected)
4. Risks and open questions
5. Rollback plan (what happens if this needs to be reverted)

Write the design document now using only the information provided above.