Files
multica/server/internal/daemon/poisoned.go
Bohan Jiang 30318b79bc MUL-5426: fix(daemon): retire sessions whose history the provider refuses to replay (#6083)
* fix(daemon): retire sessions whose history the provider refuses to replay

A run killed mid-reply (machine shutdown, force-quit, SIGKILL) can leave an
empty assistant message in the agent CLI's transcript. Every later resume
replays it, the provider rejects the request, and the (agent, issue) pair is
bricked with no self-healing and no user-facing recovery.

Multica already has the mechanism for this — poisoned-session classification —
but its detector paired "400" with "invalid_request_error", which is the
Anthropic wire shape. The same defect reported by any other provider carried
neither token, so it classified as agent_error.unknown: resume-safe by
omission. GetLastTaskSession kept handing back the dead session on every
follow-up, manual Rerun resolved it through the same predicate, and the
in-turn fresh-session retry never fired because ResumeRejected is false here
(nothing rejected the resume — the transcript loaded and the provider refused
to replay it).

Add taskfailure.UnresumableHistory, which recognises the defect by what the
provider says is wrong — some content is empty, and here is which message in
the history — rather than by status code or provider name. Both signals are
required, so a tool reporting "field must not be empty" does not match.

Wire it into the four places that decide whether a session survives:

- classifyPoisonedError, so the task is written as api_invalid_request
- shouldRetryWithFreshSession, so the turn recovers on all 17 backends
  instead of the subset whose adapter learned to detect it; the tools == 0
  gate is unchanged, so a run that already used a tool is never re-run
- ResumeUnsafeFailure, covering the manual-Rerun path
- both resume queries, as defense-in-depth for hosts whose daemon predates
  this (self-host daemons upgrade on their own cadence)

Fixes #6066. Also covers the daemon half of #5760.

Co-authored-by: multica-agent <github@multica.ai>

* fix(session): close the Chat and fresh-retry paths that resurrect a poisoned session

Review found the previous commit stopped short in two places, both of which
put the dead transcript back in play.

Chat never consulted the guarded query. The claim handler reads
chat_session.session_id first and only falls back to GetLastChatTaskSession
when it is empty, so a poisoned pointer there bypasses every filter that query
applies. The fail path merely declined to OVERWRITE the pointer, leaving it in
place. It now clears it in the same transaction, matched on session and
runtime so a concurrent turn's newer pointer survives. The promote guard moves
to ResumeUnsafeFailure as well — the reason-only check passed an un-upgraded
daemon's agent_error.unknown row and re-pinned what the clear had just removed.

GetLastChatTaskSession also kept the row-level filter the issue query dropped
in GH #5975: it discarded the newest poisoned row and fell back to an older
completed row carrying the same dead session. It now judges each session by
its latest terminal state, matching GetLastTaskSession.

A recovered turn could not retire anything. A terminal report carried one
session_id, and an empty one meant both "nothing to report" and "forget the
old session", so a fresh-session retry that SUCCEEDED left the id it retried
away from selectable — through an older completed row on the issue, or through
the chat pointer. agent_task_queue.retired_session_id records the abandonment
itself, reported on every terminal path including completed, and both resume
lookups exclude it. This is the contract gap the previous PR deferred; the
fresh-retry path now runs on all backends, so deferring it is not safe.

Also narrows what the cross-backend test claims: it pins the shared decision,
not that all 17 adapters surface the error into Result.Error (#5760 is the
counter-example), and says so.

Co-authored-by: multica-agent <github@multica.ai>

* test(session): require pgx.ErrNoRows in the resume-exclusion assertions

The `if err == nil && prior.SessionID.Valid` form these tests shared is
false-green: any real fault — undefined column, syntax error, dead connection
— makes err non-nil, so the condition is false and the test passes. Run
against a database missing this branch's new column, the exclusion tests
reported PASS on a SQLSTATE 42703, meaning they could not have caught a broken
query.

requireSessionExcluded demands pgx.ErrNoRows specifically and fails loudly on
anything else, so a green run now means the filter worked rather than the
query never ran.

Applied to all nine sites, not just the four this branch added: the other five
guard the same GetLastTaskSession exclusion behaviour that this branch
changes, so leaving them false-green would leave the change under-tested. All
nine pass on a correctly migrated database.

Co-authored-by: multica-agent <github@multica.ai>

---------

Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
2026-07-29 15:54:51 +08:00

166 lines
8.0 KiB
Go

package daemon
import (
"strings"
"github.com/multica-ai/multica/server/pkg/agent"
"github.com/multica-ai/multica/server/pkg/taskfailure"
)
// FailureReason values for tasks whose session is "poisoned" — i.e.
// resuming the same conversation on a follow-up task would deterministically
// reproduce the same failure. Listed here so the server-side query
// GetLastTaskSession can filter them out and the next task starts from
// a fresh agent session instead of inheriting the bad state.
//
// Two flavors:
// - Output-side: agent "completed" with output that is actually a known
// fallback marker (gave up mid-thought, emitted a meta message). Detected
// via classifyPoisonedOutput.
// - Error-side: the LLM API itself rejected the request with a 400
// invalid_request_error (oversized payload, malformed image, etc.).
// The bad message is already baked into the conversation history, so
// every resume hits the same 400. Detected via classifyPoisonedError.
// - Timeout-side: Codex reported semantic inactivity after the session got
// stuck without agent progress. Resuming that Codex session can replay the
// same stuck state, while a fresh manual rerun may succeed. Detected via
// classifyResumeUnsafeTimeout.
//
// MUL-2946: ReasonIterationLimit and ReasonAPIInvalidRequest are aliased
// to the canonical taskfailure values so the daemon and the in-flight
// classifier (used by every other failure path) share a single source
// of truth. agent_fallback_message and codex_semantic_inactivity are
// pre-existing operational reasons not in the canonical 21 — kept as
// string literals here until a follow-up PR migrates them or extends
// the taxonomy.
const (
FailureReasonIterationLimit = string(taskfailure.ReasonIterationLimit)
FailureReasonAgentFallbackMsg = "agent_fallback_message"
FailureReasonAPIInvalidRequest = string(taskfailure.ReasonAPIInvalidRequest)
FailureReasonCodexSemanticInactivity = "codex_semantic_inactivity"
)
// poisonedOutputMaxLen caps how long an output can be and still be
// classified as a poisoned fallback. Real fallback messages are short,
// one-sentence affairs; a long output that happens to mention a marker
// is almost certainly a real conclusion (e.g. a code-review reply
// quoting these strings, like the one currently quoting them in
// MUL-1630). The cap intentionally errs on the side of NOT classifying
// — a missed poisoned task gets retried by user action, but a
// false-positive turns a successful task into a failure and a system
// comment.
const poisonedOutputMaxLen = 320
// poisonedMarkers maps a substring fingerprint of a known agent fallback
// terminal message to its failure_reason classifier. Match is case-
// insensitive and substring-based; the cap above prevents long outputs
// that quote a marker from being misclassified.
var poisonedMarkers = []struct {
Substring string
Reason string
}{
{"i reached the iteration limit", FailureReasonIterationLimit},
{"put your final update inside the content string", FailureReasonAgentFallbackMsg},
}
// classifyPoisonedOutput reports whether output matches a known agent
// fallback terminal message and, if so, returns the failure_reason that
// should be persisted on the task row. Long outputs are never
// classified: a real fallback is the agent's only utterance for the
// turn, so anything beyond ~one paragraph is treated as a real result
// even if it contains a marker substring.
func classifyPoisonedOutput(output string) (string, bool) {
trimmed := strings.TrimSpace(output)
if trimmed == "" || len(trimmed) > poisonedOutputMaxLen {
return "", false
}
lowered := strings.ToLower(trimmed)
for _, m := range poisonedMarkers {
if strings.Contains(lowered, m.Substring) {
return m.Reason, true
}
}
return "", false
}
// classifyPoisonedError reports whether an agent error message indicates
// the LLM API itself rejected the request body — i.e. the conversation
// history contains content the API will not accept (oversized image,
// malformed base64, prompt-too-long, etc.). The conversation cannot be
// resumed: every retry replays the same body and reproduces the same 400.
// The classifier returns FailureReasonAPIInvalidRequest so GetLastTaskSession
// excludes the task from the (agent_id, issue_id) resume lookup, and the
// next task on the issue starts a fresh session instead of permanently
// inheriting the bad state.
//
// Match shape: the Claude Code SDK and similar backends surface upstream
// API failures verbatim, e.g.
//
// API Error: 400 {"type":"error","error":{"type":"invalid_request_error","message":"Could not process image"},"request_id":"..."}
//
// Matching on both "400" and "invalid_request_error" keeps the classifier
// narrow: 429 rate-limits, 5xx overloads, and tool-shaped errors are
// transient and SHOULD resume on retry.
//
// That shape is Anthropic's, though, and it is not the only way a provider
// reports an unprocessable transcript. The final clause delegates to
// taskfailure.UnresumableHistory, which detects an empty message baked into
// the conversation by ANY backend — see its doc comment for why the defect
// has to be recognised by wording rather than by status code or provider.
func classifyPoisonedError(errMsg string) (string, bool) {
if errMsg == "" {
return "", false
}
lowered := strings.ToLower(errMsg)
// Kiro/ACP replays images baked into a resumed conversation's history;
// one exceeding the provider's max pixel dimensions is rejected on every
// session/prompt and cannot be resumed away (GH #5975). The daemon's
// in-task fresh-session retry recovers the CURRENT turn, but this marks
// the conversation resume-unsafe so GetLastTaskSession excludes it and no
// later task re-selects the poisoned session. The offending
// messages[n].content[m] path and the pixel limit stay in the surfaced
// error; base64 payloads are never logged. Requiring the image-content
// marker alongside the dimension phrase keeps this narrow — an unrelated
// error mentioning dimensions won't trip it.
if strings.Contains(lowered, "image dimensions exceed max allowed size") &&
strings.Contains(lowered, "image.source.base64.data") {
return FailureReasonAPIInvalidRequest, true
}
// Both markers must be present: "400" alone is too generic (a tool
// could surface a 400 from anywhere) and "invalid_request_error"
// alone could in theory appear in non-poisoning contexts. The
// combination is the canonical Anthropic error shape and indicates
// the request body — i.e. the conversation history — is the problem.
if strings.Contains(lowered, "invalid_request_error") && strings.Contains(lowered, "400") {
return FailureReasonAPIInvalidRequest, true
}
// The same defect reported by a provider that words it differently.
// The clause above only fires on the Anthropic shape, so an empty
// message baked into the transcript by any other backend used to fall
// through to taskfailure.Classify as agent_error.unknown — resume-safe
// by omission, which permanently bricked the (agent, issue) pair
// (GH #6066, GH #5760). taskfailure.UnresumableHistory recognises the
// defect by what the provider says is wrong rather than by which
// provider said it.
if taskfailure.UnresumableHistory(errMsg) {
return FailureReasonAPIInvalidRequest, true
}
return "", false
}
// classifyResumeUnsafeTimeout reports whether a timeout means the recorded
// session should not be resumed. Keep this intentionally provider-specific:
// ordinary daemon/backend timeouts are infrastructure-shaped and should keep
// the resume pointer so retries can continue the in-flight conversation.
func classifyResumeUnsafeTimeout(provider, errMsg string) (string, bool) {
if strings.ToLower(strings.TrimSpace(provider)) != "codex" || errMsg == "" {
return "", false
}
lowered := strings.ToLower(errMsg)
if strings.Contains(lowered, strings.ToLower(agent.CodexSemanticInactivityMarker)) ||
strings.Contains(lowered, strings.ToLower(agent.CodexFirstTurnNoProgressMarker)) {
return FailureReasonCodexSemanticInactivity, true
}
return "", false
}