mirror of
https://github.com/multica-ai/multica.git
synced 2026-08-07 03:10:59 +02:00
* fix(daemon): retire sessions whose history the provider refuses to replay A run killed mid-reply (machine shutdown, force-quit, SIGKILL) can leave an empty assistant message in the agent CLI's transcript. Every later resume replays it, the provider rejects the request, and the (agent, issue) pair is bricked with no self-healing and no user-facing recovery. Multica already has the mechanism for this — poisoned-session classification — but its detector paired "400" with "invalid_request_error", which is the Anthropic wire shape. The same defect reported by any other provider carried neither token, so it classified as agent_error.unknown: resume-safe by omission. GetLastTaskSession kept handing back the dead session on every follow-up, manual Rerun resolved it through the same predicate, and the in-turn fresh-session retry never fired because ResumeRejected is false here (nothing rejected the resume — the transcript loaded and the provider refused to replay it). Add taskfailure.UnresumableHistory, which recognises the defect by what the provider says is wrong — some content is empty, and here is which message in the history — rather than by status code or provider name. Both signals are required, so a tool reporting "field must not be empty" does not match. Wire it into the four places that decide whether a session survives: - classifyPoisonedError, so the task is written as api_invalid_request - shouldRetryWithFreshSession, so the turn recovers on all 17 backends instead of the subset whose adapter learned to detect it; the tools == 0 gate is unchanged, so a run that already used a tool is never re-run - ResumeUnsafeFailure, covering the manual-Rerun path - both resume queries, as defense-in-depth for hosts whose daemon predates this (self-host daemons upgrade on their own cadence) Fixes #6066. Also covers the daemon half of #5760. Co-authored-by: multica-agent <github@multica.ai> * fix(session): close the Chat and fresh-retry paths that resurrect a poisoned session Review found the previous commit stopped short in two places, both of which put the dead transcript back in play. Chat never consulted the guarded query. The claim handler reads chat_session.session_id first and only falls back to GetLastChatTaskSession when it is empty, so a poisoned pointer there bypasses every filter that query applies. The fail path merely declined to OVERWRITE the pointer, leaving it in place. It now clears it in the same transaction, matched on session and runtime so a concurrent turn's newer pointer survives. The promote guard moves to ResumeUnsafeFailure as well — the reason-only check passed an un-upgraded daemon's agent_error.unknown row and re-pinned what the clear had just removed. GetLastChatTaskSession also kept the row-level filter the issue query dropped in GH #5975: it discarded the newest poisoned row and fell back to an older completed row carrying the same dead session. It now judges each session by its latest terminal state, matching GetLastTaskSession. A recovered turn could not retire anything. A terminal report carried one session_id, and an empty one meant both "nothing to report" and "forget the old session", so a fresh-session retry that SUCCEEDED left the id it retried away from selectable — through an older completed row on the issue, or through the chat pointer. agent_task_queue.retired_session_id records the abandonment itself, reported on every terminal path including completed, and both resume lookups exclude it. This is the contract gap the previous PR deferred; the fresh-retry path now runs on all backends, so deferring it is not safe. Also narrows what the cross-backend test claims: it pins the shared decision, not that all 17 adapters surface the error into Result.Error (#5760 is the counter-example), and says so. Co-authored-by: multica-agent <github@multica.ai> * test(session): require pgx.ErrNoRows in the resume-exclusion assertions The `if err == nil && prior.SessionID.Valid` form these tests shared is false-green: any real fault — undefined column, syntax error, dead connection — makes err non-nil, so the condition is false and the test passes. Run against a database missing this branch's new column, the exclusion tests reported PASS on a SQLSTATE 42703, meaning they could not have caught a broken query. requireSessionExcluded demands pgx.ErrNoRows specifically and fails loudly on anything else, so a green run now means the filter worked rather than the query never ran. Applied to all nine sites, not just the four this branch added: the other five guard the same GetLastTaskSession exclusion behaviour that this branch changes, so leaving them false-green would leave the change under-tested. All nine pass on a correctly migrated database. Co-authored-by: multica-agent <github@multica.ai> --------- Co-authored-by: Bohan-J <bohan@devv.ai> Co-authored-by: multica-agent <github@multica.ai>
166 lines
8.0 KiB
Go
166 lines
8.0 KiB
Go
package daemon
|
|
|
|
import (
|
|
"strings"
|
|
|
|
"github.com/multica-ai/multica/server/pkg/agent"
|
|
"github.com/multica-ai/multica/server/pkg/taskfailure"
|
|
)
|
|
|
|
// FailureReason values for tasks whose session is "poisoned" — i.e.
|
|
// resuming the same conversation on a follow-up task would deterministically
|
|
// reproduce the same failure. Listed here so the server-side query
|
|
// GetLastTaskSession can filter them out and the next task starts from
|
|
// a fresh agent session instead of inheriting the bad state.
|
|
//
|
|
// Two flavors:
|
|
// - Output-side: agent "completed" with output that is actually a known
|
|
// fallback marker (gave up mid-thought, emitted a meta message). Detected
|
|
// via classifyPoisonedOutput.
|
|
// - Error-side: the LLM API itself rejected the request with a 400
|
|
// invalid_request_error (oversized payload, malformed image, etc.).
|
|
// The bad message is already baked into the conversation history, so
|
|
// every resume hits the same 400. Detected via classifyPoisonedError.
|
|
// - Timeout-side: Codex reported semantic inactivity after the session got
|
|
// stuck without agent progress. Resuming that Codex session can replay the
|
|
// same stuck state, while a fresh manual rerun may succeed. Detected via
|
|
// classifyResumeUnsafeTimeout.
|
|
//
|
|
// MUL-2946: ReasonIterationLimit and ReasonAPIInvalidRequest are aliased
|
|
// to the canonical taskfailure values so the daemon and the in-flight
|
|
// classifier (used by every other failure path) share a single source
|
|
// of truth. agent_fallback_message and codex_semantic_inactivity are
|
|
// pre-existing operational reasons not in the canonical 21 — kept as
|
|
// string literals here until a follow-up PR migrates them or extends
|
|
// the taxonomy.
|
|
const (
|
|
FailureReasonIterationLimit = string(taskfailure.ReasonIterationLimit)
|
|
FailureReasonAgentFallbackMsg = "agent_fallback_message"
|
|
FailureReasonAPIInvalidRequest = string(taskfailure.ReasonAPIInvalidRequest)
|
|
FailureReasonCodexSemanticInactivity = "codex_semantic_inactivity"
|
|
)
|
|
|
|
// poisonedOutputMaxLen caps how long an output can be and still be
|
|
// classified as a poisoned fallback. Real fallback messages are short,
|
|
// one-sentence affairs; a long output that happens to mention a marker
|
|
// is almost certainly a real conclusion (e.g. a code-review reply
|
|
// quoting these strings, like the one currently quoting them in
|
|
// MUL-1630). The cap intentionally errs on the side of NOT classifying
|
|
// — a missed poisoned task gets retried by user action, but a
|
|
// false-positive turns a successful task into a failure and a system
|
|
// comment.
|
|
const poisonedOutputMaxLen = 320
|
|
|
|
// poisonedMarkers maps a substring fingerprint of a known agent fallback
|
|
// terminal message to its failure_reason classifier. Match is case-
|
|
// insensitive and substring-based; the cap above prevents long outputs
|
|
// that quote a marker from being misclassified.
|
|
var poisonedMarkers = []struct {
|
|
Substring string
|
|
Reason string
|
|
}{
|
|
{"i reached the iteration limit", FailureReasonIterationLimit},
|
|
{"put your final update inside the content string", FailureReasonAgentFallbackMsg},
|
|
}
|
|
|
|
// classifyPoisonedOutput reports whether output matches a known agent
|
|
// fallback terminal message and, if so, returns the failure_reason that
|
|
// should be persisted on the task row. Long outputs are never
|
|
// classified: a real fallback is the agent's only utterance for the
|
|
// turn, so anything beyond ~one paragraph is treated as a real result
|
|
// even if it contains a marker substring.
|
|
func classifyPoisonedOutput(output string) (string, bool) {
|
|
trimmed := strings.TrimSpace(output)
|
|
if trimmed == "" || len(trimmed) > poisonedOutputMaxLen {
|
|
return "", false
|
|
}
|
|
lowered := strings.ToLower(trimmed)
|
|
for _, m := range poisonedMarkers {
|
|
if strings.Contains(lowered, m.Substring) {
|
|
return m.Reason, true
|
|
}
|
|
}
|
|
return "", false
|
|
}
|
|
|
|
// classifyPoisonedError reports whether an agent error message indicates
|
|
// the LLM API itself rejected the request body — i.e. the conversation
|
|
// history contains content the API will not accept (oversized image,
|
|
// malformed base64, prompt-too-long, etc.). The conversation cannot be
|
|
// resumed: every retry replays the same body and reproduces the same 400.
|
|
// The classifier returns FailureReasonAPIInvalidRequest so GetLastTaskSession
|
|
// excludes the task from the (agent_id, issue_id) resume lookup, and the
|
|
// next task on the issue starts a fresh session instead of permanently
|
|
// inheriting the bad state.
|
|
//
|
|
// Match shape: the Claude Code SDK and similar backends surface upstream
|
|
// API failures verbatim, e.g.
|
|
//
|
|
// API Error: 400 {"type":"error","error":{"type":"invalid_request_error","message":"Could not process image"},"request_id":"..."}
|
|
//
|
|
// Matching on both "400" and "invalid_request_error" keeps the classifier
|
|
// narrow: 429 rate-limits, 5xx overloads, and tool-shaped errors are
|
|
// transient and SHOULD resume on retry.
|
|
//
|
|
// That shape is Anthropic's, though, and it is not the only way a provider
|
|
// reports an unprocessable transcript. The final clause delegates to
|
|
// taskfailure.UnresumableHistory, which detects an empty message baked into
|
|
// the conversation by ANY backend — see its doc comment for why the defect
|
|
// has to be recognised by wording rather than by status code or provider.
|
|
func classifyPoisonedError(errMsg string) (string, bool) {
|
|
if errMsg == "" {
|
|
return "", false
|
|
}
|
|
lowered := strings.ToLower(errMsg)
|
|
// Kiro/ACP replays images baked into a resumed conversation's history;
|
|
// one exceeding the provider's max pixel dimensions is rejected on every
|
|
// session/prompt and cannot be resumed away (GH #5975). The daemon's
|
|
// in-task fresh-session retry recovers the CURRENT turn, but this marks
|
|
// the conversation resume-unsafe so GetLastTaskSession excludes it and no
|
|
// later task re-selects the poisoned session. The offending
|
|
// messages[n].content[m] path and the pixel limit stay in the surfaced
|
|
// error; base64 payloads are never logged. Requiring the image-content
|
|
// marker alongside the dimension phrase keeps this narrow — an unrelated
|
|
// error mentioning dimensions won't trip it.
|
|
if strings.Contains(lowered, "image dimensions exceed max allowed size") &&
|
|
strings.Contains(lowered, "image.source.base64.data") {
|
|
return FailureReasonAPIInvalidRequest, true
|
|
}
|
|
// Both markers must be present: "400" alone is too generic (a tool
|
|
// could surface a 400 from anywhere) and "invalid_request_error"
|
|
// alone could in theory appear in non-poisoning contexts. The
|
|
// combination is the canonical Anthropic error shape and indicates
|
|
// the request body — i.e. the conversation history — is the problem.
|
|
if strings.Contains(lowered, "invalid_request_error") && strings.Contains(lowered, "400") {
|
|
return FailureReasonAPIInvalidRequest, true
|
|
}
|
|
// The same defect reported by a provider that words it differently.
|
|
// The clause above only fires on the Anthropic shape, so an empty
|
|
// message baked into the transcript by any other backend used to fall
|
|
// through to taskfailure.Classify as agent_error.unknown — resume-safe
|
|
// by omission, which permanently bricked the (agent, issue) pair
|
|
// (GH #6066, GH #5760). taskfailure.UnresumableHistory recognises the
|
|
// defect by what the provider says is wrong rather than by which
|
|
// provider said it.
|
|
if taskfailure.UnresumableHistory(errMsg) {
|
|
return FailureReasonAPIInvalidRequest, true
|
|
}
|
|
return "", false
|
|
}
|
|
|
|
// classifyResumeUnsafeTimeout reports whether a timeout means the recorded
|
|
// session should not be resumed. Keep this intentionally provider-specific:
|
|
// ordinary daemon/backend timeouts are infrastructure-shaped and should keep
|
|
// the resume pointer so retries can continue the in-flight conversation.
|
|
func classifyResumeUnsafeTimeout(provider, errMsg string) (string, bool) {
|
|
if strings.ToLower(strings.TrimSpace(provider)) != "codex" || errMsg == "" {
|
|
return "", false
|
|
}
|
|
lowered := strings.ToLower(errMsg)
|
|
if strings.Contains(lowered, strings.ToLower(agent.CodexSemanticInactivityMarker)) ||
|
|
strings.Contains(lowered, strings.ToLower(agent.CodexFirstTurnNoProgressMarker)) {
|
|
return FailureReasonCodexSemanticInactivity, true
|
|
}
|
|
return "", false
|
|
}
|