* fix(agent): clear stale session id when a resumed ACP session is gone
When an agent's stored ACP session no longer exists on the runtime side,
session/resume still succeeds — hermes echoes the requested sessionId
back — so the failure only surfaces when session/prompt returns JSON-RPC
-32603 "Session not found". The backend then reported Status=failed with
the stale SessionID still set, which kept the daemon's resume-failure
fallback (gated on SessionID == "") from ever firing. The failed task
never updates the stored session, so every future mention on the same
(agent, issue) dispatched against the same dead id, forever (#4010).
handleResponse now returns a structured acpRPCError instead of a flat
string (rendered text unchanged), and the hermes/kimi/kiro prompt-error
paths clear the session id when the error is session-not-found class on
a resumed session. The daemon's existing retry then re-executes with a
fresh session and stores the replacement id, healing the mapping.
* fix(agent): clear stale session id when set_model hits a dead resumed session
With a model override, session/set_model runs before session/prompt,
so a resumed session that is gone on the agent side surfaces there
instead of at the prompt — and the error branch returned the stale
SessionID, so the daemon's fresh-session retry (gated on
SessionID == "") never fired. Apply the same clear-the-id fix in the
set_model error branch of all three backends.
Also relax isACPSessionNotFound to accept -32602: kimi-cli raises
RequestError.invalid_params({"session_id": "Session not found"}) for
every unknown-session path (src/kimi_cli/acp/server.py), so pinning
-32603 made the fix dead code for kimi. The wording gate keeps
unrelated invalid_params errors (e.g. "model not available") on the
preserve-the-id path.
Regression tests for all three backends: resumed session + model
override + set_model failing with each runtime's observed
session-not-found shape must yield status=failed with an empty
SessionID.
Active long-running sessions are no longer killed by a fixed wall-clock deadline. Liveness is delegated to the idle watchdog (MULTICA_AGENT_IDLE_WATCHDOG, default 30m) with a larger in-flight-tool budget (MULTICA_AGENT_TOOL_WATCHDOG, default 2h). MULTICA_AGENT_TIMEOUT is an opt-in absolute cap (default 0 = no cap). The server-side 2.5h sweeper is unchanged as a coarse backstop.
Fixes#3745.
* MUL-2764 feat(agent): wire mcp_config through ACP runtimes (Hermes / Kimi / Kiro)
The MCP config Tab (#3419) already lets admins save mcp_config on an
agent, and the daemon plumbs it through to `agent.ExecOptions.McpConfig`
for every runtime. Claude and Codex consume it; the three ACP runtimes
(Hermes / Kimi / Kiro) ignored the field and hardcoded an empty
`mcpServers: []` in their `session/new` requests.
Add `buildACPMcpServers` to translate the Claude-style `{"mcpServers":
{"<name>": {...}}}` object-of-objects into the array shape ACP requires
(`[{name, command, args, env: [{name,value}, ...]}, ...]` for stdio;
`[{type, name, url, headers: [...]}, ...]` for http/sse), then pass the
translated array on `session/new` (all three) and `session/load` (kiro
resume). Malformed JSON fails the launch closed — same contract Codex's
`renderCodexMcpServersBlock` uses — so users see a real error instead of
silently running with no MCP servers. Individual unclassifiable entries
(no command, no url) are skipped with a warning so one bad row can't
take MCP down for the rest of the agent.
Co-authored-by: multica-agent <github@multica.ai>
* MUL-2764 fix(agent): wire mcp_config through ACP resume + gate http/sse on capability
Addresses the two blockers Elon raised on #3439:
1. session/resume now carries mcpServers for Hermes and Kimi (Kiro's
session/load already did). Per the ACP Session Setup spec the resume
path re-attaches MCP servers, and without this a resumed task lost
access to MCP tools that a fresh task on the same agent would have
had. Pinned with new TestHermesResumeIncludesMcpServers and
TestKimiResumeIncludesMcpServers integration tests that inspect the
recorded wire request.
2. Added extractACPMcpCapabilities + filterACPMcpServersByCapability so
http/sse MCP entries get dropped (with a daemon-log warning naming
the entry) when the runtime's initialize response doesn't advertise
mcpCapabilities.http / .sse. Sending those entries to a stdio-only
runtime is a spec violation and reliably tanks session/new; now they
get filtered and the rest of the session still starts. Stdio entries
pass through unconditionally. Both backends wire the filter in right
after initialize so session/new and session/resume see the same
filtered list.
Also added TestKiroLoadIncludesMcpServersFromConfig — Elon flagged that
no test pinned "non-empty mcp_config actually reaches the wire" for
Kimi/Kiro, so the wire assertions go in for all three runtimes.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: J <j@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
`hermes`, `kimi`, and `kiro` all wired stderr through
`cmd.Stderr = io.MultiWriter(logWriter, providerErrSniffer)`.
The OS-pipe → MultiWriter copy goroutine that exec spawns for
that form is only joined by `cmd.Wait()`, which the lifecycle
goroutine fires in deferred cleanup — *after*
`promoteACPResultOnProviderError` already consulted the sniffer.
When stopReason=end_turn (success) raced ahead of the stderr
drain, the sniffer's `lines` slice was empty, the helper fell
through to the synthetic agent-text fallback ("hermes provider
error: API call failed after 3 retries"), and the actionable
upstream signal (HTTP 429 / usage limit) was lost.
This was visible as a flaky
`TestHermesBackendPromotesProviderErrorWithNonEmptyOutput` in CI
under high parallelism — a real prod bug, not a test issue: live
runs hit the same race when an upstream LLM returns 429 and
hermes' synthetic agent turn beats the stderr drain to the
parent.
Replace the MultiWriter wiring with `cmd.StderrPipe()` + an
explicit copier goroutine that signals on `stderrDone`. The
lifecycle goroutine already awaits `<-readerDone` for stdout;
add `<-stderrDone` next to it before `promoteACPResultOnProviderError`
runs. The deferred `cmd.Wait()` ordering is unchanged — it just
becomes a cheap reap by the time it fires.
Verified: `go test ./pkg/agent/ -run "TestHermes|TestKimi|TestKiro"
-count=10 -race`, then full package `-count=3 -race`, all green.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): mark provider 429 / out-of-credit runs as failed, not completed
Two bugs combined to silently report failed agent runs as
"Completed" in the UI when the upstream LLM returned a 4xx (e.g.
HTTP 429 rate-limit / no credit on the account).
1. ACP backends (hermes, kimi, kiro) only promoted the run status to
"failed" when their stderr sniffer fired AND the agent output
buffer was empty. But hermes injects a synthetic agent text turn
("API call failed after 3 retries: HTTP 429...") on retry
exhaustion, so the buffer was never empty in the rate-limit
case and the promotion never ran. Drop the empty-output
precondition: the sniffer's regex (HTTP-status markers, named
error types) is specific enough to trust on its own.
2. The daemon's task-result switch only routed "blocked" through
FailTask; every other status — including "cancelled", and any
future status we forget to enumerate — fell through to
CompleteTask. Invert it so only an explicit "completed" status
reports success, and extract the switch into reportTaskResult
for direct testing. Cancelled now defaults to failure_reason
"cancelled" instead of being silently completed.
Closes GitHub multica#1952.
Co-authored-by: multica-agent <github@multica.ai>
* fix(agent): only promote ACP run to failed on terminal provider error
Address GPT-Boy's review on the multica#1952 fix. The previous
promotion rule ("any sniffer line → fail") was too broad: the
existing sniffer also captures transient per-attempt warnings
("API call failed (attempt 1/3): RateLimitError [HTTP 429]"), and
those lines stay in the buffer for the rest of the run. A retry
sequence whose first attempt blipped but whose third attempt
succeeded would have been wrongly reported as failed.
Tighten the criteria with two additional signals, both defined on
the existing acpProviderErrorSniffer / output buffer:
- acpTerminalErrorRe — sticky `terminal` flag set when stderr shows
an exhausted/non-retryable marker (❌, [ERROR], "after N retries",
Non-retryable, BadRequestError, AuthenticationError). Per-attempt
warnings deliberately don't match.
- acpAgentOutputTerminalRe — matches the synthetic "API call failed
after N retries..." turn that hermes-style adapters inject into
the agent text stream when they give up; this catches multica#1952
even if hermes' stderr only logged transient attempts.
Promotion logic becomes a shared helper, promoteACPResultOnProviderError,
called from hermes / kimi / kiro. Promotes when (a) terminalMessage
is non-empty, (b) output contains the synthetic give-up turn, or
(c) output is empty and the sniffer captured anything at all
(preserves the original empty-output safety net for transient-only
sequences with no real result to fall back on).
Tests:
- TestHermesProviderErrorSnifferTerminalVsTransient — transient
attempt 1/3 alone returns terminalMessage="" but message!="";
a follow-on terminal marker flips terminal on.
- TestHermesProviderErrorSnifferTerminalNonRetryable — confirms
BadRequest / Authentication / Non-retryable / ❌ / [ERROR] are
classified terminal even on the very first attempt.
- TestHermesBackendDoesNotPromoteOnTransientRetry — fake hermes
emits attempt 1/3 to stderr then a normal agent text turn and
end_turn; resulting Status must stay "completed".
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
When the local state.db of an ACP backend (hermes, kimi, kiro) is wiped
— crash, config change, manual kill, container reset — the backend's
session/resume (or session/load, in kiro's case) silently creates a
brand-new session rather than failing, and returns the new id in the
response. Today the daemon ignores the response and stamps
sessionID = opts.ResumeSessionID across all three backends, so every
subsequent session/prompt is addressed to a session id the backend has
no record of. The task fails with JSON-RPC -32603 (Internal error) on
the very first turn, with no operator-visible signal that the problem
is a session-id mismatch one layer down.
The behavior is invisible: agent shows "started", then "failed" with a
generic Internal error. Reproducing in production took repeated runs
because nothing in the logs pointed at the silent reset.
Fix: route all three ACP backends through a small `resolveResumedSessionID`
helper that:
- prefers the id the backend returned in its response (the canonical
id; the one the backend will accept on the next call)
- falls back to the requested id when the response is malformed,
empty, or omits sessionId — defensive fallback so older / non-
conforming backends (notably kiro's current session/load shape)
behave identically to today
- signals (via a bool) when the id changed, so the caller logs a Warn
with `backend=<hermes|kimi|kiro>` and operators can grep for silent
state resets to correlate them with task failures
Why this is at the backend layer rather than the daemon's existing
session-resume fallback: server/internal/daemon/daemon.go:1554-1566
already retries with a fresh session when resume fails, but it gates
on `result.Status == "failed" && result.SessionID == ""`. The backend
WILL hand back a result.SessionID — just the new one it silently
committed to — so the daemon-level fallback never fires for this
failure mode.
The helper is also what session/new already uses (extractACPSessionID,
documented in code as "Shared by all ACP backends"). session/new
extracts the canonical id from the response; session/resume just
didn't, until now.
Coverage:
- hermes.go: confirmed bug, root cause of -32603 in production
- kimi.go: same code shape, same protocol method, same response
schema as hermes (per extractACPSessionID's comment) — same bug
- kiro.go: same code shape, different method (session/load). Current
observed response doesn't include sessionId, so the defensive
fallback means today's behavior is preserved. Routing through the
same helper means a future kiro release that DOES return a sessionId
on silent reset works the same way as hermes/kimi without another
diff.
Tests (server/pkg/agent/hermes_test.go — helper covers all three
backends, no per-backend duplication):
- TestResolveResumedSessionIDMatching — backend confirms requested id
- TestResolveResumedSessionIDDifferent — backend returned a new id;
caller is told to switch
- TestResolveResumedSessionIDEmptyResponse — older / malformed body;
defensive fallback to requested id (covers kiro's current shape)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ignore Kiro ACP session/load history replay before the active prompt starts; keep task messages, usage, and tool state scoped to the current Kiro turn. Verified with go test ./pkg/agent -run TestKiro, go test ./pkg/agent, and git diff --check origin/main...HEAD.
