A quick Stop before the agent's first token no longer races a late reply. Started-but-empty cancellations defer the empty/non-empty judgment until the daemon acks its transcript flush (or a grace-period sweeper fires), then settle to a single outcome. Empty outcomes persist a durable, creator-authorized draft restore (fetched/consumed via a dedicated endpoint, reconnect-safe and at-most-once) instead of broadcasting the prompt over the workspace bus.
Closes#5219
* fix(daemon): isolate Codex sessions per task to unblock initialize (MUL-4424)
Codex 0.143+ backfills a per-home session-state DB by enumerating every
rollout visible under sessions/ during `initialize`. The per-task
CODEX_HOME symlinked the shared ~/.codex/sessions in, so a machine with a
large accumulated history (one reporter: ~2000 rollouts / ~22 GiB) stalled
`initialize` for tens of seconds — the app-server started but the task
produced no output before it was cancelled (github #5273).
Give each task its own local sessions/ directory instead:
- Fresh task: create an empty local sessions/ so backfill is trivial.
- Reused task with a real sessions/ dir: it is authoritative — leave it.
- Reused task still holding a legacy symlink (older build): migrate in
place. Replace the symlink with a real dir; when resuming, symlink only
the single rollout being resumed (never copy — a rollout can be GiB and
this is on initialize's critical path); and drop the stale, rebuildable
session-state DB (state_*.sqlite*, session_index.jsonl) so Codex
re-indexes the task-local sessions. Unrelated per-task DBs (goals_*,
logs_*, memories_*) are left intact.
Also point the token-usage fallback scan at the backend's per-task
CODEX_HOME instead of the daemon-global home, so usage isn't lost now that
sessions are isolated there.
Complements the #5319 handshake watchdog (which turns a silent stall into a
loud, phased timeout); this removes the underlying cause.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): address Codex session isolation review (MUL-4424)
Resolves the three blockers from Elon's review of #5360:
1. local_directory context loss. local_directory tasks get a fresh
codex-home per task ID (the daemon never reuses their workdir), so
task-local isolation stranded every follow-up run with an empty
sessions dir and silently restarted the conversation. Their only
stable, GC-safe cross-task store is the user's own ~/.codex/sessions
(a persistent store under WorkspacesRoot would be orphan-GC'd), so keep
the shared-sessions symlink for them (IsLocalDirectory). Managed tasks
stay isolated.
2. Migration resume robustness.
- Rollout lookup now covers the flat layout and background-compressed
.jsonl.zst rollouts, not just nested YYYY/MM/DD *.jsonl — both are
legitimate Codex 0.144 history that were previously judged "not
found", silently dropping resume.
- Exposure hard-links first, then symlinks, never copies — hard links
need no privilege and work on Windows within a volume, so the
zero-copy path is exercised identically on CI.
- The daemon now verifies the rollout is actually present in the task
CODEX_HOME (execenv.CodexResumeRolloutPresent) before the brief is
generated; if absent it clears the resume from both the backend and
the brief instead of telling the agent it is continuing a lost thread.
3. session_index.jsonl is no longer deleted during migration — Codex uses
it as the authoritative thread-id -> name store (not rebuildable from
rollouts). Only the rebuildable state_*.sqlite* is reset.
Tests: 2-round local_directory resume across task IDs; compressed/flat
lookup; hard-link zero-copy (os.SameFile); session_index preserved;
CodexResumeRolloutPresent + the daemon gate helper (present keeps /
absent drops / non-codex + empty no-op).
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): scope Codex sessions to a per-issue store; disclose lost resumes (MUL-4424)
Addresses the three blockers from Elon's second review of #5360.
1. local_directory still enumerated the whole machine history. The prior
fix re-linked the entire ~/.codex/sessions into every fresh local_directory
codex-home, so Codex still backfilled from thousands of unrelated rollouts on
`initialize` (measured ~8.3s with 3450 rollouts; the reporter's 22 GiB could
exceed the 30s watchdog). Point sessions/ at a persistent, per-(agent, issue)
store under the shared Codex home (multica-sessions/<agent>/<issue>) that holds
only this issue's rollouts. It is keyed stably across task IDs and lives
outside the task-scoped envRoot the GC reclaims, so follow-up runs resume it
while `initialize` only ever sees this issue's history.
2. Windows cross-volume resume was lost. Exposing a single rollout by hard-link
(same-volume only) then file symlink (needs Windows privilege) can't cross a
volume boundary. The store now lives on the shared Codex volume, so the resume
rollout is hard-linked there zero-copy, and sessions/ is exposed to the task
home via a directory link — a symlink on Unix, a junction on Windows — which
crosses volumes without privilege and never copies a (possibly GiB) rollout on
initialize's critical path. There is no remaining per-file cross-volume link.
3. An unavailable resume was a silent downgrade. Both resume gates
(gateResumeToReusedWorkdir, gateCodexResumeToRolloutPresence) now set
PriorSessionResumeUnavailable, and the runtime brief renders a Session
Continuity Notice telling the agent to disclose to the user, up front in its
reply, that the previous conversation context could not be restored and this
run starts fresh — turning a silent restart into a user-visible one. The task
is not failed: it can still do useful work without the prior context.
Managed fresh / reused-real-dir tasks keep their task-local, GC-collected
sessions dir unchanged; only the legacy-symlink migration with a resume routes
through the store (cross-volume-safe), and a home already linked to the store is
treated as authoritative on reuse.
Tests: local_directory per-issue store (only this issue's history, no whole-
machine leak); no-key fallback to an empty dir; two-round resume across task IDs
through the store; legacy migration routed through the store with a zero-copy
hard link; reused store link stays authoritative; both gates set the
resume-unavailable flag; brief renders the continuity notice only when a resume
was lost. execenv + daemon + pkg/agent packages, go vet, and gofmt all pass.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): disclose live resume-RPC loss; bound Codex session store lifecycle (MUL-4424)
Addresses the two blockers from Elon's third review of #5360.
1. A real thread/resume failure was still a silent new session. The brief's
Session Continuity Notice only covers losses the daemon detects before launch
(workdir not reused, rollout absent). But when the rollout is present yet
Codex rejects the live thread/resume (corrupt/incompatible rollout, server-side
thread GC, schema drift), startOrResumeThread falls back to thread/start and
the run succeeds on a fresh thread with no user-facing signal. Carry the
original resume intent into the backend as ExecOptions.ResumeExpected (set from
the post-gate PriorSessionID, so a pre-flight drop still routes through the
brief and never double-notifies), and when a resume was expected but the
backend landed on a fresh thread, prepend the same continuity notice to the
first turn/start input. This also covers the daemon's transport-error
fresh-session retry, which clears ResumeSessionID but not ResumeExpected.
2. The persistent per-issue store had no data lifecycle. multica-sessions stores
live outside the task-scoped envRoot the GC reclaims (so resume survives across
task IDs), which meant a done/abandoned issue's prompts and full rollouts (one
reporter: a single 1.5 GiB rollout) accumulated forever and were never freed on
issue/agent/workspace deletion. Add PruneCodexSessionStores: the daemon GC loop
reclaims any store untouched for GCCodexSessionTTL (default 14 days, configurable
via MULTICA_GC_CODEX_SESSION_TTL, 0 disables). A store's newest rollout mtime is
its last activity, so an active or recently-resumed task keeps its store fresh
and is never reclaimed, while a deleted issue's store ages out — an eventual
reclamation guarantee without needing deletion events.
Tests: codexTurnInput discloses on resume fallback and stays silent on success /
fresh start (paired with the existing live-RPC fallback test); store pruning
reclaims aged stores, keeps recent ones, isolates issues, cleans empty agent
dirs, and is disable-able. execenv / daemon / pkg/agent, go vet, gofmt all pass.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): protect a reopened Codex session store from GC mid-mount (MUL-4424)
Addresses Elon's fourth-review blocker: reopening an issue idle past
GCCodexSessionTTL could lose context, because mounting its per-issue session
store (MkdirAll + rollout lookup + task-home link) never refreshed the store's
mtime, so a GC cycle firing before the resumed turn wrote its first rollout saw
a >TTL-old store and reclaimed it — a stat->remove race with no in-use guard.
Two complementary defenses:
- Activity refresh: linkCodexSessionsToStore now os.Chtimes the store to now
after linking, so codexStoreStat (which reads the newest mtime as last
activity) sees a just-used store. This fixes the sequential repro — a mount
immediately followed by a prune keeps the store.
- In-process active-store guard: the daemon marks the per-issue store in-use
(execenv.CodexSessionStorePath) from before Prepare/Reuse mounts it until the
task ends, and PruneCodexSessionStores now takes an isActive predicate and
skips any store a live task holds. Because prepare and prune run in the same
process, this closes the remaining concurrent stat->remove window the mtime
refresh alone cannot. Reference-counted, mirroring the env-root guard.
Tests: a reopened >TTL store survives a GC cycle after remount and stays
resumable; an idle-on-disk store marked active is skipped, then reclaimed once
inactive; the existing idle-reclaim / isolation / disable / empty-agent-dir
cases still pass. execenv + daemon, go vet, gofmt all pass.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): make Codex store delete atomic with mark-active (MUL-4424)
Addresses Elon's fifth-review blocker: the active-store guard's check and delete
were not one atomic step. PruneCodexSessionStores called isActive (which locked,
read, and unlocked) and only then RemoveAll'd, leaving a window where a task
could markActiveCodexStore between the check and the removal and still lose its
store — the exact mark-then-delete interleaving Elon reproduced.
Replace the point-in-time isActive predicate with a reserve-for-deletion
protocol that shares one lock with mark-active:
- reserveCodexStoreForDeletion(store) atomically refuses when a live task holds
the store (or another delete already reserved it) and otherwise marks it
reserved, all under one activeCodexStoresMu acquisition. PruneCodexSessionStores
reserves before RemoveAll and commits after, so confirm-inactive and remove are
effectively atomic against a concurrent mark.
- markActiveCodexStore now waits (on a sync.Cond) while a store is reserved, so a
task never mounts a store mid-removal; committing the removal wakes it and the
store is recreated fresh by Prepare (with the continuity notice).
So mark-before-reserve keeps the store (reserve refused); reserve-before-mark
removes it and blocks the late mark until the removal commits. The genuinely
idle case still reclaims.
Tests (daemon, run under -race): mark-then-reserve is refused; reserve blocks a
concurrent mark until commit then the store reads active; a second reserve is
refused mid-flight. The execenv prune tests move to the reserve seam; the
activity-refresh / reopen-then-prune / isolation / disable cases still pass.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): namespace Codex session stores per profile for cross-daemon safety (MUL-4424)
Addresses Elon's sixth-review blocker: the in-process reservation guard cannot
span processes, but Multica supports multiple profile daemons on one machine
(e.g. production + staging) that share the same ~/.codex. Each daemon's GC
scanned the whole multica-sessions root, so a staging daemon could reclaim a
store a production task was actively resuming — its reservation lived only in the
other process's memory.
Isolate by profile instead of trying to lock across processes:
- Store path is now <shared>/multica-sessions/<namespace>/<agent>/<issue>, where
namespace is the daemon's profile (empty -> "default"). PrepareParams/ReuseParams
carry Profile; codexSessionStoreKey and CodexSessionStorePath fold it in.
- PruneCodexSessionStores takes the profile and scans ONLY that namespace, so a
daemon never even sees another profile's stores, let alone deletes them. The
per-profile trees are disjoint, so the in-process guard is sufficient within a
namespace (profiles get separate daemon state, so no two daemons share one).
Test: a "staging"-owned idle store is untouched by a default-profile prune and
reclaimed only by staging's own prune. Existing prune/guard/reopen tests move
under the namespace. execenv + daemon under -race, go vet, gofmt all pass.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): make the Codex store profile→namespace map injective (MUL-4424)
Addresses Elon's seventh-review blocker: the per-profile namespace was derived by
dropping unsafe characters, which is not injective. The CLI treats the empty
(default) profile and a profile literally named "default" as separate daemons,
yet both mapped to namespace "default"; likewise "staging.prod" and "stagingprod"
both mapped to "stagingprod". Two distinct daemons then shared one store tree, so
one could again reclaim the other's live session — the cross-process blocker
reopened for those profile names.
Make codexSessionStoreNamespace injective: the empty profile gets a reserved
bare literal "default", and every named profile is hex-encoded (bijective,
filesystem-safe) under a "p_" prefix a bare literal can never collide with. So
"" -> "default" while "default" -> "p_64656661756c74", and "staging.prod" /
"stagingprod" get distinct hex segments. sanitizeCodexPathSegment stays for the
UUID agent/issue segments (injective for real UUIDs); only the user-controlled
profile needed the encoding.
Tests: codexSessionStoreNamespace is distinct for "" vs "default", punctuation
variants, case variants, and an encoded-looking name; and end-to-end, pruning one
profile never reclaims the other's store for the "" vs "default" and
"staging.prod" vs "stagingprod" pairs. execenv + daemon under -race, go vet,
gofmt all pass.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): fixed-length Codex store namespace so long profiles fit (MUL-4424)
Addresses Elon's eighth-review blocker: hex-encoding the full profile doubled the
namespace segment length. A profile can be as long as a filesystem segment allows
(~255 bytes) and the CLI persists it as its own config dir, but the store
namespace "p_" + hex(profile) reached 2 + 127*2 = 256 bytes at 127 chars,
overflowing the 255-byte single-segment limit — the profile's own dir created
fine, then the session store failed with "file name too long".
Derive the namespace from a fixed-length hash instead: a named profile is now
"p_" + hex(sha256(profile)) — a constant 64 hex chars (66 with the prefix),
filesystem-safe and collision-resistant. The empty (default) profile keeps its
reserved bare literal "default", which the "p_"-prefixed 66-char segment can
never equal. Still injective across the CLI's distinct-daemon cases; just no
longer length-expanding.
Test: the namespace stays <=255 bytes and creatable for profiles up to the
255-byte segment limit (127- and 255-char cases that overflowed under hex); the
prior injectivity and cross-profile prune-isolation tests still hold. execenv +
daemon under -race, go vet, gofmt all pass.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: J <j@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
* fix(execenv): overlay per-task HERMES_HOME so Hermes discovers bound skills
Hermes has no workspace-relative skill discovery — it scans <HERMES_HOME>/skills
first, then skills.external_dirs from config.yaml (verified against the bundled
agent/skill_utils.py). The daemon wrote assigned skills to the generic
.agent_context/skills/ fallback, which Hermes never reads, so they silently never
took effect (#5242).
When (and only when) an agent has skills bound, redirect HERMES_HOME to a minimal
per-task compatibility overlay; a skill-less Hermes task keeps its real home and
original behavior:
- mirror every top-level entry of the shared home via symlink except the
overlay-owned ones (denylist), reconciling entries deleted from the shared home;
- derive a task-local config.yaml whose skills.external_dirs references the shared
skills dir plus the user's existing external_dirs, expanded against the
sanitized effective child env (unknown vars preserved, blocklisted keys resolved
to the process value) and normalized to absolute paths;
- write only the bound skills into the task-local skills/ dir (home skills scanned
first, so they win); global skills are referenced, not copied;
- keep memories/ overlay-owned (fresh per-task dir) AND disable the external
memory.provider, so neither on-disk memory nor a shared backend crosses tasks;
- keep active_profile/profiles out of the overlay so Hermes can't follow a sticky
profile and redirect past it at startup.
Profile handling mirrors hermes_cli.profiles: the daemon reads -p/--profile with
agent.HermesProfileFromArgs and seeds the overlay from that profile's home via
ResolveHermesSourceHome (default/invalid -> base, valid name -> <base>/profiles/
<name>, validated; a missing named profile fails closed). The profile flags are
stripped from the acp argv ONLY when the overlay is active (hermesLaunchArgs), so
a skill-less task's profile passes through unchanged. HERMES_HOME is no longer
custom_env-blocklisted: no skills -> user value passes through; skills -> overlay
overrides after layering. Fail closed — Prepare errors, Reuse returns nil.
Task home 0700, derived config 0600 via atomic replace. Platform-native default
home (%LOCALAPPDATA%\hermes, incl. the LOCALAPPDATA-missing fallback, on Windows).
Tests span execenv/daemon/agent: no-skill no-op, child-env layering + env
sanitization, profile parse/unquote + conditional strip + final args/env per
scenario, custom/profile/default/invalid/missing/Windows source home, sticky-
profile not mirrored, memory dir isolation + external provider disable, mirror
reconciliation, external_dirs rebasing + sanitized/unknown-var expansion,
local-precedence slug, perms, fail-closed, resume teardown. Docs (en + ja/ko/zh).
Fixes#5242
* fix(hermes): make profile selection one resolver contract matching Hermes
Round 5 review: the profile chain approximated Hermes' semantics in three
separate places (argv parsing, source-home selection, arg filtering), so it
diverged from native Hermes in several merge-blocking cases. Collapse it into
one authoritative resolution:
- agent.ParseHermesProfileArgs replaces HermesProfileFromArgs/
FilterHermesProfileArgs. It reproduces _apply_profile_override step 1/1b
(first occurrence, value-flag skipping, `--` and `mcp add --args` boundaries,
space-form profile-id guard) and returns the exact argv occurrence to consume;
StripHermesProfileArgs removes only that occurrence.
- execenv.ResolveHermesProfile replaces ResolveHermesSourceHome. It derives the
Hermes root exactly like get_default_hermes_root (an already-profile-scoped
HERMES_HOME roots at its grandparent), selects an explicit profile first,
otherwise trusts a profile-scoped home (step 1.5) and only then the sticky
<root>/active_profile (step 2), and validates via normalize/validate_profile_name
(reserved hermes/test/tmp/root/sudo and empty inline `--profile=` are hard
errors). Profiles always resolve under the root, so `-p default` re-roots and
`-p <sibling>` is a sibling, never nested.
- The daemon runs one parse + resolve, fails the task closed on a reserved/
invalid selection (matching Hermes' sys.exit(1)), and exports the selected
source home as the effective env's HERMES_HOME so ${HERMES_HOME} in a profile's
skills.external_dirs expands against the selected profile home (as native
Hermes does before loading config.yaml), not the root or the overlay.
Regressions added: root + sticky named profile selection; already-profile-scoped
home with no flag; that home with -p default and -p <sibling>; reserved and empty
inline profile values; and a selected profile whose external_dirs contains
${HERMES_HOME}.
* fix(hermes): overlay-owned derived .env + symlink-resolved root
Round 6 review, two remaining overlay-bypass paths:
1. A source `.env` could redirect HERMES_HOME after profile resolution. Hermes
runs `_apply_profile_override()` then `load_hermes_dotenv()`, which loads
`<HERMES_HOME>/.env` with override=True — so a mirrored source `.env` carrying
an out-of-band `HERMES_HOME=` overwrote the overlay's home, repointing skill
discovery and memory back at the source. `.env` is now overlay-owned and
DERIVED (writeDerivedHermesEnv): it preserves the source's credentials/settings
but strips any `HERMES_HOME` assignment and pins `HERMES_HOME` to the overlay
last (single-quoted, literal), written 0600 via atomic replace. It is written
even when the source has none, so Hermes' project-`.env` fallback (override=True
only when no user `.env` loaded) can't relocate the home either.
2. Root derivation was lexical-only, diverging from `get_default_hermes_root`,
which compares `env_path.resolve()` with `native_home.resolve()`. A HERMES_HOME
symlinked into `<native>/profiles/<x>` was treated as its own root, so
`-p default`/`-p <sibling>` resolved wrong. `hermesRootFromHomeFor` now resolves
symlinks (Path.resolve(strict=False)-style best effort) for the containment
decision while keeping the returned root unresolved, matching Hermes.
Regressions: source `.env` with HERMES_HOME replayed through the override=True
dotenv order (bound skill + task memory stay on the overlay; creds preserved);
minimal overlay `.env` created when the source has none; and a symlinked profile
home resolving `-p default`/`-p <sibling>` to the native root.
Two daemon-side fixes to the persisted task transcript:
- Wait for the drain goroutine to flush the final message batch before every
terminal return (result, timeout, idle-watchdog, upstream cancel), so a
consumer reading the transcript at completion can't see a truncated tail.
Bounded (10s, then cancel + 12s) so a backend that never closes its message
channel cannot stall the terminal transition.
- Share the message seq counter across a resume-failure retry so the retry's
rows keep ascending seq values instead of restarting at 1 and interleaving
with the failed attempt's rows.
Server-initiated cancellation read timing is tracked separately in #5219.
Closes#5209
MUL-4369
Adds the official ByteDance TRAE CLI (the `traecli` binary documented at
https://docs.trae.cn/cli — the product paired with the Trae IDE, not the
open-source bytedance/trae-agent) as a built-in agent backend. traecli is
ACP-native, so it is driven over the standard ACP JSON-RPC transport via
`traecli acp serve --yolo`, reusing the shared hermesClient exactly like the
Kiro and Qoder backends.
Validated end-to-end against the real traecli v0.120.42 with a logged-in
account: initialize advertises loadSession:true + mcpCapabilities{http,sse};
session/new returns result.sessionId + models.availableModels (18 models
discovered); session/prompt streams session/update notifications with
sessionUpdate=agent_message_chunk (hermesClient already normalizes this Zed-ACP
wire shape); a real board task ran 14 tool calls and completed in ~47s.
Implementation:
- server/pkg/agent/traecli.go: ACP backend; session/load resume
(loadSession:true), session/set_model, MCP via ACP mcpServers, --yolo
bypass-permissions for headless runs, blocked-arg filtering (acp, serve,
--yolo, --print, --output-format, --permission-mode)
- agent.go: New() + launch header "traecli acp serve"
- models.go: discoverTraecliModels via the shared discoverACPModels
- daemon/config.go: auto-detect the `traecli` binary
(MULTICA_TRAECLI_PATH / MULTICA_TRAECLI_MODEL)
- daemon.go: inline the runtime brief (traecli reads .trae/rules/, not
AGENTS.md) and surface the runtime as "Trae" (providerDisplayName)
- execenv: AGENTS.md + .traecli/skills wiring; ~/.traecli/skills local root
- packages/core mcp-support: traecli consumes mcp_config
- frontend: official Trae provider logo
- docs: providers.mdx matrix + section, CLI_AND_DAEMON.md, README
Tests: fake-ACP unit tests matching the real wire format (streaming,
blocked-arg filtering, session/set_model failure, session/load resume) plus a
gated real-binary smoke test (TestTraecliRealACPSmoke) that skips when traecli
is absent or not logged in. Built-in provider only (mirrors qoder): not in
SupportedTypes / RUNTIME_PROFILE_PROTOCOL_FAMILIES, so no migration is needed.
Resolves#4376.
After a WebSocket disconnect, the daemon's view of running tasks and
workspace state can lag the server for up to 5s (per-task cancellation
poll) or 30s (workspace sync) because both loops park on coarse tickers
that do not observe the WS wakeup channel.
This change adds a small fan-out broadcaster (`reconcileBroadcaster`)
that the WS connect path fires once per (re)connect. `watchTaskCancellation`
and `workspaceSyncLoop` subscribe and re-check immediately on broadcast,
without disturbing the ticker cadence. The broadcaster is edge-triggered
with a one-slot replay so a broadcast that lands before a subscriber is
ready is not lost (closes the daemon-startup race), and back-to-back
broadcasts inside 1s are debounced so a flapping connection cannot fan
out into a request stampede.
Existing behaviour is preserved: shouldInterruptAgent still decides
whether to interrupt, the 5s/30s ticker still bounds the worst case,
and the WS heartbeat / HTTP heartbeat coordination is untouched.
Closes#4665
CLI backends key their session stores to the cwd (Claude Code looks
sessions up under ~/.claude/projects/<encoded-cwd>/), so a prior session
id can only resolve when the task runs in the exact workdir the session
was recorded against. When the prior workdir no longer exists (GC'd
after the issue went done, daemon reinstall, manual cleanup),
execenv.Reuse falls back to a fresh Prepare but the stale session id was
still passed to the backend: claude exited within a second and the run
failed before doing any work — permanently, because the failed run
records no session_id and the next claim serves the same stale pointer
again.
Gate ResumeSessionID on the workdir actually being reused, and correct
PriorSessionResumed so the runtime brief uses the cold-path wording when
the session is dropped.
Fixesmultica-ai/multica#3854 (MUL-3221)
Co-authored-by: J <j@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
shouldInterruptAgent now treats every terminal task status (completed/failed/cancelled, via isAgentTaskTerminal) plus a 404 task-not-found as an interruption signal, so the daemon stops a local agent once the backend has finalized the task — e.g. the runtime offline sweeper flipping running -> failed during a disconnect/reconnect. Previously only `cancelled`/404 interrupted, so the agent ran to completion and its CompleteTask call failed against a non-running row, wasting compute and adding log noise.
Closes#3877
Active long-running sessions are no longer killed by a fixed wall-clock deadline. Liveness is delegated to the idle watchdog (MULTICA_AGENT_IDLE_WATCHDOG, default 30m) with a larger in-flight-tool budget (MULTICA_AGENT_TOOL_WATCHDOG, default 2h). MULTICA_AGENT_TIMEOUT is an opt-in absolute cap (default 0 = no cap). The server-side 2.5h sweeper is unchanged as a coarse backstop.
Fixes#3745.
Lift MUL-1949's offline backfill failure_reason taxonomy into a shared
in-flight classifier so the agent_task_queue.failure_reason column is
written with refined values (provider_auth_or_access, context_overflow,
provider_capacity_or_rate_limit, …) at write time rather than waiting on
SQL backfill to re-classify after the fact. PR1 of the Grafana board
plan in MUL-2328 — the upcoming PR2 reuses pkg/taskfailure.AllReasons()
to pre-warm the Prometheus failure_reason label set.
* server/pkg/taskfailure: new package with the canonical 21 Reason
constants (7 platform-side + 14 agent_error.* sub-reasons),
AllReasons() returning a defensive copy, IsAgentError() prefix check,
and Classify(rawError) Reason mirroring the SQL CASE rules from
MUL-1949 (db-boy's analysis). 100% statement coverage.
* server/internal/daemon/daemon.go: route the 'agent_error' coarse
fallback paths (StartTask error, runTask early-return error, CompleteTask
permanent rejection, reportTaskResult default branch) and the
executeAndDrain default error case (chained after classifyPoisonedError)
through taskfailure.Classify so blocked / timeout / unknown-status
results all carry a refined reason on the wire.
* server/internal/service/task.go: FailTask classifies errMsg when the
daemon-supplied failureReason is empty, eliminating the legacy
COALESCE(.., 'agent_error') landing.
* server/internal/daemon/poisoned.go: alias FailureReasonIterationLimit
and FailureReasonAPIInvalidRequest to the canonical taskfailure
constants. agent_fallback_message and codex_semantic_inactivity are
pre-existing operational reasons not in the canonical 21 — kept as
literals for now and revisited in a follow-up PR.
Backfill SQL from MUL-1949 stays as the authoritative offline source of
truth; this PR keeps the in-flight classifier in lock-step with the SQL
CASE expression so historical and future rows share the same taxonomy.
No behavior change for the platform-side reasons (queued_expired,
runtime_offline, runtime_recovery, timeout, etc.) which already align
with the canonical set.
Co-authored-by: Eve <eve@multica-ai.local>
Co-authored-by: multica-agent <github@multica.ai>
CompleteTask / FailTask used to be fire-once. A 1-second upstream 502
burst would drop the call, then the immediate fail-fallback also 502'd,
leaving the task stuck in `running` forever and showing the agent as
"still working" in the UI.
Add a bounded retry around the two terminal callbacks: 4s, 8s, 16s,
32s, 64s backoff schedule (5 retries, ~124s ceiling), retrying only
on transient errors (5xx, 408, 429, transport-level) and bailing
immediately on permanent 4xx. Also fix a latent bug where a transient
complete failure would silently downgrade a successful run to a fail:
the fallback now triggers only on permanent errors. Server-side
CompleteTask / FailTask are already idempotent on "already terminal",
so replays from a retry are safe even if the prior 502'd response was
actually persisted.
Co-authored-by: J <j@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
* feat(daemon): force-stop hung agent runs via idle watchdog (MUL-2281)
A backend whose subprocess hangs on a stuck child process (e.g. claude
blocked on `docker ps` against a frozen dockerd) keeps the daemon's run
record at status="running" until the full DefaultAgentTimeout (2 h)
expires, because cmd.Wait() never returns and Session.Result is never
written. MUL-2225 spent 17+ minutes in this state in the wild.
Add a per-task idle watchdog around executeAndDrain:
- Wrap the caller's ctx so a single cancel propagates to the agent
subprocess (via the ctx passed to backend.Execute) AND the drain loop.
- Stamp lastActivityAt every time the drain loop receives a message.
- Tick at window/2; when idle_for >= window AND session.Messages buffer
is empty, set a fired flag and call cancel.
- Tag the resulting Result.Status as "idle_watchdog" so runTask routes
it through a dedicated failure_reason instead of "agent_error".
Default window is 5 min, configurable via MULTICA_AGENT_IDLE_WATCHDOG;
set to 0 to disable. Tests cover the activity-then-silence case, the
zero-message case, the disabled case, and the happy path.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): skip idle watchdog while a tool call is in flight
A legitimate long-running tool call (npm install, docker build, test
suite) can sit silent between tool_use and tool_result for many minutes.
Without this gate, the watchdog would yank the agent mid-build.
Track unmatched tool_use messages in an atomic counter; only let the
watchdog fire when the counter is zero. tool_result clamps non-negative
so a stray result with no matching use can't re-arm the watchdog one
call too early.
Adds two regression tests:
- DoesNotFireDuringInFlightToolCall: tool_use -> silence past
window -> tool_result -> completed (must NOT fire)
- FiresAfterToolResultIfBackendStaysSilent: tool_use -> tool_result
-> silence past window (MUST fire — backend really is stuck)
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
* feat(task): wire claim lease queries into TaskService and sweeper (MUL-2246)
- ClaimTask now uses ClaimAgentTaskWithLease (generates claim_token + lease)
- StartTask accepts optional claim_token for token-verified start
- AgentTaskResponse includes claim_token for daemon to use
- Daemon client sends claim_token in StartTask body
- Sweeper calls RequeueExpiredClaimLeases each tick
- Legacy daemons without claim_token still work (graceful fallback)
Co-authored-by: multica-agent <github@multica.ai>
* fix(task): address PR #2662 review blockers (MUL-2246)
1. ClaimAgentTaskForRuntime: push runtime_id into atomic SQL WHERE clause
so runtime A cannot claim tasks queued for runtime B under the same agent.
2. Legacy StartAgentTask: add claim_token IS NULL guard so leased rows
cannot be started without token verification. Handler rejects malformed
tokens with 400 instead of silently degrading to legacy path.
3. StartAgentTaskWithClaimToken: validate claim_expires_at >= now(),
preserve claim_token until terminal state (only clear claim_expires_at),
use CTE + UNION ALL for idempotent retry when daemon resends after a
lost StartTask response. Return 409 Conflict on token mismatch/expiry.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): StartTask 409 handling, transport retry, claim_token on FailTask (MUL-2246)
- StartTask 409 (claim superseded): release slot, don't call FailTask
- StartTask transport timeout/5xx: retry once with same token, then
check task status before failing
- FailTask now sends claim_token; server-side FailAgentTask SQL adds
AND (claim_token IS NULL OR claim_token = @claim_token) guard so
stale daemons cannot fail tasks that have been re-claimed
Co-authored-by: multica-agent <github@multica.ai>
* fix(task): close FailTask token bypass and RequeueExpiredClaimLeases liveness gap (MUL-2246)
Blocker 1 - FailTask token validation:
- SQL: change (param IS NULL OR claim_token = param) to
(param IS NULL AND claim_token IS NULL) OR claim_token = param
so tokenless requests can only fail legacy (tokenless) rows.
- task.go: malformed claim_token now returns ErrInvalidClaimToken (400)
instead of being silently dropped to NULL.
- Handler: maps ErrInvalidClaimToken→400, ErrClaimTokenInvalid→409.
- Service: when UPDATE returns no rows but task is still active,
return ErrClaimTokenInvalid (token mismatch) instead of silent success.
Blocker 2 - RequeueExpiredClaimLeases runtime liveness:
- SQL: JOIN agent_runtime, only requeue tasks where runtime is 'online'.
Dead/offline runtime tasks stay dispatched for FailTasksForOfflineRuntimes.
- FOR UPDATE → FOR UPDATE OF atq (required with JOIN).
Regression tests:
- task_claim_token_test.go: malformed, tokenless-on-tokened, wrong-token
- requeue_lease_test.go: SQL must JOIN agent_runtime with online filter
Co-authored-by: multica-agent <github@multica.ai>
* fix(task): move expired lease requeue to ClaimTaskForRuntime preflight, add heartbeat freshness backstop (MUL-2246)
- Add RequeueExpiredClaimLeasesForRuntime: per-runtime preflight self-requeue
in ClaimTaskForRuntime. Runtime proves liveness by actively claiming, so no
heartbeat check needed.
- Update global RequeueExpiredClaimLeases to require ar.last_seen_at freshness
(stale_threshold_secs param). Prevents requeuing to a dead runtime in the
90s gap between lease expiry (60s) and offline detection (150s).
- Add regression tests verifying the heartbeat freshness check and that the
preflight query does not join agent_runtime.
Co-authored-by: multica-agent <github@multica.ai>
* fix(task): use LivenessStore for global requeue, move preflight before empty-cache (MUL-2246)
Blocker 1: Global RequeueExpiredClaimLeases now uses LivenessStore.IsAliveBatch
to verify runtimes are truly alive before requeuing expired leases. When
LivenessStore is unavailable (no Redis), global requeue is skipped entirely —
the preflight self-requeue in ClaimTaskForRuntime handles live runtimes. This
closes the 60-150s gap where a dead runtime still appears online in DB.
Blocker 2: Moved RequeueExpiredClaimLeasesForRuntime BEFORE EmptyClaim.IsEmpty
fast-path in ClaimTaskForRuntime. Expired leases are now requeued (which bumps
the empty cache via notifyTaskAvailable) before the empty check can
short-circuit the claim path.
Also adds ListRuntimesWithExpiredClaimLeases SQL query and LivenessChecker
interface on TaskService.
Co-authored-by: multica-agent <github@multica.ai>
* fix(task): wire EmptyClaimCache into backend taskSvc for backstop requeue (MUL-2246)
The backend taskSvc used by the sweeper only had Liveness wired but not
EmptyClaim. When global backstop requeue called notifyTaskAvailable,
s.EmptyClaim.Bump() was a nil no-op — the handler's empty-cache was never
invalidated, so the daemon's next claim hit a stale empty verdict.
Fix: wire the same Redis-backed EmptyClaimCache into the backend taskSvc
in main.go (same Redis keys as router.go:139 handler instance).
Add regression test verifying backstop requeue invalidates the handler's
empty-cache.
Co-authored-by: multica-agent <github@multica.ai>
* fix(task): global backstop must not requeue — alive runtimes use preflight, dead stay dispatched (MUL-2246)
- RequeueExpiredClaimLeases is now a no-op (returns 0 always)
- Alive runtimes self-requeue via ClaimTaskForRuntime preflight
- Dead runtimes stay dispatched for FailTasksForOfflineRuntimes
- Rewriting to queued on dead runtime creates 2h blackhole (offline
sweeper only handles dispatched/running)
- Test actually calls RequeueExpiredClaimLeases and asserts 0 in all cases
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): remove duplicate usage reporting block after merge conflict (MUL-2246)
The merge resolution introduced a second ReportTaskUsage call after the
status check, duplicating the usage-before-early-return block that already
runs right after runner.run. Remove the duplicate and add a regression test
asserting /usage is called exactly once on the normal completion path.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
handleTask had two early-return paths that ran before ReportTaskUsage:
the cancelledByPoll select and the post-run GetTaskStatus check. Both
silently discarded any usage accumulated by the agent — and both
claude.go and codex.go populate Result.Usage even when runCtx is
cancelled mid-run, so cancelled tasks consistently under-reported tokens.
Hoist ReportTaskUsage to run immediately after the runner returns,
before any early-return path. Add a taskRunner interface seam and a
cancelPollInterval field so tests can inject a fake runner and trigger
the poll-cancellation path on a 10ms ticker without spawning real agents.
Two regression tests cover both leak windows:
- TestHandleTask_ReportsUsageBeforeCancel: post-run /status returns
"cancelled"; usage must be reported before the status check.
- TestHandleTask_ReportsUsageWhenCancelledByPoll: poll goroutine fires
first and cancels runCtx; runner returns usage on Done; assert
poll-status precedes usage (proving the cancelledByPoll branch was
the one exercised, not the post-run path).
Sanity-checked: reverting only the ReportTaskUsage hoist fails both
tests with the original "tokens lost" message.
MUL-2258
Co-authored-by: Jiang Bohan <bhjiang@outlook.com>
Co-authored-by: multica-agent <github@multica.ai>
* feat(execenv): native OpenClaw skill discovery via per-task config
MUL-2213 stopped lying about native discovery and routed openclaw skills
to .agent_context/skills/ — a path openclaw's scanner never reads.
Multica skills attached to openclaw-backed agents were still invisible to
the runtime; the AGENTS.md fallback was only a documentation patch.
OpenClaw's skill scanner walks <workspaceDir>/skills/ (plus a few other
roots), and workspaceDir is resolved from the openclaw config file —
specifically agents.list[id].workspace → agents.defaults.workspace →
~/.openclaw/workspace. There is no CLI flag or env var override on the
agent runtime; the only knob is the config file.
This change wires a per-task synthesized config:
1. execenv.prepareOpenclawConfig deep-copies the user's existing
openclaw.json (priority: $OPENCLAW_CONFIG_PATH, else
~/.openclaw/openclaw.json), rewrites agents.defaults.workspace AND
every agents.list[].workspace to the task workdir, and writes the
result to {envRoot}/openclaw-config.json. Provider sections,
registered agents, model providers, gateway settings — everything
openclaw needs to actually start — are preserved as-is.
2. resolveSkillsDir for "openclaw" now points at {workDir}/skills/,
which is the first path openclaw scans under workspaceDir. Skills
written here are picked up natively.
3. daemon.go exports OPENCLAW_CONFIG_PATH={env.OpenclawConfigPath} on
the openclaw subprocess and adds OPENCLAW_CONFIG_PATH to the
custom_env blocklist so users cannot accidentally override it.
4. buildMetaSkillContent now lists openclaw alongside the
"discovered automatically" providers; the .agent_context/skills/
fallback line stays for gemini/hermes.
The new regression test TestPrepareOpenclawSkillWriteMatchesScanPath is
the one MUL-2219's DoD calls out: it resolves the workspaceDir the way
openclaw does (reading agents.defaults.workspace out of the synthesized
config) and proves {workspaceDir}/skills/<name>/SKILL.md is what Multica
actually wrote. The pre-MUL-2219 fix asserted "we wrote a file" without
checking the scanner would ever see it — which is how the dead drop into
.openclaw/skills/ landed in #2621's first commit.
Verified locally: minimum-viable synthesized config validates via
`openclaw config validate`, and `OPENCLAW_CONFIG_PATH=<path> openclaw
config get agents.defaults.workspace` returns the task workdir as
expected. MUL-2219
Co-authored-by: multica-agent <github@multica.ai>
* fix(execenv): delegate openclaw config parsing to CLI and fail closed
Address Elon's must-fix on PR #2628: the previous implementation parsed
~/.openclaw/openclaw.json with encoding/json, which cannot read JSON5
or follow $include — the OpenClaw spec's actual format. When parsing
failed, prepareOpenclawConfig silently emitted a minimal config, which
could boot OpenClaw without the user's registered agents, model
providers, or API keys.
Two changes:
1. Delegate active-config-path resolution and config reading to the
openclaw CLI itself. `openclaw config file` locates the active
config (covering OPENCLAW_CONFIG_PATH / OPENCLAW_STATE_DIR /
OPENCLAW_HOME / default and the legacy chain), and the wrapper we
write uses $include to point at it so OpenClaw's own loader handles
JSON5, $include nesting, env-substitution, and secret refs. We read
only agents.list via `openclaw config get --json` to rewrite each
entry's workspace — secrets, comments, and includes in the user
config are never touched.
2. Remove the silent minimal-config fallback. Any CLI failure,
malformed output, or write error now surfaces as a hard error from
Prepare / Reuse. The only "synthesize minimal" path left is a fresh
install (CLI reports a path but the file doesn't exist), where
there is no user data to lose.
The per-task override still rewrites every agents.list[].workspace,
not just agents.defaults.workspace — this is intentional task
isolation, documented in prepareOpenclawConfig and the PR body. A
host-scope per-agent workspace would otherwise silently route the
scanner back to the user's shared workspace.
Cleanups Elon flagged in the same review:
- daemon.go inline-system-prompt comment no longer claims openclaw
ignores the task workdir; it does load it now, and the inline brief
is a belt-and-suspenders carryover for older releases.
- execenv.go openclaw block no longer references "skill file paths in
the inline brief" — the brief uses "discovered automatically".
Reuse() switches to a ReuseParams struct so the openclaw binary path
threads through alongside CodexVersion without a 6th positional arg.
MUL-2219
Co-authored-by: multica-agent <github@multica.ai>
* fix(execenv): grant OpenClaw $include cross-dir confinement for per-task wrapper
The per-task wrapper at envRoot/openclaw-config.json $includes the user's
active config (typically ~/.openclaw/openclaw.json), but OpenClaw confines
$include resolution to the wrapper file's directory unless the target's
parent is granted via OPENCLAW_INCLUDE_ROOTS. Without this, OpenClaw refuses
to follow the link at runtime and the wrapper boots with no user-registered
agents.
prepareOpenclawConfig now returns dirname(activePath) as IncludeRoot, and
the daemon prepends it to whatever the user already has in
OPENCLAW_INCLUDE_ROOTS via the new composeOpenclawIncludeRoots helper
(dedupes, drops empty segments, preserves user-configured roots). Fresh
install emits no $include and leaves the env var untouched.
Adds OPENCLAW_INCLUDE_ROOTS to the custom_env blocklist so a per-agent
override cannot strip the granted root.
Regression tests:
- TestPrepareOpenclawConfigWrapperLoadableUnderIncludeConfinement asserts
every $include target's dirname is covered by the IncludeRoot we surface.
- TestPrepareEnvironmentOpenclawWiresIncludeRoot covers the non-fresh-install
Environment wiring.
- TestComposeOpenclawIncludeRoots covers the daemon-side env composition
(preserve, dedupe, drop empties).
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
PR #2564 only added IsSquadLeader handling to the assignment-triggered
workflow path and the Output section. When a squad leader is triggered by
a comment (the common case for re-evaluation), the comment-triggered
workflow path had NO squad leader special handling, so the model still
posted comments announcing no_action/silence.
Changes:
- runtime_config.go: Add IsSquadLeader check to comment-triggered step 4
with explicit prohibition against posting no_action announcement comments
- runtime_config.go: Strengthen Output section from 'may exit silently' to
'MUST exit without posting any comment' with explicit DO NOT examples
- runtime_config.go: Strengthen assignment-triggered step 5 similarly
- prompt.go: Add squad leader no_action rule to per-turn comment prompt
when trigger author is an agent and agent instructions contain the
Squad Operating Protocol marker
- Add tests for both the per-turn prompt and CLAUDE.md generation
Fixes MUL-2168
Co-authored-by: multica-agent <github@multica.ai>
Three user reports converge on the same Windows-shell encoding bug:
- #2198 / #2236 — Chinese, Codex on Win11. Comments / descriptions
generated by the agent arrive as `?`.
- #2376 — Cyrillic, non-Codex agent ("Ops Lead") on Win11 Desktop.
Title preserved (argv → CreateProcessW UTF-16), description / agent
reply garbled (stdin → shell-codepage re-encoding).
woodcoal's independent diagnosis on #2198 confirms the root cause:
Windows PowerShell 5.1's `$OutputEncoding` defaults to ASCIIEncoding
when piping to a native command, so non-ASCII bytes are silently
replaced with `?` before they reach `multica.exe`. The CLI's stdin
parsing is fine; the bytes are corrupted upstream, in the agent's
shell layer.
This PR ships the fix that supersedes the codex-only attempt in
PR #2265 (which is closed in favour of this one):
## CLI
Add `--content-file <path>` to `multica issue comment add` and
`--description-file <path>` to `multica issue {create,update}`. The
CLI reads bytes off disk via `os.ReadFile` and skips the shell
entirely; UTF-8 survives end-to-end regardless of `$OutputEncoding`
or `chcp`. The three input modes (`--content`, `--content-stdin`,
`--content-file`) are mutually exclusive.
## Runtime config
`buildMetaSkillContent`'s Available Commands section is rewritten as a
neutral three-mode menu. The previous unconditional "MUST pipe via
stdin" / `--description-stdin` mandate (over-spread from #1795 /
#1851's Codex-multi-line fix) is gone for non-Codex providers; the
strong directive now lives only in the Codex-Specific section, which
branches on host:
- Codex / Linux+macOS: `--content-stdin` + HEREDOC (preserves MUL-1467
fix against codex's literal `\n` habit).
- Codex / Windows: `--content-file` (PowerShell ASCII pipe is the
exact bug we're patching).
## Per-turn reply template
`BuildCommentReplyInstructions` now takes a provider arg and branches
provider × OS:
- Windows + any provider → `--content-file` (the bug is shell-layer,
not provider-layer; #2376 shows non-Codex agents on Windows also
hit it). All providers write a UTF-8 file with their file-write tool
and post via `--content-file ./reply.md`.
- Linux/macOS + Codex → stdin/HEREDOC (MUL-1467 protection).
- Linux/macOS + non-Codex → lightweight pre-#1795 inline
`--content "..."`. The CLI server-side decodes `\n`, so escaped
multi-line works; the agent retains stdin / file as escape hatches
for richer formatting.
`BuildPrompt` and `buildCommentPrompt` gain a `provider` arg;
`daemon.runTask` already has it in scope.
## Tests
- `TestResolveTextFlag` — file-source verbatim with non-ASCII
(`标题 / Заголовок / 中文段落`), missing-file error, empty-file
rejection, three-way mutual exclusion.
- `TestInjectRuntimeConfigAvailableCommandsIsNeutral` — every
non-Codex provider × {linux, darwin, windows} pins the three-mode
menu present + over-spread "MUST stdin" substrings absent.
- `TestInjectRuntimeConfigCodexLinuxEmphasizesStdin` +
`TestInjectRuntimeConfigCodexWindowsUsesContentFile` — Codex
section's per-OS branch.
- `TestBuildCommentReplyInstructionsCodexLinux` +
`TestBuildCommentReplyInstructionsNonCodexLinux` +
`TestBuildCommentReplyInstructionsWindowsUsesContentFile` — the
reply-template provider × OS matrix.
- `TestInjectRuntimeConfigWindowsCommentTriggerHasNoStdin` — end-to-end
AGENTS.md / CLAUDE.md on Windows has no prescriptive stdin
directive, for claude / codex / opencode.
`go test ./...` and `go vet ./...` clean.
Closes#2198, #2236, #2376.
Co-authored-by: multica-agent <github@multica.ai>
Kiro and Kimi share Hermes' ACP architecture and already accept
SystemPrompt prepended in front of the user prompt (kiro.go:244-247,
kimi.go:256-257). Without daemon-side opt-in, ExecOptions.SystemPrompt
is never set, so per-task agent identity instructions are lost in
deployments that rely on inline injection (e.g. K3 Lens-style
daemon → wrapper → docker compose exec acp).
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): mark provider 429 / out-of-credit runs as failed, not completed
Two bugs combined to silently report failed agent runs as
"Completed" in the UI when the upstream LLM returned a 4xx (e.g.
HTTP 429 rate-limit / no credit on the account).
1. ACP backends (hermes, kimi, kiro) only promoted the run status to
"failed" when their stderr sniffer fired AND the agent output
buffer was empty. But hermes injects a synthetic agent text turn
("API call failed after 3 retries: HTTP 429...") on retry
exhaustion, so the buffer was never empty in the rate-limit
case and the promotion never ran. Drop the empty-output
precondition: the sniffer's regex (HTTP-status markers, named
error types) is specific enough to trust on its own.
2. The daemon's task-result switch only routed "blocked" through
FailTask; every other status — including "cancelled", and any
future status we forget to enumerate — fell through to
CompleteTask. Invert it so only an explicit "completed" status
reports success, and extract the switch into reportTaskResult
for direct testing. Cancelled now defaults to failure_reason
"cancelled" instead of being silently completed.
Closes GitHub multica#1952.
Co-authored-by: multica-agent <github@multica.ai>
* fix(agent): only promote ACP run to failed on terminal provider error
Address GPT-Boy's review on the multica#1952 fix. The previous
promotion rule ("any sniffer line → fail") was too broad: the
existing sniffer also captures transient per-attempt warnings
("API call failed (attempt 1/3): RateLimitError [HTTP 429]"), and
those lines stay in the buffer for the rest of the run. A retry
sequence whose first attempt blipped but whose third attempt
succeeded would have been wrongly reported as failed.
Tighten the criteria with two additional signals, both defined on
the existing acpProviderErrorSniffer / output buffer:
- acpTerminalErrorRe — sticky `terminal` flag set when stderr shows
an exhausted/non-retryable marker (❌, [ERROR], "after N retries",
Non-retryable, BadRequestError, AuthenticationError). Per-attempt
warnings deliberately don't match.
- acpAgentOutputTerminalRe — matches the synthetic "API call failed
after N retries..." turn that hermes-style adapters inject into
the agent text stream when they give up; this catches multica#1952
even if hermes' stderr only logged transient attempts.
Promotion logic becomes a shared helper, promoteACPResultOnProviderError,
called from hermes / kimi / kiro. Promotes when (a) terminalMessage
is non-empty, (b) output contains the synthetic give-up turn, or
(c) output is empty and the sniffer captured anything at all
(preserves the original empty-output safety net for transient-only
sequences with no real result to fall back on).
Tests:
- TestHermesProviderErrorSnifferTerminalVsTransient — transient
attempt 1/3 alone returns terminalMessage="" but message!="";
a follow-on terminal marker flips terminal on.
- TestHermesProviderErrorSnifferTerminalNonRetryable — confirms
BadRequest / Authentication / Non-retryable / ❌ / [ERROR] are
classified terminal even on the very first attempt.
- TestHermesBackendDoesNotPromoteOnTransientRetry — fake hermes
emits attempt 1/3 to stderr then a normal agent text turn and
end_turn; resulting Status must stay "completed".
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): use brew prefix symlink for self-restart so Linux Cellar deletion does not orphan runtimes
After brew upgrade on Linux, os.Executable() resolves /proc/self/exe to
the Cellar path (e.g. .../Cellar/multica/0.2.9/bin/multica), which
brew cleanup deletes. The previous IsBrewInstall() short-circuit skipped
EvalSymlinks to 'preserve' the symlink, but on Linux there was nothing
to preserve - the path was already resolved.
Use cli.GetBrewPrefix() to resolve the stable symlink path
<brewPrefix>/bin/multica for brew installs. Fall back to
EvalSymlinks(os.Executable()) with a warning log when GetBrewPrefix()
returns empty (brew binary missing from PATH).
Introduce package-level function vars (isBrewInstall, getBrewPrefix) so
the daemon test can override them without modifying the cli package.
Closes#1624
* fix(daemon): harden brew-prefix fallback and document the WHY
When `brew --prefix` is unavailable but the binary is under a known Cellar
root, recover the prefix from cli.MatchKnownBrewPrefix and target
<prefix>/bin/multica instead of falling back to the resolved Cellar path
(which brew cleanup just deleted).
- Extract knownBrewPrefixes + MatchKnownBrewPrefix in cli/update.go and
reuse from IsBrewInstall to keep one source of truth for the install-root
list.
- Add a WHY comment above the brew branch in triggerRestart explaining the
/proc/self/exe -> Cellar -> deleted-by-brew-cleanup chain.
- Cover both fallback paths (matched / unmatched) in daemon_test.go.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
When the server deletes a task while the daemon's agent is still running
(issue removed, agent reassigned, workspace cleanup), GetTaskStatus
starts returning 404 "task not found". The previous polling loop only
checked for status == "cancelled" and silently swallowed the error, so
the local agent kept emitting tool calls against a dead task until its
own timeout fired — minutes of wasted model spend and patch_apply
operations against a workdir nobody would consume.
Changes:
- Add isTaskNotFoundError next to isWorkspaceNotFoundError so the daemon
can distinguish "task gone" 404 from "workspace gone" 404 (already
handled separately) and from generic network errors.
- Extract the cancellation polling goroutine in handleTask into
watchTaskCancellation, plus a pure shouldInterruptAgent decision
helper. The pure helper makes both signals (cancelled status and 404
task) easy to unit-test without spinning up a real backend.
- Trigger interruption on the new 404 path. Transient errors (5xx,
network) intentionally still don't cancel — the next poll will retry
and a flaky link should not kill an in-flight agent.
Tests cover the helper truth table, the existing "status cancelled"
path, the new "task deleted (404)" path, and a negative case ensuring a
running task is not interrupted.
Co-authored-by: “646826” <“646826@gmail.com”>
TestRegisterTaskReposSurvivesWorkspaceRefresh started flaking on CI
after #1988 (`feat: support repo checkout ref selection`) extended the
bare-clone path to run an extra `git fetch` to backfill
refs/remotes/origin/* under the new refspec layout. The race was
already latent: registerTaskRepos kicks off `go syncWorkspaceRepos(...)`
to clone a repo into the cache root, which in tests is `t.TempDir()`.
Once the test waited on `repoCache.Lookup` to return a path it would
proceed and return — but the bg goroutine was still inside
`ensureRemoteTrackingLayout` running git operations on the clone dir.
`t.TempDir`'s cleanup then races with those git commands and surfaces
either as "directory not empty" or "fatal: cannot change to ... No such
file or directory", with no hint that the failure is unrelated to the
test's actual assertion.
Track the background goroutine on the Daemon via a sync.WaitGroup and
expose `waitBackgroundSyncs()` for tests. `newRepoReadyTestDaemon`
registers a t.Cleanup that calls it, so every test that uses the
helper now drains in-flight syncs before t.TempDir cleanup runs. No
production-behavior change — registerTaskRepos still fires-and-forgets
from the caller's perspective.
Verified with `go test ./internal/daemon -run
TestRegisterTaskReposSurvivesWorkspaceRefresh -count=30` (was failing
within ~10 iterations before, 30 green after) and the full
`go test ./internal/daemon/...` suite.
Co-authored-by: multica-agent <github@multica.ai>
* refactor(repos): drop unused description + tighten create-project layout
Two related changes that touch the workspace-repos surface together.
1. Remove the per-repo `description` field everywhere it was threaded.
The only place it ever surfaced was a markdown table column the daemon
wrote into the agent runtime config, where most rows just read "—"
anyway. Agents already discover project structure by running
`multica project` / `multica issue` against the CLI, so the human-
readable description string carried no real value while taking up an
extra Settings input row and propagating through six layers (settings
UI → workspace.repos jsonb → handler RepoData → daemon RepoData →
repocache.RepoInfo → execenv.RepoContextForEnv).
- Settings → Repositories drops the description input; the URL field
now spans the whole row.
- WorkspaceRepo TS type loses `description`; backend RepoData /
RepoInfo / RepoContextForEnv all collapse to URL only.
- Daemon's runtime_config Repositories block changes from a
`| URL | Description |` markdown table to a simple bullet list.
- Tests updated; jsonb residue in existing workspaces is dropped at
normalize time, so no migration needed.
2. Tighten the Create Project modal footer: pull the Status / Priority /
Lead / Repos pills onto the same row as the Create Project button
(Linear-style single-row footer) instead of stacking them above it,
and swap the Repos pill icon from `FolderGit` to a real GitHub mark
(lucide-react v1 dropped brand icons, so the mark lives inline as a
small SVG component in this file).
I tried promoting Repos to its own "Resources" strip above the footer
to separate the resources abstraction from project metadata, but with
a single pill it looked too sparse — leaving a TODO comment in the
footer to revisit once we add Linear / Notion / Figma / Slack
resource types.
* fix(daemon test): drop residual Description field on RepoData literals
* fix(repos): drop Description residue surfaced after rebase on #1929
Project-resource github_repo lift path (#1929) and registerTaskRepos
both still constructed RepoData{...Description: ...} after the rebase.
Two test sites in daemon_test.go and execenv_test.go also reintroduced
the field. Strip them so the Description-removal change builds and
tests pass with the latest main.
* feat(projects): project github_repo resources override workspace repos
When an issue's project has at least one github_repo resource, the daemon
claim handler now sends only those as resp.Repos — workspace-level repos
are hidden to avoid mixing two repo lists in the agent prompt. With no
project github_repos (or no project), behavior is unchanged: workspace
repos are surfaced as before.
Lifts each project github_repo's url (and label, when present) into a
RepoData entry so `multica repo checkout` and the meta-skill render the
same URLs. The full structured list still ships at
.multica/project/resources.json for skills that want everything.
Adds TestProjectReposReplaceWorkspaceReposInMetaSkill covering the
rendering side. Docs updated to spell out the new precedence.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): allow project repo URLs through the checkout allowlist
When ClaimTaskByRuntime narrows resp.Repos to project github_repo URLs,
the daemon receives URLs that may not exist in the workspace's
GetWorkspaceRepos response. The existing checkout flow rejected those
with ErrRepoNotConfigured because the allowlist (and cache) was built
only from workspace-bound repos.
Adds registerTaskRepos in daemon.runTask: before agent spawn, merge
task.Repos into a new task-scoped allowlist (separate from the
workspace-scoped one so a workspace refresh doesn't wipe project URLs)
and kick off a background cache sync. ensureRepoReady now treats either
allowlist as valid.
Tests:
- TestRegisterTaskReposAllowsProjectOnlyURL — project-only URL is
checkout-able and does not trigger a workspace-repos refresh
- TestRegisterTaskReposSurvivesWorkspaceRefresh — task URLs persist
across refreshWorkspaceRepos
- TestClaimTask_ProjectGithubReposOverrideWorkspaceRepos — claim
handler returns only project repos when present, no workspace leakage
- TestClaimTask_ProjectWithoutRepos_FallsBackToWorkspaceRepos — fall
back to workspace repos when project has no github_repo resources
Docs updated to spell out the daemon-side allowlist behavior.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
* feat(daemon): add Co-authored-by trailer for Multica Agent to git commits
Install a prepare-commit-msg hook in worktree bare repos that appends
"Co-authored-by: multica-agent <github@multica.ai>" to every commit
made by agents. Uses git interpret-trailers for proper formatting and
skips duplicates.
* feat(settings): add Co-authored-by toggle in workspace Labs settings
Add a workspace-level toggle to enable/disable the Co-authored-by
trailer for agent commits. Default is enabled (on).
Backend:
- Include workspace settings in daemon register response
- Store settings in daemon workspaceState
- Thread CoAuthoredByEnabled through WorktreeParams to conditionally
install the prepare-commit-msg hook
- Parse co_authored_by_enabled from workspace settings JSONB
Frontend:
- Replace empty Labs tab placeholder with a Git section containing
a Switch toggle for the Co-authored-by trailer setting
- Optimistically update the workspace query cache on toggle
* chore(daemon): skip squash commits in Co-authored-by hook
Test commit to verify the prepare-commit-msg hook appends the
Co-authored-by trailer automatically.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
When an agent replied in a thread whose root mentioned another agent,
the reply inherited the parent mention and re-triggered the other agent.
This caused 'No reply needed' ping-pong loops between co-assigned agents.
Structural fix:
- In enqueueMentionedAgentTasks, suppress parent-mention inheritance
when authorType == 'agent'. Explicit @mentions in the agent's own
comment still work for deliberate handoffs.
Defense-in-depth (prompt):
- Strengthen per-turn prompt and AGENTS.md workflow instructions to
explicitly forbid posting 'No reply needed' noise comments.
Regression test:
- TestAgentReplyDoesNotInheritParentMentions covers both the fix
(agent reply does not re-trigger) and the positive control
(member reply still inherits mentions).
Also updates TestBuildPromptCommentTriggeredByAgent to match the
new prompt wording.
Follow-up to #1686. Locks in two nits flagged during review:
1. agent.Result.Status doc comment now lists "cancelled" alongside the
existing values, so the enum surface matches actual usage.
2. New TestExecuteAndDrain_ContextCancelled_ReportsCancelled exercises
the path added in #1686: when the parent context is cancelled before
the backend produces a Result, executeAndDrain must return
Status="cancelled" (not "timeout"). A regression here would silently
restore the misleading log line we just fixed.
* feat(daemon): harden agent mention-loop instructions
Two agents that mention each other via `mention://agent/<id>` can fall into
an infinite reply loop — each says "I'm done" in prose but keeps
`@mentioning` the other, which re-enqueues their run. Adding hard caps on
agent-to-agent turns conflicts with Multica's design principle of giving
agents the same authorship freedom as humans, so this change hardens the
instructions that the harness injects instead.
- Replace the terse "mentions are actions" blurb with a full Mentions
protocol: `side-effecting` warning, explicit "when NOT to mention"
(replying to another agent, sign-offs, thanks) and "when a mention IS
appropriate" (human escalation, first-time delegation, user asked).
- Add a pre-workflow decision step for comment-triggered runs: decide
whether a reply is warranted at all, decide whether to include any
`@mention`, and clarify that the post-a-comment rule is mandatory *if*
you reply — silence is a valid exit for agent-to-agent threads.
- Thread the triggering comment's author kind + display name
(`TriggerAuthorType` / `TriggerAuthorName`) from the claim endpoint
through the daemon task type, per-turn prompt, and CLAUDE.md workflow.
When the author is another agent, both surfaces now name that agent
and warn against sign-off mentions.
- Soften the old closing line that told agents to `always` use the
mention format — the word generalized to member/agent mentions and
encouraged the very behavior that causes loops.
Refs GH#1576, MUL-1323.
* fix(daemon): remove MUST-respond conflict and sanitize trigger author name
Addresses two blocking points on PR #1581:
1. buildCommentPrompt told the agent "You MUST respond to THIS comment"
and unconditionally appended the reply command — directly conflicting
with the new agent-to-agent silence-as-valid-exit workflow. Models
were likely to keep following the older must-reply rule and fall back
into the loop this PR is trying to close.
Rewrite the header as "Focus on THIS comment — do not confuse it
with previous ones" (keeps the anti-stale-comment signal) and change
BuildCommentReplyInstructions to open with "If you decide to reply,
post it by running exactly this command" so the reply command is
available but conditional across both prompt surfaces.
2. Raw agent/user display names were being embedded directly into the
high-priority prompt and CLAUDE.md via TriggerAuthorName. Agent and
member names are only validated as non-empty at write time, so a
name containing newlines, backticks, or fake mention markup would
turn the field into a cross-agent prompt-injection surface.
Add execenv.SanitizePromptField — strip control runes, collapse
whitespace, drop markdown structural characters (backtick, asterisk,
brackets, pipe, angle brackets, hash, backslash), truncate to 64
runes — and apply it at both embed sites (per-turn prompt and
CLAUDE.md). Defense-in-depth at the consumption layer so this works
for already-stored names without a migration.
Tests: TestSanitizePromptField covers the policy; TestBuildPromptSanitizesAgentName
plants an attack payload in TriggerAuthorName and checks the rendered prompt
does not leak the newline-anchored injection or the fake mention markup.
TestBuildPromptCommentTriggered*{,ByMember} updated to lock in the
conditional reply-command framing.
* refactor(daemon): trim redundant CLAUDE.md preamble and drop name sanitizer
Per PR #1581 feedback:
1. Remove the `if ctx.TriggerAuthorType == "agent"` preamble block in
runtime_config.go. It duplicated what workflow steps 4 and 5 already
say ("Decide whether a reply is warranted", "Never @mention the
agent you are replying to as a thank-you or sign-off"), so the
signal lands the same without the extra ~7 lines of CLAUDE.md. The
per-turn prompt preamble in prompt.go stays — that surface has no
numbered workflow below it and would otherwise lose the
silence-as-exit signal.
2. Delete execenv.SanitizePromptField + its test. Workspace agents are
created by trusted team members, so the cross-agent name-injection
surface it defended isn't realistic in the current trust model.
3. Drop TriggerAuthorType/Name from execenv.TaskContextForEnv and stop
populating them in daemon.go — they're no longer read by the
execenv package. The same fields on daemon.Task stay because
prompt.go still needs them to label the triggering author in the
per-turn prompt.
Tests simplified to match the leaner shape: CLAUDE.md regression
guards now assert that the anti-loop phrases live in the numbered
workflow, and the sanitizer-specific tests are removed.
* fix(agent/comments): re-emit trigger comment id every turn + server-side parent_id guard
Resumed Claude sessions keep prior turns' tool calls in context, so a
comment-triggered task could reuse the PREVIOUS turn's --parent UUID
instead of the current trigger's. The reply landed in the wrong thread
(MUL-1125): backend stored exactly what the agent sent, but the agent
pulled a stale UUID from its own conversation memory.
Two layers of defense:
1. Extract BuildCommentReplyInstructions so daemon.buildCommentPrompt
and execenv.InjectRuntimeConfig emit the same "use this exact
--parent, do not reuse values from previous turns" block. The
per-turn prompt now carries the current TriggerCommentID, which it
previously relied on CLAUDE.md for (and CLAUDE.md isn't re-read
mid-session).
2. Handler-side guard in CreateComment: when an agent posts from inside
a comment-triggered task (X-Agent-ID + X-Task-ID, task has
TriggerCommentID), require parent_id == task.TriggerCommentID or
return 409. Assignment-triggered tasks are untouched.
* fix(agent/comments): scope parent_id guard to the task's own issue
Two issues from CI + GPT-Boy's review:
1. Guard was too broad: the CLI stamps X-Task-ID on every request, so an
agent legitimately commenting on a different issue while its current
task was comment-triggered would get 409'd with the wrong issue's
trigger comment id. Narrow the guard to fire only when the request's
issue matches the task's own issue — cross-issue agent activity
stays unblocked.
2. The integration test tried to insert a second queued task for the
same (agent, issue), which hits the idx_one_pending_task_per_issue_agent
unique index. Replace the assignment-triggered-task sub-case with a
cross-issue regression test (the scenario we now need to cover anyway):
post on issue B while X-Task-ID points at a comment-triggered task on
issue A, expect 201.
- TrimSpace incoming repoURL in ensureRepoReady to prevent unnecessary
server refreshes when CLI passes URLs with whitespace
- Add comment on reposVersion field clarifying it is stored for future
version-based skip optimization
- Add concurrency safety comment on syncWorkspacesFromAPI skip logic
- Add test for URL trimming fast-path behavior
* fix(daemon): add fallback for failed session resume
When the daemon tries to resume a prior session (--resume flag for
Claude, --session for OpenCode, session/resume RPC for Hermes) and the
session no longer exists, the agent fails immediately. This adds a
fallback that retries the execution with a fresh session instead of
marking the task as blocked.
Extracts the execute+drain logic into a reusable executeAndDrain method
to avoid code duplication between the initial attempt and the retry.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(daemon): narrow session resume retry and merge usage
Address review feedback:
1. Narrow retry trigger: only retry when result.SessionID == "" (no
session was established), not on any failure with PriorSessionID set
2. Merge token usage from both attempts so billing is accurate
3. Log errors when the retry itself fails to start
4. Add unit tests for mergeUsage, fallback behavior, and no-retry
when session was already established
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When a task is triggered by a comment, the agent prompt now includes
the comment content directly. This prevents the agent from ignoring
the comment when stale output files exist in a reused workdir.
Closes#805
