* perf(daemon): parallelize runtime version detection during registration (MUL-5119)
Registration probed each agent CLI's `--version` serially, so total latency
was the sum of every probe. On an onboarding host with several coding tools
installed that stacked into many seconds before runtimes registered — long
enough that the desktop runtime step timed out into its empty 'no runtime
found' state while the daemon was still working.
Fan the probes out with a bounded errgroup so total latency tracks the slowest
single probe instead of their sum. Each probe still self-heals a vanished
pinned path and re-detects the live version (no cross-registration caching, so
an in-place upgrade is still reported correctly); failures are logged and
skipped as before. Results are sorted by provider for a deterministic payload.
Co-authored-by: multica-agent <github@multica.ai>
* fix(onboarding): stop the runtime step flashing 'no runtime found' while the daemon probes (MUL-5119)
The runtime step flipped from scanning to the empty 'no runtime found' state on
a fixed 5s wall-clock, so a machine that does have coding tools installed saw a
false-negative flash whenever registration outlasted the timeout (cold start,
slow/wedged CLI, many CLIs).
Gate the empty flip on a desktop-only `runtimesPending` signal derived from the
local daemon's live status (booting, or running with agent CLIs detected on the
host): while pending, keep the scanning skeleton past the soft timeout. An
absolute hard-timeout ceiling still guarantees a fallback so a wedged probe
can't pin the step on the skeleton forever. Web omits the signal and keeps the
plain wall-clock timeout.
Also drop the two dead/duplicated affordances on the step: the permanently
disabled 'Start exploring' button now renders only in the found phase, and the
empty state's duplicate footer 'Skip for now' is removed in favour of its own
Skip card.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Lambda <lambda@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
* fix(squads): align parent issue status ownership with agent-managed model
Squad leaders now open assigned parents to in_progress on first dispatch, keep them there while members work, and only move to in_review when overall completion is confirmed—matching ordinary agent status semantics without server auto-flips.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(squads): scope leader parent-status ownership to squad-assigned issues
Review follow-up on the parent-status alignment change. Two boundaries were
left ambiguous, both of which the change's own premise ("don't make the model
resolve a contradiction in the prompt") argues should be closed in-place.
1. Status ownership was granted too widely. The leader briefing is injected on
every leader path, keyed off is_leader_task — including the MUL-3724 case
where an issue is assigned to a plain agent and a squad was merely
@mentioned for help. The unqualified "Own the parent issue status"
responsibility therefore also reached guest leaders, who could push another
assignee's in-flight issue to in_review.
buildSquadLeaderBriefing now takes ownsIssueStatus and selects between two
variants of responsibility 6: the grant only when the issue's assignee is
this squad, otherwise an explicit "do NOT change this issue's status".
Quick-create passes false — no issue exists on that turn. Everything else in
the protocol (roster, delegation, evaluation) is unchanged for both.
2. The comment-triggered path still contradicted itself. The runtime brief says
"do not change status unless the comment explicitly asks", and a member's
delivery comment never asks. Squads that dispatch by @mention create no
child issues, so no child-done system comment exists to carry the explicit
ask either — that parent would sit in in_progress indefinitely.
writeWorkflowComment now names the protocol responsibility as the one
exception for squad leaders. It is safe to state unconditionally because the
grant is only present in the instructions when the server decided this squad
owns the issue; for a guest leader the sentence has nothing to activate.
Tests: two composition tests assemble both halves (server-side briefing +
daemon-side CLAUDE.md) for one real scenario each, since asserting each half
alone is how the original contradiction shipped. Plus execenv coverage that the
carve-out appears only for leaders and the ordinary-agent rule stays absolute.
Docs and the multica-squads skill / source map record the narrower contract.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: J <j@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
The isolated-checkout path used by Linux Codex builds a task-local
repository with `git clone --local` from the workspace's bare cache.
That command has two properties that combine badly when the cache is a
partial clone: it does not carry the promisor remote configuration
across, and it does not treat an incomplete source object store as an
error. The result is a checkout that exits 0 with every tracked file
reported as deleted, so an agent starts work in what looks like a
repository someone emptied.
Swap origin to the real remote and restore
`remote.origin.promisor` / `remote.origin.partialclonefilter` before the
first checkout, so git can lazily fetch the blobs it needs. Do the same
on the reuse path, where a workdir created against a complete cache can
later be resumed against a partial one.
No cache is created as a partial clone today, so this changes nothing
for existing installs; it is a prerequisite for the on-demand clone mode
in MUL-4983 and hardens a path that fails silently rather than loudly.
Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
* feat(agents): add per-agent runtime skill controls
Co-authored-by: multica-agent <github@multica.ai>
* fix(agents): renumber runtime-skill migration and broadcast agent:status on toggle
Address the MUL-5101 review blockers on PR #5686:
- Rebase onto main and renumber the runtime-skill-disable migration
202 -> 203. main added 202_runtime_profile_add_qwen, so the pair
collided on prefix 202 and migrations_lint_test would reject the
duplicate. 203 is the next free prefix.
- Publish an "agent:status" event after persisting a
disabled_runtime_skills override, mirroring the workspace-skill toggle
in writeUpdatedAgentSkills. The realtime layer keys off this event to
invalidate workspaceKeys.agents, so other open web/desktop/mobile
clients now drop their stale toggle state instead of only the
initiating tab refreshing. Reload junction-table skills before the
broadcast so it doesn't signal cleared skills (#3459).
- Add a handler regression test proving the broadcast fires on both
disable and enable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
Co-authored-by: Walt <walt@multica.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Rework skills.sh and github.com skill imports around a single recursive
git-tree fetch to stop the 504 on large mono-repos (e.g. api-gateway-skill):
- One tree call replaces the per-directory contents crawl.
- Import caps checked arithmetically from tree metadata before any download
(fail fast with 413 instead of timing out).
- Most-specific skill-dir resolution; repo root only as a last resort, which
fixes the root SKILL.md name collision.
- Concurrent downloads (errgroup, limit 8).
- Overall 45s fetch deadline; cancellation is fatal on every supporting-file
path (tree downloader, crawl listing/recursion/download, ClawHub) so a
mid-download abort never persists a half-populated bundle.
- A skills.sh tree-fetch failure returns a retryable 503 instead of an unsafe
root-directory fallback.
- Lenient conventional-path acceptance restored for both complete and truncated
trees.
- maxImportFileCount 128 -> 256 (aligned daemon cap); 8 MiB bundle cap remains
the real guard.
* fix(daemon): gate fresh-session retry on tools executed, not session id (MUL-4966)
Switching provider accounts leaves the stored session id pointing at a
conversation the new account does not own. The daemon still passes it to
--resume, the provider rejects it, and the task dies before doing any work.
The existing fresh-session fallback was supposed to catch this but was
gated on `result.SessionID == ""`, which is not a lifecycle fact:
- Too narrow: a backend that echoes the requested id back when it rejects
a resume keeps SessionID non-empty, so the fallback never fired — the
reported bug.
- Too broad: a provider 401 before the first stream message also leaves
SessionID empty, so an unrecoverable auth failure burned a second full
run.
Gate on `tools == 0` instead. That states the property that actually makes
a retry safe — the agent executed no tool, so it mutated nothing, so
re-running cannot double-post a comment (comment creation has no
idempotency key and a duplicate re-fires its @mention triggers), reopen a
PR, or re-plan on top of its own half-finished work in the reused workdir.
Auth failures are excluded, mirroring retryableReasons in service/task.go.
The predicate is extracted to shouldRetryWithFreshSession so the tests
exercise production logic; both existing fallback tests re-implemented the
condition inline and would not have caught a regression in it.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): gate fresh-session retry on a positive resume-rejected signal (MUL-4966)
Review of the previous commit was right: `tools == 0` plus "not an auth
error" answers whether re-running is *safe*, not whether a new session can
*fix* the failure. Those are orthogonal, and answering the second by
exclusion inverts the burden of proof — the failures a fresh session cures
are a small enumerable set, while the ones it cannot are open-ended.
Concretely, the previous predicate fresh-retried on provider_network,
429/529, quota, 5xx and unclassified startup failures. provider_network is
the sharpest conflict: internal/service/task.go marks it resume-safe
(MUL-4910) specifically so the platform retry inherits the session and
continues the truncated conversation. Resetting the session first made that
contract unsatisfiable, silently discarding conversation context on a
transient blip — and rate limits got an immediate no-backoff re-run.
Replace the inference with positive evidence: agent.Result gains an explicit
ResumeRejected field, set only when a backend has proof the resume itself
was refused. claude/codebuddy/qwen derive it from resumeWasRejected, which
promotes the predicate resolveSessionID was already computing and encoding
as the side effect of blanking SessionID — using an empty string to carry
that meaning is what made the original bug possible. SessionID keeps being
dropped for a rejected resume (a dead pointer must not be persisted), but it
is no longer the signal the daemon reads to decide *why* a run failed.
The six ACP backends that recover from "session not found" set the flag at
the same points they already clear the id, so their existing recovery is not
caught by the narrower gate. codex needs nothing: thread/resume already falls
back to thread/start in-process, and deliberately does not on transport
errors.
Matching now includes the account-switch guardrail reported in #5704
(Claude Code 2.1.207, zh-CN): "400 此 session 已绑定另外的ai账号，请执行
/new 开启新 session". The en-US wording of the same guardrail has not been
captured yet, so those variants are marked inferred in the source; a miss
degrades to a terminal failure carrying the provider's raw text rather than
a mis-routed run.
Tests: backend-level fixtures drive ResumeRejected from real stream-json for
both the account-binding 400 and a network drop, and the predicate now
covers network/rate-limit/quota/5xx/auth/unclassified as explicit
non-retries.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): restore fresh-session recovery for backends with no rejection signal (MUL-4966)
Final review caught qwen regressing: its verified rejection string
("No saved session found with ID ...", already captured in
testdata/qwen-code-0.20.0-resume-not-found.stderr.txt) was not in the phrase
list, and qwen reports no session id on that path, so the new inclusion gate
turned a working auto-recovery into a terminal failure.
Auditing the other 17 resume-capable backends showed qwen was not alone.
antigravity, copilot, cursor, deveco and opencode all recovered from a
refused resume purely by reporting an empty SessionID, and none of them has
any rejection detection to convert into ResumeRejected — copilot's own
comment documents the hole (session.error before session.start), and
antigravity's helper returns "" when "the CLI exited before dispatching".
Making ResumeRejected the sole gate silently removed recovery from all five.
Fixing that by guessing rejection phrases for five more CLIs is the wrong
trade: no real output has been captured for any of them, and a false
positive discards a recoverable session pointer. So the gate is now two
tiers. Positive evidence (ResumeRejected) decides on its own where a backend
can produce it. Where none is available, an empty SessionID still gates the
retry — it proves no session was established, which is exactly what the
pre-change behaviour relied on — minus the classes a fresh session provably
cannot cure (network, rate limit, quota, provider 5xx, auth). That keeps the
resume-safe contract in internal/service/task.go intact while restoring what
these five backends had.
Also renames claudeResumeRejectedPhrases to resumeRejectedPhrases: it is
matched by claude, codebuddy and qwen, so a qwen-only string living under a
claude-prefixed name would be actively misleading.
Tests: qwen's existing missing-resume fixture now asserts ResumeRejected
(verified failing without the phrase), and the predicate covers the
no-signal tiers — retry when nothing was established, no retry once a
session exists or the failure classifies as uncurable.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): scope the no-signal fallback to backends that cannot detect rejections (MUL-4966)
Final review caught the compatibility path applying to every backend, not
just the five it was justified for. shouldRetryWithFreshSession only saw
(Result, priorSessionID, tools), so a false ResumeRejected could not be told
apart from a backend that has no way to answer — and claude/codebuddy/qwen/ACP
startup failures with no session id still fell through to the exclusion
branch. That contradicted both the stated intent and the function's own doc
comment ("where a backend can produce it, it is the whole answer").
Make the capability explicit. agent.ResumeRejectionUndetectable names the
five backends that scrape SessionID out of stream output and have no
rejection detection at all; the daemon takes provider and consults it, so a
capable backend reporting false is now taken at its word. Membership is
opt-in, so a new backend fails closed instead of silently inheriting a
guess-based retry.
Also completes the exclusion set: missing config, unavailable model, missing
executable, unsupported runtime version and (defensively) agent timeout all
have defined non-session remedies and were reaching `default: true`. What is
left through stays narrow — unknown, process failure, unparseable output,
context overflow — because a real rejection from these five most likely
surfaces as a non-zero exit or unparseable output, none of them reporting one
explicitly.
Tests: one identical result asserted across all five undetectable backends
(retries), twelve capable ones (no retry), and an unregistered provider
(fails closed), plus table cases for each newly excluded reason. Classifier
inputs were verified to map to the intended reasons rather than passing by
accident.
Also updates the ResumeRejected doc comment, which still said the daemon
gates on it alone.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): run Codex unsandboxed on Windows to stop reject-by-policy (MUL-4957)
Windows has no Landlock/Seatbelt-equivalent filesystem sandbox that the
daemon configures, so the per-task `sandbox_mode = "workspace-write"` it
wrote was unenforceable. Worse than having no sandbox, it pushed Codex
into rejecting non-safe mutation commands "by policy": `multica issue
create` fails with "was rejected by policy" because Codex can neither
sandbox the command nor (under approval_policy = "never") escalate it to
the daemon's auto-approver, so the request never reaches the approver.
Mirror the existing macOS fallback and give Windows danger-full-access so
those commands run. Also generalize the danger-full-access warn log so it
no longer hardcodes "on macOS" and only surfaces the macOS-specific
upgrade hint on macOS (new codexSandboxPolicy.Hint field).
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): correct Windows sandbox rationale and respect user windows.sandbox (MUL-4957)
Addresses two review must-fixes on #5672:
1. Correct a false security fact. The comments and log Reason claimed
Windows has no filesystem sandbox backend. Codex 0.144.5 does ship a
native Windows sandbox (windows.sandbox = "unelevated"/"elevated"); it
is experimental with open upstream reliability bugs, so the daemon
defaults to danger-full-access as a deliberate compatibility choice.
Enabling the native sandbox is tracked as separate follow-up work.
2. Stop silently downgrading users who opted into isolation. The fallback
was unconditional. Add codexSandboxPolicyForConfig: on Windows an
explicit windows.sandbox = unelevated|elevated keeps workspace-write so
Codex enforces task isolation with the user's chosen backend;
danger-full-access applies only when windows.sandbox is absent,
disabled, or unparseable. This is also the branch point for a future
native-sandbox rollout (flip the default; callers unchanged).
Adds fixture tests locking the priority (user opt-in kept vs. unconfigured
fallback) plus predicate coverage for codexSandboxPolicyForConfig.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): fail closed on undecidable Windows sandbox config, honor -c windows.sandbox (MUL-4957)
Second review round on #5672. Two must-fixes.
1. Undecidable config no longer fails open. The old bool detector collapsed
"unparseable / invalid value / failed copy" into "unconfigured" and then
loosened to danger-full-access. Replaced with a tri-state
(absent/native/undecidable): only exact-lowercase unelevated|elevated (the
sole values Codex accepts — verified: any other value makes Codex refuse to
load the config) counts as native; any other present value, unparseable
TOML, a read error, or a missing per-task config when a shared
~/.codex/config.toml exists (i.e. the copy failed) is undecidable and fails
closed to workspace-write — it never loosens — logged at error level.
2. windows.sandbox set via `-c`/`--config` custom args is now honored. Such
args never land in config.toml, so config-only detection silently
downgraded those users' isolation. The effective Codex args (daemon
defaults + profile-fixed + per-agent custom_args) are threaded through
PrepareParams/ReuseParams/CodexHomeOptions into the sandbox decision and
scanned for a windows.sandbox override (inline, two-token, quoted, spaced;
last-wins).
Also drops issue-status-bound source comments (openai/codex#24098 has since
closed). Adds unit coverage for config/args classification, the fold
precedence (undecidable > native > absent), and the copy-failed fail-closed
path.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): fail closed on config-sync errors and honor shell-quoted -c windows.sandbox (MUL-4957)
Round-3 review must-fixes:
1. resolveWindowsSandboxState now takes the config.toml sync error and a
tri-state shared-config presence instead of re-stat-ing inside. A failed
sync (stale/absent per-task copy) or an un-stat-able shared source is
undecidable and keeps workspace-write, closing the fail-open where a failed
sync was read as "unconfigured". Splits IO from the decision so the paths
are unit-testable without faulting the filesystem.
2. The Windows sandbox decision consumes agent.NormalizeCodexLaunchArgs (the
shared helper buildCodexArgs now uses) so a shell-quoted -c windows.sandbox
opt-in is normalized identically to launch, instead of being missed by a
raw-token scan and silently downgraded.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): abort when the Codex sandbox block cannot be written (MUL-4957)
Round-4 review must-fix: ensureCodexSandboxConfig failures were warn-and-continue,
so a computed fail-closed workspace-write policy could stay only in memory while
config.toml kept a stale danger-full-access from a prior run — the decision
failed closed but the effective config failed open.
prepareCodexHomeWithOpts now returns the error, which blocks startup on both
paths: fresh Prepare fails the task, and Reuse leaves env.CodexHome unset, which
configureCodexTaskShellEnvironment already refuses to start.
Regression covers the full reuse scenario (stale danger-full-access + failed
config sync + failed managed-block write); it fails with "got nil" without the
fix.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
Co-authored-by: J <j@multica.ai>
The windows-execenv job's TestPrepareIsolated_WindowsKillsDescendantBeforeRetry
flakes on the Windows runner: its helper subprocess ends in a bare select{},
which the Go runtime can reap with 'all goroutines are asleep - deadlock!'
(exit status 2) once every goroutine is parked with no wakeup source. That
races the parent's Job Object kill, so PrepareIsolated returns 'helper failed'
instead of the context.Canceled the test asserts.
Block on a timer-backed sleep loop instead: a pending timer is a wakeable
source, so the runtime never declares a deadlock, and the process still dies
the instant the Job Object tears the tree down.
Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
* feat(issues): bump issue updated_at when a comment is added (MUL-5009)
A new comment now counts as activity on its issue and advances
updated_at, so the "Updated date" Kanban/list sort surfaces
recently-discussed cards — not only cards whose status changed.
Applies to all three comment-creation paths (user/agent HTTP,
agent task delivery, and the child-done system comment) via a
best-effort TouchIssue query. The bump never fails an already-
persisted comment; it self-heals on the next activity if it errors.
Co-authored-by: multica-agent <github@multica.ai>
* fix(issues): make comment updated_at bump atomic (MUL-5009 review)
Address Elon's review. Move the updated_at bump into CreateComment as a
leading data-modifying CTE so the comment insert and the timestamp bump
commit or roll back together — closing the non-atomic window where a
comment could persist while updated_at stayed stale. That window also
skewed the daemon GC TTL, which reads issue.updated_at to reclaim
done/cancelled workdirs.
Centralizing the bump in the query drops the three per-caller TouchIssue
calls and guarantees any future comment entrypoint inherits it.
Also refresh the now-stale gc.go / gc_test.go comments that asserted
'CreateComment does not bump issue.updated_at'.
Co-authored-by: multica-agent <github@multica.ai>
* fix(issues): make comment/issue workspace match a query-level guarantee (MUL-5009 nit2)
The touch CTE now RETURNING id, workspace_id and the INSERT SELECTs from
it, so the comment insert depends on the issue actually existing in the
passed workspace. A mismatched (issue, workspace) pair matches 0 rows in
the CTE, the dependent INSERT selects nothing, and the :one query returns
pgx.ErrNoRows — no mis-attributed comment is written and the issue is not
touched.
CreateComment is now the single carrier of the 'a comment belongs to an
issue in the same workspace and always bumps it' invariant, so no future
caller can break it by passing the wrong workspace. Signature unchanged;
no migration or foreign key.
Add TestCreateComment_WorkspaceMismatchPersistsNothing (error returned,
no comment persisted, updated_at unchanged).
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): bound pre-start task preparation
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): isolate pre-start env preparation
Run execution-environment Prepare and Reuse in a killable helper process so a timed-out attempt cannot keep writing after retry. Add FIFO lifecycle and squad Stage retry regression coverage.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): terminate Windows prepare process trees
Assign the pre-start helper to a kill-on-close Job Object before releasing its request, wait for all job members to exit on cancellation, and add a Windows runtime regression job.
Co-authored-by: multica-agent <github@multica.ai>
* ci: target Windows prepare tree regression
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): isolate Linux Codex git metadata (MUL-4925)
Co-authored-by: multica-agent <github@multica.ai>
* refactor(daemon): address isolated-checkout review nits (MUL-4925)
- rename sameFilesystemPath -> sameResolvedPath (it compares resolved
paths for equality, not same-device), with a clarifying doc comment
- prune earlier tasks' agent/* branches when reusing an isolated
checkout so a long-lived reused workdir stops accumulating one local
branch per checkout; deleteLocalBranches now takes a keepBranch arg
and the prune is non-fatal
- cover the prune in TestCreateWorktreeReusesIsolatedGitMetadata
* fix(repocache): preserve user branches on reuse (MUL-4925)
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Eve <eve@multica-ai.local>
Co-authored-by: multica-agent <github@multica.ai>
Agents were writing runtime-local paths into deliverables as clickable
links (`[screenshot](/Users/agent/work/shot.png)`). Two root causes, both
fixed here.
A. The brief never stated the delivery contract. Add an always-on delivery
invariant (outside writeOutput's kind switch, so no task kind can inherit
none) plus a per-surface file-delivery line for each of the five surfaces.
Chat splits into two: `attachment upload` works only on web/mobile chat,
never on an IM channel, so ChatChannelType is now threaded into
TaskContextForEnv.
The claim path only ever looked up Slack bindings, so a Feishu session
reported as a web chat and got upload guidance for a channel that cannot
carry attachments. Probe every channel type. The chat policy is two
independent layers and stays that way: delivery keys off "is there a
channel at all"; the `chat history` / `chat thread` commands stay
Slack-only because both endpoints are hardwired to h.SlackHistory and
there is no Feishu reader — ChatInThread only selects between those two
commands, so it stays Slack-only too.
Add a CLI hard-fail lint on `issue comment add` / `issue create` /
`issue update` as the enforcement backstop. Scoped narrowly, since a false
positive blocks a real deliverable: agent task context only (a human's PAT
run is untouched), real CommonMark link/image/autolink destinations only
via goldmark (a path in a code span or fence — how an agent quotes a path
it is discussing — is structurally invisible), and three high-confidence
signals only (`file://`, inside the workdir, or an existing local file).
A bare `/foo` is a valid origin-relative URI and is deliberately allowed.
`issue update` has no --attachment flag, so its hint redirects to
`comment add` rather than naming an argument it rejects.
B. Desktop presented the resulting router 404 as an unknown crash. 8 of 18
desktop_route_error reports were users clicking such a link and being told
the app broke and to file a bug. Split the 404 into a first-class Not Found
view: no crash framing, no Report error. Its recovery entry comes from the
tab store's active workspace, never from the failed pathname — deriving a
slug from `/Users/me/shot.png` yields "Users" and a button to `/Users/issues`,
a second 404.
Also add a will-navigate trusted-origin guard via the shared loadRenderer
(main + issue windows). This is origin hardening only, NOT the mechanism for
in-app links: client-side routing never fires will-navigate, so app paths
never reach it. Issue windows need no 404 work — their router only accepts
paths validated by parseIssueWindowPath and they do not listen for
multica:navigate, so a bad path cannot reach them.
Server-side completion observation is metric/log only and never blocks: it
is lexical (`file://` + task work_dir prefix) because the server cannot stat
the daemon's filesystem, and the metric label is a closed enum so no path or
reply text reaches Prometheus.
Verified: pnpm typecheck/lint/test (3582 tests), go vet, full Go suite
including new claim-path integration tests. cmd/multica was verified outside
the daemon workdir — inside one, 93 of its tests fail identically on
origin/main because the suite walks up and finds the runtime's own task marker.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: multica-agent <github@multica.ai>
Squad-leader follow-ups on the same issue now reuse the prior daemon-managed workdir and provider session instead of starting fresh, while never binding or locking a user-provided local_directory. Reuse eligibility is keyed off a Prepare-time .managed_env.json provenance marker, so it does not race the completion→GC-metadata write.
Closes#5535
Co-authored-by: Bohan <bohan@devv.ai>
Give Linux codex tasks a writable per-task HOME (+ XDG/npm_config_cache) so npm/Prisma stop hitting EROFS on the read-only sandbox home. Gated to Linux; macOS/Windows and non-codex providers unaffected.
Note: does NOT resolve the worktree git-metadata read-only problem tracked in multica-ai/multica#2925 — Codex workspace-write resolves the worktree .git pointer and force-protects the real gitdir read-only even inside writable_roots, so that needs a separate Codex metadata-write permission path.
* fix(daemon): stop routing CodeBuddy skills/memory through Claude's .claude paths
CodeBuddy Code is a Claude Code fork but ships its own native config
directory (~/.codebuddy, .codebuddy/) with its own memory filename
(CODEBUDDY.md). It only reads .claude/skills or CLAUDE.md if a user
manually symlinks/copies them during migration
(https://www.codebuddy.ai/docs/cli/troubleshooting#migrating-from-claude-code).
Multica's daemon/execenv code treated "codebuddy" as an alias for
"claude" in three places, so skills synced by Multica landed in
.claude/skills/ and CLAUDE.md — paths the default CodeBuddy install
never reads — instead of ~/.codebuddy/skills, .codebuddy/skills, and
CODEBUDDY.md as documented at
https://www.codebuddy.ai/docs/cli/codebuddy-dir and
https://www.codebuddy.ai/docs/cli/skills.
Split the "claude", "codebuddy" switch cases in:
- daemon/local_skills.go (user-level local skill discovery/import)
- daemon/execenv/context.go (per-task skill materialization)
- daemon/execenv/runtime_config.go (runtime brief target file)
Added regression tests locking in the new paths and updated the
install-agent-runtime / providers docs (all 4 locales) that had
documented the old .claude/skills behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(daemon): cover CodeBuddy sidecar hygiene and lifecycle
Address PR #5224 review feedback: exclude CODEBUDDY.md/.codebuddy from
repo-cache worktrees, extend sidecar lifecycle matrices to codebuddy, and
make the local-skills CodeBuddy test exercise a true same-key collision.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Eve <eve@multica-ai.local>
Add xAI Grok Build (`grok`) as a first-class Multica runtime over ACP
(`grok --no-auto-update agent --always-approve stdio`), reusing
hermesClient like traecli and kimi. Includes daemon discovery,
protocol_family migration 174, model discovery, MCP passthrough,
thinking effort, frontend branding, and product docs.
Follows xAI's documented headless ACP flow: after `initialize`, read the
advertised `authMethods` and send `authenticate` (preferring xai.api_key
when XAI_API_KEY is set, else the cached login token) before any session
operation — a real, logged-in CLI rejects session/new and session/load
without it. Model discovery performs the same handshake so it returns the
live catalog instead of the static fallback.
`--no-auto-update` is passed as a global flag (and kept daemon-owned in
the blocked custom-arg set) so a background update check can't stall an
unattended ACP task. Thinking effort uses the current `--effort` flag,
and the minimum grok version is 0.2.89 (ACP + authenticate + session/load
+ session/set_model + MCP + --effort).
Closes#2895
A quick Stop before the agent's first token no longer races a late reply. Started-but-empty cancellations defer the empty/non-empty judgment until the daemon acks its transcript flush (or a grace-period sweeper fires), then settle to a single outcome. Empty outcomes persist a durable, creator-authorized draft restore (fetched/consumed via a dedicated endpoint, reconnect-safe and at-most-once) instead of broadcasting the prompt over the workspace bus.
Closes#5219
* fix(daemon): isolate Codex sessions per task to unblock initialize (MUL-4424)
Codex 0.143+ backfills a per-home session-state DB by enumerating every
rollout visible under sessions/ during `initialize`. The per-task
CODEX_HOME symlinked the shared ~/.codex/sessions in, so a machine with a
large accumulated history (one reporter: ~2000 rollouts / ~22 GiB) stalled
`initialize` for tens of seconds — the app-server started but the task
produced no output before it was cancelled (github #5273).
Give each task its own local sessions/ directory instead:
- Fresh task: create an empty local sessions/ so backfill is trivial.
- Reused task with a real sessions/ dir: it is authoritative — leave it.
- Reused task still holding a legacy symlink (older build): migrate in
place. Replace the symlink with a real dir; when resuming, symlink only
the single rollout being resumed (never copy — a rollout can be GiB and
this is on initialize's critical path); and drop the stale, rebuildable
session-state DB (state_*.sqlite*, session_index.jsonl) so Codex
re-indexes the task-local sessions. Unrelated per-task DBs (goals_*,
logs_*, memories_*) are left intact.
Also point the token-usage fallback scan at the backend's per-task
CODEX_HOME instead of the daemon-global home, so usage isn't lost now that
sessions are isolated there.
Complements the #5319 handshake watchdog (which turns a silent stall into a
loud, phased timeout); this removes the underlying cause.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): address Codex session isolation review (MUL-4424)
Resolves the three blockers from Elon's review of #5360:
1. local_directory context loss. local_directory tasks get a fresh
codex-home per task ID (the daemon never reuses their workdir), so
task-local isolation stranded every follow-up run with an empty
sessions dir and silently restarted the conversation. Their only
stable, GC-safe cross-task store is the user's own ~/.codex/sessions
(a persistent store under WorkspacesRoot would be orphan-GC'd), so keep
the shared-sessions symlink for them (IsLocalDirectory). Managed tasks
stay isolated.
2. Migration resume robustness.
- Rollout lookup now covers the flat layout and background-compressed
.jsonl.zst rollouts, not just nested YYYY/MM/DD *.jsonl — both are
legitimate Codex 0.144 history that were previously judged "not
found", silently dropping resume.
- Exposure hard-links first, then symlinks, never copies — hard links
need no privilege and work on Windows within a volume, so the
zero-copy path is exercised identically on CI.
- The daemon now verifies the rollout is actually present in the task
CODEX_HOME (execenv.CodexResumeRolloutPresent) before the brief is
generated; if absent it clears the resume from both the backend and
the brief instead of telling the agent it is continuing a lost thread.
3. session_index.jsonl is no longer deleted during migration — Codex uses
it as the authoritative thread-id -> name store (not rebuildable from
rollouts). Only the rebuildable state_*.sqlite* is reset.
Tests: 2-round local_directory resume across task IDs; compressed/flat
lookup; hard-link zero-copy (os.SameFile); session_index preserved;
CodexResumeRolloutPresent + the daemon gate helper (present keeps /
absent drops / non-codex + empty no-op).
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): scope Codex sessions to a per-issue store; disclose lost resumes (MUL-4424)
Addresses the three blockers from Elon's second review of #5360.
1. local_directory still enumerated the whole machine history. The prior
fix re-linked the entire ~/.codex/sessions into every fresh local_directory
codex-home, so Codex still backfilled from thousands of unrelated rollouts on
`initialize` (measured ~8.3s with 3450 rollouts; the reporter's 22 GiB could
exceed the 30s watchdog). Point sessions/ at a persistent, per-(agent, issue)
store under the shared Codex home (multica-sessions/<agent>/<issue>) that holds
only this issue's rollouts. It is keyed stably across task IDs and lives
outside the task-scoped envRoot the GC reclaims, so follow-up runs resume it
while `initialize` only ever sees this issue's history.
2. Windows cross-volume resume was lost. Exposing a single rollout by hard-link
(same-volume only) then file symlink (needs Windows privilege) can't cross a
volume boundary. The store now lives on the shared Codex volume, so the resume
rollout is hard-linked there zero-copy, and sessions/ is exposed to the task
home via a directory link — a symlink on Unix, a junction on Windows — which
crosses volumes without privilege and never copies a (possibly GiB) rollout on
initialize's critical path. There is no remaining per-file cross-volume link.
3. An unavailable resume was a silent downgrade. Both resume gates
(gateResumeToReusedWorkdir, gateCodexResumeToRolloutPresence) now set
PriorSessionResumeUnavailable, and the runtime brief renders a Session
Continuity Notice telling the agent to disclose to the user, up front in its
reply, that the previous conversation context could not be restored and this
run starts fresh — turning a silent restart into a user-visible one. The task
is not failed: it can still do useful work without the prior context.
Managed fresh / reused-real-dir tasks keep their task-local, GC-collected
sessions dir unchanged; only the legacy-symlink migration with a resume routes
through the store (cross-volume-safe), and a home already linked to the store is
treated as authoritative on reuse.
Tests: local_directory per-issue store (only this issue's history, no whole-
machine leak); no-key fallback to an empty dir; two-round resume across task IDs
through the store; legacy migration routed through the store with a zero-copy
hard link; reused store link stays authoritative; both gates set the
resume-unavailable flag; brief renders the continuity notice only when a resume
was lost. execenv + daemon + pkg/agent packages, go vet, and gofmt all pass.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): disclose live resume-RPC loss; bound Codex session store lifecycle (MUL-4424)
Addresses the two blockers from Elon's third review of #5360.
1. A real thread/resume failure was still a silent new session. The brief's
Session Continuity Notice only covers losses the daemon detects before launch
(workdir not reused, rollout absent). But when the rollout is present yet
Codex rejects the live thread/resume (corrupt/incompatible rollout, server-side
thread GC, schema drift), startOrResumeThread falls back to thread/start and
the run succeeds on a fresh thread with no user-facing signal. Carry the
original resume intent into the backend as ExecOptions.ResumeExpected (set from
the post-gate PriorSessionID, so a pre-flight drop still routes through the
brief and never double-notifies), and when a resume was expected but the
backend landed on a fresh thread, prepend the same continuity notice to the
first turn/start input. This also covers the daemon's transport-error
fresh-session retry, which clears ResumeSessionID but not ResumeExpected.
2. The persistent per-issue store had no data lifecycle. multica-sessions stores
live outside the task-scoped envRoot the GC reclaims (so resume survives across
task IDs), which meant a done/abandoned issue's prompts and full rollouts (one
reporter: a single 1.5 GiB rollout) accumulated forever and were never freed on
issue/agent/workspace deletion. Add PruneCodexSessionStores: the daemon GC loop
reclaims any store untouched for GCCodexSessionTTL (default 14 days, configurable
via MULTICA_GC_CODEX_SESSION_TTL, 0 disables). A store's newest rollout mtime is
its last activity, so an active or recently-resumed task keeps its store fresh
and is never reclaimed, while a deleted issue's store ages out — an eventual
reclamation guarantee without needing deletion events.
Tests: codexTurnInput discloses on resume fallback and stays silent on success /
fresh start (paired with the existing live-RPC fallback test); store pruning
reclaims aged stores, keeps recent ones, isolates issues, cleans empty agent
dirs, and is disable-able. execenv / daemon / pkg/agent, go vet, gofmt all pass.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): protect a reopened Codex session store from GC mid-mount (MUL-4424)
Addresses Elon's fourth-review blocker: reopening an issue idle past
GCCodexSessionTTL could lose context, because mounting its per-issue session
store (MkdirAll + rollout lookup + task-home link) never refreshed the store's
mtime, so a GC cycle firing before the resumed turn wrote its first rollout saw
a >TTL-old store and reclaimed it — a stat->remove race with no in-use guard.
Two complementary defenses:
- Activity refresh: linkCodexSessionsToStore now os.Chtimes the store to now
after linking, so codexStoreStat (which reads the newest mtime as last
activity) sees a just-used store. This fixes the sequential repro — a mount
immediately followed by a prune keeps the store.
- In-process active-store guard: the daemon marks the per-issue store in-use
(execenv.CodexSessionStorePath) from before Prepare/Reuse mounts it until the
task ends, and PruneCodexSessionStores now takes an isActive predicate and
skips any store a live task holds. Because prepare and prune run in the same
process, this closes the remaining concurrent stat->remove window the mtime
refresh alone cannot. Reference-counted, mirroring the env-root guard.
Tests: a reopened >TTL store survives a GC cycle after remount and stays
resumable; an idle-on-disk store marked active is skipped, then reclaimed once
inactive; the existing idle-reclaim / isolation / disable / empty-agent-dir
cases still pass. execenv + daemon, go vet, gofmt all pass.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): make Codex store delete atomic with mark-active (MUL-4424)
Addresses Elon's fifth-review blocker: the active-store guard's check and delete
were not one atomic step. PruneCodexSessionStores called isActive (which locked,
read, and unlocked) and only then RemoveAll'd, leaving a window where a task
could markActiveCodexStore between the check and the removal and still lose its
store — the exact mark-then-delete interleaving Elon reproduced.
Replace the point-in-time isActive predicate with a reserve-for-deletion
protocol that shares one lock with mark-active:
- reserveCodexStoreForDeletion(store) atomically refuses when a live task holds
the store (or another delete already reserved it) and otherwise marks it
reserved, all under one activeCodexStoresMu acquisition. PruneCodexSessionStores
reserves before RemoveAll and commits after, so confirm-inactive and remove are
effectively atomic against a concurrent mark.
- markActiveCodexStore now waits (on a sync.Cond) while a store is reserved, so a
task never mounts a store mid-removal; committing the removal wakes it and the
store is recreated fresh by Prepare (with the continuity notice).
So mark-before-reserve keeps the store (reserve refused); reserve-before-mark
removes it and blocks the late mark until the removal commits. The genuinely
idle case still reclaims.
Tests (daemon, run under -race): mark-then-reserve is refused; reserve blocks a
concurrent mark until commit then the store reads active; a second reserve is
refused mid-flight. The execenv prune tests move to the reserve seam; the
activity-refresh / reopen-then-prune / isolation / disable cases still pass.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): namespace Codex session stores per profile for cross-daemon safety (MUL-4424)
Addresses Elon's sixth-review blocker: the in-process reservation guard cannot
span processes, but Multica supports multiple profile daemons on one machine
(e.g. production + staging) that share the same ~/.codex. Each daemon's GC
scanned the whole multica-sessions root, so a staging daemon could reclaim a
store a production task was actively resuming — its reservation lived only in the
other process's memory.
Isolate by profile instead of trying to lock across processes:
- Store path is now <shared>/multica-sessions/<namespace>/<agent>/<issue>, where
namespace is the daemon's profile (empty -> "default"). PrepareParams/ReuseParams
carry Profile; codexSessionStoreKey and CodexSessionStorePath fold it in.
- PruneCodexSessionStores takes the profile and scans ONLY that namespace, so a
daemon never even sees another profile's stores, let alone deletes them. The
per-profile trees are disjoint, so the in-process guard is sufficient within a
namespace (profiles get separate daemon state, so no two daemons share one).
Test: a "staging"-owned idle store is untouched by a default-profile prune and
reclaimed only by staging's own prune. Existing prune/guard/reopen tests move
under the namespace. execenv + daemon under -race, go vet, gofmt all pass.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): make the Codex store profile→namespace map injective (MUL-4424)
Addresses Elon's seventh-review blocker: the per-profile namespace was derived by
dropping unsafe characters, which is not injective. The CLI treats the empty
(default) profile and a profile literally named "default" as separate daemons,
yet both mapped to namespace "default"; likewise "staging.prod" and "stagingprod"
both mapped to "stagingprod". Two distinct daemons then shared one store tree, so
one could again reclaim the other's live session — the cross-process blocker
reopened for those profile names.
Make codexSessionStoreNamespace injective: the empty profile gets a reserved
bare literal "default", and every named profile is hex-encoded (bijective,
filesystem-safe) under a "p_" prefix a bare literal can never collide with. So
"" -> "default" while "default" -> "p_64656661756c74", and "staging.prod" /
"stagingprod" get distinct hex segments. sanitizeCodexPathSegment stays for the
UUID agent/issue segments (injective for real UUIDs); only the user-controlled
profile needed the encoding.
Tests: codexSessionStoreNamespace is distinct for "" vs "default", punctuation
variants, case variants, and an encoded-looking name; and end-to-end, pruning one
profile never reclaims the other's store for the "" vs "default" and
"staging.prod" vs "stagingprod" pairs. execenv + daemon under -race, go vet,
gofmt all pass.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): fixed-length Codex store namespace so long profiles fit (MUL-4424)
Addresses Elon's eighth-review blocker: hex-encoding the full profile doubled the
namespace segment length. A profile can be as long as a filesystem segment allows
(~255 bytes) and the CLI persists it as its own config dir, but the store
namespace "p_" + hex(profile) reached 2 + 127*2 = 256 bytes at 127 chars,
overflowing the 255-byte single-segment limit — the profile's own dir created
fine, then the session store failed with "file name too long".
Derive the namespace from a fixed-length hash instead: a named profile is now
"p_" + hex(sha256(profile)) — a constant 64 hex chars (66 with the prefix),
filesystem-safe and collision-resistant. The empty (default) profile keeps its
reserved bare literal "default", which the "p_"-prefixed 66-char segment can
never equal. Still injective across the CLI's distinct-daemon cases; just no
longer length-expanding.
Test: the namespace stays <=255 bytes and creatable for profiles up to the
255-byte segment limit (127- and 255-char cases that overflowed under hex); the
prior injectivity and cross-profile prune-isolation tests still hold. execenv +
daemon under -race, go vet, gofmt all pass.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: J <j@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
* fix(execenv): overlay per-task HERMES_HOME so Hermes discovers bound skills
Hermes has no workspace-relative skill discovery — it scans <HERMES_HOME>/skills
first, then skills.external_dirs from config.yaml (verified against the bundled
agent/skill_utils.py). The daemon wrote assigned skills to the generic
.agent_context/skills/ fallback, which Hermes never reads, so they silently never
took effect (#5242).
When (and only when) an agent has skills bound, redirect HERMES_HOME to a minimal
per-task compatibility overlay; a skill-less Hermes task keeps its real home and
original behavior:
- mirror every top-level entry of the shared home via symlink except the
overlay-owned ones (denylist), reconciling entries deleted from the shared home;
- derive a task-local config.yaml whose skills.external_dirs references the shared
skills dir plus the user's existing external_dirs, expanded against the
sanitized effective child env (unknown vars preserved, blocklisted keys resolved
to the process value) and normalized to absolute paths;
- write only the bound skills into the task-local skills/ dir (home skills scanned
first, so they win); global skills are referenced, not copied;
- keep memories/ overlay-owned (fresh per-task dir) AND disable the external
memory.provider, so neither on-disk memory nor a shared backend crosses tasks;
- keep active_profile/profiles out of the overlay so Hermes can't follow a sticky
profile and redirect past it at startup.
Profile handling mirrors hermes_cli.profiles: the daemon reads -p/--profile with
agent.HermesProfileFromArgs and seeds the overlay from that profile's home via
ResolveHermesSourceHome (default/invalid -> base, valid name -> <base>/profiles/
<name>, validated; a missing named profile fails closed). The profile flags are
stripped from the acp argv ONLY when the overlay is active (hermesLaunchArgs), so
a skill-less task's profile passes through unchanged. HERMES_HOME is no longer
custom_env-blocklisted: no skills -> user value passes through; skills -> overlay
overrides after layering. Fail closed — Prepare errors, Reuse returns nil.
Task home 0700, derived config 0600 via atomic replace. Platform-native default
home (%LOCALAPPDATA%\hermes, incl. the LOCALAPPDATA-missing fallback, on Windows).
Tests span execenv/daemon/agent: no-skill no-op, child-env layering + env
sanitization, profile parse/unquote + conditional strip + final args/env per
scenario, custom/profile/default/invalid/missing/Windows source home, sticky-
profile not mirrored, memory dir isolation + external provider disable, mirror
reconciliation, external_dirs rebasing + sanitized/unknown-var expansion,
local-precedence slug, perms, fail-closed, resume teardown. Docs (en + ja/ko/zh).
Fixes#5242
* fix(hermes): make profile selection one resolver contract matching Hermes
Round 5 review: the profile chain approximated Hermes' semantics in three
separate places (argv parsing, source-home selection, arg filtering), so it
diverged from native Hermes in several merge-blocking cases. Collapse it into
one authoritative resolution:
- agent.ParseHermesProfileArgs replaces HermesProfileFromArgs/
FilterHermesProfileArgs. It reproduces _apply_profile_override step 1/1b
(first occurrence, value-flag skipping, `--` and `mcp add --args` boundaries,
space-form profile-id guard) and returns the exact argv occurrence to consume;
StripHermesProfileArgs removes only that occurrence.
- execenv.ResolveHermesProfile replaces ResolveHermesSourceHome. It derives the
Hermes root exactly like get_default_hermes_root (an already-profile-scoped
HERMES_HOME roots at its grandparent), selects an explicit profile first,
otherwise trusts a profile-scoped home (step 1.5) and only then the sticky
<root>/active_profile (step 2), and validates via normalize/validate_profile_name
(reserved hermes/test/tmp/root/sudo and empty inline `--profile=` are hard
errors). Profiles always resolve under the root, so `-p default` re-roots and
`-p <sibling>` is a sibling, never nested.
- The daemon runs one parse + resolve, fails the task closed on a reserved/
invalid selection (matching Hermes' sys.exit(1)), and exports the selected
source home as the effective env's HERMES_HOME so ${HERMES_HOME} in a profile's
skills.external_dirs expands against the selected profile home (as native
Hermes does before loading config.yaml), not the root or the overlay.
Regressions added: root + sticky named profile selection; already-profile-scoped
home with no flag; that home with -p default and -p <sibling>; reserved and empty
inline profile values; and a selected profile whose external_dirs contains
${HERMES_HOME}.
* fix(hermes): overlay-owned derived .env + symlink-resolved root
Round 6 review, two remaining overlay-bypass paths:
1. A source `.env` could redirect HERMES_HOME after profile resolution. Hermes
runs `_apply_profile_override()` then `load_hermes_dotenv()`, which loads
`<HERMES_HOME>/.env` with override=True — so a mirrored source `.env` carrying
an out-of-band `HERMES_HOME=` overwrote the overlay's home, repointing skill
discovery and memory back at the source. `.env` is now overlay-owned and
DERIVED (writeDerivedHermesEnv): it preserves the source's credentials/settings
but strips any `HERMES_HOME` assignment and pins `HERMES_HOME` to the overlay
last (single-quoted, literal), written 0600 via atomic replace. It is written
even when the source has none, so Hermes' project-`.env` fallback (override=True
only when no user `.env` loaded) can't relocate the home either.
2. Root derivation was lexical-only, diverging from `get_default_hermes_root`,
which compares `env_path.resolve()` with `native_home.resolve()`. A HERMES_HOME
symlinked into `<native>/profiles/<x>` was treated as its own root, so
`-p default`/`-p <sibling>` resolved wrong. `hermesRootFromHomeFor` now resolves
symlinks (Path.resolve(strict=False)-style best effort) for the containment
decision while keeping the returned root unresolved, matching Hermes.
Regressions: source `.env` with HERMES_HOME replayed through the override=True
dotenv order (bound skill + task memory stay on the overlay; creds preserved);
minimal overlay `.env` created when the source has none; and a symlinked profile
home resolving `-p default`/`-p <sibling>` to the native root.
The daemon pins each agent CLI's symlink-resolved absolute path at startup to
block PATH-redirect of a task launch. A version manager (Homebrew Cask, nvm/fnm)
upgrading in place deletes the pinned versioned directory and repoints the stable
name, leaving the daemon on a path that no longer exists — every codex task, model
list, and version detection then hard-fails with "executable not found" until the
daemon restarts.
resolveAgentEntry now self-heals a vanished pin by re-resolving the recorded
command once, version-detecting and min-version-gating the candidate before
adopting it, and publishing {path, version} atomically so callers key policy off
the binary that actually runs. Coalesced with singleflight; a live heal wins over
a reappearing stale path; custom runtimes and custom-only hosts are untouched.
Applied at task launch, model listing, and registration.
MUL-4486
* feat(daemon-claim): machine-level batch task claim endpoint (MUL-4257)
Collapse the per-runtime /tasks/claim poll fan-out into a single machine-level
batch claim to cut /api/daemon claim request volume.
Server:
- agent.sql: = ANY(runtime_ids) batch variants of the claim queries
(ListQueuedClaimCandidatesByRuntimes, PromoteDueDeferredTasksForRuntimes,
ReclaimStaleDispatchedTasksForRuntimes); runtime.sql: GetAgentRuntimes(= ANY)
so a whole machine's runtimes are resolved/promoted/reclaimed/listed in a
constant number of queries instead of N.
- service.ClaimTasksForRuntimes: claim up to max_tasks across a runtime set,
preserving per-(issue,agent) serialization, the concurrency cap, the
empty-claim cache short-circuit, and every dispatch side effect. Batch
promote replays the per-row side effects (task:queued + empty-cache Bump).
- handler.ClaimTasksByRuntime (canonical POST /api/daemon/tasks/claim, with a
transitional /claim alias): validates daemon_id (required; must match the
mdt_ token) and rejects runtimes bound to a different daemon (group-ownership
check mirroring the WS path); resolves+authorizes each runtime_id; claims;
and finalizes each task through the SAME FinalizeTaskClaim as the per-runtime
endpoint (atomic token + delivered_comment_ids receipt), requeueing the exact
claim and omitting it on failure. buildClaimedTaskResponse is extracted from
the per-runtime handler and returns the delivered-comment ids plus a
structured *claimBuildFailure so both paths share identical payload building
and failure semantics (workspace-isolation, chat-input load/empty).
- max_tasks: negative -> 400, zero -> empty (never coerce to 1), positive
capped at 32. runtime_ids parsed with non-panicking util.ParseUUID.
Daemon:
- Client.ClaimTasks posts daemon_id + runtime set + free-slot count to the
canonical path under a short request-scoped timeout, bounding the
head-of-line coupling the per-runtime pollers avoid (MUL-1744).
Tests: service batch drain / max_tasks cap / deferred-promote receipt /
finalize-failure rollback+requeue; handler routing + token, cross-workspace
skip, cross-daemon skip, daemon_id required, owner-missing cancel,
max_tasks=0/negative, invalid-uuid skip, comment delivery receipt, stale-reclaim
replacement receipt; client posts/parses (daemon_id + canonical path).
Follow-up: cut the daemon pollLoop over to a single batched poller (flips the
MUL-1744 isolation contract; needs its concurrency tests redesigned).
Co-authored-by: multica-agent <github@multica.ai>
* feat(daemon-ws): generic WS request/response transport for daemon RPC (MUL-4257)
Add a generic daemon->server request/response layer over the existing WS
control connection, the transport for WS-first claim (HTTP fallback):
- protocol: daemon:rpc_request / daemon:rpc_response envelopes with a
correlation request_id + method + body, and an rpc-v1 capability gate.
- daemonws.Hub: SetRPCHandler + goroutine-dispatched handleRPCFrame (bounded
by a per-connection in-flight cap) that echoes the request_id; missing
handler / saturation return non-2xx so the daemon falls back to HTTP.
Read limit raised to 64KB for rpc requests carrying a runtime set.
- hub tests: round-trip, handler-error->non-2xx, no-handler->503.
Co-authored-by: multica-agent <github@multica.ai>
* feat(daemon-ws): WS-first task claim over the generic RPC transport (MUL-4257)
Bind claim to the WS request/response layer, with HTTP fallback:
- server: handler.DaemonRPCHandler adapts a daemon:rpc_request (method
tasks.claim) to the existing HTTP ClaimTasksByRuntime via a synthetic
in-process request carrying the WS connection's identity (daemon_id +
workspace + capabilities), so all auth / payload-building / finalization is
reused unchanged. Wired via daemonHub.SetRPCHandler. ClientIdentity now
captures X-Client-Capabilities so capability gating matches the HTTP path.
- daemon: wsRPCClient correlates responses by request_id over the shared WS
connection; attached to the live connection's write channel (guarded so a
Call racing teardown never sends on a closed channel) and detached on
disconnect. rpc_response frames are routed in the read loop.
Daemon.ClaimTasksWSFirst issues tasks.claim over WS and falls back to the
HTTP claim endpoint on any transport failure (no conn / buffer full /
timeout) — wired into the poller at the poller cutover.
- tests: handler tasks.claim RPC end-to-end (claims + dispatches) + unknown
method 404; daemon wsRPCClient round-trip / timeout / unavailable /
server-error / detach-fails-pending (all under -race).
Co-authored-by: multica-agent <github@multica.ai>
* feat(daemon): cut claim poller over to machine-level ClaimTasksWSFirst (MUL-4257)
Replace the per-runtime HTTP poll loop with a single batch poller: each cycle
acquires all free execution slots (slot-before-claim) and issues ONE
ClaimTasksWSFirst across every runtime the daemon hosts (WS-first, HTTP
fallback), dispatching each returned task to its runtime. Wakeups (targeted /
catch-up / runtime-set change) collapse to one nudge. Removes runRuntimePoller
+ runtimePollOffset. The WS handshake now advertises the same capabilities as
HTTP (+ rpc-v1) so WS-built claim payloads keep skill-ref / coalesced-comment
gating.
Trades per-runtime isolation (MUL-1744) for one request, bounded by the short
per-request WS timeout / client timeout. Tests: batch poller claims across
runtimes + skips-at-capacity + pollLoop shutdown drain (replacing the
per-runtime poller tests); heartbeat isolation + runtime-set watcher kept.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon-ws): WS RPC disconnect-race panic + batch stale-comment-plan repair (MUL-4257)
Two PR #5193 review blockers:
1) WS RPC send-on-closed-channel race, both ends:
- server: give each connection a cancelable ctx (cancelled on readPump
teardown) and run the RPC handler under it, so a slow claim stops on
disconnect; guard c.send with sendMu/sendClosed (trySend) so a late RPC
response goroutine never writes to the closed channel. Heartbeat ack routed
through the same guard.
- daemon: wsRPCClient.deliver now sends under the mutex, serialized with
attach(nil)'s close+delete, so a delivered response can't hit a channel
the detach path just closed.
- regressions (-race): daemon deliver-vs-detach; server
disconnect-during-handler-response.
2) batch claim now runs the stale-comment-plan repair: extracted the
per-runtime handler's repair (trigger deleted, only coalesced survive ->
cancel + replay survivors) into shared repairStaleCommentPlanIfNeeded, called
by both claim paths. Prevents the batch path (now the default poller) from
finalizing+dispatching a task with no comment input and silently dropping the
surviving user comment. Regression: batch omits the stale task, cancels it,
and rebuilds the survivor into a new trigger plan.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon-ws): server-side RPC deadline + legacy claim fallback (MUL-4257)
Two review blockers:
1) WS RPC timeout/fallback (GPT-Boy): the daemon's WS wait didn't cancel
server-side claim, so a slow WS claim could commit after the daemon fell
back to HTTP, leaking dispatched tasks and breaking the free-slot bound.
Fix: RPC envelope carries TimeoutMs; the server bounds the handler ctx by it
(so ClaimTasksByRuntime's tx is cancelled/rolled back at the deadline), and
the daemon waits budget + grace so a claim that committed before the deadline
still reports back. A committed-then-unreported claim degrades to the same
stale-reclaim safety net as HTTP, never a double effective claim. Regression:
server-side TimeoutMs cancels the handler.
2) Backward compat (Terra-Boy): a new daemon against a server without the batch
route (/api/daemon/tasks/claim 404) couldn't claim. Fix: ClaimTasksWSFirst
falls back to the legacy per-runtime ClaimTask loop on a batch 404 and caches
'batch unsupported' (reset on WS reconnect to re-probe after a server
upgrade). Regression: server exposing only the legacy route.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon-ws): no double-claim on WS teardown/detach (MUL-4257)
Sol-Boy review blocker: on reconnect, teardown failed the pending RPC (→ HTTP
fallback) but then flushed the queued tasks.claim frame to the still-alive
socket, so the server committed the WS claim on top of the HTTP one — double
claim, WS batch orphaned to stale reclaim, breaking the free-slot bound.
- Teardown now closes the connection FIRST, so runWSWriter discards the queued
RPC frame (write error path) instead of delivering it.
- A detach while a claim's frame is already in flight now returns a distinct
errWSRPCUncertain; ClaimTasksWSFirst does NOT HTTP-fall-back on uncertain (the
WS claim may have committed) — it skips the cycle and lets reclaim / the next
poll recover. Genuine 'not sent' / timeout still fall back (safe: the
server-side deadline guarantees no uncommitted claim by budget+grace).
- Regression: detach during an in-flight WS claim asserts zero HTTP claims
(at most one path claims); plus the existing detach/deliver-race and
server-timeout tests.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon-ws): cancelable RPC frames close the backpressure double-claim (MUL-4257)
Sol-Boy review blocker: the client's response budget starts at enqueue, but
the socket write is async (10s write deadline). A backpressured writer could
hold a tasks.claim in the local queue past the client timeout — the daemon
HTTP-fell-back, then the writer woke and delivered the stale WS frame, so the
server committed it too: same free slots claimed twice. No detach occurs, so
the prior errWSRPCUncertain fix did not cover it.
- WS frames are now cancelable (wsOutbound{sent,canceled} under a mutex). The
writer calls beginWrite() before WriteMessage and skips cancelled frames.
- On give-up (timeout / detach / ctx), Call cancels the queued frame: if it was
still pending the cancel wins and the frame is guaranteed never delivered
(errWSRPCUnavailable → safe HTTP fallback); if the writer already began
sending it the cancel loses and the outcome is errWSRPCUncertain (no
fallback). The decision is atomic, so at most one transport claims.
Tests: wsOutbound cancel-before-write vs write-before-cancel; Call timeout
cancels an unsent frame (writer then drops it) vs uncertain when already sent;
plus the updated detach and existing timeout/race tests.
Co-authored-by: multica-agent <github@multica.ai>
* fix(batch-claim): return partial success instead of dropping committed claims (MUL-4257)
Sol-Boy review blocker: ClaimTasksForRuntimes reclaims (step 2) and claims per
agent (step 6) in independent transactions, but a step-4 candidate-SELECT error
or a mid-loop ClaimTask error did 'return nil, err' — discarding tasks already
committed as dispatched. The handler 500s; the daemon sees a definite (non-
uncertain) 500 and HTTP-falls-back, claiming a SECOND batch into the same free
slots while the first batch waits for stale reclaim — the double-claim this PR
removes.
- Both error paths now prefer partial success: if any task has already
committed (claimed non-empty), return it (nil error) so the handler finalizes
and returns 200; the errored candidates stay queued for the next poll. The
remaining error is logged. Only a genuinely empty result still returns the
error (safe: no committed claim to lose, HTTP fallback just re-fails).
Regression (internal/service, DB-backed, fault-injected):
- PartialSuccessOnSecondAgentClaimFailure: fail the 2nd ClaimTask's Begin →
the first agent's committed task is returned, not dropped.
- PartialSuccessOnCandidateQueryFailureAfterReclaim: a stale dispatched task is
reclaimed, then the candidate SELECT fails → the reclaimed task is returned.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Eve <eve@multica-ai.local>
Co-authored-by: multica-agent <github@multica.ai>
* fix(codex): bound app-server startup RPCs
Co-authored-by: multica-agent <github@multica.ai>
* test(codex): de-flake bounded-handshake test
The single 500ms handshake bound was shared by the successful preamble
RPCs, so a slow fork/exec of the /bin/sh fake app-server could make
initialize spuriously time out under parallel load. Raise the test bound
to 3s (still below the 5s semantic timeout and 10s harness ceiling) and
loosen the elapsed assertion to match.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Eve <eve@multica-ai.local>
Co-authored-by: multica-agent <github@multica.ai>
Co-authored-by: J <j@multica.ai>
