Adds self-hosted Git provider support (Forgejo, Gitea, GitLab) alongside GitHub:
per-workspace token connection, a provider-dispatched webhook, PR/MR and CI
mirroring, and the shared issue auto-link / auto-close machinery. Off until
MULTICA_VCS_SECRET_KEY is set, so existing deployments are unaffected.
Co-authored-by: Bohan <bohan@devv.ai>
* feat(chat): add project context
Co-authored-by: multica-agent <github@multica.ai>
* fix(chat): resolve MUL-5150 review blockers
- Renumber project-context migrations to unique prefixes after current main:
206_chat_session_project -> 212 (column), 207_chat_session_project_index ->
213 (concurrent index). 206/207 collided with 206_agent_disabled_runtime_skills
and main's 207-211 client_usage_daily set.
- Add the 4 missing chat input.project_context keys to ja/ko locales so the
locale parity test passes (en/zh-Hans already had them).
- Lock the project-context control while a send is in flight (isSubmitting),
not just while the agent is running. A brand-new chat creates its session
lazily during send bound to the project at click time; switching project
mid-send would create the session against the stale project and clear the
editor as if the send landed on the new selection. Add a regression test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: multica-agent <github@multica.ai>
* fix(chat): complete project context handling
* fix(chat): pin fresh chat to open session's agent on project switch
Switching an existing session to a different project opens a fresh chat but
only cleared the active session, dropping selection back to the stored
`selectedAgentId`. When that preference was stale (open session belongs to
agent B while the persisted pick is still agent A), the lazily-created session
and its first send bound to the wrong agent (agent A).
Extract the project-switch decision into a shared `planProjectContextChange`
pure helper in use-chat-controller.ts and route both chat surfaces (the chat
tab controller and the floating ChatWindow) through it, so the fresh chat is
pinned to the open session's agent and the rule cannot drift between the two
copies. Add a dual-entry regression test (pure-fn guard + controller
integration) covering the stale selectedAgentId case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: multica-agent <github@multica.ai>
* chore(ci): re-trigger required checks on latest head
The prior push updated the branch ref but GitHub did not emit a pull_request
synchronize for it (PR head-sync lag), so CI/Mobile Verify never ran on the
commit carrying the stale-agent project-switch fix. Empty commit to force a
fresh synchronize on a head that includes it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: multica-agent <github@multica.ai>
* fix(chat): renumber project migrations to 213/214 after main added 212
Current main added 212_agent_service_tier; the PR's 212/213 chat migrations
collided with it on the merge ref, failing TestMigrationNumericPrefixesStay
UniqueAfterLegacySet. Merge current main and move the chat column migration to
213 and the concurrent index migration to 214 (column before index preserved).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: multica-agent <github@multica.ai>
* fix(chat): lock ProjectPicker clear control during send (keyboard path)
The send-pending lock only put pointer-events-none on the wrapper, which
blocks the mouse but leaves ProjectPicker's inline clear button in the tab
order — a keyboard user could Tab to "Remove from project" and press Enter
mid-send, detaching the project after the lazily-created session already went
out with the old one (reopens the mid-send retarget path via keyboard).
Add an explicit `disabled` capability to the shared ProjectPicker that locks
the trigger, the menu (forced closed), and the inline clear button (disabled +
out of the tab order). Defaults to false, so issue/create/autopilot callers
keep their hover/keyboard clear. ChatInput passes disabled while the project
selection is locked.
Tests: real-ProjectPicker regression (keyboard activation of the clear control
is inert when disabled; still works when enabled) + ChatInput wiring assertion
that the picker is disabled mid-send.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Lambda <lambda@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
Co-authored-by: Walt <walt@multica.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Naiyuan Qing <145280634+NevilleQingNY@users.noreply.github.com>
Co-authored-by: NevilleQingNY <nevilleqing@gmail.com>
* fix(agent): attribute Grok usage from the turn's own model id
A resumed Grok session with no configured model recorded its entire spend
under the model id "unknown", which matches no pricing row — so the task
reported $0 cost instead of its real spend.
grok.go only learned the model from the session handshake, and ACP's
`session/load` carries no model id (only `session/new` does). When neither
the agent nor MULTICA_GROK_MODEL pins a model, `daemon.go` legitimately
passes an empty model, leaving nothing to attribute the usage to.
Every Grok turn stamps `result._meta.modelId` with what it actually billed
against. Parse it in the shared ACP result parser and use it as the fallback
in grok.go. Other ACP backends are untouched — they keep whatever the
handshake gave them.
Co-authored-by: J <agent@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
* fix(metrics): price the Grok catalog in server-side cost metrics
server/internal/metrics/pricing.go carried no Grok rows at all, so
RecordLLMUsage took the unpriced branch for every Grok turn: llm_cost_usd
reported zero Grok spend while the tokens accumulated in
llm_unpriced_tokens. Internal cost monitoring simply could not see Grok.
Add the six SKUs xAI publishes rates for, mirroring the frontend table in
packages/views/runtimes/utils.ts. Aliases are anchored exact matches like
the gpt-5.6 rows, so `grok-composer-*` (in the catalog, absent from the
price sheet) stays unmapped instead of inheriting a guessed rate.
Short-context tier on purpose: xAI bills a request at 2x once its prompt
reaches 200K tokens, but a usage record aggregates every model call in a
turn and cannot say which tier an individual request hit.
A regression test re-derives the cost of a real grok 0.2.106 turn from the
table and checks it against the costUsdTicks xAI returned for that turn.
Co-authored-by: J <agent@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
* docs(changelog): scope the Grok cost claim to what was actually fixed
The v0.4.9 entry promised "accurate cost" in all four languages, but the
fix corrected catalog pricing and cached-input double-counting — it did not
implement xAI's 2x long-context tier, so a turn whose requests reach 200K
prompt tokens still under-reports by up to 50%. Say what was fixed instead.
Also correct two stale claims in the pricing comment: the daemon tags usage
rows with the runtime provider `grok`, not `xai` (the bare `grok-*` keys are
what make them resolve), and record why thresholding the long-context tier
on an aggregated row would be worse than not pricing it at all.
Co-authored-by: J <agent@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
* feat(usage): carry the provider's own cost through to the usage record
Cost has always been derived client-side as tokens x a static rate, which
cannot express request-level pricing rules. xAI bills a Grok request at 2x
once its prompt reaches 200K tokens, and a task_usage row aggregates every
model call in a turn — so the stored token counts genuinely cannot say which
tier any individual request hit. Thresholding on the aggregate would be worse
than the status quo: it turns a bounded 50% under-estimate into an unbounded
over-estimate for turns made of many short requests.
Grok already reports what it charged, per turn, in `_meta.usage.costUsdTicks`.
Parse it, carry it through agent -> daemon -> API, and store it on task_usage
as a nullable BIGINT of 1e-10 USD ticks (integer, so sub-cent turns stay exact
end to end). NULL means the provider reported no cost — every pre-existing row
and every provider that doesn't return one. No backfill: there is no
authoritative figure to recover for those, and inventing one is the guess this
removes.
A single hourly bucket can mix rows that carry a cost with rows that don't, so
task_usage_hourly gains both halves: `cost_usd_ticks` sums the authoritative
side, and `uncosted_*_tokens` carry exactly the tokens that still need a
rate-table estimate. Consumers report authoritative + estimate(uncosted),
which degrades to today's behaviour when nothing in the bucket is
authoritative. The existing token columns keep covering every row, so token
displays are untouched. The new columns are additive with defaults, so the
unique key, the dirty-queue shape, and migration 102's triggers are unaffected.
Co-authored-by: J <agent@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
* feat(usage): prefer the provider's own cost over the rate table
With the authoritative figure now stored, both cost consumers use it: the
usage dashboard (estimateCost / estimateCostBreakdown) and the server-side
llm_cost_usd metric. Each reports `authoritative + estimate(uncosted tokens)`,
so a row or bucket that mixes priced and unpriced sources stays whole.
The static rate tables remain, but for Grok they are now a fallback — they
still price usage recorded by a daemon too old to report cost, and every
provider that reports none. Custom pricing overrides likewise apply only to
the estimated half: they are a user's guess at a rate, and the authoritative
half is not a guess. A model with no rate-table row but a provider-reported
cost now also drops out of the "unmapped models" banner, since asking the user
to supply a rate for it would invite overriding a real bill.
llm_cost_usd is labelled by token_type and the provider reports one number per
turn, so the charge is distributed across the buckets in the rate table's own
proportions. Only the total is authoritative; the split stays an estimate,
which is why this scales the existing buckets rather than inventing a label.
estimateCostBreakdown does the same, keeping the stacked chart summing to the
headline figure instead of silently under-drawing every Grok row.
Co-authored-by: J <agent@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
* docs(changelog): say Grok cost now follows xAI's actual charge
The earlier wording scoped the claim down to catalog pricing and cached input
because the long-context tier was still unhandled. It is handled now — the
cost comes from what xAI charged for the turn — so the entry can say so.
Co-authored-by: J <agent@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
* fix(usage): keep the provider's cost when the model has no rate row
Both cost consumers bailed out before reading the authoritative figure when
the rate table had no row for the model. A `grok-composer-*` turn — in the
Grok Build catalog, absent from xAI's price sheet — was therefore reported as
$0 spend even though xAI told us exactly what it charged.
Worse on the client: estimateCost returned the real cost while
estimateCostBreakdown returned zeros, so the headline and the stacked chart
disagreed on precisely the rows whose cost is exact — and the unmapped-models
banner was (correctly) hidden, so nothing explained the discrepancy.
Handle the charge before the rate lookup in both places. Without rates there
is nothing to split a total by, so it lands whole in the `input` bucket, the
same fallback distributeAuthoritativeCost already uses when it has no shape to
scale. Tokens with no rate keep going to llm_unpriced_tokens: "unpriced"
describes the rate table, not the money.
Co-authored-by: J <agent@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
* perf(usage): drop the historical rewrite from the cost-split migration
Migration 213 rewrote every existing task_usage_hourly row to seed the
uncosted counters. That is a full-table UPDATE inside a schema migration —
lock time, WAL and bloat all scaling with table size — for rows this issue
explicitly does not care about.
Deleting the UPDATE alone would have zeroed historical cost: with
`NOT NULL DEFAULT 0`, an untouched row asserts "nothing here needs
estimating", so every pre-split bucket would report $0 until the rollup
happened to touch it. Make the uncosted columns nullable with no default
instead. NULL means "never recomputed since the split existed", readers
COALESCE it to the row's own token total ("estimate all of it"), and the
pre-split behaviour is preserved exactly — with nothing to seed, so no
rewrite. A bare ADD COLUMN is metadata-only, so this is now fast DDL.
Rows heal into the split naturally as the rollup recomputes their buckets.
Verified on a fresh database: a legacy-shaped row reads back as its full
tokens to estimate, and a group mixing legacy and post-split buckets sums to
the authoritative cost plus both rows' estimable tokens.
Co-authored-by: J <agent@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: J <agent@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
* fix(runtime): forbid blocking on external CI in the brief (MUL-5223)
The external-work boundary added in #5803 did not stop agents from
waiting on GitHub Actions. Two holes: the section's only concrete
"how to wait" example was a blocking foreground call, which is exactly
the shape of `gh pr checks --watch`; and the "unless acceptance
criteria require it" escape was satisfied by the repo's own merge
requirement that CI be green.
Name the banned tool shapes, allow a single non-blocking status
snapshot, deny branch protection as an acceptance criterion, and give
the replacement hand-off phrasing (local test result + PR link).
Co-authored-by: multica-agent <github@multica.ai>
* fix(runtime): scope the CI-wait ban so the explicit exception stays executable (MUL-5223)
Review feedback on #5840:
- The ban read as absolute ("Blocking on external CI is never part of
your deliverable") while the next bullet allowed waiting when the
task explicitly asks for the CI result, leaving no way to satisfy
both. The ban is now scoped to "unless the explicit exception below
applies", and the exception names the one executable shape: a single
foreground blocking watch inside the same turn.
- `gh pr merge --auto` enables auto-merge and returns; it is not a
wait. Only waiting for it to land is banned.
Both hard-pin tests now also pin the exception so it cannot be dropped
or re-absolutised.
Co-authored-by: multica-agent <github@multica.ai>
* polish(runtime): group Background Task Safety into run-owned and external-CI clusters (MUL-5223)
Co-authored-by: multica-agent <github@multica.ai>
* polish(runtime): cut redundant phrasing from the external-CI cluster (MUL-5223)
The cluster said "report and finish" three different ways and carried
two rhetorical tails. Fold the delivery template into the post-push
playbook bullet, tighten the merge-gate denial, and drop filler.
5 bullets -> 4, -36 words, every behavioral fact and test pin intact.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
* perf(daemon): parallelize runtime version detection during registration (MUL-5119)
Registration probed each agent CLI's `--version` serially, so total latency
was the sum of every probe. On an onboarding host with several coding tools
installed that stacked into many seconds before runtimes registered — long
enough that the desktop runtime step timed out into its empty 'no runtime
found' state while the daemon was still working.
Fan the probes out with a bounded errgroup so total latency tracks the slowest
single probe instead of their sum. Each probe still self-heals a vanished
pinned path and re-detects the live version (no cross-registration caching, so
an in-place upgrade is still reported correctly); failures are logged and
skipped as before. Results are sorted by provider for a deterministic payload.
Co-authored-by: multica-agent <github@multica.ai>
* fix(onboarding): stop the runtime step flashing 'no runtime found' while the daemon probes (MUL-5119)
The runtime step flipped from scanning to the empty 'no runtime found' state on
a fixed 5s wall-clock, so a machine that does have coding tools installed saw a
false-negative flash whenever registration outlasted the timeout (cold start,
slow/wedged CLI, many CLIs).
Gate the empty flip on a desktop-only `runtimesPending` signal derived from the
local daemon's live status (booting, or running with agent CLIs detected on the
host): while pending, keep the scanning skeleton past the soft timeout. An
absolute hard-timeout ceiling still guarantees a fallback so a wedged probe
can't pin the step on the skeleton forever. Web omits the signal and keeps the
plain wall-clock timeout.
Also drop the two dead/duplicated affordances on the step: the permanently
disabled 'Start exploring' button now renders only in the found phase, and the
empty state's duplicate footer 'Skip for now' is removed in favour of its own
Skip card.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Lambda <lambda@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
* fix(squads): align parent issue status ownership with agent-managed model
Squad leaders now open assigned parents to in_progress on first dispatch, keep them there while members work, and only move to in_review when overall completion is confirmed—matching ordinary agent status semantics without server auto-flips.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(squads): scope leader parent-status ownership to squad-assigned issues
Review follow-up on the parent-status alignment change. Two boundaries were
left ambiguous, both of which the change's own premise ("don't make the model
resolve a contradiction in the prompt") argues should be closed in-place.
1. Status ownership was granted too widely. The leader briefing is injected on
every leader path, keyed off is_leader_task — including the MUL-3724 case
where an issue is assigned to a plain agent and a squad was merely
@mentioned for help. The unqualified "Own the parent issue status"
responsibility therefore also reached guest leaders, who could push another
assignee's in-flight issue to in_review.
buildSquadLeaderBriefing now takes ownsIssueStatus and selects between two
variants of responsibility 6: the grant only when the issue's assignee is
this squad, otherwise an explicit "do NOT change this issue's status".
Quick-create passes false — no issue exists on that turn. Everything else in
the protocol (roster, delegation, evaluation) is unchanged for both.
2. The comment-triggered path still contradicted itself. The runtime brief says
"do not change status unless the comment explicitly asks", and a member's
delivery comment never asks. Squads that dispatch by @mention create no
child issues, so no child-done system comment exists to carry the explicit
ask either — that parent would sit in in_progress indefinitely.
writeWorkflowComment now names the protocol responsibility as the one
exception for squad leaders. It is safe to state unconditionally because the
grant is only present in the instructions when the server decided this squad
owns the issue; for a guest leader the sentence has nothing to activate.
Tests: two composition tests assemble both halves (server-side briefing +
daemon-side CLAUDE.md) for one real scenario each, since asserting each half
alone is how the original contradiction shipped. Plus execenv coverage that the
carve-out appears only for leaders and the ordinary-agent rule stays absolute.
Docs and the multica-squads skill / source map record the narrower contract.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: J <j@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
The isolated-checkout path used by Linux Codex builds a task-local
repository with `git clone --local` from the workspace's bare cache.
That command has two properties that combine badly when the cache is a
partial clone: it does not carry the promisor remote configuration
across, and it does not treat an incomplete source object store as an
error. The result is a checkout that exits 0 with every tracked file
reported as deleted, so an agent starts work in what looks like a
repository someone emptied.
Swap origin to the real remote and restore
`remote.origin.promisor` / `remote.origin.partialclonefilter` before the
first checkout, so git can lazily fetch the blobs it needs. Do the same
on the reuse path, where a workdir created against a complete cache can
later be resumed against a partial one.
No cache is created as a partial clone today, so this changes nothing
for existing installs; it is a prerequisite for the on-demand clone mode
in MUL-4983 and hardens a path that fails silently rather than loudly.
Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
* feat(agents): add per-agent runtime skill controls
Co-authored-by: multica-agent <github@multica.ai>
* fix(agents): renumber runtime-skill migration and broadcast agent:status on toggle
Address the MUL-5101 review blockers on PR #5686:
- Rebase onto main and renumber the runtime-skill-disable migration
202 -> 203. main added 202_runtime_profile_add_qwen, so the pair
collided on prefix 202 and migrations_lint_test would reject the
duplicate. 203 is the next free prefix.
- Publish an "agent:status" event after persisting a
disabled_runtime_skills override, mirroring the workspace-skill toggle
in writeUpdatedAgentSkills. The realtime layer keys off this event to
invalidate workspaceKeys.agents, so other open web/desktop/mobile
clients now drop their stale toggle state instead of only the
initiating tab refreshing. Reload junction-table skills before the
broadcast so it doesn't signal cleared skills (#3459).
- Add a handler regression test proving the broadcast fires on both
disable and enable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
Co-authored-by: Walt <walt@multica.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Rework skills.sh and github.com skill imports around a single recursive
git-tree fetch to stop the 504 on large mono-repos (e.g. api-gateway-skill):
- One tree call replaces the per-directory contents crawl.
- Import caps checked arithmetically from tree metadata before any download
(fail fast with 413 instead of timing out).
- Most-specific skill-dir resolution; repo root only as a last resort, which
fixes the root SKILL.md name collision.
- Concurrent downloads (errgroup, limit 8).
- Overall 45s fetch deadline; cancellation is fatal on every supporting-file
path (tree downloader, crawl listing/recursion/download, ClawHub) so a
mid-download abort never persists a half-populated bundle.
- A skills.sh tree-fetch failure returns a retryable 503 instead of an unsafe
root-directory fallback.
- Lenient conventional-path acceptance restored for both complete and truncated
trees.
- maxImportFileCount 128 -> 256 (aligned daemon cap); 8 MiB bundle cap remains
the real guard.
* fix(daemon): gate fresh-session retry on tools executed, not session id (MUL-4966)
Switching provider accounts leaves the stored session id pointing at a
conversation the new account does not own. The daemon still passes it to
--resume, the provider rejects it, and the task dies before doing any work.
The existing fresh-session fallback was supposed to catch this but was
gated on `result.SessionID == ""`, which is not a lifecycle fact:
- Too narrow: a backend that echoes the requested id back when it rejects
a resume keeps SessionID non-empty, so the fallback never fired — the
reported bug.
- Too broad: a provider 401 before the first stream message also leaves
SessionID empty, so an unrecoverable auth failure burned a second full
run.
Gate on `tools == 0` instead. That states the property that actually makes
a retry safe — the agent executed no tool, so it mutated nothing, so
re-running cannot double-post a comment (comment creation has no
idempotency key and a duplicate re-fires its @mention triggers), reopen a
PR, or re-plan on top of its own half-finished work in the reused workdir.
Auth failures are excluded, mirroring retryableReasons in service/task.go.
The predicate is extracted to shouldRetryWithFreshSession so the tests
exercise production logic; both existing fallback tests re-implemented the
condition inline and would not have caught a regression in it.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): gate fresh-session retry on a positive resume-rejected signal (MUL-4966)
Review of the previous commit was right: `tools == 0` plus "not an auth
error" answers whether re-running is *safe*, not whether a new session can
*fix* the failure. Those are orthogonal, and answering the second by
exclusion inverts the burden of proof — the failures a fresh session cures
are a small enumerable set, while the ones it cannot are open-ended.
Concretely, the previous predicate fresh-retried on provider_network,
429/529, quota, 5xx and unclassified startup failures. provider_network is
the sharpest conflict: internal/service/task.go marks it resume-safe
(MUL-4910) specifically so the platform retry inherits the session and
continues the truncated conversation. Resetting the session first made that
contract unsatisfiable, silently discarding conversation context on a
transient blip — and rate limits got an immediate no-backoff re-run.
Replace the inference with positive evidence: agent.Result gains an explicit
ResumeRejected field, set only when a backend has proof the resume itself
was refused. claude/codebuddy/qwen derive it from resumeWasRejected, which
promotes the predicate resolveSessionID was already computing and encoding
as the side effect of blanking SessionID — using an empty string to carry
that meaning is what made the original bug possible. SessionID keeps being
dropped for a rejected resume (a dead pointer must not be persisted), but it
is no longer the signal the daemon reads to decide *why* a run failed.
The six ACP backends that recover from "session not found" set the flag at
the same points they already clear the id, so their existing recovery is not
caught by the narrower gate. codex needs nothing: thread/resume already falls
back to thread/start in-process, and deliberately does not on transport
errors.
Matching now includes the account-switch guardrail reported in #5704
(Claude Code 2.1.207, zh-CN): "400 此 session 已绑定另外的ai账号，请执行
/new 开启新 session". The en-US wording of the same guardrail has not been
captured yet, so those variants are marked inferred in the source; a miss
degrades to a terminal failure carrying the provider's raw text rather than
a mis-routed run.
Tests: backend-level fixtures drive ResumeRejected from real stream-json for
both the account-binding 400 and a network drop, and the predicate now
covers network/rate-limit/quota/5xx/auth/unclassified as explicit
non-retries.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): restore fresh-session recovery for backends with no rejection signal (MUL-4966)
Final review caught qwen regressing: its verified rejection string
("No saved session found with ID ...", already captured in
testdata/qwen-code-0.20.0-resume-not-found.stderr.txt) was not in the phrase
list, and qwen reports no session id on that path, so the new inclusion gate
turned a working auto-recovery into a terminal failure.
Auditing the other 17 resume-capable backends showed qwen was not alone.
antigravity, copilot, cursor, deveco and opencode all recovered from a
refused resume purely by reporting an empty SessionID, and none of them has
any rejection detection to convert into ResumeRejected — copilot's own
comment documents the hole (session.error before session.start), and
antigravity's helper returns "" when "the CLI exited before dispatching".
Making ResumeRejected the sole gate silently removed recovery from all five.
Fixing that by guessing rejection phrases for five more CLIs is the wrong
trade: no real output has been captured for any of them, and a false
positive discards a recoverable session pointer. So the gate is now two
tiers. Positive evidence (ResumeRejected) decides on its own where a backend
can produce it. Where none is available, an empty SessionID still gates the
retry — it proves no session was established, which is exactly what the
pre-change behaviour relied on — minus the classes a fresh session provably
cannot cure (network, rate limit, quota, provider 5xx, auth). That keeps the
resume-safe contract in internal/service/task.go intact while restoring what
these five backends had.
Also renames claudeResumeRejectedPhrases to resumeRejectedPhrases: it is
matched by claude, codebuddy and qwen, so a qwen-only string living under a
claude-prefixed name would be actively misleading.
Tests: qwen's existing missing-resume fixture now asserts ResumeRejected
(verified failing without the phrase), and the predicate covers the
no-signal tiers — retry when nothing was established, no retry once a
session exists or the failure classifies as uncurable.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): scope the no-signal fallback to backends that cannot detect rejections (MUL-4966)
Final review caught the compatibility path applying to every backend, not
just the five it was justified for. shouldRetryWithFreshSession only saw
(Result, priorSessionID, tools), so a false ResumeRejected could not be told
apart from a backend that has no way to answer — and claude/codebuddy/qwen/ACP
startup failures with no session id still fell through to the exclusion
branch. That contradicted both the stated intent and the function's own doc
comment ("where a backend can produce it, it is the whole answer").
Make the capability explicit. agent.ResumeRejectionUndetectable names the
five backends that scrape SessionID out of stream output and have no
rejection detection at all; the daemon takes provider and consults it, so a
capable backend reporting false is now taken at its word. Membership is
opt-in, so a new backend fails closed instead of silently inheriting a
guess-based retry.
Also completes the exclusion set: missing config, unavailable model, missing
executable, unsupported runtime version and (defensively) agent timeout all
have defined non-session remedies and were reaching `default: true`. What is
left through stays narrow — unknown, process failure, unparseable output,
context overflow — because a real rejection from these five most likely
surfaces as a non-zero exit or unparseable output, none of them reporting one
explicitly.
Tests: one identical result asserted across all five undetectable backends
(retries), twelve capable ones (no retry), and an unregistered provider
(fails closed), plus table cases for each newly excluded reason. Classifier
inputs were verified to map to the intended reasons rather than passing by
accident.
Also updates the ResumeRejected doc comment, which still said the daemon
gates on it alone.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): run Codex unsandboxed on Windows to stop reject-by-policy (MUL-4957)
Windows has no Landlock/Seatbelt-equivalent filesystem sandbox that the
daemon configures, so the per-task `sandbox_mode = "workspace-write"` it
wrote was unenforceable. Worse than having no sandbox, it pushed Codex
into rejecting non-safe mutation commands "by policy": `multica issue
create` fails with "was rejected by policy" because Codex can neither
sandbox the command nor (under approval_policy = "never") escalate it to
the daemon's auto-approver, so the request never reaches the approver.
Mirror the existing macOS fallback and give Windows danger-full-access so
those commands run. Also generalize the danger-full-access warn log so it
no longer hardcodes "on macOS" and only surfaces the macOS-specific
upgrade hint on macOS (new codexSandboxPolicy.Hint field).
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): correct Windows sandbox rationale and respect user windows.sandbox (MUL-4957)
Addresses two review must-fixes on #5672:
1. Correct a false security fact. The comments and log Reason claimed
Windows has no filesystem sandbox backend. Codex 0.144.5 does ship a
native Windows sandbox (windows.sandbox = "unelevated"/"elevated"); it
is experimental with open upstream reliability bugs, so the daemon
defaults to danger-full-access as a deliberate compatibility choice.
Enabling the native sandbox is tracked as separate follow-up work.
2. Stop silently downgrading users who opted into isolation. The fallback
was unconditional. Add codexSandboxPolicyForConfig: on Windows an
explicit windows.sandbox = unelevated|elevated keeps workspace-write so
Codex enforces task isolation with the user's chosen backend;
danger-full-access applies only when windows.sandbox is absent,
disabled, or unparseable. This is also the branch point for a future
native-sandbox rollout (flip the default; callers unchanged).
Adds fixture tests locking the priority (user opt-in kept vs. unconfigured
fallback) plus predicate coverage for codexSandboxPolicyForConfig.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): fail closed on undecidable Windows sandbox config, honor -c windows.sandbox (MUL-4957)
Second review round on #5672. Two must-fixes.
1. Undecidable config no longer fails open. The old bool detector collapsed
"unparseable / invalid value / failed copy" into "unconfigured" and then
loosened to danger-full-access. Replaced with a tri-state
(absent/native/undecidable): only exact-lowercase unelevated|elevated (the
sole values Codex accepts — verified: any other value makes Codex refuse to
load the config) counts as native; any other present value, unparseable
TOML, a read error, or a missing per-task config when a shared
~/.codex/config.toml exists (i.e. the copy failed) is undecidable and fails
closed to workspace-write — it never loosens — logged at error level.
2. windows.sandbox set via `-c`/`--config` custom args is now honored. Such
args never land in config.toml, so config-only detection silently
downgraded those users' isolation. The effective Codex args (daemon
defaults + profile-fixed + per-agent custom_args) are threaded through
PrepareParams/ReuseParams/CodexHomeOptions into the sandbox decision and
scanned for a windows.sandbox override (inline, two-token, quoted, spaced;
last-wins).
Also drops issue-status-bound source comments (openai/codex#24098 has since
closed). Adds unit coverage for config/args classification, the fold
precedence (undecidable > native > absent), and the copy-failed fail-closed
path.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): fail closed on config-sync errors and honor shell-quoted -c windows.sandbox (MUL-4957)
Round-3 review must-fixes:
1. resolveWindowsSandboxState now takes the config.toml sync error and a
tri-state shared-config presence instead of re-stat-ing inside. A failed
sync (stale/absent per-task copy) or an un-stat-able shared source is
undecidable and keeps workspace-write, closing the fail-open where a failed
sync was read as "unconfigured". Splits IO from the decision so the paths
are unit-testable without faulting the filesystem.
2. The Windows sandbox decision consumes agent.NormalizeCodexLaunchArgs (the
shared helper buildCodexArgs now uses) so a shell-quoted -c windows.sandbox
opt-in is normalized identically to launch, instead of being missed by a
raw-token scan and silently downgraded.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): abort when the Codex sandbox block cannot be written (MUL-4957)
Round-4 review must-fix: ensureCodexSandboxConfig failures were warn-and-continue,
so a computed fail-closed workspace-write policy could stay only in memory while
config.toml kept a stale danger-full-access from a prior run — the decision
failed closed but the effective config failed open.
prepareCodexHomeWithOpts now returns the error, which blocks startup on both
paths: fresh Prepare fails the task, and Reuse leaves env.CodexHome unset, which
configureCodexTaskShellEnvironment already refuses to start.
Regression covers the full reuse scenario (stale danger-full-access + failed
config sync + failed managed-block write); it fails with "got nil" without the
fix.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
Co-authored-by: J <j@multica.ai>
The windows-execenv job's TestPrepareIsolated_WindowsKillsDescendantBeforeRetry
flakes on the Windows runner: its helper subprocess ends in a bare select{},
which the Go runtime can reap with 'all goroutines are asleep - deadlock!'
(exit status 2) once every goroutine is parked with no wakeup source. That
races the parent's Job Object kill, so PrepareIsolated returns 'helper failed'
instead of the context.Canceled the test asserts.
Block on a timer-backed sleep loop instead: a pending timer is a wakeable
source, so the runtime never declares a deadlock, and the process still dies
the instant the Job Object tears the tree down.
Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
* feat(issues): bump issue updated_at when a comment is added (MUL-5009)
A new comment now counts as activity on its issue and advances
updated_at, so the "Updated date" Kanban/list sort surfaces
recently-discussed cards — not only cards whose status changed.
Applies to all three comment-creation paths (user/agent HTTP,
agent task delivery, and the child-done system comment) via a
best-effort TouchIssue query. The bump never fails an already-
persisted comment; it self-heals on the next activity if it errors.
Co-authored-by: multica-agent <github@multica.ai>
* fix(issues): make comment updated_at bump atomic (MUL-5009 review)
Address Elon's review. Move the updated_at bump into CreateComment as a
leading data-modifying CTE so the comment insert and the timestamp bump
commit or roll back together — closing the non-atomic window where a
comment could persist while updated_at stayed stale. That window also
skewed the daemon GC TTL, which reads issue.updated_at to reclaim
done/cancelled workdirs.
Centralizing the bump in the query drops the three per-caller TouchIssue
calls and guarantees any future comment entrypoint inherits it.
Also refresh the now-stale gc.go / gc_test.go comments that asserted
'CreateComment does not bump issue.updated_at'.
Co-authored-by: multica-agent <github@multica.ai>
* fix(issues): make comment/issue workspace match a query-level guarantee (MUL-5009 nit2)
The touch CTE now RETURNING id, workspace_id and the INSERT SELECTs from
it, so the comment insert depends on the issue actually existing in the
passed workspace. A mismatched (issue, workspace) pair matches 0 rows in
the CTE, the dependent INSERT selects nothing, and the :one query returns
pgx.ErrNoRows — no mis-attributed comment is written and the issue is not
touched.
CreateComment is now the single carrier of the 'a comment belongs to an
issue in the same workspace and always bumps it' invariant, so no future
caller can break it by passing the wrong workspace. Signature unchanged;
no migration or foreign key.
Add TestCreateComment_WorkspaceMismatchPersistsNothing (error returned,
no comment persisted, updated_at unchanged).
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): bound pre-start task preparation
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): isolate pre-start env preparation
Run execution-environment Prepare and Reuse in a killable helper process so a timed-out attempt cannot keep writing after retry. Add FIFO lifecycle and squad Stage retry regression coverage.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): terminate Windows prepare process trees
Assign the pre-start helper to a kill-on-close Job Object before releasing its request, wait for all job members to exit on cancellation, and add a Windows runtime regression job.
Co-authored-by: multica-agent <github@multica.ai>
* ci: target Windows prepare tree regression
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): isolate Linux Codex git metadata (MUL-4925)
Co-authored-by: multica-agent <github@multica.ai>
* refactor(daemon): address isolated-checkout review nits (MUL-4925)
- rename sameFilesystemPath -> sameResolvedPath (it compares resolved
paths for equality, not same-device), with a clarifying doc comment
- prune earlier tasks' agent/* branches when reusing an isolated
checkout so a long-lived reused workdir stops accumulating one local
branch per checkout; deleteLocalBranches now takes a keepBranch arg
and the prune is non-fatal
- cover the prune in TestCreateWorktreeReusesIsolatedGitMetadata
* fix(repocache): preserve user branches on reuse (MUL-4925)
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Eve <eve@multica-ai.local>
Co-authored-by: multica-agent <github@multica.ai>
Agents were writing runtime-local paths into deliverables as clickable
links (`[screenshot](/Users/agent/work/shot.png)`). Two root causes, both
fixed here.
A. The brief never stated the delivery contract. Add an always-on delivery
invariant (outside writeOutput's kind switch, so no task kind can inherit
none) plus a per-surface file-delivery line for each of the five surfaces.
Chat splits into two: `attachment upload` works only on web/mobile chat,
never on an IM channel, so ChatChannelType is now threaded into
TaskContextForEnv.
The claim path only ever looked up Slack bindings, so a Feishu session
reported as a web chat and got upload guidance for a channel that cannot
carry attachments. Probe every channel type. The chat policy is two
independent layers and stays that way: delivery keys off "is there a
channel at all"; the `chat history` / `chat thread` commands stay
Slack-only because both endpoints are hardwired to h.SlackHistory and
there is no Feishu reader — ChatInThread only selects between those two
commands, so it stays Slack-only too.
Add a CLI hard-fail lint on `issue comment add` / `issue create` /
`issue update` as the enforcement backstop. Scoped narrowly, since a false
positive blocks a real deliverable: agent task context only (a human's PAT
run is untouched), real CommonMark link/image/autolink destinations only
via goldmark (a path in a code span or fence — how an agent quotes a path
it is discussing — is structurally invisible), and three high-confidence
signals only (`file://`, inside the workdir, or an existing local file).
A bare `/foo` is a valid origin-relative URI and is deliberately allowed.
`issue update` has no --attachment flag, so its hint redirects to
`comment add` rather than naming an argument it rejects.
B. Desktop presented the resulting router 404 as an unknown crash. 8 of 18
desktop_route_error reports were users clicking such a link and being told
the app broke and to file a bug. Split the 404 into a first-class Not Found
view: no crash framing, no Report error. Its recovery entry comes from the
tab store's active workspace, never from the failed pathname — deriving a
slug from `/Users/me/shot.png` yields "Users" and a button to `/Users/issues`,
a second 404.
Also add a will-navigate trusted-origin guard via the shared loadRenderer
(main + issue windows). This is origin hardening only, NOT the mechanism for
in-app links: client-side routing never fires will-navigate, so app paths
never reach it. Issue windows need no 404 work — their router only accepts
paths validated by parseIssueWindowPath and they do not listen for
multica:navigate, so a bad path cannot reach them.
Server-side completion observation is metric/log only and never blocks: it
is lexical (`file://` + task work_dir prefix) because the server cannot stat
the daemon's filesystem, and the metric label is a closed enum so no path or
reply text reaches Prometheus.
Verified: pnpm typecheck/lint/test (3582 tests), go vet, full Go suite
including new claim-path integration tests. cmd/multica was verified outside
the daemon workdir — inside one, 93 of its tests fail identically on
origin/main because the suite walks up and finds the runtime's own task marker.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: multica-agent <github@multica.ai>
Squad-leader follow-ups on the same issue now reuse the prior daemon-managed workdir and provider session instead of starting fresh, while never binding or locking a user-provided local_directory. Reuse eligibility is keyed off a Prepare-time .managed_env.json provenance marker, so it does not race the completion→GC-metadata write.
Closes#5535
Co-authored-by: Bohan <bohan@devv.ai>
Give Linux codex tasks a writable per-task HOME (+ XDG/npm_config_cache) so npm/Prisma stop hitting EROFS on the read-only sandbox home. Gated to Linux; macOS/Windows and non-codex providers unaffected.
Note: does NOT resolve the worktree git-metadata read-only problem tracked in multica-ai/multica#2925 — Codex workspace-write resolves the worktree .git pointer and force-protects the real gitdir read-only even inside writable_roots, so that needs a separate Codex metadata-write permission path.
* fix(daemon): stop routing CodeBuddy skills/memory through Claude's .claude paths
CodeBuddy Code is a Claude Code fork but ships its own native config
directory (~/.codebuddy, .codebuddy/) with its own memory filename
(CODEBUDDY.md). It only reads .claude/skills or CLAUDE.md if a user
manually symlinks/copies them during migration
(https://www.codebuddy.ai/docs/cli/troubleshooting#migrating-from-claude-code).
Multica's daemon/execenv code treated "codebuddy" as an alias for
"claude" in three places, so skills synced by Multica landed in
.claude/skills/ and CLAUDE.md — paths the default CodeBuddy install
never reads — instead of ~/.codebuddy/skills, .codebuddy/skills, and
CODEBUDDY.md as documented at
https://www.codebuddy.ai/docs/cli/codebuddy-dir and
https://www.codebuddy.ai/docs/cli/skills.
Split the "claude", "codebuddy" switch cases in:
- daemon/local_skills.go (user-level local skill discovery/import)
- daemon/execenv/context.go (per-task skill materialization)
- daemon/execenv/runtime_config.go (runtime brief target file)
Added regression tests locking in the new paths and updated the
install-agent-runtime / providers docs (all 4 locales) that had
documented the old .claude/skills behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(daemon): cover CodeBuddy sidecar hygiene and lifecycle
Address PR #5224 review feedback: exclude CODEBUDDY.md/.codebuddy from
repo-cache worktrees, extend sidecar lifecycle matrices to codebuddy, and
make the local-skills CodeBuddy test exercise a true same-key collision.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Eve <eve@multica-ai.local>
Add xAI Grok Build (`grok`) as a first-class Multica runtime over ACP
(`grok --no-auto-update agent --always-approve stdio`), reusing
hermesClient like traecli and kimi. Includes daemon discovery,
protocol_family migration 174, model discovery, MCP passthrough,
thinking effort, frontend branding, and product docs.
Follows xAI's documented headless ACP flow: after `initialize`, read the
advertised `authMethods` and send `authenticate` (preferring xai.api_key
when XAI_API_KEY is set, else the cached login token) before any session
operation — a real, logged-in CLI rejects session/new and session/load
without it. Model discovery performs the same handshake so it returns the
live catalog instead of the static fallback.
`--no-auto-update` is passed as a global flag (and kept daemon-owned in
the blocked custom-arg set) so a background update check can't stall an
unattended ACP task. Thinking effort uses the current `--effort` flag,
and the minimum grok version is 0.2.89 (ACP + authenticate + session/load
+ session/set_model + MCP + --effort).
Closes#2895
A quick Stop before the agent's first token no longer races a late reply. Started-but-empty cancellations defer the empty/non-empty judgment until the daemon acks its transcript flush (or a grace-period sweeper fires), then settle to a single outcome. Empty outcomes persist a durable, creator-authorized draft restore (fetched/consumed via a dedicated endpoint, reconnect-safe and at-most-once) instead of broadcasting the prompt over the workspace bus.
Closes#5219
* fix(daemon): isolate Codex sessions per task to unblock initialize (MUL-4424)
Codex 0.143+ backfills a per-home session-state DB by enumerating every
rollout visible under sessions/ during `initialize`. The per-task
CODEX_HOME symlinked the shared ~/.codex/sessions in, so a machine with a
large accumulated history (one reporter: ~2000 rollouts / ~22 GiB) stalled
`initialize` for tens of seconds — the app-server started but the task
produced no output before it was cancelled (github #5273).
Give each task its own local sessions/ directory instead:
- Fresh task: create an empty local sessions/ so backfill is trivial.
- Reused task with a real sessions/ dir: it is authoritative — leave it.
- Reused task still holding a legacy symlink (older build): migrate in
place. Replace the symlink with a real dir; when resuming, symlink only
the single rollout being resumed (never copy — a rollout can be GiB and
this is on initialize's critical path); and drop the stale, rebuildable
session-state DB (state_*.sqlite*, session_index.jsonl) so Codex
re-indexes the task-local sessions. Unrelated per-task DBs (goals_*,
logs_*, memories_*) are left intact.
Also point the token-usage fallback scan at the backend's per-task
CODEX_HOME instead of the daemon-global home, so usage isn't lost now that
sessions are isolated there.
Complements the #5319 handshake watchdog (which turns a silent stall into a
loud, phased timeout); this removes the underlying cause.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): address Codex session isolation review (MUL-4424)
Resolves the three blockers from Elon's review of #5360:
1. local_directory context loss. local_directory tasks get a fresh
codex-home per task ID (the daemon never reuses their workdir), so
task-local isolation stranded every follow-up run with an empty
sessions dir and silently restarted the conversation. Their only
stable, GC-safe cross-task store is the user's own ~/.codex/sessions
(a persistent store under WorkspacesRoot would be orphan-GC'd), so keep
the shared-sessions symlink for them (IsLocalDirectory). Managed tasks
stay isolated.
2. Migration resume robustness.
- Rollout lookup now covers the flat layout and background-compressed
.jsonl.zst rollouts, not just nested YYYY/MM/DD *.jsonl — both are
legitimate Codex 0.144 history that were previously judged "not
found", silently dropping resume.
- Exposure hard-links first, then symlinks, never copies — hard links
need no privilege and work on Windows within a volume, so the
zero-copy path is exercised identically on CI.
- The daemon now verifies the rollout is actually present in the task
CODEX_HOME (execenv.CodexResumeRolloutPresent) before the brief is
generated; if absent it clears the resume from both the backend and
the brief instead of telling the agent it is continuing a lost thread.
3. session_index.jsonl is no longer deleted during migration — Codex uses
it as the authoritative thread-id -> name store (not rebuildable from
rollouts). Only the rebuildable state_*.sqlite* is reset.
Tests: 2-round local_directory resume across task IDs; compressed/flat
lookup; hard-link zero-copy (os.SameFile); session_index preserved;
CodexResumeRolloutPresent + the daemon gate helper (present keeps /
absent drops / non-codex + empty no-op).
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): scope Codex sessions to a per-issue store; disclose lost resumes (MUL-4424)
Addresses the three blockers from Elon's second review of #5360.
1. local_directory still enumerated the whole machine history. The prior
fix re-linked the entire ~/.codex/sessions into every fresh local_directory
codex-home, so Codex still backfilled from thousands of unrelated rollouts on
`initialize` (measured ~8.3s with 3450 rollouts; the reporter's 22 GiB could
exceed the 30s watchdog). Point sessions/ at a persistent, per-(agent, issue)
store under the shared Codex home (multica-sessions/<agent>/<issue>) that holds
only this issue's rollouts. It is keyed stably across task IDs and lives
outside the task-scoped envRoot the GC reclaims, so follow-up runs resume it
while `initialize` only ever sees this issue's history.
2. Windows cross-volume resume was lost. Exposing a single rollout by hard-link
(same-volume only) then file symlink (needs Windows privilege) can't cross a
volume boundary. The store now lives on the shared Codex volume, so the resume
rollout is hard-linked there zero-copy, and sessions/ is exposed to the task
home via a directory link — a symlink on Unix, a junction on Windows — which
crosses volumes without privilege and never copies a (possibly GiB) rollout on
initialize's critical path. There is no remaining per-file cross-volume link.
3. An unavailable resume was a silent downgrade. Both resume gates
(gateResumeToReusedWorkdir, gateCodexResumeToRolloutPresence) now set
PriorSessionResumeUnavailable, and the runtime brief renders a Session
Continuity Notice telling the agent to disclose to the user, up front in its
reply, that the previous conversation context could not be restored and this
run starts fresh — turning a silent restart into a user-visible one. The task
is not failed: it can still do useful work without the prior context.
Managed fresh / reused-real-dir tasks keep their task-local, GC-collected
sessions dir unchanged; only the legacy-symlink migration with a resume routes
through the store (cross-volume-safe), and a home already linked to the store is
treated as authoritative on reuse.
Tests: local_directory per-issue store (only this issue's history, no whole-
machine leak); no-key fallback to an empty dir; two-round resume across task IDs
through the store; legacy migration routed through the store with a zero-copy
hard link; reused store link stays authoritative; both gates set the
resume-unavailable flag; brief renders the continuity notice only when a resume
was lost. execenv + daemon + pkg/agent packages, go vet, and gofmt all pass.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): disclose live resume-RPC loss; bound Codex session store lifecycle (MUL-4424)
Addresses the two blockers from Elon's third review of #5360.
1. A real thread/resume failure was still a silent new session. The brief's
Session Continuity Notice only covers losses the daemon detects before launch
(workdir not reused, rollout absent). But when the rollout is present yet
Codex rejects the live thread/resume (corrupt/incompatible rollout, server-side
thread GC, schema drift), startOrResumeThread falls back to thread/start and
the run succeeds on a fresh thread with no user-facing signal. Carry the
original resume intent into the backend as ExecOptions.ResumeExpected (set from
the post-gate PriorSessionID, so a pre-flight drop still routes through the
brief and never double-notifies), and when a resume was expected but the
backend landed on a fresh thread, prepend the same continuity notice to the
first turn/start input. This also covers the daemon's transport-error
fresh-session retry, which clears ResumeSessionID but not ResumeExpected.
2. The persistent per-issue store had no data lifecycle. multica-sessions stores
live outside the task-scoped envRoot the GC reclaims (so resume survives across
task IDs), which meant a done/abandoned issue's prompts and full rollouts (one
reporter: a single 1.5 GiB rollout) accumulated forever and were never freed on
issue/agent/workspace deletion. Add PruneCodexSessionStores: the daemon GC loop
reclaims any store untouched for GCCodexSessionTTL (default 14 days, configurable
via MULTICA_GC_CODEX_SESSION_TTL, 0 disables). A store's newest rollout mtime is
its last activity, so an active or recently-resumed task keeps its store fresh
and is never reclaimed, while a deleted issue's store ages out — an eventual
reclamation guarantee without needing deletion events.
Tests: codexTurnInput discloses on resume fallback and stays silent on success /
fresh start (paired with the existing live-RPC fallback test); store pruning
reclaims aged stores, keeps recent ones, isolates issues, cleans empty agent
dirs, and is disable-able. execenv / daemon / pkg/agent, go vet, gofmt all pass.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): protect a reopened Codex session store from GC mid-mount (MUL-4424)
Addresses Elon's fourth-review blocker: reopening an issue idle past
GCCodexSessionTTL could lose context, because mounting its per-issue session
store (MkdirAll + rollout lookup + task-home link) never refreshed the store's
mtime, so a GC cycle firing before the resumed turn wrote its first rollout saw
a >TTL-old store and reclaimed it — a stat->remove race with no in-use guard.
Two complementary defenses:
- Activity refresh: linkCodexSessionsToStore now os.Chtimes the store to now
after linking, so codexStoreStat (which reads the newest mtime as last
activity) sees a just-used store. This fixes the sequential repro — a mount
immediately followed by a prune keeps the store.
- In-process active-store guard: the daemon marks the per-issue store in-use
(execenv.CodexSessionStorePath) from before Prepare/Reuse mounts it until the
task ends, and PruneCodexSessionStores now takes an isActive predicate and
skips any store a live task holds. Because prepare and prune run in the same
process, this closes the remaining concurrent stat->remove window the mtime
refresh alone cannot. Reference-counted, mirroring the env-root guard.
Tests: a reopened >TTL store survives a GC cycle after remount and stays
resumable; an idle-on-disk store marked active is skipped, then reclaimed once
inactive; the existing idle-reclaim / isolation / disable / empty-agent-dir
cases still pass. execenv + daemon, go vet, gofmt all pass.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): make Codex store delete atomic with mark-active (MUL-4424)
Addresses Elon's fifth-review blocker: the active-store guard's check and delete
were not one atomic step. PruneCodexSessionStores called isActive (which locked,
read, and unlocked) and only then RemoveAll'd, leaving a window where a task
could markActiveCodexStore between the check and the removal and still lose its
store — the exact mark-then-delete interleaving Elon reproduced.
Replace the point-in-time isActive predicate with a reserve-for-deletion
protocol that shares one lock with mark-active:
- reserveCodexStoreForDeletion(store) atomically refuses when a live task holds
the store (or another delete already reserved it) and otherwise marks it
reserved, all under one activeCodexStoresMu acquisition. PruneCodexSessionStores
reserves before RemoveAll and commits after, so confirm-inactive and remove are
effectively atomic against a concurrent mark.
- markActiveCodexStore now waits (on a sync.Cond) while a store is reserved, so a
task never mounts a store mid-removal; committing the removal wakes it and the
store is recreated fresh by Prepare (with the continuity notice).
So mark-before-reserve keeps the store (reserve refused); reserve-before-mark
removes it and blocks the late mark until the removal commits. The genuinely
idle case still reclaims.
Tests (daemon, run under -race): mark-then-reserve is refused; reserve blocks a
concurrent mark until commit then the store reads active; a second reserve is
refused mid-flight. The execenv prune tests move to the reserve seam; the
activity-refresh / reopen-then-prune / isolation / disable cases still pass.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): namespace Codex session stores per profile for cross-daemon safety (MUL-4424)
Addresses Elon's sixth-review blocker: the in-process reservation guard cannot
span processes, but Multica supports multiple profile daemons on one machine
(e.g. production + staging) that share the same ~/.codex. Each daemon's GC
scanned the whole multica-sessions root, so a staging daemon could reclaim a
store a production task was actively resuming — its reservation lived only in the
other process's memory.
Isolate by profile instead of trying to lock across processes:
- Store path is now <shared>/multica-sessions/<namespace>/<agent>/<issue>, where
namespace is the daemon's profile (empty -> "default"). PrepareParams/ReuseParams
carry Profile; codexSessionStoreKey and CodexSessionStorePath fold it in.
- PruneCodexSessionStores takes the profile and scans ONLY that namespace, so a
daemon never even sees another profile's stores, let alone deletes them. The
per-profile trees are disjoint, so the in-process guard is sufficient within a
namespace (profiles get separate daemon state, so no two daemons share one).
Test: a "staging"-owned idle store is untouched by a default-profile prune and
reclaimed only by staging's own prune. Existing prune/guard/reopen tests move
under the namespace. execenv + daemon under -race, go vet, gofmt all pass.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): make the Codex store profile→namespace map injective (MUL-4424)
Addresses Elon's seventh-review blocker: the per-profile namespace was derived by
dropping unsafe characters, which is not injective. The CLI treats the empty
(default) profile and a profile literally named "default" as separate daemons,
yet both mapped to namespace "default"; likewise "staging.prod" and "stagingprod"
both mapped to "stagingprod". Two distinct daemons then shared one store tree, so
one could again reclaim the other's live session — the cross-process blocker
reopened for those profile names.
Make codexSessionStoreNamespace injective: the empty profile gets a reserved
bare literal "default", and every named profile is hex-encoded (bijective,
filesystem-safe) under a "p_" prefix a bare literal can never collide with. So
"" -> "default" while "default" -> "p_64656661756c74", and "staging.prod" /
"stagingprod" get distinct hex segments. sanitizeCodexPathSegment stays for the
UUID agent/issue segments (injective for real UUIDs); only the user-controlled
profile needed the encoding.
Tests: codexSessionStoreNamespace is distinct for "" vs "default", punctuation
variants, case variants, and an encoded-looking name; and end-to-end, pruning one
profile never reclaims the other's store for the "" vs "default" and
"staging.prod" vs "stagingprod" pairs. execenv + daemon under -race, go vet,
gofmt all pass.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): fixed-length Codex store namespace so long profiles fit (MUL-4424)
Addresses Elon's eighth-review blocker: hex-encoding the full profile doubled the
namespace segment length. A profile can be as long as a filesystem segment allows
(~255 bytes) and the CLI persists it as its own config dir, but the store
namespace "p_" + hex(profile) reached 2 + 127*2 = 256 bytes at 127 chars,
overflowing the 255-byte single-segment limit — the profile's own dir created
fine, then the session store failed with "file name too long".
Derive the namespace from a fixed-length hash instead: a named profile is now
"p_" + hex(sha256(profile)) — a constant 64 hex chars (66 with the prefix),
filesystem-safe and collision-resistant. The empty (default) profile keeps its
reserved bare literal "default", which the "p_"-prefixed 66-char segment can
never equal. Still injective across the CLI's distinct-daemon cases; just no
longer length-expanding.
Test: the namespace stays <=255 bytes and creatable for profiles up to the
255-byte segment limit (127- and 255-char cases that overflowed under hex); the
prior injectivity and cross-profile prune-isolation tests still hold. execenv +
daemon under -race, go vet, gofmt all pass.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: J <j@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
* fix(execenv): overlay per-task HERMES_HOME so Hermes discovers bound skills
Hermes has no workspace-relative skill discovery — it scans <HERMES_HOME>/skills
first, then skills.external_dirs from config.yaml (verified against the bundled
agent/skill_utils.py). The daemon wrote assigned skills to the generic
.agent_context/skills/ fallback, which Hermes never reads, so they silently never
took effect (#5242).
When (and only when) an agent has skills bound, redirect HERMES_HOME to a minimal
per-task compatibility overlay; a skill-less Hermes task keeps its real home and
original behavior:
- mirror every top-level entry of the shared home via symlink except the
overlay-owned ones (denylist), reconciling entries deleted from the shared home;
- derive a task-local config.yaml whose skills.external_dirs references the shared
skills dir plus the user's existing external_dirs, expanded against the
sanitized effective child env (unknown vars preserved, blocklisted keys resolved
to the process value) and normalized to absolute paths;
- write only the bound skills into the task-local skills/ dir (home skills scanned
first, so they win); global skills are referenced, not copied;
- keep memories/ overlay-owned (fresh per-task dir) AND disable the external
memory.provider, so neither on-disk memory nor a shared backend crosses tasks;
- keep active_profile/profiles out of the overlay so Hermes can't follow a sticky
profile and redirect past it at startup.
Profile handling mirrors hermes_cli.profiles: the daemon reads -p/--profile with
agent.HermesProfileFromArgs and seeds the overlay from that profile's home via
ResolveHermesSourceHome (default/invalid -> base, valid name -> <base>/profiles/
<name>, validated; a missing named profile fails closed). The profile flags are
stripped from the acp argv ONLY when the overlay is active (hermesLaunchArgs), so
a skill-less task's profile passes through unchanged. HERMES_HOME is no longer
custom_env-blocklisted: no skills -> user value passes through; skills -> overlay
overrides after layering. Fail closed — Prepare errors, Reuse returns nil.
Task home 0700, derived config 0600 via atomic replace. Platform-native default
home (%LOCALAPPDATA%\hermes, incl. the LOCALAPPDATA-missing fallback, on Windows).
Tests span execenv/daemon/agent: no-skill no-op, child-env layering + env
sanitization, profile parse/unquote + conditional strip + final args/env per
scenario, custom/profile/default/invalid/missing/Windows source home, sticky-
profile not mirrored, memory dir isolation + external provider disable, mirror
reconciliation, external_dirs rebasing + sanitized/unknown-var expansion,
local-precedence slug, perms, fail-closed, resume teardown. Docs (en + ja/ko/zh).
Fixes#5242
* fix(hermes): make profile selection one resolver contract matching Hermes
Round 5 review: the profile chain approximated Hermes' semantics in three
separate places (argv parsing, source-home selection, arg filtering), so it
diverged from native Hermes in several merge-blocking cases. Collapse it into
one authoritative resolution:
- agent.ParseHermesProfileArgs replaces HermesProfileFromArgs/
FilterHermesProfileArgs. It reproduces _apply_profile_override step 1/1b
(first occurrence, value-flag skipping, `--` and `mcp add --args` boundaries,
space-form profile-id guard) and returns the exact argv occurrence to consume;
StripHermesProfileArgs removes only that occurrence.
- execenv.ResolveHermesProfile replaces ResolveHermesSourceHome. It derives the
Hermes root exactly like get_default_hermes_root (an already-profile-scoped
HERMES_HOME roots at its grandparent), selects an explicit profile first,
otherwise trusts a profile-scoped home (step 1.5) and only then the sticky
<root>/active_profile (step 2), and validates via normalize/validate_profile_name
(reserved hermes/test/tmp/root/sudo and empty inline `--profile=` are hard
errors). Profiles always resolve under the root, so `-p default` re-roots and
`-p <sibling>` is a sibling, never nested.
- The daemon runs one parse + resolve, fails the task closed on a reserved/
invalid selection (matching Hermes' sys.exit(1)), and exports the selected
source home as the effective env's HERMES_HOME so ${HERMES_HOME} in a profile's
skills.external_dirs expands against the selected profile home (as native
Hermes does before loading config.yaml), not the root or the overlay.
Regressions added: root + sticky named profile selection; already-profile-scoped
home with no flag; that home with -p default and -p <sibling>; reserved and empty
inline profile values; and a selected profile whose external_dirs contains
${HERMES_HOME}.
* fix(hermes): overlay-owned derived .env + symlink-resolved root
Round 6 review, two remaining overlay-bypass paths:
1. A source `.env` could redirect HERMES_HOME after profile resolution. Hermes
runs `_apply_profile_override()` then `load_hermes_dotenv()`, which loads
`<HERMES_HOME>/.env` with override=True — so a mirrored source `.env` carrying
an out-of-band `HERMES_HOME=` overwrote the overlay's home, repointing skill
discovery and memory back at the source. `.env` is now overlay-owned and
DERIVED (writeDerivedHermesEnv): it preserves the source's credentials/settings
but strips any `HERMES_HOME` assignment and pins `HERMES_HOME` to the overlay
last (single-quoted, literal), written 0600 via atomic replace. It is written
even when the source has none, so Hermes' project-`.env` fallback (override=True
only when no user `.env` loaded) can't relocate the home either.
2. Root derivation was lexical-only, diverging from `get_default_hermes_root`,
which compares `env_path.resolve()` with `native_home.resolve()`. A HERMES_HOME
symlinked into `<native>/profiles/<x>` was treated as its own root, so
`-p default`/`-p <sibling>` resolved wrong. `hermesRootFromHomeFor` now resolves
symlinks (Path.resolve(strict=False)-style best effort) for the containment
decision while keeping the returned root unresolved, matching Hermes.
Regressions: source `.env` with HERMES_HOME replayed through the override=True
dotenv order (bound skill + task memory stay on the overlay; creds preserved);
minimal overlay `.env` created when the source has none; and a symlinked profile
home resolving `-p default`/`-p <sibling>` to the native root.
The daemon pins each agent CLI's symlink-resolved absolute path at startup to
block PATH-redirect of a task launch. A version manager (Homebrew Cask, nvm/fnm)
upgrading in place deletes the pinned versioned directory and repoints the stable
name, leaving the daemon on a path that no longer exists — every codex task, model
list, and version detection then hard-fails with "executable not found" until the
daemon restarts.
resolveAgentEntry now self-heals a vanished pin by re-resolving the recorded
command once, version-detecting and min-version-gating the candidate before
adopting it, and publishing {path, version} atomically so callers key policy off
the binary that actually runs. Coalesced with singleflight; a live heal wins over
a reappearing stale path; custom runtimes and custom-only hosts are untouched.
Applied at task launch, model listing, and registration.
MUL-4486
