Grok Build reports cachedReadTokens inside inputTokens (totalTokens ==
input + output on a real 0.2.106 turn, and that turn's costUsdTicks
matches xAI's rates only when the cached prefix is billed once). The
shared ACP parser persisted both counters raw, so the usage dashboard
charged the cached prefix at the full input rate *and* the cache-read
rate — ~4x the real spend on a cache-heavy turn.
Re-bucket cached reads out of input when totalTokens proves the overlap,
the same normalization codex.go already applies. Backends that report
mutually-exclusive buckets or omit totalTokens are untouched.
Co-authored-by: J <agent@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
Switching the runtime mid-conversation in Build with AI only updated local React state, so the picker could show runtime B while every subsequent message still executed on the runtime frozen at session create time.
- Add PATCH /api/agent-builder/sessions/{id}/runtime to rebind the hidden builder carrier (runtime_id/runtime_mode, model cleared since model ids are per-runtime). Creator-only, builder carriers only, target must be in-workspace, usable by the member, and online; a reply in flight returns 409.
- Serialise rebind against send: both take LockChatSessionForRuntimeBind on the chat_session row and SendDirectChatMessage re-reads the agent inside that transaction, so a send blocked behind a rebind cannot resume and stamp its task with the runtime the switch moved away from.
- Leave chat_session.runtime_id stale on purpose so the daemon starts a fresh provider session on the new runtime while Multica-side history and the draft survive.
- Frontend updates the draft only after the server reports the bound runtime, blocks sending during a rebind, disables the Mine/All filter alongside the trigger, and explains why the picker is locked during a pending reply.
Closes#5773
Flow drops from five steps to three: role + use_case merge into a
single About-you screen (one Skip covers both; Continue stamps skip
markers on whichever group was left unanswered), and the source
question leaves onboarding entirely.
Source is now collected only by the workspace source-backfill prompt,
which additionally waits until agents/squads have completed at least
SOURCE_BACKFILL_MIN_AGENT_DONE_ISSUES (3) issues in the workspace —
attribution is asked after Multica has visibly delivered value, not
before. The count rides a limit:1 issues query keyed under
issueKeys.all so realtime invalidations keep it fresh, enabled only
for users who still owe an answer.
Server: questionnaire complete() narrows to role + use_case so the
funnel step doesn't stall on the now-deferred source; a new
metrics-only onboarding_source_submitted event (+ Prometheus counter)
tracks the backfill prompt's answer/decline transition once per user.
Co-authored-by: Lambda <lambda@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
* perf(daemon): parallelize runtime version detection during registration (MUL-5119)
Registration probed each agent CLI's `--version` serially, so total latency
was the sum of every probe. On an onboarding host with several coding tools
installed that stacked into many seconds before runtimes registered — long
enough that the desktop runtime step timed out into its empty 'no runtime
found' state while the daemon was still working.
Fan the probes out with a bounded errgroup so total latency tracks the slowest
single probe instead of their sum. Each probe still self-heals a vanished
pinned path and re-detects the live version (no cross-registration caching, so
an in-place upgrade is still reported correctly); failures are logged and
skipped as before. Results are sorted by provider for a deterministic payload.
Co-authored-by: multica-agent <github@multica.ai>
* fix(onboarding): stop the runtime step flashing 'no runtime found' while the daemon probes (MUL-5119)
The runtime step flipped from scanning to the empty 'no runtime found' state on
a fixed 5s wall-clock, so a machine that does have coding tools installed saw a
false-negative flash whenever registration outlasted the timeout (cold start,
slow/wedged CLI, many CLIs).
Gate the empty flip on a desktop-only `runtimesPending` signal derived from the
local daemon's live status (booting, or running with agent CLIs detected on the
host): while pending, keep the scanning skeleton past the soft timeout. An
absolute hard-timeout ceiling still guarantees a fallback so a wedged probe
can't pin the step on the skeleton forever. Web omits the signal and keeps the
plain wall-clock timeout.
Also drop the two dead/duplicated affordances on the step: the permanently
disabled 'Start exploring' button now renders only in the found phase, and the
empty state's duplicate footer 'Skip for now' is removed in favour of its own
Skip card.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Lambda <lambda@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
* feat(issues): richer sub-issue rows in issue detail (MUL-5098)
The sub-issues panel showed only status, identifier, title and assignee —
priority, due dates, labels, live agent activity and nested breakdowns
were invisible without opening each child.
- SubIssueRow now shows priority (checkbox-slot swap like list rows),
the agent-activity indicator, label chips (+n overflow), the child's
own done/total progress ring, and an inline-editable due date with
overdue emphasis (muted when the child is done/cancelled)
- Right-click opens the shared issue actions menu via a section-level
IssueContextMenuProvider — parity with list/board surfaces
- ListChildIssues + ListChildrenByParents now bulk-load labels
(same labelsByIssue pattern as the other list endpoints)
- patchIssueLabels patches per-parent children caches;
invalidateIssueLabelDerivatives refetches the Map-shaped batched
children caches so label changes stay live everywhere
Co-authored-by: multica-agent <github@multica.ai>
* feat(issues): customizable property display for sub-issue rows (MUL-5098)
The enriched sub-issue rows were a fixed field set — no way to trim
them or surface workspace custom properties.
- New user-level persisted preference (useSubIssueDisplayStore):
built-in field toggles (priority / labels / sub-issue progress /
due date / assignee) + opted-in custom property ids. Defaults match
the previous fixed layout, so existing users see no change.
- SubIssueDisplayPopover on the section header — same switch-row
interaction as the main views' Display panel, reusing its card_*
locale keys (no new translations needed).
- Rows render opted-in custom property chips (PropertyIcon +
CustomPropertyValueDisplay, list-row parity) only when the child
carries a value; ids resolve against live non-archived definitions,
so foreign-workspace or archived ids are inert.
Co-authored-by: multica-agent <github@multica.ai>
* fix(issues): reconcile sub-issue cache updates
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Lambda <lambda@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
* fix(squads): align parent issue status ownership with agent-managed model
Squad leaders now open assigned parents to in_progress on first dispatch, keep them there while members work, and only move to in_review when overall completion is confirmed—matching ordinary agent status semantics without server auto-flips.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(squads): scope leader parent-status ownership to squad-assigned issues
Review follow-up on the parent-status alignment change. Two boundaries were
left ambiguous, both of which the change's own premise ("don't make the model
resolve a contradiction in the prompt") argues should be closed in-place.
1. Status ownership was granted too widely. The leader briefing is injected on
every leader path, keyed off is_leader_task — including the MUL-3724 case
where an issue is assigned to a plain agent and a squad was merely
@mentioned for help. The unqualified "Own the parent issue status"
responsibility therefore also reached guest leaders, who could push another
assignee's in-flight issue to in_review.
buildSquadLeaderBriefing now takes ownsIssueStatus and selects between two
variants of responsibility 6: the grant only when the issue's assignee is
this squad, otherwise an explicit "do NOT change this issue's status".
Quick-create passes false — no issue exists on that turn. Everything else in
the protocol (roster, delegation, evaluation) is unchanged for both.
2. The comment-triggered path still contradicted itself. The runtime brief says
"do not change status unless the comment explicitly asks", and a member's
delivery comment never asks. Squads that dispatch by @mention create no
child issues, so no child-done system comment exists to carry the explicit
ask either — that parent would sit in in_progress indefinitely.
writeWorkflowComment now names the protocol responsibility as the one
exception for squad leaders. It is safe to state unconditionally because the
grant is only present in the instructions when the server decided this squad
owns the issue; for a guest leader the sentence has nothing to activate.
Tests: two composition tests assemble both halves (server-side briefing +
daemon-side CLAUDE.md) for one real scenario each, since asserting each half
alone is how the original contradiction shipped. Plus execenv coverage that the
carve-out appears only for leaders and the ordinary-agent rule stays absolute.
Docs and the multica-squads skill / source map record the narrower contract.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: J <j@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
The isolated-checkout path used by Linux Codex builds a task-local
repository with `git clone --local` from the workspace's bare cache.
That command has two properties that combine badly when the cache is a
partial clone: it does not carry the promisor remote configuration
across, and it does not treat an incomplete source object store as an
error. The result is a checkout that exits 0 with every tracked file
reported as deleted, so an agent starts work in what looks like a
repository someone emptied.
Swap origin to the real remote and restore
`remote.origin.promisor` / `remote.origin.partialclonefilter` before the
first checkout, so git can lazily fetch the blobs it needs. Do the same
on the reuse path, where a workdir created against a complete cache can
later be resumed against a partial one.
No cache is created as a partial clone today, so this changes nothing
for existing installs; it is a prerequisite for the on-demand clone mode
in MUL-4983 and hardens a path that fails silently rather than loudly.
Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
* feat(agents): add per-agent runtime skill controls
Co-authored-by: multica-agent <github@multica.ai>
* fix(agents): renumber runtime-skill migration and broadcast agent:status on toggle
Address the MUL-5101 review blockers on PR #5686:
- Rebase onto main and renumber the runtime-skill-disable migration
202 -> 203. main added 202_runtime_profile_add_qwen, so the pair
collided on prefix 202 and migrations_lint_test would reject the
duplicate. 203 is the next free prefix.
- Publish an "agent:status" event after persisting a
disabled_runtime_skills override, mirroring the workspace-skill toggle
in writeUpdatedAgentSkills. The realtime layer keys off this event to
invalidate workspaceKeys.agents, so other open web/desktop/mobile
clients now drop their stale toggle state instead of only the
initiating tab refreshing. Reload junction-table skills before the
broadcast so it doesn't signal cleared skills (#3459).
- Add a handler regression test proving the broadcast fires on both
disable and enable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
Co-authored-by: Walt <walt@multica.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Rework skills.sh and github.com skill imports around a single recursive
git-tree fetch to stop the 504 on large mono-repos (e.g. api-gateway-skill):
- One tree call replaces the per-directory contents crawl.
- Import caps checked arithmetically from tree metadata before any download
(fail fast with 413 instead of timing out).
- Most-specific skill-dir resolution; repo root only as a last resort, which
fixes the root SKILL.md name collision.
- Concurrent downloads (errgroup, limit 8).
- Overall 45s fetch deadline; cancellation is fatal on every supporting-file
path (tree downloader, crawl listing/recursion/download, ClawHub) so a
mid-download abort never persists a half-populated bundle.
- A skills.sh tree-fetch failure returns a retryable 503 instead of an unsafe
root-directory fallback.
- Lenient conventional-path acceptance restored for both complete and truncated
trees.
- maxImportFileCount 128 -> 256 (aligned daemon cap); 8 MiB bundle cap remains
the real guard.
* fix(agent): inject --yolo in Qwen headless runs so shell/edit/write tools are available (Fixes#5743)
Qwen Code's non-interactive mode (`-p … --output-format stream-json`) uses a
fail-closed approval policy: it silently drops `run_shell_command`, `edit`,
`write_file`, and `monitor` from the tool registry unless bypass mode is
active. Every other Multica-supported coding adapter already injects its
equivalent permission flag as a daemon-owned argument (e.g. Claude uses
`--permission-mode bypassPermissions`, Grok uses `--always-approve`, Qoder
uses `--yolo --acp`). Qwen was the only exception.
Changes:
- `buildQwenArgs`: append `--yolo` after the protocol flags and before any
custom args so headless daemon runs always receive the full tool set.
- `qwenBlockedArgs`: add `--yolo`, `-y`, `--approval-mode`, and
`--allowed-tools` as daemon-owned flags that are stripped from custom_args.
This prevents users from accidentally or intentionally disabling bypass mode
or narrowing the allowed tool set via per-agent settings. `--exclude-tools`
is intentionally left unblocked so users can still hard-deny specific tools.
- `TestBuildQwenArgsKeepsProtocolManaged`: extend with the new blocked flags
and assert daemon-owned `--yolo` appears exactly once.
- `TestBuildQwenArgsYoloAlwaysPresent`: new test asserting `--yolo` is present
even when `ExecOptions` carries no custom args.
* fix(agent): correct Qwen permission args and docs
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Eve <eve@multica-ai.local>
Co-authored-by: multica-agent <github@multica.ai>
* fix(agent): parse Grok ACP token usage from session/prompt _meta
Grok Build places per-turn metering under result._meta (and
_meta.usage), not the top-level ACP usage field. Multica's shared
parser only read result.usage, so Grok tasks recorded empty token
usage and cost dashboards stayed at zero.
Fall back to _meta.usage (then flat _meta counters) when top-level
usage is absent. Prefer standard top-level usage when both are
present. Update the Grok fake ACP fixture and add regression tests
against the live 0.2.x payload shape.
* test(agent): cover zero ACP usage meta fallback
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Eve <eve@multica-ai.local>
Co-authored-by: multica-agent <github@multica.ai>
Retries once when Codex's model catalog refresh blocks the first turn, with a narrow safety gate: first-turn no-progress timeout, zero semantic progress, catalog-refresh evidence in stderr, and a confirmed-reaped process tree. Clears ResumeSessionID on retry so a stalled thread is never resumed, and buffers the leading session pin so a discarded attempt cannot pin the resume pointer.
* fix(daemon): gate fresh-session retry on tools executed, not session id (MUL-4966)
Switching provider accounts leaves the stored session id pointing at a
conversation the new account does not own. The daemon still passes it to
--resume, the provider rejects it, and the task dies before doing any work.
The existing fresh-session fallback was supposed to catch this but was
gated on `result.SessionID == ""`, which is not a lifecycle fact:
- Too narrow: a backend that echoes the requested id back when it rejects
a resume keeps SessionID non-empty, so the fallback never fired — the
reported bug.
- Too broad: a provider 401 before the first stream message also leaves
SessionID empty, so an unrecoverable auth failure burned a second full
run.
Gate on `tools == 0` instead. That states the property that actually makes
a retry safe — the agent executed no tool, so it mutated nothing, so
re-running cannot double-post a comment (comment creation has no
idempotency key and a duplicate re-fires its @mention triggers), reopen a
PR, or re-plan on top of its own half-finished work in the reused workdir.
Auth failures are excluded, mirroring retryableReasons in service/task.go.
The predicate is extracted to shouldRetryWithFreshSession so the tests
exercise production logic; both existing fallback tests re-implemented the
condition inline and would not have caught a regression in it.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): gate fresh-session retry on a positive resume-rejected signal (MUL-4966)
Review of the previous commit was right: `tools == 0` plus "not an auth
error" answers whether re-running is *safe*, not whether a new session can
*fix* the failure. Those are orthogonal, and answering the second by
exclusion inverts the burden of proof — the failures a fresh session cures
are a small enumerable set, while the ones it cannot are open-ended.
Concretely, the previous predicate fresh-retried on provider_network,
429/529, quota, 5xx and unclassified startup failures. provider_network is
the sharpest conflict: internal/service/task.go marks it resume-safe
(MUL-4910) specifically so the platform retry inherits the session and
continues the truncated conversation. Resetting the session first made that
contract unsatisfiable, silently discarding conversation context on a
transient blip — and rate limits got an immediate no-backoff re-run.
Replace the inference with positive evidence: agent.Result gains an explicit
ResumeRejected field, set only when a backend has proof the resume itself
was refused. claude/codebuddy/qwen derive it from resumeWasRejected, which
promotes the predicate resolveSessionID was already computing and encoding
as the side effect of blanking SessionID — using an empty string to carry
that meaning is what made the original bug possible. SessionID keeps being
dropped for a rejected resume (a dead pointer must not be persisted), but it
is no longer the signal the daemon reads to decide *why* a run failed.
The six ACP backends that recover from "session not found" set the flag at
the same points they already clear the id, so their existing recovery is not
caught by the narrower gate. codex needs nothing: thread/resume already falls
back to thread/start in-process, and deliberately does not on transport
errors.
Matching now includes the account-switch guardrail reported in #5704
(Claude Code 2.1.207, zh-CN): "400 此 session 已绑定另外的ai账号，请执行
/new 开启新 session". The en-US wording of the same guardrail has not been
captured yet, so those variants are marked inferred in the source; a miss
degrades to a terminal failure carrying the provider's raw text rather than
a mis-routed run.
Tests: backend-level fixtures drive ResumeRejected from real stream-json for
both the account-binding 400 and a network drop, and the predicate now
covers network/rate-limit/quota/5xx/auth/unclassified as explicit
non-retries.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): restore fresh-session recovery for backends with no rejection signal (MUL-4966)
Final review caught qwen regressing: its verified rejection string
("No saved session found with ID ...", already captured in
testdata/qwen-code-0.20.0-resume-not-found.stderr.txt) was not in the phrase
list, and qwen reports no session id on that path, so the new inclusion gate
turned a working auto-recovery into a terminal failure.
Auditing the other 17 resume-capable backends showed qwen was not alone.
antigravity, copilot, cursor, deveco and opencode all recovered from a
refused resume purely by reporting an empty SessionID, and none of them has
any rejection detection to convert into ResumeRejected — copilot's own
comment documents the hole (session.error before session.start), and
antigravity's helper returns "" when "the CLI exited before dispatching".
Making ResumeRejected the sole gate silently removed recovery from all five.
Fixing that by guessing rejection phrases for five more CLIs is the wrong
trade: no real output has been captured for any of them, and a false
positive discards a recoverable session pointer. So the gate is now two
tiers. Positive evidence (ResumeRejected) decides on its own where a backend
can produce it. Where none is available, an empty SessionID still gates the
retry — it proves no session was established, which is exactly what the
pre-change behaviour relied on — minus the classes a fresh session provably
cannot cure (network, rate limit, quota, provider 5xx, auth). That keeps the
resume-safe contract in internal/service/task.go intact while restoring what
these five backends had.
Also renames claudeResumeRejectedPhrases to resumeRejectedPhrases: it is
matched by claude, codebuddy and qwen, so a qwen-only string living under a
claude-prefixed name would be actively misleading.
Tests: qwen's existing missing-resume fixture now asserts ResumeRejected
(verified failing without the phrase), and the predicate covers the
no-signal tiers — retry when nothing was established, no retry once a
session exists or the failure classifies as uncurable.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): scope the no-signal fallback to backends that cannot detect rejections (MUL-4966)
Final review caught the compatibility path applying to every backend, not
just the five it was justified for. shouldRetryWithFreshSession only saw
(Result, priorSessionID, tools), so a false ResumeRejected could not be told
apart from a backend that has no way to answer — and claude/codebuddy/qwen/ACP
startup failures with no session id still fell through to the exclusion
branch. That contradicted both the stated intent and the function's own doc
comment ("where a backend can produce it, it is the whole answer").
Make the capability explicit. agent.ResumeRejectionUndetectable names the
five backends that scrape SessionID out of stream output and have no
rejection detection at all; the daemon takes provider and consults it, so a
capable backend reporting false is now taken at its word. Membership is
opt-in, so a new backend fails closed instead of silently inheriting a
guess-based retry.
Also completes the exclusion set: missing config, unavailable model, missing
executable, unsupported runtime version and (defensively) agent timeout all
have defined non-session remedies and were reaching `default: true`. What is
left through stays narrow — unknown, process failure, unparseable output,
context overflow — because a real rejection from these five most likely
surfaces as a non-zero exit or unparseable output, none of them reporting one
explicitly.
Tests: one identical result asserted across all five undetectable backends
(retries), twelve capable ones (no retry), and an unregistered provider
(fails closed), plus table cases for each newly excluded reason. Classifier
inputs were verified to map to the intended reasons rather than passing by
accident.
Also updates the ResumeRejected doc comment, which still said the daemon
gates on it alone.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
* fix(agent/cursor): send prompt on stdin so CLI-like flags cannot be re-tokenised (MUL-4992)
On Windows a Cursor task whose prompt contains CLI-like flags failed in ~2s
with `error: unknown option '-X'` and no agent output (#5649). A pasted build
log such as
go build -ldflags "-X main.version=foo" -o bin/server ./cmd/server
was enough to kill the run.
buildCursorArgs put the whole prompt in argv as the positional after -p.
The official Windows launcher chain ends in `& node.exe index.js $args`
inside cursor-agent.ps1, where PowerShell re-serialises $args onto node's
command line. Under Windows PowerShell 5.1 and pwsh <= 7.2 (Legacy native
argument passing) an argument holding embedded double quotes is not
re-escaped, so the quoted region closes early, node's argv parser re-splits
at the interior spaces, and `-X` reaches commander.js as a standalone flag.
#1709 removed the cmd.exe `%*` re-tokenisation but stopped at the
Go -> PowerShell boundary, one hop before this. Its Windows tests only
compare the argv slice Go builds and never execute a shim, which is why the
gap stayed invisible.
Fix: keep the prompt off every command line. cursor-agent's -p is a boolean
print-mode switch and the prompt is positional; with no positional prompt and
a non-TTY stdin the CLI reads stdin to EOF and uses that as the prompt. So
drop the prompt from argv on all platforms and write it to stdin, leaving
only fixed, content-free flags in argv. No shell or launcher on any platform
can re-tokenise what is not on a command line.
The write runs in its own goroutine: a prompt larger than the pipe buffer
(~64 KiB) blocks mid-write until the child drains it, and the child cannot
drain while nothing reads its stdout. Closing stdin signals end-of-prompt, so
it is closed on both success and error paths, and on cancellation to release
a blocked write. Write failures surface in the result diagnostic, ranked
below explicit agent errors so an early child exit (bad auth, bad flag) is
not masked by the resulting EPIPE.
Tests: a prompt carrying the exact `-ldflags "-X ..."` shape must arrive
byte-for-byte on stdin and appear nowhere in argv; a 512 KiB prompt must not
deadlock; the prompt is written verbatim (the CLI trims it, we do not). Both
new unix tests fail against the pre-fix code. A Windows-tagged test drives a
real PowerShell host through the same .cmd -> -File rewrite.
Closes#5649
Co-authored-by: multica-agent <github@multica.ai>
* test(agent/cursor): prove the Windows launcher fix on both PowerShell hosts in CI
The stdin fix has to hold on the host that actually exhibits the bug.
powershell.exe (5.1) and pwsh <= 7.2 default to Legacy native argument
passing; pwsh >= 7.3 defaults to Standard. A fix verified only on the newer
host would not be a fix for the reporter.
Run the shim probe against every PowerShell host on PATH rather than only the
one defaultPowerShellLookup would select, and hook the windows-tagged launcher
tests into the existing windows-execenv CI job. These tests are windows-tagged
and the backend job runs on ubuntu, so until now they ran nowhere.
Co-authored-by: multica-agent <github@multica.ai>
* test(ci): run Windows launcher tests verbosely so skips are visible
A skipped or unmatched test still reports "ok", which would make the
Windows job look like coverage it is not providing.
Co-authored-by: multica-agent <github@multica.ai>
* test(agent/cursor): close both coverage gaps in the #5649 regression tests
Two tests claimed guarantees they did not actually establish.
Windows: the fake cursor-agent.ps1 called [Console]::In.ReadToEnd() and wrote
the result itself, so it never launched a native child. The official shim ends
in `& node.exe index.js $args`, and that last hop is precisely where the bug
lives -- PowerShell re-serialises $args onto the child command line, and
whether the child inherits stdin was left unproven. The fake ps1 now
re-executes the test binary as a real native child (helper-process idiom),
which records the argv it actually received and drains stdin. Both PowerShell
hosts still run.
Interlock: the large-prompt fake drained stdin before writing any stdout, so
the child always unblocked the parent immediately and the test passed even
against a synchronous write -- it could not fail for the reason it existed.
The fake now floods stdout past pipe capacity *before* reading stdin, creating
the real mutual block. Verified: with the writer made synchronous the test
deadlocks to its 30s timeout, and passes only with the concurrent writer.
Also adds the missing cancellation case: a child that never reads stdin leaves
the writer blocked forever, so cancelling the context must close stdin,
release the writer and settle the run as aborted.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): run Codex unsandboxed on Windows to stop reject-by-policy (MUL-4957)
Windows has no Landlock/Seatbelt-equivalent filesystem sandbox that the
daemon configures, so the per-task `sandbox_mode = "workspace-write"` it
wrote was unenforceable. Worse than having no sandbox, it pushed Codex
into rejecting non-safe mutation commands "by policy": `multica issue
create` fails with "was rejected by policy" because Codex can neither
sandbox the command nor (under approval_policy = "never") escalate it to
the daemon's auto-approver, so the request never reaches the approver.
Mirror the existing macOS fallback and give Windows danger-full-access so
those commands run. Also generalize the danger-full-access warn log so it
no longer hardcodes "on macOS" and only surfaces the macOS-specific
upgrade hint on macOS (new codexSandboxPolicy.Hint field).
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): correct Windows sandbox rationale and respect user windows.sandbox (MUL-4957)
Addresses two review must-fixes on #5672:
1. Correct a false security fact. The comments and log Reason claimed
Windows has no filesystem sandbox backend. Codex 0.144.5 does ship a
native Windows sandbox (windows.sandbox = "unelevated"/"elevated"); it
is experimental with open upstream reliability bugs, so the daemon
defaults to danger-full-access as a deliberate compatibility choice.
Enabling the native sandbox is tracked as separate follow-up work.
2. Stop silently downgrading users who opted into isolation. The fallback
was unconditional. Add codexSandboxPolicyForConfig: on Windows an
explicit windows.sandbox = unelevated|elevated keeps workspace-write so
Codex enforces task isolation with the user's chosen backend;
danger-full-access applies only when windows.sandbox is absent,
disabled, or unparseable. This is also the branch point for a future
native-sandbox rollout (flip the default; callers unchanged).
Adds fixture tests locking the priority (user opt-in kept vs. unconfigured
fallback) plus predicate coverage for codexSandboxPolicyForConfig.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): fail closed on undecidable Windows sandbox config, honor -c windows.sandbox (MUL-4957)
Second review round on #5672. Two must-fixes.
1. Undecidable config no longer fails open. The old bool detector collapsed
"unparseable / invalid value / failed copy" into "unconfigured" and then
loosened to danger-full-access. Replaced with a tri-state
(absent/native/undecidable): only exact-lowercase unelevated|elevated (the
sole values Codex accepts — verified: any other value makes Codex refuse to
load the config) counts as native; any other present value, unparseable
TOML, a read error, or a missing per-task config when a shared
~/.codex/config.toml exists (i.e. the copy failed) is undecidable and fails
closed to workspace-write — it never loosens — logged at error level.
2. windows.sandbox set via `-c`/`--config` custom args is now honored. Such
args never land in config.toml, so config-only detection silently
downgraded those users' isolation. The effective Codex args (daemon
defaults + profile-fixed + per-agent custom_args) are threaded through
PrepareParams/ReuseParams/CodexHomeOptions into the sandbox decision and
scanned for a windows.sandbox override (inline, two-token, quoted, spaced;
last-wins).
Also drops issue-status-bound source comments (openai/codex#24098 has since
closed). Adds unit coverage for config/args classification, the fold
precedence (undecidable > native > absent), and the copy-failed fail-closed
path.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): fail closed on config-sync errors and honor shell-quoted -c windows.sandbox (MUL-4957)
Round-3 review must-fixes:
1. resolveWindowsSandboxState now takes the config.toml sync error and a
tri-state shared-config presence instead of re-stat-ing inside. A failed
sync (stale/absent per-task copy) or an un-stat-able shared source is
undecidable and keeps workspace-write, closing the fail-open where a failed
sync was read as "unconfigured". Splits IO from the decision so the paths
are unit-testable without faulting the filesystem.
2. The Windows sandbox decision consumes agent.NormalizeCodexLaunchArgs (the
shared helper buildCodexArgs now uses) so a shell-quoted -c windows.sandbox
opt-in is normalized identically to launch, instead of being missed by a
raw-token scan and silently downgraded.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): abort when the Codex sandbox block cannot be written (MUL-4957)
Round-4 review must-fix: ensureCodexSandboxConfig failures were warn-and-continue,
so a computed fail-closed workspace-write policy could stay only in memory while
config.toml kept a stale danger-full-access from a prior run — the decision
failed closed but the effective config failed open.
prepareCodexHomeWithOpts now returns the error, which blocks startup on both
paths: fresh Prepare fails the task, and Reuse leaves env.CodexHome unset, which
configureCodexTaskShellEnvironment already refuses to start.
Regression covers the full reuse scenario (stale danger-full-access + failed
config sync + failed managed-block write); it fails with "got nil" without the
fix.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
Co-authored-by: J <j@multica.ai>
When agent.model equals the model the Hermes ACP session already reports
as current, Multica was still replaying session/set_model. Hermes'
set_model re-runs provider auto-detection on the model id, which for a
provider:model id whose parsed provider matches the session's current
provider (e.g. a named custom endpoint exposed as "custom") can mis-route
to a different provider — custom:deepseek-v4-pro resolves into the
OpenRouter catalog and the turn fails with an auth error. Only the retry
via session resume, where the corrupted provider no longer matches the
custom: prefix, succeeds.
Capture the runtime's reported currentModelId from session/new and
session/resume and skip the redundant switch when it already equals the
requested model. An empty/unparsable current model falls through and
still sends set_model, preserving prior behaviour. The explicit
provider:model id is never rewritten. Upstream: NousResearch/hermes-agent#59089.
MUL-5029
Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
The windows-execenv job's TestPrepareIsolated_WindowsKillsDescendantBeforeRetry
flakes on the Windows runner: its helper subprocess ends in a bare select{},
which the Go runtime can reap with 'all goroutines are asleep - deadlock!'
(exit status 2) once every goroutine is parked with no wakeup source. That
races the parent's Job Object kill, so PrepareIsolated returns 'helper failed'
instead of the context.Canceled the test asserts.
Block on a timer-backed sleep loop instead: a pending timer is a wakeable
source, so the runtime never declares a deadlock, and the process still dies
the instant the Job Object tears the tree down.
Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
* feat(issues): bump issue updated_at when a comment is added (MUL-5009)
A new comment now counts as activity on its issue and advances
updated_at, so the "Updated date" Kanban/list sort surfaces
recently-discussed cards — not only cards whose status changed.
Applies to all three comment-creation paths (user/agent HTTP,
agent task delivery, and the child-done system comment) via a
best-effort TouchIssue query. The bump never fails an already-
persisted comment; it self-heals on the next activity if it errors.
Co-authored-by: multica-agent <github@multica.ai>
* fix(issues): make comment updated_at bump atomic (MUL-5009 review)
Address Elon's review. Move the updated_at bump into CreateComment as a
leading data-modifying CTE so the comment insert and the timestamp bump
commit or roll back together — closing the non-atomic window where a
comment could persist while updated_at stayed stale. That window also
skewed the daemon GC TTL, which reads issue.updated_at to reclaim
done/cancelled workdirs.
Centralizing the bump in the query drops the three per-caller TouchIssue
calls and guarantees any future comment entrypoint inherits it.
Also refresh the now-stale gc.go / gc_test.go comments that asserted
'CreateComment does not bump issue.updated_at'.
Co-authored-by: multica-agent <github@multica.ai>
* fix(issues): make comment/issue workspace match a query-level guarantee (MUL-5009 nit2)
The touch CTE now RETURNING id, workspace_id and the INSERT SELECTs from
it, so the comment insert depends on the issue actually existing in the
passed workspace. A mismatched (issue, workspace) pair matches 0 rows in
the CTE, the dependent INSERT selects nothing, and the :one query returns
pgx.ErrNoRows — no mis-attributed comment is written and the issue is not
touched.
CreateComment is now the single carrier of the 'a comment belongs to an
issue in the same workspace and always bumps it' invariant, so no future
caller can break it by passing the wrong workspace. Signature unchanged;
no migration or foreign key.
Add TestCreateComment_WorkspaceMismatchPersistsNothing (error returned,
no comment persisted, updated_at unchanged).
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): bound pre-start task preparation
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): isolate pre-start env preparation
Run execution-environment Prepare and Reuse in a killable helper process so a timed-out attempt cannot keep writing after retry. Add FIFO lifecycle and squad Stage retry regression coverage.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): terminate Windows prepare process trees
Assign the pre-start helper to a kill-on-close Job Object before releasing its request, wait for all job members to exit on cancellation, and add a Windows runtime regression job.
Co-authored-by: multica-agent <github@multica.ai>
* ci: target Windows prepare tree regression
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
Classify outbound replies by task origin (chat_input_task_id) before consulting a channel binding, so web/mobile direct-chat completions and failures stay inside Multica even when the session was originally created from Lark or Slack. Channel-created tasks continue to deliver. Closes#5644.
* feat(issues): add configurable table view
Co-authored-by: multica-agent <github@multica.ai>
* test(issues): cover table columns in page fixture
Co-authored-by: multica-agent <github@multica.ai>
* fix(issues): make table column picker interactive
Co-authored-by: multica-agent <github@multica.ai>
* fix(issues): repair quick create and virtualize table rows
Co-authored-by: multica-agent <github@multica.ai>
* fix(issues): keep pinned table cells opaque
Co-authored-by: multica-agent <github@multica.ai>
* fix(issues): anchor full-width table rows
Co-authored-by: multica-agent <github@multica.ai>
* fix(issues): consolidate table controls
Co-authored-by: multica-agent <github@multica.ai>
* fix(issues): harden table pagination and export
Co-authored-by: multica-agent <github@multica.ai>
* feat(issues): add table quick search
Co-authored-by: multica-agent <github@multica.ai>
* fix(issues): make table window filters, selection, and export authoritative
Round-2 review fixes for the issues table (MUL-4797):
- Send the agents-working filter as a server ids facet so matches on
unfetched pages surface and total/pagination/export agree; a present-
but-empty id list yields an empty window instead of an unfiltered one.
- Reset surface selection when the membership window changes and act on
selection ∩ visible rows in the batch toolbar, so batch actions,
Export selected, and the count all share one authoritative set.
- Materialize the full flat window while table grouping is active, and
suspend hierarchy nesting / parent-based grouping until the window is
complete so structure cannot reshuffle as pages arrive; suppress
header facet-count badges while the table window is partial.
- Resolve actor directories and the property catalog at export time and
fail the export instead of writing Unknown* actors or dropping
configured property columns on cold/errored lookups.
- Append a unique id tie-break to the list/grouped ORDER BY and mirror
it in compareIssuesForSort so offset pages are stable across
same-timestamp ties.
Co-authored-by: multica-agent <github@multica.ai>
* fix(issues): bound table structure window and align chip/transport/selection
Round-3 review fixes for the issues table (MUL-4797):
- Cap whole-window materialization at TABLE_STRUCTURE_MAX_WINDOW (1000):
below it the remaining pages load automatically — hierarchy applies
without scrolling to the last page — and above it grouping/hierarchy
suspend with an explicit toolbar notice instead of triggering an
unbounded workspace download from a persisted view option.
- Give the agents-working chip the authoritative in-window running set
(the ids-facet window query, shared key with the filter-on state) so
its badge can no longer say 0 while the filter would find matches on
unfetched pages; falls back to loaded-row scoping elsewhere.
- Route ids-facet windows through a new POST /api/issues/query twin —
hundreds of running-issue UUIDs overflow the ~8 KB GET request-line
budget of common proxies. The body carries the same key/value pairs;
the handler rebuilds the query string and delegates to ListIssues.
- Reset surface selection during render (key-change pattern) instead of
a post-commit effect, so no frame ever pairs new membership with the
old selection.
Co-authored-by: multica-agent <github@multica.ai>
* fix(issues): harden table auto-pagination against errors and stale totals
Round-4 review fixes for the issues table (MUL-4797):
- Stop the structure materialization loop (and the scroll sentinel) when
the window query is in error state — a persistently failing page left
hasNextPage true and isFetchingNextPage false after every attempt, so
the ungated effect refired forever. Resuming is an explicit toolbar
Retry. The advancement decision now lives in a pure, tested
shouldAutoLoadNextStructurePage helper.
- Make the structure ceiling a hard stop: the ceiling check reads the
LATEST page's total (pagination already advances on it, so a stale
small page-1 total could re-open unbounded materialization), and the
loop additionally halts on loaded count >= ceiling regardless of any
reported total.
- Drive the working (ids-facet) window to completion — it is inherently
bounded by the running set — and treat it as the chip's authoritative
scope only when complete, so >100 running issues no longer under-count
as a single page.
Co-authored-by: multica-agent <github@multica.ai>
* fix(issues): make working-window pagination capped and unknown-aware
Round-5 (final) review fixes for the issues table (MUL-4797):
- The working (ids-facet) window now advances through the same
shouldAutoLoadNextWindowPage gates as the structure loop — it shares
the main table's cache key while the agents-working filter is on, so
an uncapped chip-driven loop re-opened the very ceiling the table just
enforced. An over-ceiling window stops after page one.
- A cold-load failure of the flat window is an ERROR state, not an empty
workspace: isEmpty only claims empty on a successful zero-result
fetch, and the surface renders a dedicated failed-to-load state with a
reachable Retry (the in-table Retry never mounted without data).
- The chip scope is now tri-state honest: a COMPLETE window (or an empty
running set) yields a precise count, keepPreviousData carries the
last-known-complete set across re-keys, and everything else — cold
resolving, failed, over the ceiling — presents as an explicit unknown
('Agents working: —') instead of a number derived from whichever
incomplete window happened to be loaded.
Co-authored-by: multica-agent <github@multica.ai>
* fix(issues): single pagination owner and placeholder-honest chip scope
Round-6 review fixes for the issues table (MUL-4797):
- Exclude placeholder data from the working-window completeness gate:
on a re-key (running set or facet change) keepPreviousData leaves the
OLD key's rows visible, and pairing them with the new task snapshot
published a precise-looking number for a scope nobody fetched. The
scope now reads unknown until the new key resolves.
- Make the shared table query single-owner while the agents-working
filter is on: the chip's background loop no longer answers the same
render snapshot as TableView's structure loop, and every auto caller
(structure loop, working loop, scroll sentinel, retry) now uses
fetchNextPage({cancelRefetch: false}) so a concurrent responder
no-ops instead of cancel/restarting a fetch whose HTTP request is not
abortable — which had been duplicating every offset.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Lambda <lambda@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): isolate Linux Codex git metadata (MUL-4925)
Co-authored-by: multica-agent <github@multica.ai>
* refactor(daemon): address isolated-checkout review nits (MUL-4925)
- rename sameFilesystemPath -> sameResolvedPath (it compares resolved
paths for equality, not same-device), with a clarifying doc comment
- prune earlier tasks' agent/* branches when reusing an isolated
checkout so a long-lived reused workdir stops accumulating one local
branch per checkout; deleteLocalBranches now takes a keepBranch arg
and the prune is non-fatal
- cover the prune in TestCreateWorktreeReusesIsolatedGitMetadata
* fix(repocache): preserve user branches on reuse (MUL-4925)
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Eve <eve@multica-ai.local>
Co-authored-by: multica-agent <github@multica.ai>
* fix(task): auto-retry transient provider stream cut like chat (MUL-4910)
Claude Code's "API Error: Connection closed mid-response" is a transient
network cut. In unattended issue runs it fell through to
agent_error.unknown / process_failure — neither in retryableReasons — so
the task terminated. Interactive chat only appeared resilient because the
CLI's own in-process retry usually recovers first; there is no
chat-specific retry in Multica. Both paths share the same
finalizeStreamResult -> Classify -> retryableReasons pipeline.
- classify: route "connection closed" / "mid-response" to
agent_error.provider_network, before the exit-status rule so the
"exited with error: exit status N" variant also lands here.
- retry: add provider_network to retryableReasons. It is resume-safe, so
the retry child inherits the session and continues the truncated
conversation instead of restarting.
Tests cover the new classification (incl. exit-status variant) and the
retry/resume flags. Note: mirror the substrings into the MUL-1949 offline
backfill SQL to keep in-flight and historical taxonomies aligned.
Co-authored-by: multica-agent <github@multica.ai>
* feat(task): defer provider_network's final retry ~5s — three-tier (MUL-4910)
Follow-up to the immediate-retry fix: make the connection-closed retry a
three-tier schedule — first run + immediate retry + one retry deferred ~5s —
so a blip that survives the immediate retry gets a short cooldown before the
final attempt instead of firing back-to-back.
Reuses the existing deferred/fire_at primitive (the comment-routing escalation
mechanism) rather than adding new infrastructure: CreateRetryTask gains an
optional fire_at; when set, the child is inserted 'deferred' and the existing
PromoteDueDeferredTasksForRuntime sweeper — already run promote-first on every
claim poll — flips it to 'queued' at fire_at. No migration, no claim-query
change, no daemon change.
- retryAttemptCeiling: raise provider_network's ceiling to 3 (other reasons
keep max_attempts=2); applied in both retryEligible and
MaybeRetryFailedTask's budget pre-check so the primary and sweeper paths agree.
- retryDelayForAttempt: only provider_network's final attempt is deferred (5s);
every other retry — including provider_network's first — stays immediate.
- FailTask + MaybeRetryFailedTask pass fire_at and skip the queued
broadcast/notify for a deferred child (promotion emits them at fire time).
Timing: the deferred child fires on the first claim poll at/after fire_at, so
>= 5s; on an otherwise-idle runtime it can stretch to the poll interval — the
same behaviour deferred escalations already have.
Tests: pure schedule/eligibility coverage (TestProviderNetworkRetrySchedule)
plus a DB test asserting fire_at controls deferred-vs-queued, attempt=3, and
resume-safety.
Co-authored-by: multica-agent <github@multica.ai>
* fix(task): persist reason-aware retry budget; respect max_attempts=1 disable (MUL-4910)
Addresses the pre-merge review must-fix: retryAttemptCeiling raised
provider_network to 3 unconditionally, which (1) overrode the
max_attempts=1 "auto-retry disabled" contract and (2) persisted a
self-contradictory child (attempt=3, max_attempts=2) that leaks to the
task API, so a naive attempt < max_attempts consumer would misjudge the
budget.
- retryAttemptCeiling now returns taskMaxAttempts unchanged when it is <= 1
(disabled stays disabled) and only ever WIDENS otherwise (max(col, 3) for
provider_network) — a higher configured budget is kept.
- CreateRetryTask takes an optional max_attempts; FailTask and
MaybeRetryFailedTask write the reason-aware effective ceiling into the
child so the whole retry chain self-describes (attempt=3, max_attempts=3).
NULL inherits the parent column, so non-provider_network reasons are
unchanged.
Tests:
- pure: ceiling widens to 3, keeps a higher budget, and never revives a
disabled (max_attempts=1) task; eligibility rejects the disabled case.
- DB (end-to-end FailTask): default budget → deferred final child at
attempt=3/max_attempts=3; first failure → immediate child; max_attempts=1
→ no child. Plus CreateRetryTask persists the passed budget / inherits on NULL.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
Replace the free-form trigger-config form with a structured schedule
editor built on an orthogonal cron model: separate frequency, time, and
day-of-week/day-of-month dimensions map to and from cron expressions via
a dedicated grammar and mapping layer, with validation and a
human-readable describe() summary. The grammar suite drives the editor
against a combinatorially generated corpus of 51,755 distinct cron
expressions - every token form of every field, crossed - each judged
against a reference robfig/cron v3 parser.
Add a server-side /autopilot/cron-preview endpoint (plus schema and
React Query hook) so the editor shows upcoming run times, and echo
wildcard-carrying cron lists correctly instead of collapsing them.
Supporting pieces: timezone-aware formatting helper, segmented-toggle
and debounced-value utilities, a reworked time-input, and refreshed
en/ja/ko/zh-Hans locale strings.
Self-hosted upgrades to v0.4.3 failed closed on migration 198's VALIDATE of the strict attribution constraint, because the legacy rows migration 190 exempted were only backfilled out-of-band on cloud. Registers a pre-198 preMigrationHook that idempotently mirrors originator_user_id into accountable_user_id in batches before VALIDATE, with FOR UPDATE + repeated predicate to avoid clobbering concurrently-written rows, so a stuck-at-197 instance auto-heals on migrate up with no manual SQL. Originator-NULL rows are left untouched. Verified with unit + concurrency + end-to-end tests against real Postgres.
Fixes#5544
