- Multi-select UI for batch importing skills from a local runtime
- Server batch-dispatches up to 10 import requests per heartbeat cycle
- WS heartbeat now reads supports_batch_import from daemon payload
instead of hardcoding true, so old daemons correctly fall back to
one-at-a-time dispatch
- Raised server pending timeout to 3min and client poll timeout to 4min
to accommodate daemons that pop only one import per 15s heartbeat
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Local daemon previously logged mostly at Info, leaving startup/exit,
config resolution, registration, heartbeat ticks, agent invocation, and
result classification undiagnosable without code-reading. Add Debug
logs at those checkpoints so LOG_LEVEL=debug (the default) produces
enough detail to follow a run end-to-end without changing normal Info
output.
Co-authored-by: multica-agent <github@multica.ai>
* feat(daemon): force-stop hung agent runs via idle watchdog (MUL-2281)
A backend whose subprocess hangs on a stuck child process (e.g. claude
blocked on `docker ps` against a frozen dockerd) keeps the daemon's run
record at status="running" until the full DefaultAgentTimeout (2 h)
expires, because cmd.Wait() never returns and Session.Result is never
written. MUL-2225 spent 17+ minutes in this state in the wild.
Add a per-task idle watchdog around executeAndDrain:
- Wrap the caller's ctx so a single cancel propagates to the agent
subprocess (via the ctx passed to backend.Execute) AND the drain loop.
- Stamp lastActivityAt every time the drain loop receives a message.
- Tick at window/2; when idle_for >= window AND session.Messages buffer
is empty, set a fired flag and call cancel.
- Tag the resulting Result.Status as "idle_watchdog" so runTask routes
it through a dedicated failure_reason instead of "agent_error".
Default window is 5 min, configurable via MULTICA_AGENT_IDLE_WATCHDOG;
set to 0 to disable. Tests cover the activity-then-silence case, the
zero-message case, the disabled case, and the happy path.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): skip idle watchdog while a tool call is in flight
A legitimate long-running tool call (npm install, docker build, test
suite) can sit silent between tool_use and tool_result for many minutes.
Without this gate, the watchdog would yank the agent mid-build.
Track unmatched tool_use messages in an atomic counter; only let the
watchdog fire when the counter is zero. tool_result clamps non-negative
so a stray result with no matching use can't re-arm the watchdog one
call too early.
Adds two regression tests:
- DoesNotFireDuringInFlightToolCall: tool_use -> silence past
window -> tool_result -> completed (must NOT fire)
- FiresAfterToolResultIfBackendStaysSilent: tool_use -> tool_result
-> silence past window (MUST fire — backend really is stuck)
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
* feat(daemon): auto-update CLI when idle (MUL-2100)
Add a periodic poller that checks GitHub for a newer multica release
every hour and self-updates when the daemon is idle, reusing the same
brew-or-download upgrade path the Runtimes-page "Update" button already
runs.
- Refactor handleUpdate to call a shared runUpdate(target) helper so
both server-triggered and auto-triggered upgrades go through the same
brew detection + atomic replace + restart.
- New autoUpdateLoop gates each tick on: opt-out flag, Desktop launch
source, dev-build version, an in-flight update, and active tasks. The
idle gate guarantees we never interrupt a running agent — busy ticks
silently retry at the next interval.
- Config: MULTICA_DAEMON_AUTO_UPDATE=false to disable (also via
--no-auto-update), MULTICA_DAEMON_AUTO_UPDATE_INTERVAL to retune the
poll period.
- IsNewerVersion / IsReleaseVersion helpers in the cli package, with
tests covering patch/minor/major bumps, dev-describe strings, and
malformed input.
- Daemon-side tests cover every skip path (updating, active tasks,
fetch failure, no-newer) plus the success path that fires
triggerRestart while keeping the updating flag held to the end.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): close idle race + verify checksum in auto-update (MUL-2100)
Two issues raised in PR #2679 review:
1. The first idle check in tryAutoUpdate only ran before the release-metadata
fetch, so a poller that won the claim race during the fetch could end up
handing handleTask a task that triggerRestart was about to cancel via root-
ctx cancellation. Add a strict claim barrier: runRuntimePoller now
tryEnterClaim()s before ClaimTask, and tryAutoUpdate flips pauseClaims
under claimMu only after observing claimsInFlight + activeTasks == 0.
Pollers that were already mid-claim hold claimsInFlight > 0, so the barrier
refuses to engage and the update defers to the next tick.
2. The direct-download path replaced the running binary with whatever bytes
GitHub returned, without checking checksums.txt. Pull the manifest first,
buffer the archive, and reject on SHA-256 mismatch before extraction. The
GoReleaser config already publishes checksums.txt; we just consume it.
Also tighten parseReleaseVersion so it stops accepting dev-describe shapes
like "v0.1.13-5-gabcdef0" through the patch trim, matching its docstring.
The auto-update loop already guards on IsReleaseVersion, but the lenient
parser was a footgun and the existing test name even said "not newer" while
asserting the opposite.
Tests:
- TestTryAutoUpdate_DefersWhenClaimInFlightAtBarrier (new race coverage)
- TestTryAutoUpdate_HoldsBarrierAcrossRestart / ReleasesBarrierOnUpgradeFailure
- TestTryEnterClaim_RespectsBarrier
- TestFindChecksumManifestAsset / TestParseChecksumManifest / TestVerifyAssetSHA256
- TestIsNewerVersion: dev-describe cases now expect false (matches docstring)
Co-authored-by: multica-agent <github@multica.ai>
* chore(daemon): default auto-update poll interval to 6h (MUL-2100)
1h was overly chatty for a release that lands at most a few times a week.
Operators who want a different cadence can still set
MULTICA_DAEMON_AUTO_UPDATE_INTERVAL or --auto-update-interval.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
* feat(task): wire claim lease queries into TaskService and sweeper (MUL-2246)
- ClaimTask now uses ClaimAgentTaskWithLease (generates claim_token + lease)
- StartTask accepts optional claim_token for token-verified start
- AgentTaskResponse includes claim_token for daemon to use
- Daemon client sends claim_token in StartTask body
- Sweeper calls RequeueExpiredClaimLeases each tick
- Legacy daemons without claim_token still work (graceful fallback)
Co-authored-by: multica-agent <github@multica.ai>
* fix(task): address PR #2662 review blockers (MUL-2246)
1. ClaimAgentTaskForRuntime: push runtime_id into atomic SQL WHERE clause
so runtime A cannot claim tasks queued for runtime B under the same agent.
2. Legacy StartAgentTask: add claim_token IS NULL guard so leased rows
cannot be started without token verification. Handler rejects malformed
tokens with 400 instead of silently degrading to legacy path.
3. StartAgentTaskWithClaimToken: validate claim_expires_at >= now(),
preserve claim_token until terminal state (only clear claim_expires_at),
use CTE + UNION ALL for idempotent retry when daemon resends after a
lost StartTask response. Return 409 Conflict on token mismatch/expiry.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): StartTask 409 handling, transport retry, claim_token on FailTask (MUL-2246)
- StartTask 409 (claim superseded): release slot, don't call FailTask
- StartTask transport timeout/5xx: retry once with same token, then
check task status before failing
- FailTask now sends claim_token; server-side FailAgentTask SQL adds
AND (claim_token IS NULL OR claim_token = @claim_token) guard so
stale daemons cannot fail tasks that have been re-claimed
Co-authored-by: multica-agent <github@multica.ai>
* fix(task): close FailTask token bypass and RequeueExpiredClaimLeases liveness gap (MUL-2246)
Blocker 1 - FailTask token validation:
- SQL: change (param IS NULL OR claim_token = param) to
(param IS NULL AND claim_token IS NULL) OR claim_token = param
so tokenless requests can only fail legacy (tokenless) rows.
- task.go: malformed claim_token now returns ErrInvalidClaimToken (400)
instead of being silently dropped to NULL.
- Handler: maps ErrInvalidClaimToken→400, ErrClaimTokenInvalid→409.
- Service: when UPDATE returns no rows but task is still active,
return ErrClaimTokenInvalid (token mismatch) instead of silent success.
Blocker 2 - RequeueExpiredClaimLeases runtime liveness:
- SQL: JOIN agent_runtime, only requeue tasks where runtime is 'online'.
Dead/offline runtime tasks stay dispatched for FailTasksForOfflineRuntimes.
- FOR UPDATE → FOR UPDATE OF atq (required with JOIN).
Regression tests:
- task_claim_token_test.go: malformed, tokenless-on-tokened, wrong-token
- requeue_lease_test.go: SQL must JOIN agent_runtime with online filter
Co-authored-by: multica-agent <github@multica.ai>
* fix(task): move expired lease requeue to ClaimTaskForRuntime preflight, add heartbeat freshness backstop (MUL-2246)
- Add RequeueExpiredClaimLeasesForRuntime: per-runtime preflight self-requeue
in ClaimTaskForRuntime. Runtime proves liveness by actively claiming, so no
heartbeat check needed.
- Update global RequeueExpiredClaimLeases to require ar.last_seen_at freshness
(stale_threshold_secs param). Prevents requeuing to a dead runtime in the
90s gap between lease expiry (60s) and offline detection (150s).
- Add regression tests verifying the heartbeat freshness check and that the
preflight query does not join agent_runtime.
Co-authored-by: multica-agent <github@multica.ai>
* fix(task): use LivenessStore for global requeue, move preflight before empty-cache (MUL-2246)
Blocker 1: Global RequeueExpiredClaimLeases now uses LivenessStore.IsAliveBatch
to verify runtimes are truly alive before requeuing expired leases. When
LivenessStore is unavailable (no Redis), global requeue is skipped entirely —
the preflight self-requeue in ClaimTaskForRuntime handles live runtimes. This
closes the 60-150s gap where a dead runtime still appears online in DB.
Blocker 2: Moved RequeueExpiredClaimLeasesForRuntime BEFORE EmptyClaim.IsEmpty
fast-path in ClaimTaskForRuntime. Expired leases are now requeued (which bumps
the empty cache via notifyTaskAvailable) before the empty check can
short-circuit the claim path.
Also adds ListRuntimesWithExpiredClaimLeases SQL query and LivenessChecker
interface on TaskService.
Co-authored-by: multica-agent <github@multica.ai>
* fix(task): wire EmptyClaimCache into backend taskSvc for backstop requeue (MUL-2246)
The backend taskSvc used by the sweeper only had Liveness wired but not
EmptyClaim. When global backstop requeue called notifyTaskAvailable,
s.EmptyClaim.Bump() was a nil no-op — the handler's empty-cache was never
invalidated, so the daemon's next claim hit a stale empty verdict.
Fix: wire the same Redis-backed EmptyClaimCache into the backend taskSvc
in main.go (same Redis keys as router.go:139 handler instance).
Add regression test verifying backstop requeue invalidates the handler's
empty-cache.
Co-authored-by: multica-agent <github@multica.ai>
* fix(task): global backstop must not requeue — alive runtimes use preflight, dead stay dispatched (MUL-2246)
- RequeueExpiredClaimLeases is now a no-op (returns 0 always)
- Alive runtimes self-requeue via ClaimTaskForRuntime preflight
- Dead runtimes stay dispatched for FailTasksForOfflineRuntimes
- Rewriting to queued on dead runtime creates 2h blackhole (offline
sweeper only handles dispatched/running)
- Test actually calls RequeueExpiredClaimLeases and asserts 0 in all cases
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): remove duplicate usage reporting block after merge conflict (MUL-2246)
The merge resolution introduced a second ReportTaskUsage call after the
status check, duplicating the usage-before-early-return block that already
runs right after runner.run. Remove the duplicate and add a regression test
asserting /usage is called exactly once on the normal completion path.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
handleTask had two early-return paths that ran before ReportTaskUsage:
the cancelledByPoll select and the post-run GetTaskStatus check. Both
silently discarded any usage accumulated by the agent — and both
claude.go and codex.go populate Result.Usage even when runCtx is
cancelled mid-run, so cancelled tasks consistently under-reported tokens.
Hoist ReportTaskUsage to run immediately after the runner returns,
before any early-return path. Add a taskRunner interface seam and a
cancelPollInterval field so tests can inject a fake runner and trigger
the poll-cancellation path on a 10ms ticker without spawning real agents.
Two regression tests cover both leak windows:
- TestHandleTask_ReportsUsageBeforeCancel: post-run /status returns
"cancelled"; usage must be reported before the status check.
- TestHandleTask_ReportsUsageWhenCancelledByPoll: poll goroutine fires
first and cancels runCtx; runner returns usage on Done; assert
poll-status precedes usage (proving the cancelledByPoll branch was
the one exercised, not the post-run path).
Sanity-checked: reverting only the ReportTaskUsage hoist fails both
tests with the original "tokens lost" message.
MUL-2258
Co-authored-by: Jiang Bohan <bhjiang@outlook.com>
Co-authored-by: multica-agent <github@multica.ai>
* feat(execenv): native OpenClaw skill discovery via per-task config
MUL-2213 stopped lying about native discovery and routed openclaw skills
to .agent_context/skills/ — a path openclaw's scanner never reads.
Multica skills attached to openclaw-backed agents were still invisible to
the runtime; the AGENTS.md fallback was only a documentation patch.
OpenClaw's skill scanner walks <workspaceDir>/skills/ (plus a few other
roots), and workspaceDir is resolved from the openclaw config file —
specifically agents.list[id].workspace → agents.defaults.workspace →
~/.openclaw/workspace. There is no CLI flag or env var override on the
agent runtime; the only knob is the config file.
This change wires a per-task synthesized config:
1. execenv.prepareOpenclawConfig deep-copies the user's existing
openclaw.json (priority: $OPENCLAW_CONFIG_PATH, else
~/.openclaw/openclaw.json), rewrites agents.defaults.workspace AND
every agents.list[].workspace to the task workdir, and writes the
result to {envRoot}/openclaw-config.json. Provider sections,
registered agents, model providers, gateway settings — everything
openclaw needs to actually start — are preserved as-is.
2. resolveSkillsDir for "openclaw" now points at {workDir}/skills/,
which is the first path openclaw scans under workspaceDir. Skills
written here are picked up natively.
3. daemon.go exports OPENCLAW_CONFIG_PATH={env.OpenclawConfigPath} on
the openclaw subprocess and adds OPENCLAW_CONFIG_PATH to the
custom_env blocklist so users cannot accidentally override it.
4. buildMetaSkillContent now lists openclaw alongside the
"discovered automatically" providers; the .agent_context/skills/
fallback line stays for gemini/hermes.
The new regression test TestPrepareOpenclawSkillWriteMatchesScanPath is
the one MUL-2219's DoD calls out: it resolves the workspaceDir the way
openclaw does (reading agents.defaults.workspace out of the synthesized
config) and proves {workspaceDir}/skills/<name>/SKILL.md is what Multica
actually wrote. The pre-MUL-2219 fix asserted "we wrote a file" without
checking the scanner would ever see it — which is how the dead drop into
.openclaw/skills/ landed in #2621's first commit.
Verified locally: minimum-viable synthesized config validates via
`openclaw config validate`, and `OPENCLAW_CONFIG_PATH=<path> openclaw
config get agents.defaults.workspace` returns the task workdir as
expected. MUL-2219
Co-authored-by: multica-agent <github@multica.ai>
* fix(execenv): delegate openclaw config parsing to CLI and fail closed
Address Elon's must-fix on PR #2628: the previous implementation parsed
~/.openclaw/openclaw.json with encoding/json, which cannot read JSON5
or follow $include — the OpenClaw spec's actual format. When parsing
failed, prepareOpenclawConfig silently emitted a minimal config, which
could boot OpenClaw without the user's registered agents, model
providers, or API keys.
Two changes:
1. Delegate active-config-path resolution and config reading to the
openclaw CLI itself. `openclaw config file` locates the active
config (covering OPENCLAW_CONFIG_PATH / OPENCLAW_STATE_DIR /
OPENCLAW_HOME / default and the legacy chain), and the wrapper we
write uses $include to point at it so OpenClaw's own loader handles
JSON5, $include nesting, env-substitution, and secret refs. We read
only agents.list via `openclaw config get --json` to rewrite each
entry's workspace — secrets, comments, and includes in the user
config are never touched.
2. Remove the silent minimal-config fallback. Any CLI failure,
malformed output, or write error now surfaces as a hard error from
Prepare / Reuse. The only "synthesize minimal" path left is a fresh
install (CLI reports a path but the file doesn't exist), where
there is no user data to lose.
The per-task override still rewrites every agents.list[].workspace,
not just agents.defaults.workspace — this is intentional task
isolation, documented in prepareOpenclawConfig and the PR body. A
host-scope per-agent workspace would otherwise silently route the
scanner back to the user's shared workspace.
Cleanups Elon flagged in the same review:
- daemon.go inline-system-prompt comment no longer claims openclaw
ignores the task workdir; it does load it now, and the inline brief
is a belt-and-suspenders carryover for older releases.
- execenv.go openclaw block no longer references "skill file paths in
the inline brief" — the brief uses "discovered automatically".
Reuse() switches to a ReuseParams struct so the openclaw binary path
threads through alongside CodexVersion without a 6th positional arg.
MUL-2219
Co-authored-by: multica-agent <github@multica.ai>
* fix(execenv): grant OpenClaw $include cross-dir confinement for per-task wrapper
The per-task wrapper at envRoot/openclaw-config.json $includes the user's
active config (typically ~/.openclaw/openclaw.json), but OpenClaw confines
$include resolution to the wrapper file's directory unless the target's
parent is granted via OPENCLAW_INCLUDE_ROOTS. Without this, OpenClaw refuses
to follow the link at runtime and the wrapper boots with no user-registered
agents.
prepareOpenclawConfig now returns dirname(activePath) as IncludeRoot, and
the daemon prepends it to whatever the user already has in
OPENCLAW_INCLUDE_ROOTS via the new composeOpenclawIncludeRoots helper
(dedupes, drops empty segments, preserves user-configured roots). Fresh
install emits no $include and leaves the env var untouched.
Adds OPENCLAW_INCLUDE_ROOTS to the custom_env blocklist so a per-agent
override cannot strip the granted root.
Regression tests:
- TestPrepareOpenclawConfigWrapperLoadableUnderIncludeConfinement asserts
every $include target's dirname is covered by the IncludeRoot we surface.
- TestPrepareEnvironmentOpenclawWiresIncludeRoot covers the non-fresh-install
Environment wiring.
- TestComposeOpenclawIncludeRoots covers the daemon-side env composition
(preserve, dedupe, drop empties).
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
* MUL-2215: fix(daemon): close handleRuntimeGone success/straggler race
handleRuntimeGone coalesced concurrent recoveries with a per-workspace
`reregisterNextAttempt` slot that was deleted immediately on success. A
late-arriving goroutine whose `removeStaleRuntime` was delayed by mutex
contention could reach the coalesce gate after the winner cleared the
slot, observe no slot, re-claim, and double-register — the source of the
intermittent `register endpoint called 2 times under stampede, want 1`
failure on PR #2348.
The slot delete on success is intentional (a genuinely later distinct
deletion in the same workspace must register again, validated by
TestHandleRuntimeGone_DistinctDeletionsWithinCoalesceWindowBothRecover),
so we can't just extend the slot's lifetime.
Add a second per-workspace gate: `reregisterLastCompletedAt`. Every call
captures `entryAt` at the top of handleRuntimeGone; at the coalesce gate
a caller bails if `lastCompletedAt >= entryAt`, i.e. a peer's register
completed AFTER we entered the function. Same-wave stragglers bail
deterministically; distinct later events have `entryAt > lastCompletedAt`
and proceed.
Extracted the gate into `tryClaimRegisterSlot` / `recordRegisterCompletion`
so the race can be exercised deterministically with synthetic timestamps
instead of relying on `-count=N` to win the scheduling lottery.
- TestHandleRuntimeGone_CoalescesConcurrentCallers: -count=500 -race
clean (previously intermittent).
- New unit tests cover the straggler bail, the distinct-later-event
claim, failure backoff suppression, and peer-holds-slot coalescing.
Co-authored-by: multica-agent <github@multica.ai>
* MUL-2215: narrow completion stamp to success path
Second review caught that recordRegisterCompletion stamped
lastCompletedAt on both success and failure. A failed register has not
covered any workspace state, so a same-wave straggler whose entryAt
predates the failure must be allowed to retry once the failure backoff
expires — the previous behavior would let the failure-time stamp also
hide that straggler. workspaceSyncLoop only retries when a workspace's
runtimeIDs fully drain, so partial-deletion recovery has to come from
the straggler path.
Failure path now only updates reregisterNextAttempt; success path keeps
its existing stamp + slot clear. Add a regression test covering the
entryAt-before-failed-completion / arrival-past-backoff edge.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
The runtime prompt's Output section unconditionally required all tasks to
post a comment via 'multica issue comment add', which conflicted with the
squad leader protocol that says to 'exit silently' on no_action.
Changes:
- Add IsSquadLeader bool to TaskContextForEnv (detected via Squad Operating
Protocol marker in agent instructions)
- Relax the Output section and assignment-triggered workflow step 5 to
allow squad leaders to exit with only a 'multica squad activity' call
when the outcome is no_action
Fixes MUL-2168
Co-authored-by: multica-agent <github@multica.ai>
When an agent completes successfully (exit 0) but produces no text
output, the daemon incorrectly classified it as 'blocked'. This is
wrong — agents can legitimately complete work via tool calls (posting
comments, pushing code) without emitting text output.
Change the empty-output path to return status=completed so the task
is correctly reported as successful.
Fixes MUL-2104
Co-authored-by: yushen <ldnvnbl@gmail.com>
Co-authored-by: multica-agent <github@multica.ai>
Three user reports converge on the same Windows-shell encoding bug:
- #2198 / #2236 — Chinese, Codex on Win11. Comments / descriptions
generated by the agent arrive as `?`.
- #2376 — Cyrillic, non-Codex agent ("Ops Lead") on Win11 Desktop.
Title preserved (argv → CreateProcessW UTF-16), description / agent
reply garbled (stdin → shell-codepage re-encoding).
woodcoal's independent diagnosis on #2198 confirms the root cause:
Windows PowerShell 5.1's `$OutputEncoding` defaults to ASCIIEncoding
when piping to a native command, so non-ASCII bytes are silently
replaced with `?` before they reach `multica.exe`. The CLI's stdin
parsing is fine; the bytes are corrupted upstream, in the agent's
shell layer.
This PR ships the fix that supersedes the codex-only attempt in
PR #2265 (which is closed in favour of this one):
## CLI
Add `--content-file <path>` to `multica issue comment add` and
`--description-file <path>` to `multica issue {create,update}`. The
CLI reads bytes off disk via `os.ReadFile` and skips the shell
entirely; UTF-8 survives end-to-end regardless of `$OutputEncoding`
or `chcp`. The three input modes (`--content`, `--content-stdin`,
`--content-file`) are mutually exclusive.
## Runtime config
`buildMetaSkillContent`'s Available Commands section is rewritten as a
neutral three-mode menu. The previous unconditional "MUST pipe via
stdin" / `--description-stdin` mandate (over-spread from #1795 /
#1851's Codex-multi-line fix) is gone for non-Codex providers; the
strong directive now lives only in the Codex-Specific section, which
branches on host:
- Codex / Linux+macOS: `--content-stdin` + HEREDOC (preserves MUL-1467
fix against codex's literal `\n` habit).
- Codex / Windows: `--content-file` (PowerShell ASCII pipe is the
exact bug we're patching).
## Per-turn reply template
`BuildCommentReplyInstructions` now takes a provider arg and branches
provider × OS:
- Windows + any provider → `--content-file` (the bug is shell-layer,
not provider-layer; #2376 shows non-Codex agents on Windows also
hit it). All providers write a UTF-8 file with their file-write tool
and post via `--content-file ./reply.md`.
- Linux/macOS + Codex → stdin/HEREDOC (MUL-1467 protection).
- Linux/macOS + non-Codex → lightweight pre-#1795 inline
`--content "..."`. The CLI server-side decodes `\n`, so escaped
multi-line works; the agent retains stdin / file as escape hatches
for richer formatting.
`BuildPrompt` and `buildCommentPrompt` gain a `provider` arg;
`daemon.runTask` already has it in scope.
## Tests
- `TestResolveTextFlag` — file-source verbatim with non-ASCII
(`标题 / Заголовок / 中文段落`), missing-file error, empty-file
rejection, three-way mutual exclusion.
- `TestInjectRuntimeConfigAvailableCommandsIsNeutral` — every
non-Codex provider × {linux, darwin, windows} pins the three-mode
menu present + over-spread "MUST stdin" substrings absent.
- `TestInjectRuntimeConfigCodexLinuxEmphasizesStdin` +
`TestInjectRuntimeConfigCodexWindowsUsesContentFile` — Codex
section's per-OS branch.
- `TestBuildCommentReplyInstructionsCodexLinux` +
`TestBuildCommentReplyInstructionsNonCodexLinux` +
`TestBuildCommentReplyInstructionsWindowsUsesContentFile` — the
reply-template provider × OS matrix.
- `TestInjectRuntimeConfigWindowsCommentTriggerHasNoStdin` — end-to-end
AGENTS.md / CLAUDE.md on Windows has no prescriptive stdin
directive, for claude / codex / opencode.
`go test ./...` and `go vet ./...` clean.
Closes#2198, #2236, #2376.
Co-authored-by: multica-agent <github@multica.ai>
* feat: per-runtime timezone for token usage aggregation
The runtime token-usage charts (daily and hourly tabs on the
runtime-detail page) bucketed every event by the Postgres session
timezone, which is UTC in production. For an operator in UTC+8 that
meant a Tuesday afternoon's tasks landed in Tuesday early-morning's
bar — the chart was always one off.
Fix: store an IANA timezone on agent_runtime and aggregate under it.
* migrations 081 / 082 add agent_runtime.timezone (TEXT NOT NULL
DEFAULT 'UTC') and rebuild the rollup pipeline (window function
and both trigger functions) to compute bucket_date with
AT TIME ZONE rt.timezone instead of bare DATE().
* No historical backfill — task_usage_daily rows already on disk
keep their UTC bucket_date; only future writes / re-touches
recompute under the new tz. (Product call from MUL-1950: 'guarantee
future correctness'.)
* runtime_usage.sql gains a @tz parameter on ListRuntimeUsage and
GetRuntimeUsageByHour and threads tz through GetRuntimeTaskHourly Activity. ListRuntimeUsageDaily reads bucket_date as-is since the
rollup already wrote it in tz.
* parseSinceParamInTZ replaces the raw N×24h cutoff with start-of-
day-N in the runtime's tz so 'last 7 days' lines up with bucket
boundaries.
* Daemon registration sends the host's IANA tz (TZ env, then
time.Local), and UpsertAgentRuntime preserves any user override
via a CASE-on-existing-value pattern so a daemon reconnect can't
silently revert the operator's setting.
* New PATCH /api/runtimes/:id endpoint (UpdateAgentRuntime) lets
the runtime detail page edit the tz; the editor seeds with the
browser tz on first interaction.
Refs: MUL-1950
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: multica-agent <github@multica.ai>
* fix: harden runtime timezone rollups
Co-authored-by: multica-agent <github@multica.ai>
* fix: address runtime timezone review nits
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Eve <eve@multica.ai>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: multica-agent <github@multica.ai>
Co-authored-by: Eve <eve@multica-ai.local>
The inline path now carries the full runtime brief (CLI catalog,
workflow steps, persona, skills, project context) rather than just
identity/persona instructions, after #2353 / #2355. The pre-existing
comment still described it as "identity/persona instructions inline",
which would mislead future maintainers about why the inline payload is
load-bearing.
Also call out kiro/kimi alongside openclaw/hermes since they were added
to providerNeedsInlineSystemPrompt in #2328, and document the concrete
failure mode (issues stuck in todo) so the rationale is searchable.
Co-authored-by: multica-agent <github@multica.ai>
InjectRuntimeConfig writes the full meta skill content (CLI catalog,
workflow instructions, project context, skills) to workdir/AGENTS.md,
but providers like OpenClaw, Hermes, Kiro, and Kimi read bootstrap
files from their own agent workspace — not the task workdir. The
inline system prompt path (providerNeedsInlineSystemPrompt) only
passed the agent persona instructions, so these providers never
received the runtime brief.
Have InjectRuntimeConfig return the rendered content so the daemon can
both write it to disk (for file-reading providers) and pass it inline
(for workspace-isolated providers). This avoids double-rendering and
keeps the file and inline payloads identical.
Fixes#2353
Kiro and Kimi share Hermes' ACP architecture and already accept
SystemPrompt prepended in front of the user prompt (kiro.go:244-247,
kimi.go:256-257). Without daemon-side opt-in, ExecOptions.SystemPrompt
is never set, so per-task agent identity instructions are lost in
deployments that rely on inline injection (e.g. K3 Lens-style
daemon → wrapper → docker compose exec acp).
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): mark provider 429 / out-of-credit runs as failed, not completed
Two bugs combined to silently report failed agent runs as
"Completed" in the UI when the upstream LLM returned a 4xx (e.g.
HTTP 429 rate-limit / no credit on the account).
1. ACP backends (hermes, kimi, kiro) only promoted the run status to
"failed" when their stderr sniffer fired AND the agent output
buffer was empty. But hermes injects a synthetic agent text turn
("API call failed after 3 retries: HTTP 429...") on retry
exhaustion, so the buffer was never empty in the rate-limit
case and the promotion never ran. Drop the empty-output
precondition: the sniffer's regex (HTTP-status markers, named
error types) is specific enough to trust on its own.
2. The daemon's task-result switch only routed "blocked" through
FailTask; every other status — including "cancelled", and any
future status we forget to enumerate — fell through to
CompleteTask. Invert it so only an explicit "completed" status
reports success, and extract the switch into reportTaskResult
for direct testing. Cancelled now defaults to failure_reason
"cancelled" instead of being silently completed.
Closes GitHub multica#1952.
Co-authored-by: multica-agent <github@multica.ai>
* fix(agent): only promote ACP run to failed on terminal provider error
Address GPT-Boy's review on the multica#1952 fix. The previous
promotion rule ("any sniffer line → fail") was too broad: the
existing sniffer also captures transient per-attempt warnings
("API call failed (attempt 1/3): RateLimitError [HTTP 429]"), and
those lines stay in the buffer for the rest of the run. A retry
sequence whose first attempt blipped but whose third attempt
succeeded would have been wrongly reported as failed.
Tighten the criteria with two additional signals, both defined on
the existing acpProviderErrorSniffer / output buffer:
- acpTerminalErrorRe — sticky `terminal` flag set when stderr shows
an exhausted/non-retryable marker (❌, [ERROR], "after N retries",
Non-retryable, BadRequestError, AuthenticationError). Per-attempt
warnings deliberately don't match.
- acpAgentOutputTerminalRe — matches the synthetic "API call failed
after N retries..." turn that hermes-style adapters inject into
the agent text stream when they give up; this catches multica#1952
even if hermes' stderr only logged transient attempts.
Promotion logic becomes a shared helper, promoteACPResultOnProviderError,
called from hermes / kimi / kiro. Promotes when (a) terminalMessage
is non-empty, (b) output contains the synthetic give-up turn, or
(c) output is empty and the sniffer captured anything at all
(preserves the original empty-output safety net for transient-only
sequences with no real result to fall back on).
Tests:
- TestHermesProviderErrorSnifferTerminalVsTransient — transient
attempt 1/3 alone returns terminalMessage="" but message!="";
a follow-on terminal marker flips terminal on.
- TestHermesProviderErrorSnifferTerminalNonRetryable — confirms
BadRequest / Authentication / Non-retryable / ❌ / [ERROR] are
classified terminal even on the very first attempt.
- TestHermesBackendDoesNotPromoteOnTransientRetry — fake hermes
emits attempt 1/3 to stderr then a normal agent text turn and
end_turn; resulting Status must stay "completed".
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): treat upstream API 400 invalid_request_error as poisoned session
A markdown-linked image in an issue description that the agent downloads as
a tiny CDN auth-error file and Read's as a PNG poisons the conversation:
the LLM API rejects the bad image with 400 invalid_request_error, the
session_id is pinned mid-flight, and every follow-up task on the issue
(comment-trigger, auto-retry) resumes the same poisoned conversation and
hits the same 400 — the issue can no longer be executed even after the
description is cleaned up.
Mirror the existing fallback-output classifier on the error side: detect
"API Error: ... 400 ... invalid_request_error" in the agent error string,
persist failure_reason='api_invalid_request', and add it to the
GetLastTaskSession exclusion list so the next task starts a fresh
session that re-reads the (now-clean) description.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): unblock issues already poisoned by API 400 invalid_request_error
The forward-only classifier from the previous commit only tags new failures.
Issues like MUL-1918 already have multiple failed-task rows whose
failure_reason is the pre-fix default 'agent_error', and GetLastTaskSession
falls back to those legacy rows on the next claim — so deploying the
classifier alone leaves existing poisoned issues stuck (GPT-Boy review
on PR #2314).
Two complementary changes:
- Migration 079 backfills failure_reason='api_invalid_request' on every
pre-existing 'agent_error' row whose error text matches the canonical
Anthropic 400 invalid_request_error shape. Keeps observability
consistent (multica issue runs / UI now report the right reason).
- GetLastTaskSession adds a defensive ILIKE clause on error text. Closes
the deploy-window gap where the old binary could write a new
'agent_error' row between the migration running and the new code
taking over, and protects against future error-format variants the
daemon classifier might miss.
Plus regression tests covering the legacy + new coexistence case GPT-Boy
flagged, and a guard rail asserting benign 'agent_error' failures
(timeouts, tool errors) still resume their session.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
Two follow-up nits from PR #2211 review:
- Rename the package-local `repoCache` interface to `repoCacheBackend`
so the field declaration `repoCache repoCacheBackend` no longer shadows
its own type name.
- Bump the `/health`-must-respond timeout in
`TestHealthHandlerRespondsWhileTaskRepoLookupWaits` from 200ms to 1s.
The regression case blocks indefinitely on the old code, so a 1s
upper bound still fail-fast detects it while leaving headroom for
loaded CI runners.
Co-authored-by: multica-agent <github@multica.ai>
* feat(daemon): extend GC to chat / autopilot / quick-create tasks
Before this change the daemon's GC was strictly issue-centric: only tasks
with a non-empty issue_id ever wrote .gc_meta.json, and shouldCleanTaskDir
called only the issue gc-check endpoint. Chat / autopilot run / quick-create
tasks fell through to the GCOrphanTTL mtime path, which mis-killed active
chat sessions while leaving deleted ones around far longer than necessary.
Schema:
- GCMeta gains a Kind discriminator and per-kind ID fields
(ChatSessionID / AutopilotRunID / TaskID). WriteGCMeta now takes a
GCMeta struct so the call site classifies the task explicitly.
- ReadGCMeta defaults empty Kind to GCKindIssue, so legacy on-disk meta
files keep flowing through the issue path with no migration required.
Server endpoints (siblings of /api/daemon/issues/{id}/gc-check, all behind
requireDaemonWorkspaceAccess for the same anti-enumeration shape):
- GET /api/daemon/chat-sessions/{id}/gc-check -> {status, updated_at}
- GET /api/daemon/autopilot-runs/{id}/gc-check -> {status, completed_at}
- GET /api/daemon/tasks/{id}/gc-check -> {status, completed_at}
shouldCleanTaskDir dispatches on Kind:
- chat: active is hard-skipped (no mtime fallback) so idle sessions are
never reclaimed; archived + GCTTL cleans; 404 falls back to mtime to
stay safe for cross-workspace tokens.
- autopilot_run: terminal (completed/failed/skipped/issue_created) +
GCTTL cleans; running/pending skips. Uses run.completed_at as the TTL
anchor since autopilot_run has no updated_at column.
- quick_create: terminal task status cleans immediately (workdir is not
reused by the linked issue task, which has its own envRoot); running
skips.
Also drops the "skipping .gc_meta.json: issue_id is empty" warn — with
the new kind dispatch, chat/autopilot/quick-create tasks now write a
proper meta file instead of triggering this log.
Refs: GC follow-up to PR #2077 (symptom fix) and #2115 (chat hard delete).
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): chat gc-check 404 cleans immediately, no mtime gate
PR review caught that the chat 404 path was routing through
orphanByMTime, which deferred reclamation to GCOrphanTTL (72h) when
acceptance #3 calls for cleanup within one GC cycle (≤ 1h) after the
user hard-deletes a session.
Every chat_session_id we ever ask about was written by this same daemon
under its current token, so the cross-workspace probe defense the issue
path needs doesn't apply here. Drop the gate and clean on 404 directly.
Test updates:
- TestShouldCleanTaskDir_KindDispatch/chat_404 flips the locked
expectation from gcActionSkip to gcActionClean.
- Adds TestShouldCleanTaskDir_ChatHardDeletedFreshMtime: GCOrphanTTL
set to a year so any mtime-based path is unmistakably out, and the
fresh-mtime workdir still cleans on the chat-404 fast path.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): use brew prefix symlink for self-restart so Linux Cellar deletion does not orphan runtimes
After brew upgrade on Linux, os.Executable() resolves /proc/self/exe to
the Cellar path (e.g. .../Cellar/multica/0.2.9/bin/multica), which
brew cleanup deletes. The previous IsBrewInstall() short-circuit skipped
EvalSymlinks to 'preserve' the symlink, but on Linux there was nothing
to preserve - the path was already resolved.
Use cli.GetBrewPrefix() to resolve the stable symlink path
<brewPrefix>/bin/multica for brew installs. Fall back to
EvalSymlinks(os.Executable()) with a warning log when GetBrewPrefix()
returns empty (brew binary missing from PATH).
Introduce package-level function vars (isBrewInstall, getBrewPrefix) so
the daemon test can override them without modifying the cli package.
Closes#1624
* fix(daemon): harden brew-prefix fallback and document the WHY
When `brew --prefix` is unavailable but the binary is under a known Cellar
root, recover the prefix from cli.MatchKnownBrewPrefix and target
<prefix>/bin/multica instead of falling back to the resolved Cellar path
(which brew cleanup just deleted).
- Extract knownBrewPrefixes + MatchKnownBrewPrefix in cli/update.go and
reuse from IsBrewInstall to keep one source of truth for the install-root
list.
- Add a WHY comment above the brew branch in triggerRestart explaining the
/proc/self/exe -> Cellar -> deleted-by-brew-cleanup chain.
- Cover both fallback paths (matched / unmatched) in daemon_test.go.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* fix(server): return 500 for transient DB errors in daemon task lookup
requireDaemonTaskAccess used to turn any GetAgentTask error into
404 "task not found", including transient DB connection / pool errors.
Combined with PR #2107 — which added 404+"task not found" as a daemon
cancellation trigger — that means a single DB hiccup could kill an
in-flight agent run.
Distinguish pgx.ErrNoRows (real "task gone", 404) from other errors
(transient, 500 + warn log) using the existing isNotFound helper.
Tests cover both paths via the mockDB pattern already used by
TestFindOrCreateUserGating.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): honor task-deleted signal in post-runTask completion guard
The final pre-completion check in handleTask only looked for
status == "cancelled" and ignored errors. After PR #2107 added a 404
task-deleted cancellation path to the in-flight watcher, this trailing
guard fell out of sync — if the task was deleted between the watcher's
last poll and runTask returning, handleTask would still try to call
CompleteTask and only learn about the deletion via the 404 from that
callback.
Reuse shouldInterruptAgent so the same truth table (cancelled OR
404 task-not-found, but NOT transient errors) drives both polling and
the final guard.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
When the server deletes a task while the daemon's agent is still running
(issue removed, agent reassigned, workspace cleanup), GetTaskStatus
starts returning 404 "task not found". The previous polling loop only
checked for status == "cancelled" and silently swallowed the error, so
the local agent kept emitting tool calls against a dead task until its
own timeout fired — minutes of wasted model spend and patch_apply
operations against a workdir nobody would consume.
Changes:
- Add isTaskNotFoundError next to isWorkspaceNotFoundError so the daemon
can distinguish "task gone" 404 from "workspace gone" 404 (already
handled separately) and from generic network errors.
- Extract the cancellation polling goroutine in handleTask into
watchTaskCancellation, plus a pure shouldInterruptAgent decision
helper. The pure helper makes both signals (cancelled status and 404
task) easy to unit-test without spinning up a real backend.
- Trigger interruption on the new 404 path. Transient errors (5xx,
network) intentionally still don't cancel — the next poll will retry
and a flaky link should not kill an in-flight agent.
Tests cover the helper truth table, the existing "status cancelled"
path, the new "task deleted (404)" path, and a negative case ensuring a
running task is not interrupted.
Co-authored-by: “646826” <“646826@gmail.com”>
A non-trivial fraction of completed task workdirs (~28% in field reports)
end up with .gc_meta.json files containing issue_id: "". Empty issue_id
defeats the daemon's own GC loop (gc.go:139 calls
GetIssueGCCheck(meta.IssueID)) and external retention scripts that
cross-reference issue status before deleting orphaned workdirs.
Refuse to write the file when issueID is empty, logging a Warn so
operators have a starting point for debugging the upstream race
condition. Skip is preferred over a sentinel-marker file: it keeps the
data invariant clean (a .gc_meta.json file always carries a valid
issue_id) and matches the repo CLAUDE.md preference for not preserving
dual-state behavior.
WriteGCMeta now takes a *slog.Logger so it can emit the warning. The
package already uses log/slog (Prepare/reuseEnv), and daemon.go:884 has
taskLog in scope at the only call site.
Closes#1913
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* fix(daemon): isolate runtime poll & heartbeat schedules per runtime
A daemon serving multiple workspaces ran a single round-robin poll loop
and a single HTTP heartbeat loop across every registered runtime. A 30s
HTTP timeout for any one runtime serialized that delay across all the
others — observed in production as one workspace's runtimes wedging
every other workspace's runtimes on the same daemon.
This change:
- Replaces the shared runtime-set channel with a multi-subscriber
watcher so taskWakeupLoop, heartbeatLoop, and pollLoop can each
react to runtime-set changes independently.
- Splits heartbeatLoop and pollLoop into supervisor + per-runtime
worker goroutines. Each runtime owns its claim cadence and its
heartbeat ticker, so a slow request on one runtime no longer blocks
any other.
- Stagers the per-runtime heartbeat first tick by a jittered delay up
to one full interval to avoid a thundering herd at startup.
- Sizes the WS writer channel to scale with the runtime count
(max(16, 2*N)) so a full per-runtime heartbeat batch always fits;
the previous fixed 8-slot buffer dropped heartbeats whenever a
daemon watched more than ~8 runtimes.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): acquire execution slot only after ClaimTask, drain pollers before taskWG
Two issues from review on the previous commit:
1. Acquiring the shared task slot before ClaimTask reintroduced the very
head-of-line blocking the refactor was meant to remove. With
MaxConcurrentTasks=1, a slow claim on one runtime parked the only slot
for the duration of the HTTP timeout (up to 30s), starving every other
runtime's claim attempts. Slots are now acquired after the claim
returns a task; other runtimes' pollers stay free to claim. The
already-dispatched task waits for a slot under MaxConcurrentTasks
bounds, which is the same backpressure shape we had before.
2. pollLoop's shutdown path called taskWG.Wait immediately after
cancelling pollers, but a poller could still be between ClaimTask
returning a task and taskWG.Add(1). When taskWG's counter is zero
that races with Wait — undefined sync.WaitGroup misuse, sometimes
panic. Added a pollerWG so the supervisor blocks until every poller
goroutine has actually returned before reaching taskWG.Wait.
Tests:
- TestRunRuntimePollerIsolatesSlowRuntime now uses MaxConcurrentTasks=1
(was 4) so it would have failed under the old slot-before-claim path.
- New TestPollLoopShutdownWaitsForPollersBeforeTaskWG drives the exact
race window — claim returns a task at the same moment shutdown fires —
under -race.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): acquire slot before ClaimTask so capacity-waiters never enter dispatched
The previous commit moved slot acquisition AFTER ClaimTask to address a
review concern about head-of-line blocking with MaxConcurrentTasks=1.
That introduced a strictly worse failure mode: server-side ClaimTask
flips the task to `dispatched` immediately (agent.sql:174-176), and the
runtime sweeper fails any task in `dispatched` for >300s with
`failed/timeout` (runtime_sweeper.go:25-28). When local execution
capacity is full and the next claimed task can't acquire a slot within
5 minutes, the user sees the exact failure this issue is fixing —
`dispatched_at` set, `started_at` NULL, `failure_reason=timeout`.
Reverted to slot-before-claim. The trade-off is the original review
concern: with MaxConcurrentTasks=1 and a slow ClaimTask, other
runtimes' claims are delayed by up to client.Timeout=30s. That's a
30s polling delay, not a failure — server-side those tasks remain
`queued` (no timeout in that state) until a slot frees. 30s ≪ 300s,
so other runtimes' tasks cannot get sweeper-failed because of this.
The pollerWG fix from the previous commit (avoiding sync.WaitGroup
misuse on shutdown) is preserved.
Tests:
- TestRunRuntimePollerIsolatesSlowRuntime: MaxConcurrentTasks back to
4 (the pre-issue baseline) — the headroom case where slot-before-
claim still gives full per-runtime isolation.
- New TestRunRuntimePollerSkipsClaimWhenAtCapacity: holds the only
slot and verifies the poller never calls ClaimTask while sem is
empty. The previous "claim first" path would have failed this.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
TestRegisterTaskReposSurvivesWorkspaceRefresh started flaking on CI
after #1988 (`feat: support repo checkout ref selection`) extended the
bare-clone path to run an extra `git fetch` to backfill
refs/remotes/origin/* under the new refspec layout. The race was
already latent: registerTaskRepos kicks off `go syncWorkspaceRepos(...)`
to clone a repo into the cache root, which in tests is `t.TempDir()`.
Once the test waited on `repoCache.Lookup` to return a path it would
proceed and return — but the bg goroutine was still inside
`ensureRemoteTrackingLayout` running git operations on the clone dir.
`t.TempDir`'s cleanup then races with those git commands and surfaces
either as "directory not empty" or "fatal: cannot change to ... No such
file or directory", with no hint that the failure is unrelated to the
test's actual assertion.
Track the background goroutine on the Daemon via a sync.WaitGroup and
expose `waitBackgroundSyncs()` for tests. `newRepoReadyTestDaemon`
registers a t.Cleanup that calls it, so every test that uses the
helper now drains in-flight syncs before t.TempDir cleanup runs. No
production-behavior change — registerTaskRepos still fires-and-forgets
from the caller's perspective.
Verified with `go test ./internal/daemon -run
TestRegisterTaskReposSurvivesWorkspaceRefresh -count=30` (was failing
within ~10 iterations before, 30 green after) and the full
`go test ./internal/daemon/...` suite.
Co-authored-by: multica-agent <github@multica.ai>
* fix(server): persist ModelListStore across replicas via Redis
The model picker uses a pending-request pattern: the frontend POSTs to
create a request, the daemon pops it on its next heartbeat, runs
agent.ListModels locally, and reports back. Until now the store was a
plain in-memory map per Handler instance.
That works for self-hosted single-instance deploys but fails in any
multi-replica environment (Multica Cloud). Each replica has its own
map, so:
POST /runtimes/:id/models → request stored in replica A
GET /runtimes/:id/models/<requestId> → polls land on B/C → 404
daemon heartbeat → only A sees PendingModelList
POST .../<requestId>/result → daemon's report has to land on A
Success probability ~1/N². The visible symptom is "No models available"
in the picker for every provider, even those (Claude/Codex) whose
catalog is statically populated end-to-end.
Same shape of bug, same Redis-backed fix as multica-ai/multica#1557 did
for LocalSkillListStore / LocalSkillImportStore. Reuse the operational
playbook (namespaced keys, ZSET-backed pending queue, atomic
ZREM+SET-running via the shared Lua script) so we don't introduce a
second concurrency model for the same primitive.
Changes:
- Convert ModelListStore from struct to interface with context-aware
methods. Add HasPending for cheap heartbeat-side probing.
- InMemoryModelListStore — single-node fallback, used when REDIS_URL
is unset (self-hosted dev / tests).
- RedisModelListStore — multi-node implementation using the same key
layout and Lua atomic claim as RedisLocalSkillListStore.
- Use RunStartedAt (not UpdatedAt) as the running-timeout reference
point, matching the local-skill stores so subsequent UpdatedAt
bumps don't reset the running clock.
- Heartbeat now uses the probe-then-pop pattern for the model queue
(matching local-skills) so a slow Redis can't stall every connected
daemon. Extends heartbeatMetrics + slow-log with probe_model_ms /
pop_model_ms / probe_model_timed_out for parity.
- Wire the Redis backend in NewRouterWithOptions when rdb != nil.
- Tests for both backends. Redis tests gate on REDIS_TEST_URL so
laptop runs without Redis still pass; CI provides it.
Co-authored-by: multica-agent <github@multica.ai>
* fix(server): persist RunStartedAt + retry model report on transient failures
Two follow-ups from PR #2022 review:
1. RedisModelListStore was dropping ModelListRequest.RunStartedAt on
persistence — the field is tagged json:"-" so it doesn't leak into
the HTTP response, which made plain json.Marshal(req) silently
discard it. Across-node readers saw RunStartedAt=nil and
applyModelListTimeout's running branch became a no-op, so the 60s
running-timeout escape hatch never fired. CI's
TestRedisModelListStore_RunningTimeout was failing on this exact
case. Fix mirrors RedisLocalSkillImportStore's envelope pattern —
wrap in an internal struct that re-promotes the field. HTTP shape
stays clean. Adds a no-Redis unit test that pins the round trip.
2. Daemon's handleModelList called d.client.ReportModelListResult
directly and swallowed any 5xx, leaving the pending request
stranded in "running" until its 60s server-side timeout — exactly
the failure mode the multi-node store fix was meant to eliminate.
Generalize the existing local-skill retry helper into
reportRuntimeResultWithRetry (kind: model_list / local_skill_list /
local_skill_import) and wire handleModelList through a new
reportModelListResult helper. Renames the test-overridable
var localSkillReportBackoffs → runtimeReportBackoffs to match.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): reclaim disk on long-open issues + correct cancelled-status check
Two related fixes for GitHub #1890 (self-hosted disk space growth):
- The GC's done/cancelled branch compared `status.Status` against `"canceled"`
(single l), but the issue schema and the rest of the daemon use `"cancelled"`
(double l). Cancelled issues therefore never matched and only fell out via the
72h orphan TTL, which itself doesn't fire because cancelled issues are still
reachable. Aligning the spelling lets cancelled-issue task dirs be reclaimed
on the normal TTL path.
- Add a third GC mode, artifact-only cleanup, for the common case the report
flagged: an issue stays open for days while many tasks complete on it, so
per-task `node_modules`, `.next` and `.turbo` directories accumulate without
ever becoming GC-eligible. The new branch fires when `.gc_meta.completed_at`
is older than `MULTICA_GC_ARTIFACT_TTL` (default 12h), the env root is not
currently in use by an active task, and the issue is still alive. It removes
only directories whose basename matches `MULTICA_GC_ARTIFACT_PATTERNS`
(default narrow: `node_modules,.next,.turbo`); source, `.git`, `output/`,
`logs/` and the meta file are preserved so subsequent tasks can still resume
the workdir. Patterns containing path separators are dropped, `.git` subtrees
are never descended into, symlinked matches are not followed, and every
removal target is verified to live inside the task dir.
Bookkeeping: `Daemon` now tracks active env roots with a refcounted set so the
GC loop never reclaims a directory that is mid-execution; `runTask` claims the
predicted root early plus the prior workdir on reuse paths. The cycle log is
extended with bytes reclaimed and per-pattern counts so self-hosted operators
can see what was freed.
Docs: extend the daemon configuration table in CLI_AND_DAEMON.md with the new
GC env vars and add a Workspace garbage collection section explaining the
three modes and the artifact-pattern contract.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): protect active env root from full GC removal too
Address GPT-Boy's PR #1931 review: the active-root guard only fired in the
artifact-cleanup branch, leaving a real race on the full-removal paths. A
follow-up comment on a long-done issue dispatches a task that reuses the prior
workdir, but `CreateComment` does not bump issue.updated_at — so the issue
still satisfies the done+stale GCTTL window and `gcActionClean` would
`RemoveAll` the directory mid-execution. The orphan-404 path is similarly
exposed when a token's workspace access is in flux.
Move the `isActiveEnvRoot` check to the top of `shouldCleanTaskDir` so all
three delete actions (clean, orphan, artifact) skip an in-use env root in one
place, and drop the now-redundant guard from the artifact branch.
Add tests covering the three at-risk paths: active root + done/stale issue,
active root + 404 issue past orphan TTL, active root + no-meta orphan past
TTL.
Also align two stale comments noted in the same review: cleanTaskArtifacts now
documents that symlinks are skipped entirely (the previous note implied the
link itself was removed), and GCOrphanTTL no longer claims that 404s are
cleaned immediately — the implementation gates them on the same TTL.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
* refactor(repos): drop unused description + tighten create-project layout
Two related changes that touch the workspace-repos surface together.
1. Remove the per-repo `description` field everywhere it was threaded.
The only place it ever surfaced was a markdown table column the daemon
wrote into the agent runtime config, where most rows just read "—"
anyway. Agents already discover project structure by running
`multica project` / `multica issue` against the CLI, so the human-
readable description string carried no real value while taking up an
extra Settings input row and propagating through six layers (settings
UI → workspace.repos jsonb → handler RepoData → daemon RepoData →
repocache.RepoInfo → execenv.RepoContextForEnv).
- Settings → Repositories drops the description input; the URL field
now spans the whole row.
- WorkspaceRepo TS type loses `description`; backend RepoData /
RepoInfo / RepoContextForEnv all collapse to URL only.
- Daemon's runtime_config Repositories block changes from a
`| URL | Description |` markdown table to a simple bullet list.
- Tests updated; jsonb residue in existing workspaces is dropped at
normalize time, so no migration needed.
2. Tighten the Create Project modal footer: pull the Status / Priority /
Lead / Repos pills onto the same row as the Create Project button
(Linear-style single-row footer) instead of stacking them above it,
and swap the Repos pill icon from `FolderGit` to a real GitHub mark
(lucide-react v1 dropped brand icons, so the mark lives inline as a
small SVG component in this file).
I tried promoting Repos to its own "Resources" strip above the footer
to separate the resources abstraction from project metadata, but with
a single pill it looked too sparse — leaving a TODO comment in the
footer to revisit once we add Linear / Notion / Figma / Slack
resource types.
* fix(daemon test): drop residual Description field on RepoData literals
* fix(repos): drop Description residue surfaced after rebase on #1929
Project-resource github_repo lift path (#1929) and registerTaskRepos
both still constructed RepoData{...Description: ...} after the rebase.
Two test sites in daemon_test.go and execenv_test.go also reintroduced
the field. Strip them so the Description-removal change builds and
tests pass with the latest main.
* feat(projects): project github_repo resources override workspace repos
When an issue's project has at least one github_repo resource, the daemon
claim handler now sends only those as resp.Repos — workspace-level repos
are hidden to avoid mixing two repo lists in the agent prompt. With no
project github_repos (or no project), behavior is unchanged: workspace
repos are surfaced as before.
Lifts each project github_repo's url (and label, when present) into a
RepoData entry so `multica repo checkout` and the meta-skill render the
same URLs. The full structured list still ships at
.multica/project/resources.json for skills that want everything.
Adds TestProjectReposReplaceWorkspaceReposInMetaSkill covering the
rendering side. Docs updated to spell out the new precedence.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): allow project repo URLs through the checkout allowlist
When ClaimTaskByRuntime narrows resp.Repos to project github_repo URLs,
the daemon receives URLs that may not exist in the workspace's
GetWorkspaceRepos response. The existing checkout flow rejected those
with ErrRepoNotConfigured because the allowlist (and cache) was built
only from workspace-bound repos.
Adds registerTaskRepos in daemon.runTask: before agent spawn, merge
task.Repos into a new task-scoped allowlist (separate from the
workspace-scoped one so a workspace refresh doesn't wipe project URLs)
and kick off a background cache sync. ensureRepoReady now treats either
allowlist as valid.
Tests:
- TestRegisterTaskReposAllowsProjectOnlyURL — project-only URL is
checkout-able and does not trigger a workspace-repos refresh
- TestRegisterTaskReposSurvivesWorkspaceRefresh — task URLs persist
across refreshWorkspaceRepos
- TestClaimTask_ProjectGithubReposOverrideWorkspaceRepos — claim
handler returns only project repos when present, no workspace leakage
- TestClaimTask_ProjectWithoutRepos_FallsBackToWorkspaceRepos — fall
back to workspace repos when project has no github_repo resources
Docs updated to spell out the daemon-side allowlist behavior.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
* fix(task): rerun starts a fresh session, skip poisoned resume
When a task ended in a known agent fallback ("I reached the iteration
limit and couldn't generate a summary.", "Put your final update inside
the content string. Keep it concise.") the (agent_id, issue_id) resume
lookup would still pick that session, so a manual rerun inherited the
poisoned state and reproduced the same bad output.
Two complementary guards:
1. Daemon classifies poisoned terminal output and routes it through the
blocked path with failure_reason set ('iteration_limit' /
'agent_fallback_message'). GetLastTaskSession excludes failed tasks
with those reasons, so even comment-triggered tasks no longer resume
them. Tasks that failed mid-flight (timeout, runtime_recovery, etc.)
are still resumable, preserving MUL-1128's auto-retry contract.
2. Manual rerun marks the new task force_fresh_session=true. The daemon
claim handler skips the resume lookup entirely when the flag is set,
capturing the user-intent signal that "the prior output was bad" even
when poisoned classification misses a future fallback wording.
Auto-retry of orphaned mid-flight failures (MaybeRetryFailedTask →
CreateRetryTask) does not take this path, so it keeps resuming.
Tests: classifyPoisonedOutput unit test; integration tests assert the
SQL filter excludes poisoned classifiers, RerunIssue flips the flag,
and the normal enqueue path leaves it false.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): cap poisoned-output matcher to short trimmed text
GPT-Boy review on MUL-1630: the previous strings.Contains match would
classify any output that quoted the marker substring — including a
review/analysis that simply discussed the marker itself. Real fallback
messages are short single-sentence affairs, so cap the candidate at
~one paragraph and trim whitespace before matching. Adds regression
tests covering a long quoting review and a marker buried in a long
real conclusion; both must stay classified as completed.
Co-authored-by: multica-agent <github@multica.ai>
* fix(migrations): rename 065 force_fresh_session → 066 to clear collision
main introduced 065_project_resources after this branch was cut, so
both files shared the 065_ prefix. The readiness check
(server/cmd/server/health.go → migrations.LatestVersion) takes the
last entry by lexical order, which is 065_project_resources, leaving
this branch's 065_force_fresh_session unguarded — a deploy that
applied project_resources but not force_fresh_session would still
report ready, and the next enqueue / rerun / claim would crash on
"column force_fresh_session does not exist".
Renaming to 066_force_fresh_session puts it strictly after
project_resources so readiness blocks until it's applied.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
* feat(projects): typed project resources + agent runtime injection
Adds a `project_resource` table that lets a project carry typed pointers
(github_repo today, more later) and surfaces them at agent runtime.
Server
- migration 065: project_resource (resource_type TEXT + resource_ref JSONB)
- sqlc CRUD + handler at /api/projects/{id}/resources
- claim handler attaches project_id/title + resources to issue tasks
Daemon
- TaskContextForEnv carries project context
- writes .multica/project/resources.json into workdir
- adds "## Project Context" block to CLAUDE.md / AGENTS.md / GEMINI.md
via type-dispatched formatter so new resource types just add a case
CLI
- multica project create --repo <url> attaches repos in one step
- multica project resource add/list/remove
Frontend
- Project create modal: Repos pill (workspace repos + ad-hoc URL)
- Project detail sidebar: collapsible Resources section with attach/remove
Docs
- New "Project Resources" chapter explaining the abstraction and
exactly what code to touch when adding a new resource type
Co-authored-by: multica-agent <github@multica.ai>
* fix(projects): transactional resources[] on create + generic CLI ref + test fix
Addresses review feedback on PR #1926:
1. CI red: TestProjectResourceLifecycle delete step called withURLParam
twice, which replaced the chi route context and dropped the project id.
Switched to the existing withURLParams helper from daemon_test.go.
2. POST /api/projects now accepts resources[] and attaches them in the
same transaction as the project. Invalid refs roll back the whole
create — no more half-attached projects on failure. Web modal + CLI
`project create --repo` both use the new bundled payload.
3. CLI `project resource add` now accepts a generic --ref '<json>' flag
so a new resource_type works without a CLI change. Per-type
shortcuts (--url for github_repo) remain as a convenience but are no
longer the only way in. Docs updated to drop the CLI from the
"files you must touch" list.
Adds two new server handler tests:
- TestCreateProjectAttachesResources (resources[] happy path)
- TestCreateProjectRollsBackOnInvalidResource (transactional rollback)
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
* feat(daemon): add Co-authored-by trailer for Multica Agent to git commits
Install a prepare-commit-msg hook in worktree bare repos that appends
"Co-authored-by: multica-agent <github@multica.ai>" to every commit
made by agents. Uses git interpret-trailers for proper formatting and
skips duplicates.
* feat(settings): add Co-authored-by toggle in workspace Labs settings
Add a workspace-level toggle to enable/disable the Co-authored-by
trailer for agent commits. Default is enabled (on).
Backend:
- Include workspace settings in daemon register response
- Store settings in daemon workspaceState
- Thread CoAuthoredByEnabled through WorktreeParams to conditionally
install the prepare-commit-msg hook
- Parse co_authored_by_enabled from workspace settings JSONB
Frontend:
- Replace empty Labs tab placeholder with a Git section containing
a Switch toggle for the Co-authored-by trailer setting
- Optimistically update the workspace query cache on toggle
* chore(daemon): skip squash commits in Co-authored-by hook
Test commit to verify the prepare-commit-msg hook appends the
Co-authored-by trailer automatically.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
* feat(server): add quick-create issue async task path
Adds POST /api/issues/quick-create which validates the picked agent's
reachability up front (not archived, has runtime, runtime online) then
queues an issue-less agent task whose context JSONB carries the user's
natural-language prompt + requester + workspace. Daemon claim resolves
the workspace from the context, and the prompt builder switches to a
quick-create template instructing the agent to translate the prompt
into a single multica issue create call.
Task completion writes a success inbox item to the requester pointing at
the newly-created issue (located by querying the agent's most recent
issue in the workspace since task start, so we don't depend on agent
stdout shape). Failures write an action_required inbox item carrying the
original prompt + agent id so the frontend can offer "Edit as advanced
form" without losing input.
* feat(views): quick-create issue modal + inbox failure CTA
Adds a streamlined create-issue UI bound to the c shortcut: pick an
agent, type one line, submit. The modal closes immediately and the
agent translates the prompt into a multica issue create call in the
background. Shift+c keeps the legacy advanced form for users who want
every field. The "Advanced" button inside the new modal seeds the
shared issue-draft store with the prompt + picked agent so switching
mid-flow doesn't lose input.
Last-used agent persists per (user, workspace) via a workspace-aware
zustand store so frequent users skip the picker on every open.
Inbox renders quick_create_done items with a status pin to the new
issue and quick_create_failed items with an "Edit as advanced form"
CTA that re-seeds the legacy modal with the original prompt.
ApiError now carries the parsed JSON body so the modal can branch on
the structured agent_unavailable code without parsing the error
message.
* fix(quick-create): execenv injection, claim race, private-agent permission
Addresses GPT-Boy review on #1786:
1. execenv was rendering the assignment-task issue_context.md / runtime
workflow even for quick-create, telling the agent to call
`multica issue get/status/comment add` against an empty IssueID.
Adds QuickCreatePrompt to TaskContextForEnv, plus a quick-create
branch in renderIssueContext + the runtime_config workflow that
instructs the agent to run a single `multica issue create` and
exit, with explicit "do NOT call issue get/status/comment add"
guards.
2. ClaimAgentTask serialized only on issue_id / chat_session_id, so
concurrent quick-creates on the same agent (both NULL on those
columns) ran in parallel — making the success-inbox lookup race
over "most recent issue by this agent". Adds a third OR clause
that treats "all four FKs NULL" as a serialization key for the
same agent, so quick-create tasks on a given agent run one at a
time.
3. QuickCreateIssue handler bypassed the private-agent ownership rule
that validateAssigneePair enforces elsewhere — a user could POST a
private agent_id they didn't own and trigger it. Now routes the
picked agent through validateAssigneePair before the runtime
liveness check.
4. Clarifies the quick-create-store namespacing comment to match the
actual workspace-aware StateStorage convention used by the other
issue stores (per-user is browser-profile-local).
* fix(quick-create): branch Output section + deterministic origin lookup
Addresses GPT-Boy's second-pass review on #1786:
1. The runtime_config.go Output section forced "Final results MUST be
delivered via multica issue comment add" for every non-autopilot
task — quick-create still got this conflicting instruction even
though there's no issue to comment on. Switched the Output block
to a three-way switch so quick-create gets a tailored "stdout is
captured automatically; do NOT call comment add" branch matching
the autopilot variant.
2. Completion lookup was "most recent issue created by this agent
since task.started_at", which races against concurrent issue
creates by the same agent (assignment task running alongside
quick-create when max_concurrent_tasks > 1). Replaced with a
deterministic origin link:
- Migration 060 extends issue.origin_type CHECK to allow
'quick_create'.
- Daemon sets MULTICA_QUICK_CREATE_TASK_ID env var when running a
quick-create task.
- multica issue create CLI reads the env var and stamps the new
issue with origin_type=quick_create + origin_id=<task_id>.
- Server CreateIssue handler accepts (origin_type, origin_id)
from trusted callers (only "quick_create" is allowed; the pair
is rejected unless both fields are provided together).
- notifyQuickCreateCompleted now calls GetIssueByOrigin keyed on
(workspace_id, "quick_create", task.ID) — no more time-window
racing against parallel agent activity.
The old GetRecentIssueByCreatorSince query is removed.
When the server cancels a task (e.g. assignee changes during execution,
explicit user cancel, or workspace_isolation check fail), the daemon's
cancellation poll fires runCancel() on the run context. The drainCtx
derived from runCtx then signals Done(), but executeAndDrain() was
returning Status: "timeout" regardless of *why* the context ended.
The "agent finished status=timeout" log line is then misleading — it
suggests an actual deadline timeout when really the task was cancelled
by upstream. We spent hours misdiagnosing a healthy handoff as a
broken timeout because of this.
Distinguish context.Canceled from context.DeadlineExceeded in
executeAndDrain, and add a "cancelled" case to runTask so the status
propagates through the existing log path.
No behaviour change for genuine timeouts; no behaviour change for
the cancelled-by-poll discard path in handleTask. Only the daemon
log line and TaskResult.Status get the more accurate label.
Review follow-up on PR #1557: the server-side change started returning
500 when the store write failed, but the daemon's handleLocalSkillList /
handleLocalSkillImport were discarding the ReportLocalSkill*Result error
return. Net effect was a silent drop — the daemon moved on, the request
stayed in "running" on the server, and the user saw the same "daemon did
not respond within 30 seconds" timeout the store refactor was supposed
to kill.
Fix: route both report calls through reportLocalSkillResultWithRetry,
which retries on 5xx + network errors with 0 / 0.5s / 2s / 4s backoff
(total ~6.5s, well inside the 60s server-side running timeout), stops
on 4xx (request expired / cross-workspace rejection — retry won't help),
bails on context cancel, and logs Error on exhaustion so ops has a
footprint to grep for.
Tests (server/internal/daemon/local_skill_report_test.go, 6 new cases):
- 500 twice then success -> 3 attempts, second retry lands
- 404 -> exactly 1 attempt (permanent, no retry)
- import 502 then success -> 2 attempts
- All-500 -> burns through all backoff slots then gives up with ERROR log
- Context cancel mid-backoff -> exactly 1 attempt, cancellation logged
- Smoke: report paths hit /api/daemon/runtimes/<rt>/local-skills{,import}/<req>/result
localSkillReportBackoffs is var-assignable so tests can swap in zero-delay
schedules without paying real sleep latency.
* feat(runtimes): remove Test Connection / runtime ping feature
The Test Connection action invoked a real single-turn agent run to verify
runtime connectivity. In practice it was expensive (reuses none of the
normal task exec env, so it also gave misleading results) and low value —
daemon heartbeat + Online status already covers the "is the runtime
alive" question. Dropping the whole end-to-end probe path:
- deletes server handler and in-memory PingStore
- drops pending_ping from the heartbeat response and daemon poll loop
- removes daemon.handlePing, PendingPing, ReportPingResult
- removes the CLI `multica runtime ping` command
- removes the PingSection UI block and RuntimePing types / api methods
* docs: fix runtime CLI subcommand list in product-overview
