Codex telemetry was never reaching the OTLP collector for tasks run by the
daemon. The per-task config (including the [otel] block) is copied into
CODEX_HOME correctly, but the lifecycle goroutine closed stdin and then
immediately cancelled the run context, which SIGKILLs the app-server. Codex's
OTEL batch exporters only force-flush on a graceful shutdown, so the buffered
spans/metrics/logs were dropped before they could be exported — short tasks
lost everything, long tasks lost the final batch.
Let codex exit on its own after stdin EOF (running its shutdown + flush path)
and only force-cancel after a bounded grace period if it doesn't, so the reader
goroutine still can't block forever. Also set cmd.WaitDelay, matching the other
long-lived backends (claude, copilot, cursor, …).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Active long-running sessions are no longer killed by a fixed wall-clock deadline. Liveness is delegated to the idle watchdog (MULTICA_AGENT_IDLE_WATCHDOG, default 30m) with a larger in-flight-tool budget (MULTICA_AGENT_TOOL_WATCHDOG, default 2h). MULTICA_AGENT_TIMEOUT is an opt-in absolute cap (default 0 = no cap). The server-side 2.5h sweeper is unchanged as a coarse backstop.
Fixes#3745.
* MUL-2764: feat(agents): add MCP config tab to agent detail page
Backend already stores `mcp_config` and the daemon forwards it to the
runtime CLI via `--mcp-config`; this only adds the UI entry point.
The new tab presents a JSON editor that pretty-prints the existing
config, validates the buffer on every keystroke, and saves through the
existing `PUT /api/agents/{id}` path. Clearing the editor sends
`mcp_config: null`, which the handler reads as "wipe the column" and
the daemon falls back to the CLI's own default.
When the caller can't see secrets (agent actor, or a non-owner
non-admin member), the server already returns `mcp_config: null` with
`mcp_config_redacted: true`; the tab renders a read-only "configured
but hidden" state in that case so a non-privileged member cannot
silently overwrite an admin-owned config by saving an empty editor.
Co-authored-by: multica-agent <github@multica.ai>
* fix(agents): MCP tab — preserve in-flight edits + warn non-Claude runtimes
- Fix stale-editor sync: compare the local draft against the *previous*
original via a ref, so a background agent refetch updates an untouched
editor instead of being silently ignored. Without this, a draft equal to
the OLD original was treated as user-edited after the prop changed, and
the next Save would write the old config back over a concurrent admin
edit.
- Surface a notice inside the tab when the agent's runtime provider is not
Claude — today's daemon only forwards mcp_config via Claude's
--mcp-config, so saving on e.g. a Codex agent was silent but ineffective.
- Tests for both: rerender resyncs an untouched editor, rerender preserves
an in-flight edit, warning renders on non-Claude / hides on Claude.
MUL-2764
Co-authored-by: multica-agent <github@multica.ai>
* MUL-2764: feat(agents): codex MCP support + hide MCP tab on unsupported runtimes
- Backend: codex.go now translates agent.mcp_config (Claude-style
`{"mcpServers": {...}}`) into `-c mcp_servers.<name>=<inline-toml>`
flags for `codex app-server`, so MCP servers configured in the UI
reach Codex's per-task config layer. Bad mcp_config JSON downgrades
to a warn-and-skip so it can't break the agent launch.
- Frontend: AgentOverviewPane hides the MCP tab when the agent's
runtime provider doesn't read mcp_config — only `claude` and `codex`
are supported today, every other provider sees no MCP tab. The
previous in-tab warning is removed (no longer reachable).
- New shared helper `providerSupportsMcpConfig` lives in
`@multica/core/agents` so views and any future caller share one list
of MCP-aware providers.
- Tests: new go-side coverage for stdio + url + multi-server inputs,
TOML string escaping, malformed-input fallback, and arg ordering vs
custom_args; new views-side coverage for which providers surface the
MCP tab. En + zh-Hans copy and parity test refreshed.
Co-authored-by: multica-agent <github@multica.ai>
* MUL-2764: fix(agents): keep codex mcp_config secrets out of argv/logs
Move the agent's mcp_config from a `-c mcp_servers.<id>=<inline-toml>`
argv flag into a daemon-managed `[mcp_servers.*]` block inside the
per-task `$CODEX_HOME/config.toml`. mcp_servers.<id>.env is a documented
Codex config field and the UI already treats mcp_config as redacted for
non-admins; argv would have leaked those values into `ps aux` and the
`agent command` log line. The file is forced to 0600 to keep secrets in
the daemon owner's lane regardless of the seed file's mode.
Also drop user-supplied `-c/--config mcp_servers.*` entries from
custom_args. Codex `-c` is last-wins (verified against codex-cli 0.132.0),
so without filtering, a custom_args entry could silently shadow whatever
the MCP Tab saved.
Strip inherited `[mcp_servers.*]` tables from the per-task config.toml
when the agent has its own mcp_config, mirroring Claude's
`--strict-mcp-config`: avoids TOML "table already exists" errors on
name collisions and matches admin expectations that the MCP Tab is the
authoritative source for that task.
Co-authored-by: multica-agent <github@multica.ai>
* MUL-2764: fix(agents): codex mcp_config three-state semantics + custom_args compat
Address the third review pass:
1. Distinguish nil vs present-but-empty mcp_config. `{}` and
`{"mcpServers":{}}` now count as "admin saved an explicit (empty)
managed set" — strip inherited user `[mcp_servers.*]` and pin an
empty managed marker block. Only SQL NULL / JSON `null` map to
"absent" and fall back to the user's global `~/.codex/config.toml`.
This aligns Codex with the API's three-state contract (omit / null
/ object) and with Claude's `--strict-mcp-config` semantics.
2. Fail closed on `ensureCodexMcpConfig` errors and on managed
mcp_config without CODEX_HOME. Previous warn-and-launch would
silently inherit the user's global MCP servers and look identical
to a successful apply — exactly the surprise the MCP Tab is meant
to remove.
3. Only filter `-c mcp_servers.*` from `custom_args`/`extra_args`
when the agent has a managed mcp_config. Pre-MUL-2764 agents that
configured MCP via custom_args keep working; once an admin opts
in via the MCP Tab the daemon owns the `mcp_servers` namespace
and overrides are dropped (last-wins safety).
4. Update mcp_config locale intro to mention $CODEX_HOME/config.toml
instead of the now-removed `-c mcp_servers.*` argv path.
Tests:
- Split `TestEnsureCodexMcpConfigEmptyInputsAreNoop` into
`TestEnsureCodexMcpConfigAbsentLeavesUserTablesAlone` (nil/null)
and `TestEnsureCodexMcpConfigEmptyManagedSetStripsUserMcp` (`{}`,
`{"mcpServers":{}}`).
- Add `TestEnsureCodexMcpConfigEmptyManagedSetIdempotent` to pin
byte-identical reruns on the empty managed marker block.
- Add `TestHasManagedCodexMcpConfig` covering the eight relevant
inputs.
- Add `TestBuildCodexArgsPreservesCustomMcpOverridesWhenUnmanaged`
and `TestBuildCodexArgsDropsCustomMcpOverridesWhenManaged` to
pin the new gating.
- Add `TestCodexExecuteFailsClosedWhenMcpConfigInvalid` and
`TestCodexExecuteFailsClosedWhenManagedMcpButNoCodexHome` for the
Execute paths.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: J <j@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
* feat(agent): persist thinking_level per agent (MUL-2339)
Adds a nullable `thinking_level` column to the `agent` table so the
backend can route a runtime-native reasoning/effort token (e.g. Claude's
`xhigh`, Codex's `minimal`) through to the agent CLI on every dispatch.
The column is intentionally TEXT rather than an enum — Claude and Codex
publish overlapping but distinct vocabularies and we want the persisted
value to round-trip exactly through whichever CLI receives it. NULL is
the "use runtime default" sentinel that every downstream consumer reads
as "do not inject --effort / reasoning_effort".
This commit is just the storage layer (migration + sqlc); subsequent
commits wire it through the API, daemon, and agent backends.
Co-authored-by: multica-agent <github@multica.ai>
* feat(agent-backend): inject reasoning effort for claude + codex (MUL-2339)
Extends ExecOptions with a runtime-native ThinkingLevel string and wires
it into the Claude and Codex backends. Discovery is driven by the local
CLI so the daemon advertises whatever the host install supports rather
than a hand-maintained list that goes stale.
Per Elon's PR1 review:
- Claude: parses `claude --help` to learn the `--effort` superset and
projects through a per-model allow-list (xhigh is Opus-only; max is
session-only on the smaller models). Falls back to a conservative
static list when the binary is missing or help drift hides the line.
- Codex: drives `codex debug models --output json` so per-model
reasoning subsets and the documented default come directly from the
CLI. The older config-error probe trick is gone — the JSON path is
stable and doesn't pollute stderr with an intentional misconfig.
- Cache key includes (provider, executablePath, cliVersion) so a CLI
upgrade invalidates entries that referenced the older help / catalog.
Per Trump's PR1 constraint, all three Codex injection points
(thread/start.config, thread/resume.config, turn/start.effort) flow
through one helper (`applyCodexReasoningEffort`) so they cannot drift
independently. The shared `codexReasoningCases` fixture in
`thinking_test.go` asserts the same value→{shape, key} contract at
each site for every level the runtimes know about.
Claude's `--effort` is also added to `claudeBlockedArgs` so a user
custom_args entry can't silently outvote the daemon-injected value.
Co-authored-by: multica-agent <github@multica.ai>
* feat(api): wire thinking_level through API + daemon contract (MUL-2339)
End-to-end plumbing for the per-agent reasoning/effort setting:
- AgentResponse / TaskAgentData now carry `thinking_level`; the daemon's
claim response includes it and the daemon's executor passes it through
to agent.ExecOptions, where the Claude and Codex backends already know
what to do with it.
- ModelEntry on the runtime-models wire format gains a `thinking` block
carrying `supported_levels` + `default_level` per model so the UI can
render a runtime-aware picker without the server having to know about
the local CLI install. `handleModelList` projects the agent-package
catalog (including the new Thinking field) into the wire shape.
- CreateAgent / UpdateAgent gate the field with a synchronous provider
enum check (claude / codex only today). UpdateAgent is tri-state:
field omitted = no change, "" = explicit clear (new
`ClearAgentThinkingLevel` query, mirrors the existing mcp_config null
pattern), non-empty = validate then set.
Per Trump's PR1 review, the API NEVER auto-clears on a runtime/model
swap and ALWAYS returns 400 on an unknown literal value — same shape
across CreateAgent, UpdateAgent, and combined patches that move
runtime + level in one request. Per-model combination failures (e.g.
`xhigh` against a model that only supports up to `high`) surface as a
daemon-side task error, not a silent server-side rewrite.
TS types follow the same shape: `Agent.thinking_level`,
`CreateAgentRequest`/`UpdateAgentRequest` add the field, `RuntimeModel`
grows a `thinking` block. Older backends omit the field, which the
front-end treats as "no picker for this model" — installed desktop
builds keep working.
Co-authored-by: multica-agent <github@multica.ai>
* fix(agent): correct codex debug models argv + pin via runner test (MUL-2339)
`codex debug models --output json` is rejected by codex-cli 0.131.0 —
the subcommand emits JSON on stdout by default and has no `--output`
flag. Drop the flag and add `--bundled` to skip the network refresh
discovery doesn't need. Move the argv to a package-level var and add
a test that runs a fake `codex` to assert the binary actually
receives exactly `debug models --bundled`, so the contract can't
silently drift on the next refactor.
Also teach ValidateThinkingLevel to resolve an empty model to the
provider's default model entry. Without this, every default-model
task with a persisted thinking_level would be misjudged "unknown
model" by the daemon guard.
Co-authored-by: multica-agent <github@multica.ai>
* fix(api): reject runtime switch that would leave invalid thinking_level (MUL-2339)
A PATCH that changed `runtime_id` without touching `thinking_level`
used to silently keep the existing value, so a Claude agent storing
`max` could land on a Codex runtime where `max` is not a recognised
token at all, and the daemon would receive a literal-invalid level.
Hold the same "always 400 on literal-invalid, never silent coerce"
rule on this implicit path. When runtime_id changes and the existing
value is not in the new provider's enum, return 400 with the
recovery options (clear via `thinking_level=""` or re-set in the
same PATCH).
Add coverage for both the kept-when-still-valid and the rejected
cases, plus the two recovery paths (clear and replace).
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): guard runTask with per-model thinking_level validator (MUL-2339)
ValidateThinkingLevel existed but had no call site — `task.Agent.
ThinkingLevel` flowed straight into ExecOptions, so `xhigh` configured
on a non-Opus Claude model, or API-side stale values that escaped the
provider enum gate, would be injected anyway.
Run the validator before building ExecOptions. Invalid combinations
log a warning and drop the level instead of failing the task: the
agent still runs, just at the runtime's default reasoning effort.
Discovery errors fail open (keep the level, let the CLI surface any
objection) so a transient `claude --help` failure can't strand work.
Empty model is forwarded as-is; the validator resolves it to the
provider's default model internally per the cross-package contract.
Co-authored-by: multica-agent <github@multica.ai>
* chore(agent): drop stale `--output json` comments + unused scanner (MUL-2339)
Codex CLI's `debug models` subcommand emits JSON without an `--output`
flag, and `parseCodexDebugModels` never read from the bufio.Scanner.
Sync the comments with the actual invocation and remove the dead init.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
The watchdog fires on a "no progress" window, so the default mainly
matters for commands that go fully silent (no outputDelta). Bumping
from 2m → 3m leaves more headroom for legitimately slow silent
commands before treating them as a dropped function_call_output, at
a modest cost to recovery latency.
MUL-2337
Co-authored-by: multica-agent <github@multica.ai>
* feat(codex): add per-exec_command watchdog to escape dropped function_call_output (MUL-2337)
Codex app-server can drop the second function_call_output when two
exec_command calls fan out in the same turn and both async-yield through
the yield_time_ms boundary (observed 2026-05-18, MUL-2334 — Trump Agent
wedged for 6+ min with no semantic activity events to drive any existing
timer). The model then waits forever for the missing output; only the
10-minute semantic inactivity timeout would eventually rescue the run.
Add a per-call watchdog in the codex client that tracks open
exec_command / commandExecution items by call_id and fails the turn
quickly (default 2 min, configurable via ExecOptions.ExecCommandStuckTimeout)
when one stays open without progress. outputDelta events reset the
per-call progress timestamp so long-running streaming commands aren't
flagged.
This is a daemon-side mitigation only — codex itself still has the
upstream race, but the daemon no longer burns the full inactivity budget
before the run is marked failed and a new run can recover.
Co-authored-by: multica-agent <github@multica.ai>
* feat(codex): track legacy exec_command_output_delta in watchdog (MUL-2337)
Mirrors the raw v2 item/commandExecution/outputDelta refresh on the legacy
codex/event protocol so a long-running streaming exec doesn't get falsely
flagged as stuck after begin + 2 min.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
* fix(codex): handle MCP elicitation server requests correctly
Fixes#1942.
handleServerRequest responded with {} to unrecognized Codex server
requests including mcpServer/elicitation/request. Codex 0.125+ expects
{action, content, _meta} for elicitation — the empty object causes a
deserialization error and the MCP tool call is reported as user-rejected.
Changes:
- Add mcpServer/elicitation/request case with correct response schema
- Add respondError helper for JSON-RPC error responses
- Return proper JSON-RPC method-not-found error for unknown server
requests instead of silent empty object
- Add tests for MCP elicitation and unknown method handling
* fix: use cfg.Logger instead of global slog in codex handleServerRequest
Switch the unhandled-server-request warning from global slog.Warn to
c.cfg.Logger.Warn for consistency with all other log calls in codex.go.
This ensures the warning appears in daemon run-logs and per-task
pipelines where operators look during triage.
Hoist the existing stderrTail ring-buffer (previously codex-only) into
a shared pkg/agent helper so every Backend that supervises a child CLI
can include the last ~2 KB of that CLI's stderr in Result.Error. Wire
the claude backend through the same path.
Motivation: claude on Windows occasionally exits with a non-zero status
after ~5–8 minutes of a single long-running tool_use, and right now the
daemon only reports "claude exited with error: exit status 3" /
"exit status 0x80000003" — useless for root-causing V8 aborts, Bun
panics, native-module OOMs, or any other CLI-side crash. With the tail
attached, the failure message carries the real signal (panic line, V8
assertion, stderr-printed HTTP error) all the way into the task row's
error field that users see in the API.
Renames withCodexStderr to withAgentStderr(msg, label, tail) so the
helper is self-documenting across providers.
* fix(daemon): suppress agent terminal windows on Windows (#1471)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: add hideAgentWindow to detectCLIVersion and avoid SysProcAttr overwrite
- Add missing hideAgentWindow(cmd) call in detectCLIVersion (claude.go:554)
so --version checks don't flash console windows on Windows.
- Refactor hideAgentWindow to preserve existing SysProcAttr fields
instead of overwriting the entire struct.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat(server): orphan-task recovery + auto-retry + manual rerun (MUL-1128)
When the daemon process crashed mid-task the issue was stuck at
in_progress for up to 2.5h: the in-flight task timeout was the only
mechanism that ever moved the row, and the runtime heartbeat sweeper
only fires after the runtime stays offline for 45s — a quick restart
beats both windows.
This change implements the A+B plan from the issue thread:
A. lifecycle hygiene
- migration 055 adds attempt / max_attempts / parent_task_id /
failure_reason / last_heartbeat_at to agent_task_queue
- new daemon-auth endpoint POST /runtimes/{id}/recover-orphans:
daemon calls it on every register so the server fails any
dispatched/running tasks the previous process left behind
- new daemon-auth endpoint POST /tasks/{id}/session: persists the
agent's session_id + work_dir mid-flight so a crash doesn't
lose the resume pointer (claude+codex emit MessageStatus with
SessionID; daemon forwards on the first one it sees)
- FailAgentTask / FailStaleTasks / FailTasksForOfflineRuntimes
now set failure_reason ('agent_error' / 'timeout' /
'runtime_offline')
B. auto-retry with resume context
- TaskService.MaybeRetryFailedTask spawns a fresh queued attempt
carrying parent's session_id/work_dir when the failure reason
is infrastructure-shaped (timeout, runtime_offline,
runtime_recovery) and attempt < max_attempts; skips autopilot
- wired into the runtime sweeper paths and TaskService.FailTask
so the user transparently sees a new in_progress run instead of
a stuck row
- new user-auth POST /api/issues/{id}/rerun + multica issue rerun
CLI for the manual escape hatch
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(server): address PR review for orphan-task recovery (MUL-1128)
Three review-must-fix items on top of the A+B implementation:
1. recover-orphans now funnels through TaskService.HandleFailedTasks,
the same shared post-failure pipeline used by the runtime sweeper.
This guarantees task:failed events are emitted, agent status is
reconciled, and issues stuck in_progress with no remaining active
task are reset to todo even when no auto-retry is created
(max_attempts exhausted, autopilot, non-retryable reason).
2. RerunIssue now uses CancelAgentTasksByIssueAndAgent, scoped to the
issue's current assignee. The previous implementation called
CancelAgentTasksByIssue, which would collateral-cancel parallel
@-mention agents on the same issue.
3. GetLastTaskSession now considers both completed and failed tasks
(mirroring GetLastChatTaskSession), ordering by the most recent
timestamp. With UpdateAgentTaskSession pinning session_id/work_dir
mid-flight, an auto-retry or manual rerun of a daemon-crash failure
now actually resumes the prior conversation context instead of
starting fresh — matching the stated B-branch behaviour.
go build / go vet pass; the existing service and agent test suites pass.
runtime_sweeper / handler integration tests require a local DB with the
055 migration (and the pre-existing 050 first_executed_at column).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Surface the actual exec path + argv for every agent backend at INFO
so operators can see the exact command without flipping to debug.
Also add the missing log line in pi.go for consistency with the
other nine backends.
* fix(agent/codex): surface stderr tail in initialize / turn startup errors
When codex app-server exits before the JSON-RPC handshake completes —
e.g. because the user put a flag in custom_args that the subcommand
rejects — the Result.Error users see is `codex initialize failed:
codex process exited`, while codex's actual complaint (typically
something like `error: unexpected argument '-m' found`) only lives in
daemon logs.
Wrap the stderr writer with a bounded stderrTail that still forwards
to the slog logWriter but also retains the last 2 KiB of bytes
written. Include that tail on the three startup failure paths
(initialize, startOrResumeThread, turn/start). Runtime cancellation
paths are left untouched — they're our own abort and the stderr
context isn't a clear signal there.
Refs #1308. Complement to #1310 / #1312 — lets "bad custom_args fail
loudly" actually be workable by giving the failure a real message.
* fix(agent/codex): join cmd.Wait() before sampling stderr tail
Addressing review of #1314: reading stderrBuf.Tail() right after
c.request returns "codex process exited" was racy. Nothing in that
path synchronizes with os/exec's internal stderr copy goroutine —
cmd.Wait() is the only documented join point. The original defer ran
cmd.Wait() later, but by then we had already built Result.Error from
a potentially-empty Tail().
Replace the ad-hoc deferred stdin.Close()/cmd.Wait() with a
sync.Once-wrapped drainAndWait closure. Call it explicitly on the
three startup failure paths before sampling the tail; keep it as the
cleanup defer so the success path behaves identically.
Also add TestCodexExecuteSurfacesStderrWhenChildExitsEarly: spawns a
real subprocess that prints to stderr and exits before responding to
initialize, runs it through Execute, and asserts Result.Error
contains the stderr hint. This covers the full timing path the
reviewer flagged, which the helper-level tests in this PR did not.
Follow-up to #1192. Document the v2 protocol contract that the
dispatch-level threadId guard relies on, and lock down the two leakage
paths the guard closes:
- turn/completed from a subagent thread must not call onTurnDone
- item/completed (agentMessage, final_answer) from a subagent thread
must neither leak text into the output builder nor terminate the turn
Without these tests a future refactor that drops or relocates the guard
would not be caught by CI, since existing notification tests omit the
top-level threadId field and pass through unfiltered.
* fix(daemon): filter thread/status/changed by threadId to prevent subagent interference
When Codex CLI has memories enabled, the app-server spawns a memory
consolidation subagent as a separate thread within the same stdio
connection. When that subagent thread finishes and transitions to idle,
the daemon's codex backend mistakenly interprets the idle signal as the
main turn completing, causing it to close stdin and cancel the context
before the real turn produces any output.
Add a threadId check to the thread/status/changed handler so only
status changes from the tracked thread trigger turn completion. Signals
from subagent threads (threadId != c.threadID) are now ignored.
Fixes#1181
* fix(codex): dispatch-level threadId filter for subagent notifications
Codex multiplexes subagent threads (e.g. memory consolidation) on
the same stdio pipe. Previously only thread/status/changed had a
threadId guard, but item/completed (agentMessage + final_answer),
turn/completed, and turn/started from subagent threads could still
trigger onTurnDone or contaminate output.
Move the threadId check to the top of handleRawNotification so all
notification handlers are protected. Remove the now-redundant
per-handler check on thread/status/changed.
Fixesmultica-ai/multica#1181
---------
Co-authored-by: fuxiao <fuxiao@zyql.com>
Every other backend (Claude, Gemini, OpenCode, OpenClaw, Hermes) honors
ExecOptions.ResumeSessionID — only Codex didn't. That's why users on
the Codex runtime saw each new comment on an issue start a fresh Codex
conversation: the daemon persists Result.SessionID per (agent, issue)
and passes it back as PriorSessionID, but codex.go always called
thread/start and never populated SessionID, so the value round-tripped
as empty.
Wire the missing half:
- Extract startOrResumeThread on codexClient. When ResumeSessionID is
set, call thread/resume (per the Codex app-server protocol), passing
only cwd / model / developerInstructions overrides so the thread
keeps its persisted model and reasoning effort. If resume fails
(unknown thread, schema drift, transport error) fall back to
thread/start so the task still runs on a fresh thread.
- Surface the live threadID as Result.SessionID on the final emit so
the daemon stores it and feeds it back into ResumeSessionID on the
next claim.
Tests drive the new helper through the fake stdin harness, covering:
fresh start, successful resume, fallback on resume error, fallback
when resume returns no thread ID, and surfacing of thread/start
failures.
When codex emits `turn/completed` with `status="failed"` or a terminal
top-level `error` notification, the daemon previously treated the turn
as successfully completed, saw no accumulated text, and surfaced the
generic "codex returned empty output" — hiding the real reason (auth,
sandbox, API error, etc.).
Capture `turn.error.message` on failed turns and the `error.message`
from non-retrying top-level error notifications, then propagate them
through `Result.Error` with `finalStatus="failed"` so the daemon's
default branch reports the actual cause.
Add a debug-level log line in every agent backend (claude, codex,
opencode, openclaw, gemini, hermes) that prints the executable path
and full argument list when spawning the agent process. Helps diagnose
custom args, model overrides, and other CLI flag issues.
* feat(agent): add custom CLI arguments support
Allow users to configure custom CLI arguments per agent that get
appended to the agent subprocess command at launch time. This enables
use cases like specifying different models (--model o3), max turns,
or other provider-specific flags without needing separate runtimes.
Changes:
- Add custom_args JSONB column to agent table (migration 041)
- Update API handler to accept/return custom_args in create/update
- Pass custom_args through claim endpoint to daemon
- Append custom_args to CLI commands for all agent backends
- Add ExecOptions.CustomArgs field in agent package
- Add Custom Args tab in agent detail UI
- Add --custom-args flag to CLI agent create/update commands
Closes MUL-802
* fix(agent): filter protocol-critical flags from custom_args
Add per-backend filtering of custom_args to prevent users from
accidentally overriding flags that the daemon hardcodes for its
communication protocol (e.g. --output-format, --input-format,
--permission-mode for Claude).
This follows the same pattern as custom_env's isBlockedEnvKey: we
only block the small, stable set of flags that would break the
daemon↔agent protocol — not every possible dangerous flag. Workspace
members are trusted for everything else.
Each backend defines its own blocked set:
- Claude: -p, --output-format, --input-format, --permission-mode
- Gemini: -p, --yolo, -o
- Codex: --listen
- OpenCode: --format
- OpenClaw: --local, --json, --session-id, --message
- Hermes: none (ACP is positional)
Includes unit tests for the filtering logic.
* fix(agent): address code review nits for custom_args
- Replace module-level `nextArgId` counter with `crypto.randomUUID()`
in custom-args-tab.tsx to avoid SSR ID conflicts
- Add unit tests for custom args passthrough and blocked-arg filtering
in both Claude and Gemini arg builders
Codex tasks running in workspace-write sandbox mode could not resolve
api.multica.ai because the hardcoded sandbox parameter in thread/start
overrode any config.toml settings, and the default sandbox policy blocks
network access.
Changes:
- Remove hardcoded `sandbox: "workspace-write"` from thread/start RPC —
let Codex read sandbox config from its own config.toml instead
- Auto-generate config.toml in per-task CODEX_HOME with
`sandbox_mode = "workspace-write"` and `network_access = true`,
preserving any existing user settings
- Fix Reuse() to restore CodexHome for Codex provider on workdir reuse
Closes#368
Codex doesn't expose token usage through its JSON-RPC app-server
protocol. The turn/completed and task_complete notifications don't
contain usage fields.
Fix: after Codex execution finishes, scan the on-disk session JSONL
files (~/.codex/sessions/YYYY/MM/DD/*.jsonl) for token_count events.
Only files modified after the task's start time are scanned, avoiding
counting unrelated sessions. This matches the same data format the
existing runtime_usage scanner reads.
1. Separate ReportTaskUsage endpoint (POST /api/daemon/tasks/{id}/usage)
so usage is captured independently of complete/fail — fixes usage loss
for failed/blocked tasks.
2. Add usage tracking for all four providers:
- Claude: already done (stream-json message.usage)
- OpenCode: extract from step_finish.part.tokens
- OpenClaw: extract from step_end.data token fields
- Codex: extract from turn/completed and task_complete usage fields
3. Remove usage from CompleteTask payload — all usage goes through the
dedicated endpoint now.
Replace raw fmt/log calls with structured slog logger (Go) and
console-based logger (TypeScript). Add request logging middleware.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove debug log.Printf calls from handler/daemon.go and service/task.go
that used the global log package instead of structured logging
- Remove unused truncate() helper from service/task.go
- Add error handling for EnqueueTaskForIssue in createAgentInitIssue
- Clean up verbose debug logging in daemon/daemon.go handleTask
- Add shutdown sequence comment to codex.go lifecycle goroutine
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The codex backend spawns a long-running app-server process that doesn't
exit after completing a turn. The lifecycle goroutine was waiting on
<-readerDone which blocks on scanner.Scan() until stdout closes — but
stdout never closes because the process stays alive. This caused the
entire poll loop to freeze, preventing any further task processing.
Fix: explicitly close stdin and cancel the context after the turn
completes, which terminates the codex process and unblocks the reader.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix data race on output strings.Builder in codex backend by adding
mutex and waiting for reader goroutine before reading final output
- Fix data race on onTurnDone by initializing it before reader starts
- Fix bug where notificationProtocol zero value "" never matched
"unknown", silently dropping all raw v2 notifications from codex
- Add round-robin polling to prevent runtime starvation in poll loop
- Log errors in claude handleControlRequest instead of silently dropping
- Add 35 tests for pkg/agent covering claude parsing, codex JSON-RPC,
protocol detection, event handling, and helper functions
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace deprecated strings.Title with manual capitalize
- Fix race: set codexClient.onMessage before starting reader goroutine
- Remove unused msgCh parameter from codexClient.handleLine
- Route agent stderr through logger instead of dumping to os.Stderr
- Use deterministic agent order in ensurePaired (prefer codex)
- Increase message channel buffer from 64 to 256
- Rename test to match function rename (buildPrompt)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add a reusable Go agent package (server/pkg/agent/) that provides a
unified Backend interface for executing prompts via either Claude Code
or Codex. The daemon now auto-detects which CLIs are available at
startup, registers a runtime for each, and routes tasks to the correct
backend based on task.Context.Runtime.Provider.
Key changes:
- server/pkg/agent/agent.go: Backend interface, Message/Result types, factory
- server/pkg/agent/claude.go: Spawns claude CLI with stream-json, parses output
- server/pkg/agent/codex.go: Spawns codex app-server, JSON-RPC 2.0 protocol
- server/cmd/daemon/daemon.go: Multi-runtime registration, round-robin polling,
provider-based backend selection. Removes old runCodexExec/codexResultSchema.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
