* feat(agent): persist thinking_level per agent (MUL-2339)
Adds a nullable `thinking_level` column to the `agent` table so the
backend can route a runtime-native reasoning/effort token (e.g. Claude's
`xhigh`, Codex's `minimal`) through to the agent CLI on every dispatch.
The column is intentionally TEXT rather than an enum — Claude and Codex
publish overlapping but distinct vocabularies and we want the persisted
value to round-trip exactly through whichever CLI receives it. NULL is
the "use runtime default" sentinel that every downstream consumer reads
as "do not inject --effort / reasoning_effort".
This commit is just the storage layer (migration + sqlc); subsequent
commits wire it through the API, daemon, and agent backends.
Co-authored-by: multica-agent <github@multica.ai>
* feat(agent-backend): inject reasoning effort for claude + codex (MUL-2339)
Extends ExecOptions with a runtime-native ThinkingLevel string and wires
it into the Claude and Codex backends. Discovery is driven by the local
CLI so the daemon advertises whatever the host install supports rather
than a hand-maintained list that goes stale.
Per Elon's PR1 review:
- Claude: parses `claude --help` to learn the `--effort` superset and
projects through a per-model allow-list (xhigh is Opus-only; max is
session-only on the smaller models). Falls back to a conservative
static list when the binary is missing or help drift hides the line.
- Codex: drives `codex debug models --output json` so per-model
reasoning subsets and the documented default come directly from the
CLI. The older config-error probe trick is gone — the JSON path is
stable and doesn't pollute stderr with an intentional misconfig.
- Cache key includes (provider, executablePath, cliVersion) so a CLI
upgrade invalidates entries that referenced the older help / catalog.
Per Trump's PR1 constraint, all three Codex injection points
(thread/start.config, thread/resume.config, turn/start.effort) flow
through one helper (`applyCodexReasoningEffort`) so they cannot drift
independently. The shared `codexReasoningCases` fixture in
`thinking_test.go` asserts the same value→{shape, key} contract at
each site for every level the runtimes know about.
Claude's `--effort` is also added to `claudeBlockedArgs` so a user
custom_args entry can't silently outvote the daemon-injected value.
Co-authored-by: multica-agent <github@multica.ai>
* feat(api): wire thinking_level through API + daemon contract (MUL-2339)
End-to-end plumbing for the per-agent reasoning/effort setting:
- AgentResponse / TaskAgentData now carry `thinking_level`; the daemon's
claim response includes it and the daemon's executor passes it through
to agent.ExecOptions, where the Claude and Codex backends already know
what to do with it.
- ModelEntry on the runtime-models wire format gains a `thinking` block
carrying `supported_levels` + `default_level` per model so the UI can
render a runtime-aware picker without the server having to know about
the local CLI install. `handleModelList` projects the agent-package
catalog (including the new Thinking field) into the wire shape.
- CreateAgent / UpdateAgent gate the field with a synchronous provider
enum check (claude / codex only today). UpdateAgent is tri-state:
field omitted = no change, "" = explicit clear (new
`ClearAgentThinkingLevel` query, mirrors the existing mcp_config null
pattern), non-empty = validate then set.
Per Trump's PR1 review, the API NEVER auto-clears on a runtime/model
swap and ALWAYS returns 400 on an unknown literal value — same shape
across CreateAgent, UpdateAgent, and combined patches that move
runtime + level in one request. Per-model combination failures (e.g.
`xhigh` against a model that only supports up to `high`) surface as a
daemon-side task error, not a silent server-side rewrite.
TS types follow the same shape: `Agent.thinking_level`,
`CreateAgentRequest`/`UpdateAgentRequest` add the field, `RuntimeModel`
grows a `thinking` block. Older backends omit the field, which the
front-end treats as "no picker for this model" — installed desktop
builds keep working.
Co-authored-by: multica-agent <github@multica.ai>
* fix(agent): correct codex debug models argv + pin via runner test (MUL-2339)
`codex debug models --output json` is rejected by codex-cli 0.131.0 —
the subcommand emits JSON on stdout by default and has no `--output`
flag. Drop the flag and add `--bundled` to skip the network refresh
discovery doesn't need. Move the argv to a package-level var and add
a test that runs a fake `codex` to assert the binary actually
receives exactly `debug models --bundled`, so the contract can't
silently drift on the next refactor.
Also teach ValidateThinkingLevel to resolve an empty model to the
provider's default model entry. Without this, every default-model
task with a persisted thinking_level would be misjudged "unknown
model" by the daemon guard.
Co-authored-by: multica-agent <github@multica.ai>
* fix(api): reject runtime switch that would leave invalid thinking_level (MUL-2339)
A PATCH that changed `runtime_id` without touching `thinking_level`
used to silently keep the existing value, so a Claude agent storing
`max` could land on a Codex runtime where `max` is not a recognised
token at all, and the daemon would receive a literal-invalid level.
Hold the same "always 400 on literal-invalid, never silent coerce"
rule on this implicit path. When runtime_id changes and the existing
value is not in the new provider's enum, return 400 with the
recovery options (clear via `thinking_level=""` or re-set in the
same PATCH).
Add coverage for both the kept-when-still-valid and the rejected
cases, plus the two recovery paths (clear and replace).
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): guard runTask with per-model thinking_level validator (MUL-2339)
ValidateThinkingLevel existed but had no call site — `task.Agent.
ThinkingLevel` flowed straight into ExecOptions, so `xhigh` configured
on a non-Opus Claude model, or API-side stale values that escaped the
provider enum gate, would be injected anyway.
Run the validator before building ExecOptions. Invalid combinations
log a warning and drop the level instead of failing the task: the
agent still runs, just at the runtime's default reasoning effort.
Discovery errors fail open (keep the level, let the CLI surface any
objection) so a transient `claude --help` failure can't strand work.
Empty model is forwarded as-is; the validator resolves it to the
provider's default model internally per the cross-package contract.
Co-authored-by: multica-agent <github@multica.ai>
* chore(agent): drop stale `--output json` comments + unused scanner (MUL-2339)
Codex CLI's `debug models` subcommand emits JSON without an `--output`
flag, and `parseCodexDebugModels` never read from the bufio.Scanner.
Sync the comments with the actual invocation and remove the dead init.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): disable Claude AskUserQuestion in non-interactive mode (MUL-2244)
GitHub #2588: when Claude Code calls its built-in AskUserQuestion tool
inside the daemon's stream-json runtime, the question never reaches the
user — there's no UI to render it — so the SDK returns an empty answer
and the agent silently "infers" and continues. From the issue's
perspective, execution looks stuck while the agent is actually charging
ahead on its own guess.
Two-part fix:
- `buildClaudeArgs` now passes `--disallowedTools AskUserQuestion` so
the tool is not exposed to the model at all.
- The Claude-specific runtime brief tells the agent to use a `blocked`
issue comment for genuine clarification, or to state an explicit
assumption and proceed.
Adds a regression test that pins both: AskUserQuestion is forbidden in
CLAUDE.md and is NOT mentioned in the AGENTS.md emitted for non-Claude
providers (the tool is Claude-specific).
Co-authored-by: multica-agent <github@multica.ai>
* refactor(daemon): drop CLAUDE.md AskUserQuestion guidance, rely on --disallowedTools
The --disallowedTools flag already prevents Claude from invoking
AskUserQuestion, so duplicating the rule in the runtime brief just bloats
the prompt without changing behavior. Removes the section and its
regression test; the argv-level test in pkg/agent already pins the flag.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
The gemini CLI's Windows shim emits `Active code page: 65001` (from
`chcp`) to stdout before the real version reaches `--version` output.
The daemon stored the raw concatenation as the runtime version, so the
runtime detail page rendered `Active code page: 65001 0.42.0` instead
of `0.42.0`.
Scan `<cli> --version` line by line and return the first line carrying
a semver-shaped token. Full strings like `2.1.5 (Claude Code)` or
`codex-cli 0.118.0` survive unchanged; unparseable output falls back to
the trimmed raw value.
Co-authored-by: multica-agent <github@multica.ai>
#1674 wired claude's post-handshake error path through withAgentStderr but
left the writeClaudeInput failure branch returning a bare "broken pipe"
error. That branch fires precisely when claude crashes during startup —
exactly when the stderr tail is most useful for root-causing V8 aborts,
Bun panics, or missing native modules. cmd.Wait() before sampling Tail()
flushes os/exec's internal stderr copy goroutine, matching the
Wait→Tail synchronization contract spelled out in stderr_tail.go.
Adds TestClaudeExecuteSurfacesStderrWhenChildExitsEarly mirroring the
codex test: a fake claude binary drains stdin, writes a V8-abort line to
stderr, and exits 3. Locks in the contract that Result.Error carries the
stderr tail in the post-handshake failure path on the claude backend too.
Hoist the existing stderrTail ring-buffer (previously codex-only) into
a shared pkg/agent helper so every Backend that supervises a child CLI
can include the last ~2 KB of that CLI's stderr in Result.Error. Wire
the claude backend through the same path.
Motivation: claude on Windows occasionally exits with a non-zero status
after ~5–8 minutes of a single long-running tool_use, and right now the
daemon only reports "claude exited with error: exit status 3" /
"exit status 0x80000003" — useless for root-causing V8 aborts, Bun
panics, native-module OOMs, or any other CLI-side crash. With the tail
attached, the failure message carries the real signal (panic line, V8
assertion, stderr-printed HTTP error) all the way into the task row's
error field that users see in the API.
Renames withCodexStderr to withAgentStderr(msg, label, tail) so the
helper is self-documenting across providers.
* fix(daemon): suppress agent terminal windows on Windows (#1471)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: add hideAgentWindow to detectCLIVersion and avoid SysProcAttr overwrite
- Add missing hideAgentWindow(cmd) call in detectCLIVersion (claude.go:554)
so --version checks don't flash console windows on Windows.
- Refactor hideAgentWindow to preserve existing SysProcAttr fields
instead of overwriting the entire struct.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat(server): orphan-task recovery + auto-retry + manual rerun (MUL-1128)
When the daemon process crashed mid-task the issue was stuck at
in_progress for up to 2.5h: the in-flight task timeout was the only
mechanism that ever moved the row, and the runtime heartbeat sweeper
only fires after the runtime stays offline for 45s — a quick restart
beats both windows.
This change implements the A+B plan from the issue thread:
A. lifecycle hygiene
- migration 055 adds attempt / max_attempts / parent_task_id /
failure_reason / last_heartbeat_at to agent_task_queue
- new daemon-auth endpoint POST /runtimes/{id}/recover-orphans:
daemon calls it on every register so the server fails any
dispatched/running tasks the previous process left behind
- new daemon-auth endpoint POST /tasks/{id}/session: persists the
agent's session_id + work_dir mid-flight so a crash doesn't
lose the resume pointer (claude+codex emit MessageStatus with
SessionID; daemon forwards on the first one it sees)
- FailAgentTask / FailStaleTasks / FailTasksForOfflineRuntimes
now set failure_reason ('agent_error' / 'timeout' /
'runtime_offline')
B. auto-retry with resume context
- TaskService.MaybeRetryFailedTask spawns a fresh queued attempt
carrying parent's session_id/work_dir when the failure reason
is infrastructure-shaped (timeout, runtime_offline,
runtime_recovery) and attempt < max_attempts; skips autopilot
- wired into the runtime sweeper paths and TaskService.FailTask
so the user transparently sees a new in_progress run instead of
a stuck row
- new user-auth POST /api/issues/{id}/rerun + multica issue rerun
CLI for the manual escape hatch
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(server): address PR review for orphan-task recovery (MUL-1128)
Three review-must-fix items on top of the A+B implementation:
1. recover-orphans now funnels through TaskService.HandleFailedTasks,
the same shared post-failure pipeline used by the runtime sweeper.
This guarantees task:failed events are emitted, agent status is
reconciled, and issues stuck in_progress with no remaining active
task are reset to todo even when no auto-retry is created
(max_attempts exhausted, autopilot, non-retryable reason).
2. RerunIssue now uses CancelAgentTasksByIssueAndAgent, scoped to the
issue's current assignee. The previous implementation called
CancelAgentTasksByIssue, which would collateral-cancel parallel
@-mention agents on the same issue.
3. GetLastTaskSession now considers both completed and failed tasks
(mirroring GetLastChatTaskSession), ordering by the most recent
timestamp. With UpdateAgentTaskSession pinning session_id/work_dir
mid-flight, an auto-retry or manual rerun of a daemon-crash failure
now actually resumes the prior conversation context instead of
starting fresh — matching the stated B-branch behaviour.
go build / go vet pass; the existing service and agent test suites pass.
runtime_sweeper / handler integration tests require a local DB with the
055 migration (and the pre-existing 050 first_executed_at column).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Surface the actual exec path + argv for every agent backend at INFO
so operators can see the exact command without flipping to debug.
Also add the missing log line in pi.go for consistency with the
other nine backends.
When --resume targets a dead session, claude prints
"No conversation found with session ID: ..." to stderr, emits a stream-json
system init with a fresh session_id, then exits with code 1. The backend
was treating that fresh id as the authoritative session, so
daemon.go's retry-with-fresh-session fallback (SessionID == "" guard)
never triggered. Every subsequent task for the same (issue, agent) pair
stayed permanently broken until the server-side session_id was cleared by
hand.
Fix: when --resume was requested but the emitted session_id differs AND
the run failed, drop the fresh id from Result so the daemon's existing
fallback can do its job. Factored into a pure helper and unit-tested.
Fixes#1284
Co-authored-by: fuxiao <fuxiao@zyql.com>
* fix(agent): add per-agent mcp_config field to restore MCP access
Closes#1111
The --strict-mcp-config flag was added defensively in #592 to prevent
Claude agents from inheriting MCP state from the outer Claude Code session.
It was meant to be paired with --mcp-config <path> to inject a controlled
set of MCPs, but that path was never implemented, which silently stripped
all user-scope MCPs from spawned agents.
This PR completes the original design by:
- Adding a nullable mcp_config jsonb column to the agents table
- Wiring mcp_config through AgentResponse, Create/Update requests
- Piping it into ExecOptions.McpConfig in the daemon
- Serializing to a temp file and passing --mcp-config <path> in buildClaudeArgs
- Blocklisting --mcp-config in claudeBlockedArgs to prevent override
via custom_args
Does not touch Codex provider (tracked separately in #674).
Does not implement Multica MCP auto-injection (out of scope).
* fix: disambiguate JSON null vs absent for mcp_config
Add a debug-level log line in every agent backend (claude, codex,
opencode, openclaw, gemini, hermes) that prints the executable path
and full argument list when spawning the agent process. Helps diagnose
custom args, model overrides, and other CLI flag issues.
* feat(agent): add custom CLI arguments support
Allow users to configure custom CLI arguments per agent that get
appended to the agent subprocess command at launch time. This enables
use cases like specifying different models (--model o3), max turns,
or other provider-specific flags without needing separate runtimes.
Changes:
- Add custom_args JSONB column to agent table (migration 041)
- Update API handler to accept/return custom_args in create/update
- Pass custom_args through claim endpoint to daemon
- Append custom_args to CLI commands for all agent backends
- Add ExecOptions.CustomArgs field in agent package
- Add Custom Args tab in agent detail UI
- Add --custom-args flag to CLI agent create/update commands
Closes MUL-802
* fix(agent): filter protocol-critical flags from custom_args
Add per-backend filtering of custom_args to prevent users from
accidentally overriding flags that the daemon hardcodes for its
communication protocol (e.g. --output-format, --input-format,
--permission-mode for Claude).
This follows the same pattern as custom_env's isBlockedEnvKey: we
only block the small, stable set of flags that would break the
daemon↔agent protocol — not every possible dangerous flag. Workspace
members are trusted for everything else.
Each backend defines its own blocked set:
- Claude: -p, --output-format, --input-format, --permission-mode
- Gemini: -p, --yolo, -o
- Codex: --listen
- OpenCode: --format
- OpenClaw: --local, --json, --session-id, --message
- Hermes: none (ACP is positional)
Includes unit tests for the filtering logic.
* fix(agent): address code review nits for custom_args
- Replace module-level `nextArgId` counter with `crypto.randomUUID()`
in custom-args-tab.tsx to avoid SSR ID conflicts
- Add unit tests for custom args passthrough and blocked-arg filtering
in both Claude and Gemini arg builders
When an agent CLI process hangs (e.g. a tool call blocks on unreachable
I/O), the daemon's scanner blocks indefinitely on stdout, preventing the
Result from ever being sent. This causes tasks to stay in "running"
state permanently with no further events.
Three-layer fix:
1. Agent backends (claude, opencode, openclaw, gemini): add a watchdog
goroutine that closes the stdout/stderr pipe when the context is
cancelled, forcing the scanner to unblock. Also set cmd.WaitDelay
so Go force-closes pipes after 10s if the process doesn't exit.
2. daemon executeAndDrain: add an independent drain timeout (backend
timeout + 30s buffer) with context-aware select on both the message
channel and the result channel, so the daemon never blocks forever.
3. daemon ping path: add context-aware select so pings don't deadlock
if the agent backend stalls.
Closes#925
Co-authored-by: Devv <devv@Devvs-Mac-mini.local>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Claude's stream-json flow can emit the terminal result event while the
child process still waits on open stdin. Closing stdin as soon as the
final result arrives lets the CLI exit cleanly instead of idling until
the daemon timeout fires.
Constraint: Must preserve the existing Claude stream-json protocol and child-process lifecycle
Rejected: Increase ping timeout only | masks the hang without fixing process exit
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep Claude stdin handling aligned with the stream-json terminal result semantics; do not defer closure until goroutine teardown
Tested: Reproduced self-hosted runtime ping timeout locally; verified ping succeeds after closing stdin on result; cd server && go test ./pkg/agent
Not-tested: Full make check; Bedrock/Vertex-specific Claude auth flows
Extract token usage from Claude Code's stream-json output in real-time
during task execution, replacing the inaccurate global JSONL log scanner.
- New `task_usage` table: tracks (task_id, provider, model) level usage
- Agent SDK: parse `message.usage` from assistant messages, accumulate
per-model and return in Result
- Daemon: convert agent usage to entries, send with CompleteTask
- Server: store usage on task completion, expose workspace-level
aggregation APIs (GET /api/usage/daily, GET /api/usage/summary)
- Fix duplicate icons in tool call rows (use chevron only for expand/collapse)
- Show detailed tool information (WebSearch queries, Agent prompts, Skill names)
- Add thinking/reasoning rows with Brain icon and expandable content
- Show tool results as separate chronological entries with previews
- Add TaskRunHistory component for viewing past agent execution logs
- Add listTasksByIssue API endpoint and task-runs route
- Support thinking content blocks in agent SDK (MessageThinking type)
- Improve callID→toolName mapping in daemon message forwarding
Store the Claude Code session ID and working directory when a task
completes. On the next task for the same (agent, issue) pair, look up
the prior session and pass --resume <session_id> to Claude Code so
the agent retains conversation context across multiple tasks on the
same issue.
Changes:
- Migration 020: add session_id and work_dir columns to agent_task_queue
- CompleteAgentTask stores session_id and work_dir on completion
- GetLastTaskSession query retrieves prior session for (agent, issue)
- ClaimTaskByRuntime handler populates prior_session_id in response
- Daemon passes ResumeSessionID through to Claude backend Execute()
- Claude backend adds --resume flag when ResumeSessionID is set
Replace raw fmt/log calls with structured slog logger (Go) and
console-based logger (TypeScript). Add request logging middleware.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix data race on output strings.Builder in codex backend by adding
mutex and waiting for reader goroutine before reading final output
- Fix data race on onTurnDone by initializing it before reader starts
- Fix bug where notificationProtocol zero value "" never matched
"unknown", silently dropping all raw v2 notifications from codex
- Add round-robin polling to prevent runtime starvation in poll loop
- Log errors in claude handleControlRequest instead of silently dropping
- Add 35 tests for pkg/agent covering claude parsing, codex JSON-RPC,
protocol detection, event handling, and helper functions
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace deprecated strings.Title with manual capitalize
- Fix race: set codexClient.onMessage before starting reader goroutine
- Remove unused msgCh parameter from codexClient.handleLine
- Route agent stderr through logger instead of dumping to os.Stderr
- Use deterministic agent order in ensurePaired (prefer codex)
- Increase message channel buffer from 64 to 256
- Rename test to match function rename (buildPrompt)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add a reusable Go agent package (server/pkg/agent/) that provides a
unified Backend interface for executing prompts via either Claude Code
or Codex. The daemon now auto-detects which CLIs are available at
startup, registers a runtime for each, and routes tasks to the correct
backend based on task.Context.Runtime.Provider.
Key changes:
- server/pkg/agent/agent.go: Backend interface, Message/Result types, factory
- server/pkg/agent/claude.go: Spawns claude CLI with stream-json, parses output
- server/pkg/agent/codex.go: Spawns codex app-server, JSON-RPC 2.0 protocol
- server/cmd/daemon/daemon.go: Multi-runtime registration, round-robin polling,
provider-based backend selection. Removes old runCodexExec/codexResultSchema.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
