* feat(runtimes): remove Test Connection / runtime ping feature
The Test Connection action invoked a real single-turn agent run to verify
runtime connectivity. In practice it was expensive (reuses none of the
normal task exec env, so it also gave misleading results) and low value —
daemon heartbeat + Online status already covers the "is the runtime
alive" question. Dropping the whole end-to-end probe path:
- deletes server handler and in-memory PingStore
- drops pending_ping from the heartbeat response and daemon poll loop
- removes daemon.handlePing, PendingPing, ReportPingResult
- removes the CLI `multica runtime ping` command
- removes the PingSection UI block and RuntimePing types / api methods
* docs: fix runtime CLI subcommand list in product-overview
* feat: identify clients via X-Client-Platform/Version/OS
Adds client identification headers (and matching WS query params) across
all first-party clients so the server can split logs/metrics/gating by
caller without parsing User-Agent.
- HTTP: X-Client-Platform, X-Client-Version, X-Client-OS
- WS: client_platform, client_version, client_os query params
- Platform ∈ {web, desktop, cli, daemon}; OS ∈ {macos, windows, linux}
Wired through the shared TS ApiClient/WSClient via a new identity option
on CoreProvider. Web reads its version from package.json/env; Desktop
captures version + OS synchronously in preload via sendSync IPC. Go CLI
and daemon clients populate the same headers using runtime.GOOS
(normalized darwin → macos).
Server-side adds a ClientMetadata middleware that stashes the headers in
request context; the request logger and logger.RequestAttrs surface them
on every access log and handler-level log. Realtime hub logs the same
fields on websocket connect.
CORS allowlist extended for the new headers.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test: address client-identity PR nits
- Memoize the CoreProvider identity object on Web and Desktop, and key
WSProvider's effect on identity primitives instead of the object
reference, so unrelated parent re-renders no longer tear down and
reconnect the WebSocket.
- Add direct header-injection tests for the CLI and daemon Go HTTP
clients (X-Client-Platform/Version/OS) and a normalizeGOOS unit test
on both packages.
- Add a TS test for WSClient that asserts client_platform/client_version/
client_os land on the upgrade URL and never leak the auth token.
- Add a hub test that dials the WS endpoint with client_* query params
and asserts the "websocket connected" log entry surfaces them as
structured attributes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat(server): orphan-task recovery + auto-retry + manual rerun (MUL-1128)
When the daemon process crashed mid-task the issue was stuck at
in_progress for up to 2.5h: the in-flight task timeout was the only
mechanism that ever moved the row, and the runtime heartbeat sweeper
only fires after the runtime stays offline for 45s — a quick restart
beats both windows.
This change implements the A+B plan from the issue thread:
A. lifecycle hygiene
- migration 055 adds attempt / max_attempts / parent_task_id /
failure_reason / last_heartbeat_at to agent_task_queue
- new daemon-auth endpoint POST /runtimes/{id}/recover-orphans:
daemon calls it on every register so the server fails any
dispatched/running tasks the previous process left behind
- new daemon-auth endpoint POST /tasks/{id}/session: persists the
agent's session_id + work_dir mid-flight so a crash doesn't
lose the resume pointer (claude+codex emit MessageStatus with
SessionID; daemon forwards on the first one it sees)
- FailAgentTask / FailStaleTasks / FailTasksForOfflineRuntimes
now set failure_reason ('agent_error' / 'timeout' /
'runtime_offline')
B. auto-retry with resume context
- TaskService.MaybeRetryFailedTask spawns a fresh queued attempt
carrying parent's session_id/work_dir when the failure reason
is infrastructure-shaped (timeout, runtime_offline,
runtime_recovery) and attempt < max_attempts; skips autopilot
- wired into the runtime sweeper paths and TaskService.FailTask
so the user transparently sees a new in_progress run instead of
a stuck row
- new user-auth POST /api/issues/{id}/rerun + multica issue rerun
CLI for the manual escape hatch
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(server): address PR review for orphan-task recovery (MUL-1128)
Three review-must-fix items on top of the A+B implementation:
1. recover-orphans now funnels through TaskService.HandleFailedTasks,
the same shared post-failure pipeline used by the runtime sweeper.
This guarantees task:failed events are emitted, agent status is
reconciled, and issues stuck in_progress with no remaining active
task are reset to todo even when no auto-retry is created
(max_attempts exhausted, autopilot, non-retryable reason).
2. RerunIssue now uses CancelAgentTasksByIssueAndAgent, scoped to the
issue's current assignee. The previous implementation called
CancelAgentTasksByIssue, which would collateral-cancel parallel
@-mention agents on the same issue.
3. GetLastTaskSession now considers both completed and failed tasks
(mirroring GetLastChatTaskSession), ordering by the most recent
timestamp. With UpdateAgentTaskSession pinning session_id/work_dir
mid-flight, an auto-retry or manual rerun of a daemon-crash failure
now actually resumes the prior conversation context instead of
starting fresh — matching the stated B-branch behaviour.
go build / go vet pass; the existing service and agent test suites pass.
runtime_sweeper / handler integration tests require a local DB with the
055 migration (and the pre-existing 050 first_executed_at column).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds a first-class `model` field on agents so users can pick the LLM model from the create / settings UI instead of editing `custom_env` / `custom_args`. Each provider's dropdown is populated from the live CLI when possible (`opencode models`, `pi --list-models`, `openclaw agents list --json`, `cursor-agent --list-models`, hermes ACP `session/new` → `SessionModelState`), with a static catalog for providers that don't enumerate.
Daemon resolves the runtime model as `agent.model → MULTICA_<PROVIDER>_MODEL → ""` — empty passes through so each backend's CLI picks its own default, avoiding static-guess drift.
Per-provider honouring:
- Claude / Codex / OpenCode / Cursor / Gemini / Pi / Copilot — CLI `--model` / thread payload.
- OpenClaw — `opts.Model` is mapped to `--agent <name>` (the CLI rejects `--model`).
- Hermes — `session/set_model` ACP RPC; stderr is sniffed for provider-level errors so HTTP 4xx from the configured LLM surfaces instead of "empty output"; explicit-model failures mark the task `failed`.
Supporting changes: migration 050 adds `agent.model`; daemon ↔ server heartbeat piggyback carries a model-discovery request; new REST endpoints under `/api/runtimes/{id}/models`; `multica agent create --model` / `update --model`; shared `ModelDropdown` in `packages/views/agents` (searchable, creatable, provider-grouped, default-badge, runtime-supported gate).
* fix(chat): preserve chat session resume pointer across failures
The chat 'forgets earlier messages' bug came from PriorSessionID being
silently lost in several edge cases:
- UpdateChatSessionSession unconditionally overwrote chat_session.session_id,
so any task that completed without a session_id (early agent crash,
missing result) wiped the resume pointer to NULL.
- CompleteAgentTask + UpdateChatSessionSession ran in separate calls. A
follow-up chat message claimed in between resumed against a stale (or
NULL) session and started over.
- FailAgentTask never wrote session_id back, so a task that established
a real session before failing lost its resume pointer.
- ClaimTaskByRuntime only trusted chat_session.session_id and never
fell back to the existing GetLastChatTaskSession query, so a single
bad turn could permanently drop the conversation memory.
This change:
- Use COALESCE in UpdateChatSessionSession so empty inputs preserve the
existing pointer; surface DB errors instead of swallowing them.
- Run CompleteAgentTask/FailAgentTask + UpdateChatSessionSession inside
the same transaction (TaskService now takes a TxStarter).
- Extend FailAgentTask + the daemon FailTask path (client, handler,
service) to forward session_id/work_dir, so failed/blocked tasks that
built a real session still record it.
- Fall back to GetLastChatTaskSession in ClaimTaskByRuntime when the
chat_session pointer is missing, and include failed tasks in that
lookup so a single failure can't lose the conversation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(daemon): forward session_id/work_dir on blocked + timeout paths
runTask previously dropped result.SessionID and env.WorkDir on the
non-completed return paths:
- timeout returned a naked error, so handleTask called FailTask with
empty session info and the chat resume pointer was either left stale
or eventually overwritten with NULL.
- blocked / failed (default branch) returned a TaskResult without
SessionID / WorkDir, so even though FailTask now COALESCEs into
chat_session, there was no value to write through.
- the empty-output completion path was the same: it raised an error
even when a real session_id had been built.
All three paths now return a TaskResult that carries the SessionID /
WorkDir the backend produced. Combined with the COALESCE-based update
in UpdateChatSessionSession and the FailTask plumbing introduced in
PR #1360, the next chat turn can always resume from the latest agent
session — even when the previous turn timed out, was rate-limited, or
returned an empty completion — instead of starting over with no memory
of the conversation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(copilot): capture session id from session.start as fallback
The Copilot backend only read sessionId from the synthetic 'result'
event, ignoring the one already present on session.start. When the CLI
was killed before result arrived (timeout, cancel, crash, or a
session.error mid-turn), the daemon reported SessionID="" and the
chat-session resume pointer could not advance — causing the chat to
silently drop conversation memory on the next turn.
Capture session.start.sessionId into state up front, and only let
'result' overwrite it when it actually carries one. result still wins
when present (it is the authoritative end-of-turn record).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(copilot): parse premiumRequests as float to preserve session id
Copilot CLI v1.0.32 serializes premiumRequests as a float (e.g. 7.5),
not an integer. Our copilotResultUsage struct typed it as int, which
made the entire 'result' line fail json.Unmarshal — silently dropping
sessionId on every turn.
This was the real cause of chat memory loss: the daemon reported
SessionID="" to the server, chat_session.session_id stayed NULL, and
the next chat turn never received --resume <id>, so each turn started
a fresh Copilot session with no prior context.
Add a regression test using the real JSON line from CLI v1.0.32 that
asserts sessionId is preserved when premiumRequests is fractional.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Devv <devv@Devvs-Mac-mini.local>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Eve <eve@multica.ai>
Co-authored-by: yushen <ldnvnbl@gmail.com>
* refactor(runtime): derive runtime usage from task_usage only
The daemon used to scan each runtime's local CLI log directory every 5
minutes (Claude Code, Codex, OpenCode, OpenClaw, Hermes) and post daily
aggregates to /api/daemon/runtimes/{id}/usage. Those directories are
shared with the user's own local CLI sessions, so the user's personal
usage was being counted as Daemon-executed usage. Cursor and Gemini had
no scanner at all, so their runtime-level aggregates were always zero.
Switch GetRuntimeUsage to aggregate task_usage (already scoped to
Daemon-executed tasks) via agent_task_queue.runtime_id. Single source of
truth; Cursor/Gemini/Copilot get runtime usage for free; no reliance on
external CLI log formats.
Removes:
- server/internal/daemon/usage/ (all scanners)
- Daemon.usageScanLoop + providerToRuntimeMap
- Client.ReportUsage
- ReportRuntimeUsage handler + POST /api/daemon/runtimes/{id}/usage
- UpsertRuntimeUsage / GetRuntimeUsageSummary queries
- runtime_usage table (migration 046)
Refs: MUL-786
* fix(runtime): bucket daily usage by task_usage.created_at, not enqueue time
ListRuntimeUsage was aggregating by DATE(atq.created_at) and filtering
on atq.created_at. agent_task_queue.created_at is the enqueue timestamp,
which drifts from actual token-production time: a task queued at 23:58
and executed at 00:05 was attributed to yesterday; a task sitting in
the queue overnight was counted on the queue day.
The ?days=N cutoff also became a rolling window (now() - N) instead of
a calendar-day boundary, silently clipping the morning of the earliest
day returned.
Switch bucket + filter to task_usage.created_at (~= task completion /
usage-report time) and snap the since cutoff to start-of-day via
DATE_TRUNC.
Add a regression test covering both scenarios: cross-midnight task
attributes to the day tokens were reported, and the earliest day's
pre-cutoff rows are still included.
Isolation directories accumulate indefinitely because they're preserved
for session reuse but never cleaned up after the issue is closed.
This adds a background GC loop that periodically scans local workspace
directories and removes those whose issue is done/canceled and hasn't
been updated for 5 days (configurable via MULTICA_GC_TTL). Orphan
directories with no metadata are cleaned after 30 days.
Changes:
- Write .gc_meta.json (issue_id, workspace_id) at task completion
- Add GET /api/daemon/issues/{issueId}/gc-check endpoint for status queries
- Add gcLoop goroutine to daemon with configurable interval/TTL
- Prune stale git worktree references from bare repo caches each cycle
- New env vars: MULTICA_GC_ENABLED, MULTICA_GC_INTERVAL, MULTICA_GC_TTL,
MULTICA_GC_ORPHAN_TTL
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. Separate ReportTaskUsage endpoint (POST /api/daemon/tasks/{id}/usage)
so usage is captured independently of complete/fail — fixes usage loss
for failed/blocked tasks.
2. Add usage tracking for all four providers:
- Claude: already done (stream-json message.usage)
- OpenCode: extract from step_finish.part.tokens
- OpenClaw: extract from step_end.data token fields
- Codex: extract from turn/completed and task_complete usage fields
3. Remove usage from CompleteTask payload — all usage goes through the
dedicated endpoint now.
Extract token usage from Claude Code's stream-json output in real-time
during task execution, replacing the inaccurate global JSONL log scanner.
- New `task_usage` table: tracks (task_id, provider, model) level usage
- Agent SDK: parse `message.usage` from assistant messages, accumulate
per-model and return in Result
- Daemon: convert agent usage to entries, send with CompleteTask
- Server: store usage on task completion, expose workspace-level
aggregation APIs (GET /api/usage/daily, GET /api/usage/summary)
* feat(runtime): support CLI update from web runtime page
Add the ability to update the CLI daemon from the web Runtime detail page.
When a newer version is available on GitHub Releases, an update button
appears. Clicking it sends an update command through the server to the
daemon via the heartbeat mechanism (same pattern as ping). The daemon
executes `brew upgrade`, reports the result, and restarts itself with the
new binary.
Changes across all three layers:
- Frontend: version display, GitHub latest check, UpdateSection component
- Server: UpdateStore (in-memory), heartbeat extension, 3 new endpoints
- CLI: shared update logic, daemon handleUpdate + graceful restart
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(runtime): handle 'running' status in ReportUpdateResult
The daemon sends {"status":"running"} when it starts executing the
update, but ReportUpdateResult treated any non-"completed" status as
failure — immediately marking the update as failed before brew upgrade
even ran.
Fix: use a switch statement to handle "running" as a no-op (status is
already "running" from PopPending), and also timeout running updates
after 120 seconds in case brew upgrade hangs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When an agent is working on an issue, users can now see real-time output
in the issue detail page instead of waiting for completion.
Backend:
- Add task_message table and migration for persisting agent messages
- Add POST /api/daemon/tasks/{id}/messages endpoint for daemon to report
structured messages (tool_use, tool_result, text, error) in batches
- Add GET /api/daemon/tasks/{id}/messages for catch-up after reconnect
- Add GET /api/issues/{id}/active-task to check for running tasks
- Broadcast task:message events via WebSocket
- Daemon forwards agent session messages with 500ms text throttling
Frontend:
- Add AgentLiveCard component showing live tool calls, text output,
and progress indicators with auto-scroll
- Wire into issue detail timeline with WS subscription and HTTP catch-up
- Card appears when agent is working, disappears on completion/failure
The daemon now periodically fetches the user's workspace list from the
API (every 30s) and adds any new workspaces to the watched config. The
existing config-watch loop then picks up the change and registers
runtimes. This fixes the issue where workspaces created after
`multica login` were not discovered until the daemon was restarted.
Agents now decide which repo to use based on issue context and check out
repos on demand via `multica repo checkout <url>`. Workspace repos are
cached locally as bare clones for fast worktree creation.
Key changes:
- Add repocache package for bare clone management (clone, fetch, worktree)
- Add `multica repo checkout` CLI command that talks to local daemon
- Add POST /repo/checkout endpoint on daemon health server
- Pass workspace repos metadata through register + task claim responses
- Remove pre-created worktrees from execenv (workdir starts empty)
- Update CLAUDE.md template to instruct agents to use `multica repo checkout`
- Pass MULTICA_DAEMON_PORT, WORKSPACE_ID, AGENT_NAME, TASK_ID env vars to agent
- Coalescing queue: use HasPendingTaskForIssue (queued/dispatched only)
instead of HasActiveTaskForIssue so comments during a running task
enqueue exactly one follow-up task that picks up all new comments.
- Stale task cleanup: runtime sweeper now fails orphaned tasks when
their runtime goes offline (daemon crash/network partition).
- Cancel-aware daemon: handleTask checks task status after execution
and discards results if the task was cancelled mid-run (e.g. reassign).
- Terminal issue guard: ClaimTaskForRuntime auto-cancels pending tasks
for done/cancelled issues instead of executing them.
- Race condition safety net: unique partial index ensures at most one
pending task per issue at the DB level.
Store the Claude Code session ID and working directory when a task
completes. On the next task for the same (agent, issue) pair, look up
the prior session and pass --resume <session_id> to Claude Code so
the agent retains conversation context across multiple tasks on the
same issue.
Changes:
- Migration 020: add session_id and work_dir columns to agent_task_queue
- CompleteAgentTask stores session_id and work_dir on completion
- GetLastTaskSession query retrieves prior session for (agent, issue)
- ClaimTaskByRuntime handler populates prior_session_id in response
- Daemon passes ResumeSessionID through to Claude backend Execute()
- Claude backend adds --resume flag when ResumeSessionID is set
Replace the frozen context snapshot pattern with a CLI-driven approach:
agents now use `multica` CLI commands to fetch issue details, comments,
and workspace context on demand, always getting the latest data.
- Remove buildContextSnapshot and snapshot generation from enqueue
- Claim endpoint now returns fresh agent name + skills from DB
- Daemon resolves provider from local runtimeIndex, not snapshot
- Prompt instructs agent to use `multica issue get` / `comment list`
- Meta skill (CLAUDE.md/AGENTS.md) documents all available CLI commands
- Skills still injected as filesystem files (static agent config)
- Simplify daemon types: remove TaskContext/IssueContext/RuntimeContext
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adapt runtime features (usage tracking, ping, heartbeat) to main's
multi-workspace architecture. Update frontend imports from @multica/types
to @/shared/types after the package consolidation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a new "Runtimes" sidebar tab to manage local agent runtimes with three
main capabilities: runtime status overview, token usage tracking (reading
Claude Code and Codex CLI local JSONL logs via daemon), and an interactive
connection test that sends a ping through the daemon to verify end-to-end
agent CLI connectivity.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove unused ListWorkspaces/Workspace from daemon client, add log when
default workspace is set implicitly, document token reload limitation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The daemon now reads the auth token from ~/.multica/config.json (set by
`multica auth login`) instead of requiring a browser-based pairing flow.
If not authenticated, it logs a message and exits.
Workspace ID is auto-resolved from the user's workspaces when not
explicitly set via flag/env.
Removed: daemon.json, pairing session flow, --config-path flag,
PairingSession type, PersistedConfig, LoadWorkspaceIDFromDaemonConfig.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Introduce the `execenv` package that creates isolated working directories
for each agent task. Supports git worktree mode (code tasks) and plain
directory mode (non-code tasks), with `.agent_context/issue_context.md`
injected into the workdir for Claude Code to discover.
Key changes:
- New `server/internal/daemon/execenv/` package (Prepare/Cleanup)
- `runTask()` now creates isolated env instead of using shared reposRoot
- Prompt updated to reference `.agent_context/` files
- Add `WorkspacesRoot` config (default ~/multica_workspaces)
- Add `KeepEnvAfterTask` config for debugging
- Default agent timeout increased from 20min to 2h
- `CompleteTask` now forwards branch name to server
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Resolve conflicts:
- CLAUDE.md: merge feature-based frontend docs from main with comprehensive
architecture docs from HEAD
- Makefile: merge .PHONY targets from both branches
- settings/page.tsx: keep Context field using main's Label component
- auth-context.test.tsx: accept main's deletion (moved to features/auth/)
- cmd/daemon/daemon.go: accept HEAD's deletion (moved to internal/daemon/)
- daemon/client.go: port requestError type and isWorkspaceNotFoundError from
main's old daemon into the new internal/daemon package
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. Add Client.SendHeartbeat/Register methods — no more direct postJSON calls
2. Use url.Values for query params to prevent URL injection
3. Unexport helpers (envOrDefault, durationFromEnv, sleepWithContext)
4. CLI resolveWorkspaceID falls back to daemon.json
5. Implement agent stop (PUT /api/agents/{id} with status=offline)
6. Add --output flag to agent get for consistent UX
7. Add server/multica to .gitignore for stray builds
8. Inject version/commit via -ldflags in Makefile build target
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract daemon logic from cmd/daemon/ into internal/daemon/ package and
create a new unified CLI entry point at cmd/multica/ using cobra. The CLI
supports `daemon` as a long-running subcommand plus ctrl subcommands for
agent/runtime management, config, status, and version.
Server, migrate, and seed binaries remain unchanged.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
