* fix(server): persist ModelListStore across replicas via Redis
The model picker uses a pending-request pattern: the frontend POSTs to
create a request, the daemon pops it on its next heartbeat, runs
agent.ListModels locally, and reports back. Until now the store was a
plain in-memory map per Handler instance.
That works for self-hosted single-instance deploys but fails in any
multi-replica environment (Multica Cloud). Each replica has its own
map, so:
POST /runtimes/:id/models → request stored in replica A
GET /runtimes/:id/models/<requestId> → polls land on B/C → 404
daemon heartbeat → only A sees PendingModelList
POST .../<requestId>/result → daemon's report has to land on A
Success probability ~1/N². The visible symptom is "No models available"
in the picker for every provider, even those (Claude/Codex) whose
catalog is statically populated end-to-end.
Same shape of bug, same Redis-backed fix as multica-ai/multica#1557 did
for LocalSkillListStore / LocalSkillImportStore. Reuse the operational
playbook (namespaced keys, ZSET-backed pending queue, atomic
ZREM+SET-running via the shared Lua script) so we don't introduce a
second concurrency model for the same primitive.
Changes:
- Convert ModelListStore from struct to interface with context-aware
methods. Add HasPending for cheap heartbeat-side probing.
- InMemoryModelListStore — single-node fallback, used when REDIS_URL
is unset (self-hosted dev / tests).
- RedisModelListStore — multi-node implementation using the same key
layout and Lua atomic claim as RedisLocalSkillListStore.
- Use RunStartedAt (not UpdatedAt) as the running-timeout reference
point, matching the local-skill stores so subsequent UpdatedAt
bumps don't reset the running clock.
- Heartbeat now uses the probe-then-pop pattern for the model queue
(matching local-skills) so a slow Redis can't stall every connected
daemon. Extends heartbeatMetrics + slow-log with probe_model_ms /
pop_model_ms / probe_model_timed_out for parity.
- Wire the Redis backend in NewRouterWithOptions when rdb != nil.
- Tests for both backends. Redis tests gate on REDIS_TEST_URL so
laptop runs without Redis still pass; CI provides it.
Co-authored-by: multica-agent <github@multica.ai>
* fix(server): persist RunStartedAt + retry model report on transient failures
Two follow-ups from PR #2022 review:
1. RedisModelListStore was dropping ModelListRequest.RunStartedAt on
persistence — the field is tagged json:"-" so it doesn't leak into
the HTTP response, which made plain json.Marshal(req) silently
discard it. Across-node readers saw RunStartedAt=nil and
applyModelListTimeout's running branch became a no-op, so the 60s
running-timeout escape hatch never fired. CI's
TestRedisModelListStore_RunningTimeout was failing on this exact
case. Fix mirrors RedisLocalSkillImportStore's envelope pattern —
wrap in an internal struct that re-promotes the field. HTTP shape
stays clean. Adds a no-Redis unit test that pins the round trip.
2. Daemon's handleModelList called d.client.ReportModelListResult
directly and swallowed any 5xx, leaving the pending request
stranded in "running" until its 60s server-side timeout — exactly
the failure mode the multi-node store fix was meant to eliminate.
Generalize the existing local-skill retry helper into
reportRuntimeResultWithRetry (kind: model_list / local_skill_list /
local_skill_import) and wire handleModelList through a new
reportModelListResult helper. Renames the test-overridable
var localSkillReportBackoffs → runtimeReportBackoffs to match.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
* refactor(server): make ParseUUID error-returning to prevent silent data loss (MUL-1410)
util.ParseUUID previously swallowed errors and returned a zero pgtype.UUID
on invalid input. When this zero UUID reached a write query (DELETE/UPDATE),
the SQL matched zero rows and the handler returned 2xx success — producing
silent data corruption. #1661 (DeleteIssue with identifier-style ID) was the
visible symptom; PR #1680 patched that one site, this commit closes the
class of bug.
Changes:
- util.ParseUUID now returns (pgtype.UUID, error). Add util.MustParseUUID
for trusted round-trips that should panic on invalid input.
- handler/handler.go: parseUUID wrapper now calls MustParseUUID — any
unguarded user-input string reaching it surfaces as a recovered panic
(chi middleware.Recoverer → 500) instead of silently corrupting data.
Add parseUUIDOrBadRequest(w, s, fieldName) for handler entry points.
- Convert every Queries.Delete*/Update* call site reachable from raw user
input (autopilot, comment, project, skill, skill_file, label, pin,
attachment, feedback, issue assignee, daemon runtime, workspace) to
validate UUIDs explicitly with parseUUIDOrBadRequest, returning 400 on
invalid input. Where a resolved entity.ID is already in scope, write
queries now use it directly instead of re-parsing the URL string.
- Update getWorkspaceMember + loadIssueForUser to handle invalid UUIDs
gracefully (404/400 instead of panic).
- Update util/middleware/cmd-level callers (subscriber_listeners,
notification_listeners, activity_listeners, scope_authorizer,
middleware/workspace) to use the error-returning API.
- Add server/internal/util/pgx_test.go covering valid/invalid input and
the MustParseUUID panic contract.
- Add TestDeleteIssueByIdentifier + TestDeleteIssueRejectsInvalidUUID
regression tests in handler_test.go (the original #1661 bug + the
invalid-input case).
- Document the handler UUID parsing convention in CLAUDE.md so the rule
is enforceable in future PR review.
* fix(server): address GPT-Boy review of #1748
P1 fixes from PR #1748 review:
1. Migrate remaining request-boundary UUIDs to parseUUIDOrBadRequest so
malformed input returns 400 instead of panic/500. Was missing on:
- issue.go: workspace_id in CreateIssue/ChildIssueProgress/ListIssues/
SearchIssues/BatchUpdateIssues/BatchDeleteIssues; project_id /
parent_issue_id / lead_id / assignee_id / assignee_ids / creator_id
filters; batch issue_ids and assignee/parent/project fields in
BatchUpdateIssues (skip on bad input via util.ParseUUID, matching
the existing per-row continue semantics).
- project.go: project id + workspace_id in GetProject/UpdateProject/
DeleteProject; lead_id in CreateProject/UpdateProject;
workspace_id in ListProjects + SearchProjects.
- handler.go: resolveActor now uses util.ParseUUID for X-Agent-ID /
X-Task-ID headers; invalid UUID falls back to "member" (matches
pre-existing semantics) instead of panicking.
- issue.go: validateAssigneePair returns 400 on invalid workspace_id
instead of panicking.
2. Fix issue:deleted WS event payloads to emit uuidToString(issue.ID)
instead of the raw URL string. After an identifier-path delete
("MUL-7"), the previous payload would have leaked the identifier to
subscribers, leaving stale entries in frontend caches that key by
UUID. Updated DeleteIssue (issue.go:1341) and BatchDeleteIssues
(issue.go:1641). The slog "issue deleted" log line also now records
the resolved UUID so logs match the WS payload.
3. Extend TestDeleteIssueByIdentifier to subscribe to the bus and
assert issue:deleted.payload.issue_id is the resolved UUID, not
the identifier.
* fix(server): validate remaining reviewed UUID inputs
* fix(server): validate remaining handler UUID inputs
* fix(server): finish request boundary UUID audit
* fix(server): validate remaining request body UUIDs
* fix(server): validate runtime path UUIDs
* fix(server): validate remaining audit UUID inputs
---------
Co-authored-by: Eve <eve@multica.ai>
Mitigates #1637 and the related model-discovery failure in MUL-1397 by bounding the /api/daemon/heartbeat hot path with an ack-safe probe/claim split, adding structured slow-log attribution, and closing the ModelListStore running-state gap. See PR description for details.
* feat(runtimes): remove Test Connection / runtime ping feature
The Test Connection action invoked a real single-turn agent run to verify
runtime connectivity. In practice it was expensive (reuses none of the
normal task exec env, so it also gave misleading results) and low value —
daemon heartbeat + Online status already covers the "is the runtime
alive" question. Dropping the whole end-to-end probe path:
- deletes server handler and in-memory PingStore
- drops pending_ping from the heartbeat response and daemon poll loop
- removes daemon.handlePing, PendingPing, ReportPingResult
- removes the CLI `multica runtime ping` command
- removes the PingSection UI block and RuntimePing types / api methods
* docs: fix runtime CLI subcommand list in product-overview
Adds a first-class `model` field on agents so users can pick the LLM model from the create / settings UI instead of editing `custom_env` / `custom_args`. Each provider's dropdown is populated from the live CLI when possible (`opencode models`, `pi --list-models`, `openclaw agents list --json`, `cursor-agent --list-models`, hermes ACP `session/new` → `SessionModelState`), with a static catalog for providers that don't enumerate.
Daemon resolves the runtime model as `agent.model → MULTICA_<PROVIDER>_MODEL → ""` — empty passes through so each backend's CLI picks its own default, avoiding static-guess drift.
Per-provider honouring:
- Claude / Codex / OpenCode / Cursor / Gemini / Pi / Copilot — CLI `--model` / thread payload.
- OpenClaw — `opts.Model` is mapped to `--agent <name>` (the CLI rejects `--model`).
- Hermes — `session/set_model` ACP RPC; stderr is sniffed for provider-level errors so HTTP 4xx from the configured LLM surfaces instead of "empty output"; explicit-model failures mark the task `failed`.
Supporting changes: migration 050 adds `agent.model`; daemon ↔ server heartbeat piggyback carries a model-discovery request; new REST endpoints under `/api/runtimes/{id}/models`; `multica agent create --model` / `update --model`; shared `ModelDropdown` in `packages/views/agents` (searchable, creatable, provider-grouped, default-badge, runtime-supported gate).
