Files
multica/server/internal/handler/daemon_batch_claim_test.go
Multica Eve c3dd9ec845 Machine-level batch task claim endpoint (MUL-4257) (#5193)
* feat(daemon-claim): machine-level batch task claim endpoint (MUL-4257)

Collapse the per-runtime /tasks/claim poll fan-out into a single machine-level
batch claim to cut /api/daemon claim request volume.

Server:
- agent.sql: = ANY(runtime_ids) batch variants of the claim queries
  (ListQueuedClaimCandidatesByRuntimes, PromoteDueDeferredTasksForRuntimes,
  ReclaimStaleDispatchedTasksForRuntimes); runtime.sql: GetAgentRuntimes(= ANY)
  so a whole machine's runtimes are resolved/promoted/reclaimed/listed in a
  constant number of queries instead of N.
- service.ClaimTasksForRuntimes: claim up to max_tasks across a runtime set,
  preserving per-(issue,agent) serialization, the concurrency cap, the
  empty-claim cache short-circuit, and every dispatch side effect. Batch
  promote replays the per-row side effects (task:queued + empty-cache Bump).
- handler.ClaimTasksByRuntime (canonical POST /api/daemon/tasks/claim, with a
  transitional /claim alias): validates daemon_id (required; must match the
  mdt_ token) and rejects runtimes bound to a different daemon (group-ownership
  check mirroring the WS path); resolves+authorizes each runtime_id; claims;
  and finalizes each task through the SAME FinalizeTaskClaim as the per-runtime
  endpoint (atomic token + delivered_comment_ids receipt), requeueing the exact
  claim and omitting it on failure. buildClaimedTaskResponse is extracted from
  the per-runtime handler and returns the delivered-comment ids plus a
  structured *claimBuildFailure so both paths share identical payload building
  and failure semantics (workspace-isolation, chat-input load/empty).
- max_tasks: negative -> 400, zero -> empty (never coerce to 1), positive
  capped at 32. runtime_ids parsed with non-panicking util.ParseUUID.

Daemon:
- Client.ClaimTasks posts daemon_id + runtime set + free-slot count to the
  canonical path under a short request-scoped timeout, bounding the
  head-of-line coupling the per-runtime pollers avoid (MUL-1744).

Tests: service batch drain / max_tasks cap / deferred-promote receipt /
finalize-failure rollback+requeue; handler routing + token, cross-workspace
skip, cross-daemon skip, daemon_id required, owner-missing cancel,
max_tasks=0/negative, invalid-uuid skip, comment delivery receipt, stale-reclaim
replacement receipt; client posts/parses (daemon_id + canonical path).

Follow-up: cut the daemon pollLoop over to a single batched poller (flips the
MUL-1744 isolation contract; needs its concurrency tests redesigned).

Co-authored-by: multica-agent <github@multica.ai>

* feat(daemon-ws): generic WS request/response transport for daemon RPC (MUL-4257)

Add a generic daemon->server request/response layer over the existing WS
control connection, the transport for WS-first claim (HTTP fallback):
- protocol: daemon:rpc_request / daemon:rpc_response envelopes with a
  correlation request_id + method + body, and an rpc-v1 capability gate.
- daemonws.Hub: SetRPCHandler + goroutine-dispatched handleRPCFrame (bounded
  by a per-connection in-flight cap) that echoes the request_id; missing
  handler / saturation return non-2xx so the daemon falls back to HTTP.
  Read limit raised to 64KB for rpc requests carrying a runtime set.
- hub tests: round-trip, handler-error->non-2xx, no-handler->503.

Co-authored-by: multica-agent <github@multica.ai>

* feat(daemon-ws): WS-first task claim over the generic RPC transport (MUL-4257)

Bind claim to the WS request/response layer, with HTTP fallback:
- server: handler.DaemonRPCHandler adapts a daemon:rpc_request (method
  tasks.claim) to the existing HTTP ClaimTasksByRuntime via a synthetic
  in-process request carrying the WS connection's identity (daemon_id +
  workspace + capabilities), so all auth / payload-building / finalization is
  reused unchanged. Wired via daemonHub.SetRPCHandler. ClientIdentity now
  captures X-Client-Capabilities so capability gating matches the HTTP path.
- daemon: wsRPCClient correlates responses by request_id over the shared WS
  connection; attached to the live connection's write channel (guarded so a
  Call racing teardown never sends on a closed channel) and detached on
  disconnect. rpc_response frames are routed in the read loop.
  Daemon.ClaimTasksWSFirst issues tasks.claim over WS and falls back to the
  HTTP claim endpoint on any transport failure (no conn / buffer full /
  timeout) — wired into the poller at the poller cutover.
- tests: handler tasks.claim RPC end-to-end (claims + dispatches) + unknown
  method 404; daemon wsRPCClient round-trip / timeout / unavailable /
  server-error / detach-fails-pending (all under -race).

Co-authored-by: multica-agent <github@multica.ai>

* feat(daemon): cut claim poller over to machine-level ClaimTasksWSFirst (MUL-4257)

Replace the per-runtime HTTP poll loop with a single batch poller: each cycle
acquires all free execution slots (slot-before-claim) and issues ONE
ClaimTasksWSFirst across every runtime the daemon hosts (WS-first, HTTP
fallback), dispatching each returned task to its runtime. Wakeups (targeted /
catch-up / runtime-set change) collapse to one nudge. Removes runRuntimePoller
+ runtimePollOffset. The WS handshake now advertises the same capabilities as
HTTP (+ rpc-v1) so WS-built claim payloads keep skill-ref / coalesced-comment
gating.

Trades per-runtime isolation (MUL-1744) for one request, bounded by the short
per-request WS timeout / client timeout. Tests: batch poller claims across
runtimes + skips-at-capacity + pollLoop shutdown drain (replacing the
per-runtime poller tests); heartbeat isolation + runtime-set watcher kept.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon-ws): WS RPC disconnect-race panic + batch stale-comment-plan repair (MUL-4257)

Two PR #5193 review blockers:

1) WS RPC send-on-closed-channel race, both ends:
   - server: give each connection a cancelable ctx (cancelled on readPump
     teardown) and run the RPC handler under it, so a slow claim stops on
     disconnect; guard c.send with sendMu/sendClosed (trySend) so a late RPC
     response goroutine never writes to the closed channel. Heartbeat ack routed
     through the same guard.
   - daemon: wsRPCClient.deliver now sends under the mutex, serialized with
     attach(nil)'s close+delete, so a delivered response can't hit a channel
     the detach path just closed.
   - regressions (-race): daemon deliver-vs-detach; server
     disconnect-during-handler-response.

2) batch claim now runs the stale-comment-plan repair: extracted the
   per-runtime handler's repair (trigger deleted, only coalesced survive ->
   cancel + replay survivors) into shared repairStaleCommentPlanIfNeeded, called
   by both claim paths. Prevents the batch path (now the default poller) from
   finalizing+dispatching a task with no comment input and silently dropping the
   surviving user comment. Regression: batch omits the stale task, cancels it,
   and rebuilds the survivor into a new trigger plan.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon-ws): server-side RPC deadline + legacy claim fallback (MUL-4257)

Two review blockers:

1) WS RPC timeout/fallback (GPT-Boy): the daemon's WS wait didn't cancel
   server-side claim, so a slow WS claim could commit after the daemon fell
   back to HTTP, leaking dispatched tasks and breaking the free-slot bound.
   Fix: RPC envelope carries TimeoutMs; the server bounds the handler ctx by it
   (so ClaimTasksByRuntime's tx is cancelled/rolled back at the deadline), and
   the daemon waits budget + grace so a claim that committed before the deadline
   still reports back. A committed-then-unreported claim degrades to the same
   stale-reclaim safety net as HTTP, never a double effective claim. Regression:
   server-side TimeoutMs cancels the handler.

2) Backward compat (Terra-Boy): a new daemon against a server without the batch
   route (/api/daemon/tasks/claim 404) couldn't claim. Fix: ClaimTasksWSFirst
   falls back to the legacy per-runtime ClaimTask loop on a batch 404 and caches
   'batch unsupported' (reset on WS reconnect to re-probe after a server
   upgrade). Regression: server exposing only the legacy route.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon-ws): no double-claim on WS teardown/detach (MUL-4257)

Sol-Boy review blocker: on reconnect, teardown failed the pending RPC (→ HTTP
fallback) but then flushed the queued tasks.claim frame to the still-alive
socket, so the server committed the WS claim on top of the HTTP one — double
claim, WS batch orphaned to stale reclaim, breaking the free-slot bound.

- Teardown now closes the connection FIRST, so runWSWriter discards the queued
  RPC frame (write error path) instead of delivering it.
- A detach while a claim's frame is already in flight now returns a distinct
  errWSRPCUncertain; ClaimTasksWSFirst does NOT HTTP-fall-back on uncertain (the
  WS claim may have committed) — it skips the cycle and lets reclaim / the next
  poll recover. Genuine 'not sent' / timeout still fall back (safe: the
  server-side deadline guarantees no uncommitted claim by budget+grace).
- Regression: detach during an in-flight WS claim asserts zero HTTP claims
  (at most one path claims); plus the existing detach/deliver-race and
  server-timeout tests.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon-ws): cancelable RPC frames close the backpressure double-claim (MUL-4257)

Sol-Boy review blocker: the client's response budget starts at enqueue, but
the socket write is async (10s write deadline). A backpressured writer could
hold a tasks.claim in the local queue past the client timeout — the daemon
HTTP-fell-back, then the writer woke and delivered the stale WS frame, so the
server committed it too: same free slots claimed twice. No detach occurs, so
the prior errWSRPCUncertain fix did not cover it.

- WS frames are now cancelable (wsOutbound{sent,canceled} under a mutex). The
  writer calls beginWrite() before WriteMessage and skips cancelled frames.
- On give-up (timeout / detach / ctx), Call cancels the queued frame: if it was
  still pending the cancel wins and the frame is guaranteed never delivered
  (errWSRPCUnavailable → safe HTTP fallback); if the writer already began
  sending it the cancel loses and the outcome is errWSRPCUncertain (no
  fallback). The decision is atomic, so at most one transport claims.

Tests: wsOutbound cancel-before-write vs write-before-cancel; Call timeout
cancels an unsent frame (writer then drops it) vs uncertain when already sent;
plus the updated detach and existing timeout/race tests.

Co-authored-by: multica-agent <github@multica.ai>

* fix(batch-claim): return partial success instead of dropping committed claims (MUL-4257)

Sol-Boy review blocker: ClaimTasksForRuntimes reclaims (step 2) and claims per
agent (step 6) in independent transactions, but a step-4 candidate-SELECT error
or a mid-loop ClaimTask error did 'return nil, err' — discarding tasks already
committed as dispatched. The handler 500s; the daemon sees a definite (non-
uncertain) 500 and HTTP-falls-back, claiming a SECOND batch into the same free
slots while the first batch waits for stale reclaim — the double-claim this PR
removes.

- Both error paths now prefer partial success: if any task has already
  committed (claimed non-empty), return it (nil error) so the handler finalizes
  and returns 200; the errored candidates stay queued for the next poll. The
  remaining error is logged. Only a genuinely empty result still returns the
  error (safe: no committed claim to lose, HTTP fallback just re-fails).

Regression (internal/service, DB-backed, fault-injected):
- PartialSuccessOnSecondAgentClaimFailure: fail the 2nd ClaimTask's Begin →
  the first agent's committed task is returned, not dropped.
- PartialSuccessOnCandidateQueryFailureAfterReclaim: a stale dispatched task is
  reclaimed, then the candidate SELECT fails → the reclaimed task is returned.

Co-authored-by: multica-agent <github@multica.ai>

---------

Co-authored-by: Eve <eve@multica-ai.local>
Co-authored-by: multica-agent <github@multica.ai>
2026-07-14 11:53:42 +08:00

195 lines
8.2 KiB
Go

package handler
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
// batchClaimResponse mirrors the {"tasks":[...]} envelope ClaimTasksByRuntime
// returns, with the few fields these tests assert on.
type batchClaimResponse struct {
Tasks []struct {
ID string `json:"id"`
RuntimeID string `json:"runtime_id"`
AuthToken string `json:"auth_token"`
} `json:"tasks"`
}
func seedQueuedIssueTask(t *testing.T, ctx context.Context, agentID, runtimeID, issueID string) string {
t.Helper()
var id string
if err := testPool.QueryRow(ctx, `
INSERT INTO agent_task_queue (agent_id, runtime_id, issue_id, status, priority)
VALUES ($1, $2, $3, 'queued', 0)
RETURNING id
`, agentID, runtimeID, issueID).Scan(&id); err != nil {
t.Fatalf("seed queued task: %v", err)
}
t.Cleanup(func() { testPool.Exec(ctx, `DELETE FROM agent_task_queue WHERE id = $1`, id) })
return id
}
func postBatchClaim(t *testing.T, workspaceID string, runtimeIDs []string, maxTasks int) *httptest.ResponseRecorder {
t.Helper()
w := httptest.NewRecorder()
req := newDaemonTokenRequest("POST", "/api/daemon/tasks/claim",
map[string]any{"daemon_id": batchClaimTestDaemonID, "runtime_ids": runtimeIDs, "max_tasks": maxTasks},
workspaceID, batchClaimTestDaemonID)
testHandler.ClaimTasksByRuntime(w, req)
return w
}
// batchClaimTestDaemonID is the daemon id used by both the mdt_ token context
// and the request body in batch-claim handler tests, so the daemon_id
// consistency check passes on the happy path.
const batchClaimTestDaemonID = "batch-claim-review"
// TestClaimTasksByRuntime_RoutesAcrossRuntimesAndMintsTokens covers the happy
// path: one call claims across two runtimes on the same machine, returns one
// task per runtime (per-agent dedup), and mints a task-scoped token for each.
func TestClaimTasksByRuntime_RoutesAcrossRuntimesAndMintsTokens(t *testing.T) {
if testHandler == nil || testPool == nil {
t.Skip("database not available")
}
ctx := context.Background()
rt1 := createClaimReclaimRuntime(t, ctx, "Batch claim rt1")
rt2 := createClaimReclaimRuntime(t, ctx, "Batch claim rt2")
a1, i1 := createClaimReclaimAgentAndIssue(t, ctx, rt1, "Batch claim a1")
a2, i2 := createClaimReclaimAgentAndIssue(t, ctx, rt2, "Batch claim a2")
seedQueuedIssueTask(t, ctx, a1, rt1, i1)
seedQueuedIssueTask(t, ctx, a2, rt2, i2)
w := postBatchClaim(t, testWorkspaceID, []string{rt1, rt2}, 5)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
}
var resp batchClaimResponse
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("decode: %v", err)
}
if len(resp.Tasks) != 2 {
t.Fatalf("claimed %d tasks, want 2: %s", len(resp.Tasks), w.Body.String())
}
seen := map[string]int{}
for _, task := range resp.Tasks {
seen[task.RuntimeID]++
if !strings.HasPrefix(task.AuthToken, "mat_") {
t.Fatalf("task %s missing mat_ task token, got %q", task.ID, task.AuthToken)
}
}
if seen[rt1] != 1 || seen[rt2] != 1 {
t.Fatalf("runtime distribution = %v, want one task each for rt1/rt2", seen)
}
}
// TestClaimTasksByRuntime_SkipsCrossWorkspaceRuntime is the security-critical
// case: a daemon token scoped to workspace A must not claim a task routed to a
// runtime in workspace B, even when B's runtime_id is included in the request.
func TestClaimTasksByRuntime_SkipsCrossWorkspaceRuntime(t *testing.T) {
if testHandler == nil || testPool == nil {
t.Skip("database not available")
}
ctx := context.Background()
// A foreign workspace with its own runtime + agent + queued task.
var foreignUser, foreignWS string
if err := testPool.QueryRow(ctx, `INSERT INTO "user" (name, email) VALUES ('Foreign User', 'batch-foreign@multica.ai') RETURNING id`).Scan(&foreignUser); err != nil {
t.Fatalf("foreign user: %v", err)
}
t.Cleanup(func() { testPool.Exec(ctx, `DELETE FROM "user" WHERE id = $1`, foreignUser) })
if err := testPool.QueryRow(ctx, `INSERT INTO workspace (name, slug, description, issue_prefix) VALUES ('Foreign WS','batch-foreign-ws','x','FGN') RETURNING id`).Scan(&foreignWS); err != nil {
t.Fatalf("foreign workspace: %v", err)
}
t.Cleanup(func() { testPool.Exec(ctx, `DELETE FROM workspace WHERE id = $1`, foreignWS) })
if _, err := testPool.Exec(ctx, `INSERT INTO member (workspace_id, user_id, role) VALUES ($1,$2,'owner')`, foreignWS, foreignUser); err != nil {
t.Fatalf("foreign member: %v", err)
}
var foreignRT, foreignAgent, foreignIssue string
if err := testPool.QueryRow(ctx, `
INSERT INTO agent_runtime (workspace_id, daemon_id, name, runtime_mode, provider, status, device_info, metadata, last_seen_at, visibility, owner_id)
VALUES ($1, NULL, 'Foreign RT', 'cloud', 'handler_test_runtime', 'online', 'x', '{}'::jsonb, now(), 'private', $2)
RETURNING id`, foreignWS, foreignUser).Scan(&foreignRT); err != nil {
t.Fatalf("foreign runtime: %v", err)
}
if err := testPool.QueryRow(ctx, `
INSERT INTO agent (workspace_id, name, description, runtime_mode, runtime_config, runtime_id, visibility, max_concurrent_tasks, owner_id)
VALUES ($1, 'Foreign Agent', '', 'cloud', '{}'::jsonb, $2, 'private', 1, $3)
RETURNING id`, foreignWS, foreignRT, foreignUser).Scan(&foreignAgent); err != nil {
t.Fatalf("foreign agent: %v", err)
}
if err := testPool.QueryRow(ctx, `
INSERT INTO issue (workspace_id, title, status, priority, creator_id, creator_type, number, position)
VALUES ($1, 'foreign issue', 'in_progress', 'none', $2, 'member', 1, 0)
RETURNING id`, foreignWS, foreignUser).Scan(&foreignIssue); err != nil {
t.Fatalf("foreign issue: %v", err)
}
foreignTask := seedQueuedIssueTask(t, ctx, foreignAgent, foreignRT, foreignIssue)
// Daemon token scoped to the (unrelated) handler-test workspace.
w := postBatchClaim(t, testWorkspaceID, []string{foreignRT}, 5)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
}
var resp batchClaimResponse
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("decode: %v", err)
}
if len(resp.Tasks) != 0 {
t.Fatalf("cross-workspace claim leaked %d tasks, want 0: %s", len(resp.Tasks), w.Body.String())
}
var status string
if err := testPool.QueryRow(ctx, `SELECT status FROM agent_task_queue WHERE id = $1`, foreignTask).Scan(&status); err != nil {
t.Fatalf("read foreign task status: %v", err)
}
if status != "queued" {
t.Fatalf("foreign task status = %s, want still queued (untouched)", status)
}
}
// TestClaimTasksByRuntime_CancelsTaskWhenRuntimeOwnerMissing pins the
// unscoped-credential guard: a runtime with no owner cannot mint a task token,
// so the claimed task must be cancelled and omitted from the response rather
// than shipped without a scoped credential.
func TestClaimTasksByRuntime_CancelsTaskWhenRuntimeOwnerMissing(t *testing.T) {
if testHandler == nil || testPool == nil {
t.Skip("database not available")
}
ctx := context.Background()
var rtNull string
if err := testPool.QueryRow(ctx, `
INSERT INTO agent_runtime (workspace_id, daemon_id, name, runtime_mode, provider, status, device_info, metadata, last_seen_at, visibility, owner_id)
VALUES ($1, NULL, 'Ownerless RT', 'cloud', 'handler_test_runtime', 'online', 'x', '{}'::jsonb, now(), 'private', NULL)
RETURNING id`, testWorkspaceID).Scan(&rtNull); err != nil {
t.Fatalf("ownerless runtime: %v", err)
}
t.Cleanup(func() { testPool.Exec(ctx, `DELETE FROM agent_runtime WHERE id = $1`, rtNull) })
agentID, issueID := createClaimReclaimAgentAndIssue(t, ctx, rtNull, "Ownerless agent")
taskID := seedQueuedIssueTask(t, ctx, agentID, rtNull, issueID)
w := postBatchClaim(t, testWorkspaceID, []string{rtNull}, 1)
if w.Code != http.StatusOK {
t.Fatalf("expected 200, got %d: %s", w.Code, w.Body.String())
}
var resp batchClaimResponse
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("decode: %v", err)
}
if len(resp.Tasks) != 0 {
t.Fatalf("claimed %d tasks from owner-less runtime, want 0: %s", len(resp.Tasks), w.Body.String())
}
var status string
if err := testPool.QueryRow(ctx, `SELECT status FROM agent_task_queue WHERE id = $1`, taskID).Scan(&status); err != nil {
t.Fatalf("read task status: %v", err)
}
if status != "cancelled" {
t.Fatalf("task status = %s, want cancelled (owner missing)", status)
}
}