mirror of
https://github.com/multica-ai/multica.git
synced 2026-07-26 12:35:35 +02:00
* feat(daemon-claim): machine-level batch task claim endpoint (MUL-4257) Collapse the per-runtime /tasks/claim poll fan-out into a single machine-level batch claim to cut /api/daemon claim request volume. Server: - agent.sql: = ANY(runtime_ids) batch variants of the claim queries (ListQueuedClaimCandidatesByRuntimes, PromoteDueDeferredTasksForRuntimes, ReclaimStaleDispatchedTasksForRuntimes); runtime.sql: GetAgentRuntimes(= ANY) so a whole machine's runtimes are resolved/promoted/reclaimed/listed in a constant number of queries instead of N. - service.ClaimTasksForRuntimes: claim up to max_tasks across a runtime set, preserving per-(issue,agent) serialization, the concurrency cap, the empty-claim cache short-circuit, and every dispatch side effect. Batch promote replays the per-row side effects (task:queued + empty-cache Bump). - handler.ClaimTasksByRuntime (canonical POST /api/daemon/tasks/claim, with a transitional /claim alias): validates daemon_id (required; must match the mdt_ token) and rejects runtimes bound to a different daemon (group-ownership check mirroring the WS path); resolves+authorizes each runtime_id; claims; and finalizes each task through the SAME FinalizeTaskClaim as the per-runtime endpoint (atomic token + delivered_comment_ids receipt), requeueing the exact claim and omitting it on failure. buildClaimedTaskResponse is extracted from the per-runtime handler and returns the delivered-comment ids plus a structured *claimBuildFailure so both paths share identical payload building and failure semantics (workspace-isolation, chat-input load/empty). - max_tasks: negative -> 400, zero -> empty (never coerce to 1), positive capped at 32. runtime_ids parsed with non-panicking util.ParseUUID. Daemon: - Client.ClaimTasks posts daemon_id + runtime set + free-slot count to the canonical path under a short request-scoped timeout, bounding the head-of-line coupling the per-runtime pollers avoid (MUL-1744). Tests: service batch drain / max_tasks cap / deferred-promote receipt / finalize-failure rollback+requeue; handler routing + token, cross-workspace skip, cross-daemon skip, daemon_id required, owner-missing cancel, max_tasks=0/negative, invalid-uuid skip, comment delivery receipt, stale-reclaim replacement receipt; client posts/parses (daemon_id + canonical path). Follow-up: cut the daemon pollLoop over to a single batched poller (flips the MUL-1744 isolation contract; needs its concurrency tests redesigned). Co-authored-by: multica-agent <github@multica.ai> * feat(daemon-ws): generic WS request/response transport for daemon RPC (MUL-4257) Add a generic daemon->server request/response layer over the existing WS control connection, the transport for WS-first claim (HTTP fallback): - protocol: daemon:rpc_request / daemon:rpc_response envelopes with a correlation request_id + method + body, and an rpc-v1 capability gate. - daemonws.Hub: SetRPCHandler + goroutine-dispatched handleRPCFrame (bounded by a per-connection in-flight cap) that echoes the request_id; missing handler / saturation return non-2xx so the daemon falls back to HTTP. Read limit raised to 64KB for rpc requests carrying a runtime set. - hub tests: round-trip, handler-error->non-2xx, no-handler->503. Co-authored-by: multica-agent <github@multica.ai> * feat(daemon-ws): WS-first task claim over the generic RPC transport (MUL-4257) Bind claim to the WS request/response layer, with HTTP fallback: - server: handler.DaemonRPCHandler adapts a daemon:rpc_request (method tasks.claim) to the existing HTTP ClaimTasksByRuntime via a synthetic in-process request carrying the WS connection's identity (daemon_id + workspace + capabilities), so all auth / payload-building / finalization is reused unchanged. Wired via daemonHub.SetRPCHandler. ClientIdentity now captures X-Client-Capabilities so capability gating matches the HTTP path. - daemon: wsRPCClient correlates responses by request_id over the shared WS connection; attached to the live connection's write channel (guarded so a Call racing teardown never sends on a closed channel) and detached on disconnect. rpc_response frames are routed in the read loop. Daemon.ClaimTasksWSFirst issues tasks.claim over WS and falls back to the HTTP claim endpoint on any transport failure (no conn / buffer full / timeout) — wired into the poller at the poller cutover. - tests: handler tasks.claim RPC end-to-end (claims + dispatches) + unknown method 404; daemon wsRPCClient round-trip / timeout / unavailable / server-error / detach-fails-pending (all under -race). Co-authored-by: multica-agent <github@multica.ai> * feat(daemon): cut claim poller over to machine-level ClaimTasksWSFirst (MUL-4257) Replace the per-runtime HTTP poll loop with a single batch poller: each cycle acquires all free execution slots (slot-before-claim) and issues ONE ClaimTasksWSFirst across every runtime the daemon hosts (WS-first, HTTP fallback), dispatching each returned task to its runtime. Wakeups (targeted / catch-up / runtime-set change) collapse to one nudge. Removes runRuntimePoller + runtimePollOffset. The WS handshake now advertises the same capabilities as HTTP (+ rpc-v1) so WS-built claim payloads keep skill-ref / coalesced-comment gating. Trades per-runtime isolation (MUL-1744) for one request, bounded by the short per-request WS timeout / client timeout. Tests: batch poller claims across runtimes + skips-at-capacity + pollLoop shutdown drain (replacing the per-runtime poller tests); heartbeat isolation + runtime-set watcher kept. Co-authored-by: multica-agent <github@multica.ai> * fix(daemon-ws): WS RPC disconnect-race panic + batch stale-comment-plan repair (MUL-4257) Two PR #5193 review blockers: 1) WS RPC send-on-closed-channel race, both ends: - server: give each connection a cancelable ctx (cancelled on readPump teardown) and run the RPC handler under it, so a slow claim stops on disconnect; guard c.send with sendMu/sendClosed (trySend) so a late RPC response goroutine never writes to the closed channel. Heartbeat ack routed through the same guard. - daemon: wsRPCClient.deliver now sends under the mutex, serialized with attach(nil)'s close+delete, so a delivered response can't hit a channel the detach path just closed. - regressions (-race): daemon deliver-vs-detach; server disconnect-during-handler-response. 2) batch claim now runs the stale-comment-plan repair: extracted the per-runtime handler's repair (trigger deleted, only coalesced survive -> cancel + replay survivors) into shared repairStaleCommentPlanIfNeeded, called by both claim paths. Prevents the batch path (now the default poller) from finalizing+dispatching a task with no comment input and silently dropping the surviving user comment. Regression: batch omits the stale task, cancels it, and rebuilds the survivor into a new trigger plan. Co-authored-by: multica-agent <github@multica.ai> * fix(daemon-ws): server-side RPC deadline + legacy claim fallback (MUL-4257) Two review blockers: 1) WS RPC timeout/fallback (GPT-Boy): the daemon's WS wait didn't cancel server-side claim, so a slow WS claim could commit after the daemon fell back to HTTP, leaking dispatched tasks and breaking the free-slot bound. Fix: RPC envelope carries TimeoutMs; the server bounds the handler ctx by it (so ClaimTasksByRuntime's tx is cancelled/rolled back at the deadline), and the daemon waits budget + grace so a claim that committed before the deadline still reports back. A committed-then-unreported claim degrades to the same stale-reclaim safety net as HTTP, never a double effective claim. Regression: server-side TimeoutMs cancels the handler. 2) Backward compat (Terra-Boy): a new daemon against a server without the batch route (/api/daemon/tasks/claim 404) couldn't claim. Fix: ClaimTasksWSFirst falls back to the legacy per-runtime ClaimTask loop on a batch 404 and caches 'batch unsupported' (reset on WS reconnect to re-probe after a server upgrade). Regression: server exposing only the legacy route. Co-authored-by: multica-agent <github@multica.ai> * fix(daemon-ws): no double-claim on WS teardown/detach (MUL-4257) Sol-Boy review blocker: on reconnect, teardown failed the pending RPC (→ HTTP fallback) but then flushed the queued tasks.claim frame to the still-alive socket, so the server committed the WS claim on top of the HTTP one — double claim, WS batch orphaned to stale reclaim, breaking the free-slot bound. - Teardown now closes the connection FIRST, so runWSWriter discards the queued RPC frame (write error path) instead of delivering it. - A detach while a claim's frame is already in flight now returns a distinct errWSRPCUncertain; ClaimTasksWSFirst does NOT HTTP-fall-back on uncertain (the WS claim may have committed) — it skips the cycle and lets reclaim / the next poll recover. Genuine 'not sent' / timeout still fall back (safe: the server-side deadline guarantees no uncommitted claim by budget+grace). - Regression: detach during an in-flight WS claim asserts zero HTTP claims (at most one path claims); plus the existing detach/deliver-race and server-timeout tests. Co-authored-by: multica-agent <github@multica.ai> * fix(daemon-ws): cancelable RPC frames close the backpressure double-claim (MUL-4257) Sol-Boy review blocker: the client's response budget starts at enqueue, but the socket write is async (10s write deadline). A backpressured writer could hold a tasks.claim in the local queue past the client timeout — the daemon HTTP-fell-back, then the writer woke and delivered the stale WS frame, so the server committed it too: same free slots claimed twice. No detach occurs, so the prior errWSRPCUncertain fix did not cover it. - WS frames are now cancelable (wsOutbound{sent,canceled} under a mutex). The writer calls beginWrite() before WriteMessage and skips cancelled frames. - On give-up (timeout / detach / ctx), Call cancels the queued frame: if it was still pending the cancel wins and the frame is guaranteed never delivered (errWSRPCUnavailable → safe HTTP fallback); if the writer already began sending it the cancel loses and the outcome is errWSRPCUncertain (no fallback). The decision is atomic, so at most one transport claims. Tests: wsOutbound cancel-before-write vs write-before-cancel; Call timeout cancels an unsent frame (writer then drops it) vs uncertain when already sent; plus the updated detach and existing timeout/race tests. Co-authored-by: multica-agent <github@multica.ai> * fix(batch-claim): return partial success instead of dropping committed claims (MUL-4257) Sol-Boy review blocker: ClaimTasksForRuntimes reclaims (step 2) and claims per agent (step 6) in independent transactions, but a step-4 candidate-SELECT error or a mid-loop ClaimTask error did 'return nil, err' — discarding tasks already committed as dispatched. The handler 500s; the daemon sees a definite (non- uncertain) 500 and HTTP-falls-back, claiming a SECOND batch into the same free slots while the first batch waits for stale reclaim — the double-claim this PR removes. - Both error paths now prefer partial success: if any task has already committed (claimed non-empty), return it (nil error) so the handler finalizes and returns 200; the errored candidates stay queued for the next poll. The remaining error is logged. Only a genuinely empty result still returns the error (safe: no committed claim to lose, HTTP fallback just re-fails). Regression (internal/service, DB-backed, fault-injected): - PartialSuccessOnSecondAgentClaimFailure: fail the 2nd ClaimTask's Begin → the first agent's committed task is returned, not dropped. - PartialSuccessOnCandidateQueryFailureAfterReclaim: a stale dispatched task is reclaimed, then the candidate SELECT fails → the reclaimed task is returned. Co-authored-by: multica-agent <github@multica.ai> --------- Co-authored-by: Eve <eve@multica-ai.local> Co-authored-by: multica-agent <github@multica.ai>
122 lines
5.1 KiB
Go
122 lines
5.1 KiB
Go
package service
|
|
|
|
import (
|
|
"context"
|
|
"fmt"
|
|
"sync"
|
|
"testing"
|
|
"time"
|
|
|
|
"github.com/jackc/pgx/v5/pgtype"
|
|
"github.com/jackc/pgx/v5/pgxpool"
|
|
"github.com/multica-ai/multica/server/internal/events"
|
|
"github.com/multica-ai/multica/server/internal/util"
|
|
db "github.com/multica-ai/multica/server/pkg/db/generated"
|
|
"github.com/multica-ai/multica/server/pkg/protocol"
|
|
)
|
|
|
|
// TestClaimTasksForRuntimes_PromotesDeferredAndEmitsQueuedEvent is the
|
|
// regression test for the PR #5193 review must-fix: the batch promote step must
|
|
// replay the singular service method's per-row side effects — emit the
|
|
// deferred→queued (`task:queued`) event AND invalidate the empty-claim verdict
|
|
// via NotifyTaskEnqueued — so a just-promoted deferred task is claimable in the
|
|
// SAME batch call rather than sitting idle until the empty key's TTL.
|
|
//
|
|
// The empty-claim cache is nil in tests (no Redis), so this asserts the two
|
|
// directly observable halves of the fix: (1) the promoted task is claimed this
|
|
// round, and (2) a `task:queued` event is published for it.
|
|
func TestClaimTasksForRuntimes_PromotesDeferredAndEmitsQueuedEvent(t *testing.T) {
|
|
ctx := context.Background()
|
|
pool := newTaskClaimRacePool(t)
|
|
bus := events.New()
|
|
|
|
var mu sync.Mutex
|
|
queuedTaskIDs := map[string]int{}
|
|
bus.Subscribe(protocol.EventTaskQueued, func(e events.Event) {
|
|
if payload, ok := e.Payload.(map[string]any); ok {
|
|
if id, ok := payload["task_id"].(string); ok {
|
|
mu.Lock()
|
|
queuedTaskIDs[id]++
|
|
mu.Unlock()
|
|
}
|
|
}
|
|
})
|
|
|
|
svc := NewTaskService(db.New(pool), pool, nil, bus)
|
|
rt, deferredTaskID := deferredBatchFixture(t, ctx, pool)
|
|
|
|
got, err := svc.ClaimTasksForRuntimes(ctx, []pgtype.UUID{util.MustParseUUID(rt)}, 5)
|
|
if err != nil {
|
|
t.Fatalf("claim: %v", err)
|
|
}
|
|
if len(got) != 1 || util.UUIDToString(got[0].ID) != deferredTaskID {
|
|
t.Fatalf("expected the promoted deferred task %s to be claimed this round, got %d tasks", deferredTaskID, len(got))
|
|
}
|
|
|
|
mu.Lock()
|
|
n := queuedTaskIDs[deferredTaskID]
|
|
mu.Unlock()
|
|
if n != 1 {
|
|
t.Fatalf("expected exactly one task:queued event for the promoted task, got %d", n)
|
|
}
|
|
}
|
|
|
|
// deferredBatchFixture provisions one runtime + agent with a single DEFERRED
|
|
// task whose fire_at is already due. Returns the runtime id and the task id.
|
|
func deferredBatchFixture(t *testing.T, ctx context.Context, pool *pgxpool.Pool) (runtimeID, taskID string) {
|
|
t.Helper()
|
|
suffix := time.Now().UnixNano()
|
|
|
|
var userID string
|
|
if err := pool.QueryRow(ctx, `INSERT INTO "user" (name, email) VALUES ($1,$2) RETURNING id`,
|
|
"Deferred Batch Test", fmt.Sprintf("deferred-batch-%d@multica.ai", suffix)).Scan(&userID); err != nil {
|
|
t.Fatalf("create user: %v", err)
|
|
}
|
|
var workspaceID string
|
|
if err := pool.QueryRow(ctx, `INSERT INTO workspace (name, slug, description, issue_prefix) VALUES ($1,$2,$3,$4) RETURNING id`,
|
|
"Deferred Batch Test", fmt.Sprintf("deferred-batch-%d", suffix), "temp deferred batch test", "DBR").Scan(&workspaceID); err != nil {
|
|
t.Fatalf("create workspace: %v", err)
|
|
}
|
|
if _, err := pool.Exec(ctx, `INSERT INTO member (workspace_id, user_id, role) VALUES ($1,$2,'owner')`, workspaceID, userID); err != nil {
|
|
t.Fatalf("create member: %v", err)
|
|
}
|
|
if err := pool.QueryRow(ctx, `
|
|
INSERT INTO agent_runtime (workspace_id, daemon_id, name, runtime_mode, provider, status, device_info, metadata, last_seen_at, visibility, owner_id)
|
|
VALUES ($1, 'daemon-deferred', 'Deferred RT', 'cloud', 'deferred_provider', 'online', 'x', '{}'::jsonb, now(), 'private', $2)
|
|
RETURNING id`, workspaceID, userID).Scan(&runtimeID); err != nil {
|
|
t.Fatalf("create runtime: %v", err)
|
|
}
|
|
var agentID string
|
|
if err := pool.QueryRow(ctx, `
|
|
INSERT INTO agent (workspace_id, name, description, runtime_mode, runtime_config, runtime_id, visibility, max_concurrent_tasks, owner_id)
|
|
VALUES ($1, 'Deferred Agent', '', 'cloud', '{}'::jsonb, $2, 'private', 5, $3)
|
|
RETURNING id`, workspaceID, runtimeID, userID).Scan(&agentID); err != nil {
|
|
t.Fatalf("create agent: %v", err)
|
|
}
|
|
var issueID string
|
|
if err := pool.QueryRow(ctx, `
|
|
INSERT INTO issue (workspace_id, title, status, priority, creator_id, creator_type, number, position)
|
|
VALUES ($1, 'deferred issue', 'in_progress', 'none', $2, 'member', 700001, 0)
|
|
RETURNING id`, workspaceID, userID).Scan(&issueID); err != nil {
|
|
t.Fatalf("create issue: %v", err)
|
|
}
|
|
if err := pool.QueryRow(ctx, `
|
|
INSERT INTO agent_task_queue (agent_id, issue_id, status, priority, context, runtime_id, fire_at)
|
|
VALUES ($1, $2, 'deferred', 0, '{}'::jsonb, $3, now() - interval '1 minute')
|
|
RETURNING id`, agentID, issueID, runtimeID).Scan(&taskID); err != nil {
|
|
t.Fatalf("create deferred task: %v", err)
|
|
}
|
|
|
|
t.Cleanup(func() {
|
|
c := context.Background()
|
|
pool.Exec(c, `DELETE FROM agent_task_queue WHERE agent_id = $1`, agentID)
|
|
pool.Exec(c, `DELETE FROM issue WHERE workspace_id = $1`, workspaceID)
|
|
pool.Exec(c, `DELETE FROM agent WHERE id = $1`, agentID)
|
|
pool.Exec(c, `DELETE FROM agent_runtime WHERE id = $1`, runtimeID)
|
|
pool.Exec(c, `DELETE FROM member WHERE workspace_id = $1 AND user_id = $2`, workspaceID, userID)
|
|
pool.Exec(c, `DELETE FROM workspace WHERE id = $1`, workspaceID)
|
|
pool.Exec(c, `DELETE FROM "user" WHERE id = $1`, userID)
|
|
})
|
|
return runtimeID, taskID
|
|
}
|