Files
multica/server/internal/handler/daemon_claim_continuity_gap_test.go
Bohan Jiang 85a14cde37 fix(daemon): gate codex session pointer writes on rollout presence (MUL-5305) (#5960)
* fix(daemon): gate codex session pointer writes on rollout presence (MUL-5305)

Codex issue follow-ups on local_directory projects intermittently lost
their session: the server sent a prior session whose rollout was not in
the task CODEX_HOME, so the daemon dropped the resume and started a fresh
thread (gateCodexResumeToRolloutPresence), losing the conversation.

Root of the bad pointer: the daemon persists a Codex session id as the
resumable pointer at two points -- the mid-flight pin and the terminal
report -- before the rollout is guaranteed on disk. A task that exits
early (crash / runtime offline / timeout) leaves a pinned/reported
session id with no rollout; GetLastTaskSession (which accepts failed
rows) then hands it to the next follow-up, which drops it.

Enforce the invariant at write time: only record a Codex session as the
resumable pointer once its rollout is present in the per-issue store,
with a short bounded wait for flush. If it never lands, don't overwrite
the last good pointer -- a blanked session_id becomes NULL server-side,
so GetLastTaskSession falls back to the most recent session whose
rollout is real. Non-Codex providers are unaffected; crash recovery is
preserved because a present rollout still pins.

- codexSessionResumable: shared write-time presence check (bounded wait)
- runTask: gate the terminal session_id before reporting
- executeAndDrain: gate the mid-flight pin (thread codexHome through)
- tests: helper cases + behavioral pin test

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon): address review — don't silently downgrade completed sessions (MUL-5305)

Follow-up to review feedback on #5960:

- Must-fix 1 (silent downgrade): limit the write-time session withholding
  to NON-completed terminal states. A missing rollout means no resumable
  conversation was persisted, so a withheld non-completed attempt loses
  nothing; a completed session is authoritative and, if its rollout is
  anomalously absent, is still recorded so the next run's resume gate
  discloses the loss (PriorSessionResumeUnavailable, MUL-4424) instead of
  silently falling back to an older session. Extracted
  resumableTerminalSessionID.
- Non-blocking risk: pin the mid-flight resume pointer with a per-status
  presence check instead of one fixed 2s window, and set sessionPinned
  only once the rollout is confirmed, so a rollout that lands shortly
  after the first status is still pinned this run.
- Must-fix 2 (regression coverage): pin skipped when rollout absent (no
  /session call); terminal helper (completed keeps / failed withholds);
  and a DB-backed GetLastTaskSession test proving the next claim falls
  back to the older recorded session when the latest was blanked.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon): disclose Codex session continuity gaps end-to-end (MUL-5305)

Addresses review feedback on #5960.

Must-fix 1 — a completed turn whose rollout is missing is exactly the
#5934 case (the reporter waits for each turn to finish), so it can no
longer be excluded from withholding. Withhold the session for ANY
terminal state, and pair the withhold with a persisted continuity-gap
signal so the next claim still discloses the loss even while resuming an
older good session:
  - new agent_task_queue.session_rollout_missing column (migration 224)
  - daemon sends session_rollout_missing on the terminal report; the
    handler clears the resume pointer (MarkTaskSessionRolloutMissing,
    overriding FailAgentTask's COALESCE) and flags the row
  - claim reads GetLatestTaskRolloutMissing and sets a new
    prior_session_resume_unavailable response field, which the daemon ORs
    into the brief's PriorSessionResumeUnavailable disclosure

Must-fix 2 — Codex reveals the session id on a single task_started
status, so a one-shot presence check missed a rollout that flushed later
and lost in-flight crash recovery. Pin via a background waiter bounded by
the run's context that pins the moment the rollout lands.

Tests: - completed + rollout missing -> next claim withholds the bad session
    AND flags the continuity gap (cross-layer DB test)
  - session pinned once its rollout appears after the status (mid-run)
  - pin skipped while the rollout is absent
Co-authored-by: multica-agent <github@multica.ai>

* fix(server): make continuity-gap write atomic + disclose on all claim paths (MUL-5305)

Addresses review round 3 of #5960.

Must-fix 1 — the previous handler-level marker ran AFTER the terminal
transaction committed, and FailTask creates + wakes the auto-retry inside
that same transaction, so a retry could claim the rollout-missing session
before the marker cleared it (and a marker failure was swallowed). Move
session_rollout_missing INTO the terminal write: CompleteAgentTask and
FailAgentTask now force session_id NULL (overriding Fail's COALESCE that
would keep a stale mid-flight pin) and set the flag in the SAME UPDATE, so
the withhold + gap flag commit atomically with the retry creation. The
flag is threaded through TaskService.CompleteTask/FailTask; the swallowed
best-effort MarkTaskSessionRolloutMissing query is removed.

Must-fix 2 — the daemon withholds for all Codex tasks, but only the issue
non-rerun claim consumed the disclosure. Now every fallback path sets
prior_session_resume_unavailable: the manual-rerun branch reads the source
task's session_rollout_missing, and the chat branch reads a new
GetLatestChatTaskRolloutMissing.

Tests (cross-layer DB):
- completed + rollout missing via the real CompleteAgentTask terminal
  write -> session withheld AND gap flagged
- failed + rollout missing forces session_id NULL over the COALESCE-
  preserved mid-flight pin in ONE statement

Deploy order: migration + server first, daemon second (new fields are
omitempty and ignored by an old peer).

Co-authored-by: multica-agent <github@multica.ai>

* fix(handler): return 5xx on FailTask error + cover claim-response gap paths (MUL-5305)

Addresses review round 4 of #5960.

Must-fix 1 — the FailTask handler returned 400 on a service/DB error, but
the daemon's terminal callback treats 400 as permanent (postJSONWithRetry
/ isTransientError bails without retrying). Since the fail transaction is
now the sole persistence point for the withheld session + continuity-gap
flag + auto-retry, a rolled-back fail must be retried, so return 5xx (an
invalid request body still returns 400), mirroring CompleteTask.
Regression: client.FailTask retries on a transient 5xx and eventually
succeeds.

Must-fix 2 — add claim-response-level regressions that drive the two new
disclosure branches through buildClaimedTaskResponse:
  - chat: the latest terminal task on the session withheld -> the next
    chat claim sets prior_session_resume_unavailable
  - manual rerun: the source task withheld -> the rerun claim discloses
These handler DB tests run under CI's fully-migrated database (the local
workspace DB cannot set up the handler fixture).

Co-authored-by: multica-agent <github@multica.ai>

---------

Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
2026-07-27 13:12:06 +08:00

147 lines
6.3 KiB
Go

package handler
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
)
// claimContinuityGapProbe decodes just the continuity-gap fields off a claim
// response so the two MUL-5305 disclosure paths can be asserted end-to-end.
type claimContinuityGapProbe struct {
Task *struct {
ID string `json:"id"`
PriorSessionID string `json:"prior_session_id"`
PriorSessionResumeUnavailable bool `json:"prior_session_resume_unavailable"`
} `json:"task"`
}
func claimOneTaskForRuntime(t *testing.T, runtimeID, daemonID string) claimContinuityGapProbe {
t.Helper()
w := httptest.NewRecorder()
req := newDaemonTokenRequest(http.MethodPost, "/api/daemon/runtimes/"+runtimeID+"/claim", nil, testWorkspaceID, daemonID)
req = withURLParam(req, "runtimeId", runtimeID)
testHandler.ClaimTaskByRuntime(w, req)
if w.Code != http.StatusOK {
t.Fatalf("ClaimTaskByRuntime: expected 200, got %d: %s", w.Code, w.Body.String())
}
var probe claimContinuityGapProbe
if err := json.NewDecoder(w.Body).Decode(&probe); err != nil {
t.Fatalf("decode claim response: %v", err)
}
if probe.Task == nil {
t.Fatal("expected a claimed task in the response")
}
return probe
}
// TestClaimTaskByRuntime_ChatRolloutMissingDisclosesGap is the claim-response
// half of MUL-5305 Must-fix 2 for chat: when the most recent terminal task on a
// chat session withheld its Codex session (rollout missing), the next chat claim
// resumes the older pointer but MUST still set prior_session_resume_unavailable
// so the run discloses the gap instead of silently continuing.
func TestClaimTaskByRuntime_ChatRolloutMissingDisclosesGap(t *testing.T) {
if testHandler == nil || testPool == nil {
t.Skip("database not available")
}
ctx := context.Background()
agentID := createHandlerTestAgent(t, "ChatGapClaimAgent", []byte("[]"))
runtimeID := handlerTestRuntimeID(t)
// Chat session that still carries a good resume pointer from an earlier turn.
var sessionID string
if err := testPool.QueryRow(ctx, `
INSERT INTO chat_session (workspace_id, agent_id, creator_id, title, status, runtime_id, session_id, work_dir)
VALUES ($1, $2, $3, 'gap chat', 'active', $4, 'OLD-GOOD-CHAT-SESSION', '/tmp/chat')
RETURNING id
`, testWorkspaceID, agentID, testUserID, runtimeID).Scan(&sessionID); err != nil {
t.Fatalf("create chat session: %v", err)
}
t.Cleanup(func() { testPool.Exec(context.Background(), `DELETE FROM chat_session WHERE id = $1`, sessionID) })
// Most recent terminal turn on this session withheld its Codex session.
if _, err := testPool.Exec(ctx, `
INSERT INTO agent_task_queue (agent_id, runtime_id, status, priority, started_at, completed_at, chat_session_id, session_rollout_missing)
VALUES ($1, $2, 'completed', 0, now() - interval '1 minute', now() - interval '1 minute', $3, TRUE)
`, agentID, runtimeID, sessionID); err != nil {
t.Fatalf("insert withheld chat task: %v", err)
}
if _, err := testPool.Exec(ctx, `
INSERT INTO chat_message (chat_session_id, role, content) VALUES ($1, 'user', 'next turn')
`, sessionID); err != nil {
t.Fatalf("insert chat message: %v", err)
}
var taskID string
if err := testPool.QueryRow(ctx, `
INSERT INTO agent_task_queue (agent_id, runtime_id, status, priority, chat_session_id)
VALUES ($1, $2, 'queued', 1000, $3) RETURNING id
`, agentID, runtimeID, sessionID).Scan(&taskID); err != nil {
t.Fatalf("create chat follow-up task: %v", err)
}
t.Cleanup(func() { testPool.Exec(context.Background(), `DELETE FROM agent_task_queue WHERE id = $1`, taskID) })
probe := claimOneTaskForRuntime(t, runtimeID, "chat-gap-claim-test")
if !probe.Task.PriorSessionResumeUnavailable {
t.Fatalf("expected chat claim to disclose the continuity gap; response task=%+v", *probe.Task)
}
}
// TestClaimTaskByRuntime_RerunSourceRolloutMissingDisclosesGap is the
// claim-response half of MUL-5305 Must-fix 2 for manual rerun: when the exact
// source task withheld its Codex session, the rerun has nothing resumable from
// it and MUST disclose the gap rather than silently start fresh.
func TestClaimTaskByRuntime_RerunSourceRolloutMissingDisclosesGap(t *testing.T) {
if testHandler == nil || testPool == nil {
t.Skip("database not available")
}
ctx := context.Background()
agentID := createHandlerTestAgent(t, "RerunGapClaimAgent", []byte("[]"))
runtimeID := handlerTestRuntimeID(t)
var issueID string
if err := testPool.QueryRow(ctx, `
INSERT INTO issue (workspace_id, title, status, priority, creator_type, creator_id, assignee_type, assignee_id, number)
VALUES ($1, 'rerun gap issue', 'todo', 'none', 'member', $2, 'agent', $3,
(SELECT COALESCE(MAX(number), 0) + 1 FROM issue WHERE workspace_id = $1))
RETURNING id
`, testWorkspaceID, testUserID, agentID).Scan(&issueID); err != nil {
t.Fatalf("create issue: %v", err)
}
t.Cleanup(func() {
testPool.Exec(context.Background(), `DELETE FROM agent_task_queue WHERE issue_id = $1`, issueID)
testPool.Exec(context.Background(), `DELETE FROM issue WHERE id = $1`, issueID)
})
// Source task whose Codex session was withheld (rollout missing).
var srcID string
if err := testPool.QueryRow(ctx, `
INSERT INTO agent_task_queue (agent_id, runtime_id, issue_id, status, priority, started_at, completed_at, session_rollout_missing)
VALUES ($1, $2, $3, 'completed', 0, now() - interval '2 minutes', now() - interval '2 minutes', TRUE)
RETURNING id
`, agentID, runtimeID, issueID).Scan(&srcID); err != nil {
t.Fatalf("insert source task: %v", err)
}
// A manual rerun of that source task, queued to claim (rerun carries
// force_fresh_session=true; the disclosure must still fire from the source).
var taskID string
if err := testPool.QueryRow(ctx, `
INSERT INTO agent_task_queue (agent_id, runtime_id, issue_id, status, priority, rerun_of_task_id, force_fresh_session)
VALUES ($1, $2, $3, 'queued', 1000, $4, TRUE) RETURNING id
`, agentID, runtimeID, issueID, srcID).Scan(&taskID); err != nil {
t.Fatalf("create rerun task: %v", err)
}
t.Cleanup(func() { testPool.Exec(context.Background(), `DELETE FROM agent_task_queue WHERE id = $1`, taskID) })
probe := claimOneTaskForRuntime(t, runtimeID, "rerun-gap-claim-test")
if !probe.Task.PriorSessionResumeUnavailable {
t.Fatalf("expected rerun claim to disclose the source task's continuity gap; response task=%+v", *probe.Task)
}
}