mirror of
https://github.com/multica-ai/multica.git
synced 2026-07-16 22:59:04 +02:00
* fix(daemon): isolate Codex sessions per task to unblock initialize (MUL-4424) Codex 0.143+ backfills a per-home session-state DB by enumerating every rollout visible under sessions/ during `initialize`. The per-task CODEX_HOME symlinked the shared ~/.codex/sessions in, so a machine with a large accumulated history (one reporter: ~2000 rollouts / ~22 GiB) stalled `initialize` for tens of seconds — the app-server started but the task produced no output before it was cancelled (github #5273). Give each task its own local sessions/ directory instead: - Fresh task: create an empty local sessions/ so backfill is trivial. - Reused task with a real sessions/ dir: it is authoritative — leave it. - Reused task still holding a legacy symlink (older build): migrate in place. Replace the symlink with a real dir; when resuming, symlink only the single rollout being resumed (never copy — a rollout can be GiB and this is on initialize's critical path); and drop the stale, rebuildable session-state DB (state_*.sqlite*, session_index.jsonl) so Codex re-indexes the task-local sessions. Unrelated per-task DBs (goals_*, logs_*, memories_*) are left intact. Also point the token-usage fallback scan at the backend's per-task CODEX_HOME instead of the daemon-global home, so usage isn't lost now that sessions are isolated there. Complements the #5319 handshake watchdog (which turns a silent stall into a loud, phased timeout); this removes the underlying cause. Co-authored-by: multica-agent <github@multica.ai> * fix(daemon): address Codex session isolation review (MUL-4424) Resolves the three blockers from Elon's review of #5360: 1. local_directory context loss. local_directory tasks get a fresh codex-home per task ID (the daemon never reuses their workdir), so task-local isolation stranded every follow-up run with an empty sessions dir and silently restarted the conversation. Their only stable, GC-safe cross-task store is the user's own ~/.codex/sessions (a persistent store under WorkspacesRoot would be orphan-GC'd), so keep the shared-sessions symlink for them (IsLocalDirectory). Managed tasks stay isolated. 2. Migration resume robustness. - Rollout lookup now covers the flat layout and background-compressed .jsonl.zst rollouts, not just nested YYYY/MM/DD *.jsonl — both are legitimate Codex 0.144 history that were previously judged "not found", silently dropping resume. - Exposure hard-links first, then symlinks, never copies — hard links need no privilege and work on Windows within a volume, so the zero-copy path is exercised identically on CI. - The daemon now verifies the rollout is actually present in the task CODEX_HOME (execenv.CodexResumeRolloutPresent) before the brief is generated; if absent it clears the resume from both the backend and the brief instead of telling the agent it is continuing a lost thread. 3. session_index.jsonl is no longer deleted during migration — Codex uses it as the authoritative thread-id -> name store (not rebuildable from rollouts). Only the rebuildable state_*.sqlite* is reset. Tests: 2-round local_directory resume across task IDs; compressed/flat lookup; hard-link zero-copy (os.SameFile); session_index preserved; CodexResumeRolloutPresent + the daemon gate helper (present keeps / absent drops / non-codex + empty no-op). Co-authored-by: multica-agent <github@multica.ai> * fix(daemon): scope Codex sessions to a per-issue store; disclose lost resumes (MUL-4424) Addresses the three blockers from Elon's second review of #5360. 1. local_directory still enumerated the whole machine history. The prior fix re-linked the entire ~/.codex/sessions into every fresh local_directory codex-home, so Codex still backfilled from thousands of unrelated rollouts on `initialize` (measured ~8.3s with 3450 rollouts; the reporter's 22 GiB could exceed the 30s watchdog). Point sessions/ at a persistent, per-(agent, issue) store under the shared Codex home (multica-sessions/<agent>/<issue>) that holds only this issue's rollouts. It is keyed stably across task IDs and lives outside the task-scoped envRoot the GC reclaims, so follow-up runs resume it while `initialize` only ever sees this issue's history. 2. Windows cross-volume resume was lost. Exposing a single rollout by hard-link (same-volume only) then file symlink (needs Windows privilege) can't cross a volume boundary. The store now lives on the shared Codex volume, so the resume rollout is hard-linked there zero-copy, and sessions/ is exposed to the task home via a directory link — a symlink on Unix, a junction on Windows — which crosses volumes without privilege and never copies a (possibly GiB) rollout on initialize's critical path. There is no remaining per-file cross-volume link. 3. An unavailable resume was a silent downgrade. Both resume gates (gateResumeToReusedWorkdir, gateCodexResumeToRolloutPresence) now set PriorSessionResumeUnavailable, and the runtime brief renders a Session Continuity Notice telling the agent to disclose to the user, up front in its reply, that the previous conversation context could not be restored and this run starts fresh — turning a silent restart into a user-visible one. The task is not failed: it can still do useful work without the prior context. Managed fresh / reused-real-dir tasks keep their task-local, GC-collected sessions dir unchanged; only the legacy-symlink migration with a resume routes through the store (cross-volume-safe), and a home already linked to the store is treated as authoritative on reuse. Tests: local_directory per-issue store (only this issue's history, no whole- machine leak); no-key fallback to an empty dir; two-round resume across task IDs through the store; legacy migration routed through the store with a zero-copy hard link; reused store link stays authoritative; both gates set the resume-unavailable flag; brief renders the continuity notice only when a resume was lost. execenv + daemon + pkg/agent packages, go vet, and gofmt all pass. Co-authored-by: multica-agent <github@multica.ai> * fix(daemon): disclose live resume-RPC loss; bound Codex session store lifecycle (MUL-4424) Addresses the two blockers from Elon's third review of #5360. 1. A real thread/resume failure was still a silent new session. The brief's Session Continuity Notice only covers losses the daemon detects before launch (workdir not reused, rollout absent). But when the rollout is present yet Codex rejects the live thread/resume (corrupt/incompatible rollout, server-side thread GC, schema drift), startOrResumeThread falls back to thread/start and the run succeeds on a fresh thread with no user-facing signal. Carry the original resume intent into the backend as ExecOptions.ResumeExpected (set from the post-gate PriorSessionID, so a pre-flight drop still routes through the brief and never double-notifies), and when a resume was expected but the backend landed on a fresh thread, prepend the same continuity notice to the first turn/start input. This also covers the daemon's transport-error fresh-session retry, which clears ResumeSessionID but not ResumeExpected. 2. The persistent per-issue store had no data lifecycle. multica-sessions stores live outside the task-scoped envRoot the GC reclaims (so resume survives across task IDs), which meant a done/abandoned issue's prompts and full rollouts (one reporter: a single 1.5 GiB rollout) accumulated forever and were never freed on issue/agent/workspace deletion. Add PruneCodexSessionStores: the daemon GC loop reclaims any store untouched for GCCodexSessionTTL (default 14 days, configurable via MULTICA_GC_CODEX_SESSION_TTL, 0 disables). A store's newest rollout mtime is its last activity, so an active or recently-resumed task keeps its store fresh and is never reclaimed, while a deleted issue's store ages out — an eventual reclamation guarantee without needing deletion events. Tests: codexTurnInput discloses on resume fallback and stays silent on success / fresh start (paired with the existing live-RPC fallback test); store pruning reclaims aged stores, keeps recent ones, isolates issues, cleans empty agent dirs, and is disable-able. execenv / daemon / pkg/agent, go vet, gofmt all pass. Co-authored-by: multica-agent <github@multica.ai> * fix(daemon): protect a reopened Codex session store from GC mid-mount (MUL-4424) Addresses Elon's fourth-review blocker: reopening an issue idle past GCCodexSessionTTL could lose context, because mounting its per-issue session store (MkdirAll + rollout lookup + task-home link) never refreshed the store's mtime, so a GC cycle firing before the resumed turn wrote its first rollout saw a >TTL-old store and reclaimed it — a stat->remove race with no in-use guard. Two complementary defenses: - Activity refresh: linkCodexSessionsToStore now os.Chtimes the store to now after linking, so codexStoreStat (which reads the newest mtime as last activity) sees a just-used store. This fixes the sequential repro — a mount immediately followed by a prune keeps the store. - In-process active-store guard: the daemon marks the per-issue store in-use (execenv.CodexSessionStorePath) from before Prepare/Reuse mounts it until the task ends, and PruneCodexSessionStores now takes an isActive predicate and skips any store a live task holds. Because prepare and prune run in the same process, this closes the remaining concurrent stat->remove window the mtime refresh alone cannot. Reference-counted, mirroring the env-root guard. Tests: a reopened >TTL store survives a GC cycle after remount and stays resumable; an idle-on-disk store marked active is skipped, then reclaimed once inactive; the existing idle-reclaim / isolation / disable / empty-agent-dir cases still pass. execenv + daemon, go vet, gofmt all pass. Co-authored-by: multica-agent <github@multica.ai> * fix(daemon): make Codex store delete atomic with mark-active (MUL-4424) Addresses Elon's fifth-review blocker: the active-store guard's check and delete were not one atomic step. PruneCodexSessionStores called isActive (which locked, read, and unlocked) and only then RemoveAll'd, leaving a window where a task could markActiveCodexStore between the check and the removal and still lose its store — the exact mark-then-delete interleaving Elon reproduced. Replace the point-in-time isActive predicate with a reserve-for-deletion protocol that shares one lock with mark-active: - reserveCodexStoreForDeletion(store) atomically refuses when a live task holds the store (or another delete already reserved it) and otherwise marks it reserved, all under one activeCodexStoresMu acquisition. PruneCodexSessionStores reserves before RemoveAll and commits after, so confirm-inactive and remove are effectively atomic against a concurrent mark. - markActiveCodexStore now waits (on a sync.Cond) while a store is reserved, so a task never mounts a store mid-removal; committing the removal wakes it and the store is recreated fresh by Prepare (with the continuity notice). So mark-before-reserve keeps the store (reserve refused); reserve-before-mark removes it and blocks the late mark until the removal commits. The genuinely idle case still reclaims. Tests (daemon, run under -race): mark-then-reserve is refused; reserve blocks a concurrent mark until commit then the store reads active; a second reserve is refused mid-flight. The execenv prune tests move to the reserve seam; the activity-refresh / reopen-then-prune / isolation / disable cases still pass. Co-authored-by: multica-agent <github@multica.ai> * fix(daemon): namespace Codex session stores per profile for cross-daemon safety (MUL-4424) Addresses Elon's sixth-review blocker: the in-process reservation guard cannot span processes, but Multica supports multiple profile daemons on one machine (e.g. production + staging) that share the same ~/.codex. Each daemon's GC scanned the whole multica-sessions root, so a staging daemon could reclaim a store a production task was actively resuming — its reservation lived only in the other process's memory. Isolate by profile instead of trying to lock across processes: - Store path is now <shared>/multica-sessions/<namespace>/<agent>/<issue>, where namespace is the daemon's profile (empty -> "default"). PrepareParams/ReuseParams carry Profile; codexSessionStoreKey and CodexSessionStorePath fold it in. - PruneCodexSessionStores takes the profile and scans ONLY that namespace, so a daemon never even sees another profile's stores, let alone deletes them. The per-profile trees are disjoint, so the in-process guard is sufficient within a namespace (profiles get separate daemon state, so no two daemons share one). Test: a "staging"-owned idle store is untouched by a default-profile prune and reclaimed only by staging's own prune. Existing prune/guard/reopen tests move under the namespace. execenv + daemon under -race, go vet, gofmt all pass. Co-authored-by: multica-agent <github@multica.ai> * fix(daemon): make the Codex store profile→namespace map injective (MUL-4424) Addresses Elon's seventh-review blocker: the per-profile namespace was derived by dropping unsafe characters, which is not injective. The CLI treats the empty (default) profile and a profile literally named "default" as separate daemons, yet both mapped to namespace "default"; likewise "staging.prod" and "stagingprod" both mapped to "stagingprod". Two distinct daemons then shared one store tree, so one could again reclaim the other's live session — the cross-process blocker reopened for those profile names. Make codexSessionStoreNamespace injective: the empty profile gets a reserved bare literal "default", and every named profile is hex-encoded (bijective, filesystem-safe) under a "p_" prefix a bare literal can never collide with. So "" -> "default" while "default" -> "p_64656661756c74", and "staging.prod" / "stagingprod" get distinct hex segments. sanitizeCodexPathSegment stays for the UUID agent/issue segments (injective for real UUIDs); only the user-controlled profile needed the encoding. Tests: codexSessionStoreNamespace is distinct for "" vs "default", punctuation variants, case variants, and an encoded-looking name; and end-to-end, pruning one profile never reclaims the other's store for the "" vs "default" and "staging.prod" vs "stagingprod" pairs. execenv + daemon under -race, go vet, gofmt all pass. Co-authored-by: multica-agent <github@multica.ai> * fix(daemon): fixed-length Codex store namespace so long profiles fit (MUL-4424) Addresses Elon's eighth-review blocker: hex-encoding the full profile doubled the namespace segment length. A profile can be as long as a filesystem segment allows (~255 bytes) and the CLI persists it as its own config dir, but the store namespace "p_" + hex(profile) reached 2 + 127*2 = 256 bytes at 127 chars, overflowing the 255-byte single-segment limit — the profile's own dir created fine, then the session store failed with "file name too long". Derive the namespace from a fixed-length hash instead: a named profile is now "p_" + hex(sha256(profile)) — a constant 64 hex chars (66 with the prefix), filesystem-safe and collision-resistant. The empty (default) profile keeps its reserved bare literal "default", which the "p_"-prefixed 66-char segment can never equal. Still injective across the CLI's distinct-daemon cases; just no longer length-expanding. Test: the namespace stays <=255 bytes and creatable for profiles up to the 255-byte segment limit (127- and 255-char cases that overflowed under hex); the prior injectivity and cross-profile prune-isolation tests still hold. execenv + daemon under -race, go vet, gofmt all pass. Co-authored-by: multica-agent <github@multica.ai> --------- Co-authored-by: J <j@multica.ai> Co-authored-by: multica-agent <github@multica.ai>
759 lines
24 KiB
Go
759 lines
24 KiB
Go
package daemon
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"net/http"
|
|
"os"
|
|
"os/exec"
|
|
"path/filepath"
|
|
"strings"
|
|
"time"
|
|
|
|
"github.com/multica-ai/multica/server/internal/daemon/execenv"
|
|
)
|
|
|
|
// gcLoop periodically scans local workspace directories and removes those
|
|
// whose issue is done/cancelled and hasn't been updated within the configured TTL.
|
|
func (d *Daemon) gcLoop(ctx context.Context) {
|
|
if !d.cfg.GCEnabled {
|
|
d.logger.Info("gc: disabled")
|
|
return
|
|
}
|
|
d.logger.Info("gc: started",
|
|
"interval", d.cfg.GCInterval,
|
|
"ttl", d.cfg.GCTTL,
|
|
"orphan_ttl", d.cfg.GCOrphanTTL,
|
|
"artifact_ttl", d.cfg.GCArtifactTTL,
|
|
"artifact_patterns", d.cfg.GCArtifactPatterns,
|
|
)
|
|
|
|
// Run once at startup after a short delay (let the daemon finish initializing).
|
|
if err := sleepWithContext(ctx, 30*time.Second); err != nil {
|
|
return
|
|
}
|
|
d.runGC(ctx)
|
|
|
|
ticker := time.NewTicker(d.cfg.GCInterval)
|
|
defer ticker.Stop()
|
|
|
|
for {
|
|
select {
|
|
case <-ctx.Done():
|
|
return
|
|
case <-ticker.C:
|
|
d.runGC(ctx)
|
|
}
|
|
}
|
|
}
|
|
|
|
// gcStats accumulates byte counts and per-pattern hit counts for one GC cycle.
|
|
type gcStats struct {
|
|
cleaned int // whole task dirs removed (issue done/cancelled)
|
|
orphaned int // whole task dirs removed (no meta / unreachable issue)
|
|
skipped int // task dirs left untouched
|
|
artifactDirs int // task dirs that had at least one artifact reclaimed
|
|
artifactRemoved int // count of removed artifact subdirs
|
|
storesReclaimed int // per-issue Codex session stores reclaimed past their TTL
|
|
bytesReclaimed int64 // total bytes freed in this cycle
|
|
byPattern map[string]int // basename -> reclaim count, for visibility
|
|
}
|
|
|
|
// runGC performs a single GC scan across all workspace directories.
|
|
func (d *Daemon) runGC(ctx context.Context) {
|
|
root := d.cfg.WorkspacesRoot
|
|
entries, err := os.ReadDir(root)
|
|
if err != nil {
|
|
if os.IsNotExist(err) {
|
|
return
|
|
}
|
|
d.logger.Warn("gc: read workspaces root failed", "error", err)
|
|
return
|
|
}
|
|
|
|
stats := &gcStats{byPattern: map[string]int{}}
|
|
for _, wsEntry := range entries {
|
|
if !wsEntry.IsDir() || wsEntry.Name() == ".repos" {
|
|
continue
|
|
}
|
|
wsDir := filepath.Join(root, wsEntry.Name())
|
|
d.gcWorkspace(ctx, wsDir, stats)
|
|
}
|
|
|
|
// Prune stale worktree references from all bare repo caches.
|
|
d.pruneRepoWorktrees(root)
|
|
|
|
// Reclaim per-issue Codex session stores idle past their TTL. These live
|
|
// under the shared ~/.codex home (outside WorkspacesRoot) so resume survives
|
|
// the task GC, which means they need their own bounded lifecycle (MUL-4424).
|
|
if storesRemoved, storeBytes := execenv.PruneCodexSessionStores(d.cfg.Profile, d.cfg.GCCodexSessionTTL, time.Now(), d.reserveCodexStoreForDeletion, d.logger); storesRemoved > 0 {
|
|
stats.storesReclaimed += storesRemoved
|
|
stats.bytesReclaimed += storeBytes
|
|
}
|
|
|
|
if stats.cleaned > 0 || stats.orphaned > 0 || stats.artifactDirs > 0 || stats.storesReclaimed > 0 {
|
|
d.logger.Info("gc: cycle complete",
|
|
"cleaned", stats.cleaned,
|
|
"orphaned", stats.orphaned,
|
|
"skipped", stats.skipped,
|
|
"artifact_dirs", stats.artifactDirs,
|
|
"artifact_removed", stats.artifactRemoved,
|
|
"codex_session_stores_reclaimed", stats.storesReclaimed,
|
|
"bytes_reclaimed", stats.bytesReclaimed,
|
|
"by_pattern", stats.byPattern,
|
|
)
|
|
}
|
|
}
|
|
|
|
// gcWorkspace scans task directories inside a single workspace directory.
|
|
func (d *Daemon) gcWorkspace(ctx context.Context, wsDir string, stats *gcStats) {
|
|
taskEntries, err := os.ReadDir(wsDir)
|
|
if err != nil {
|
|
d.logger.Warn("gc: read workspace dir failed", "dir", wsDir, "error", err)
|
|
return
|
|
}
|
|
|
|
cleanedHere := 0
|
|
for _, entry := range taskEntries {
|
|
if ctx.Err() != nil {
|
|
return
|
|
}
|
|
if !entry.IsDir() {
|
|
continue
|
|
}
|
|
taskDir := filepath.Join(wsDir, entry.Name())
|
|
action := d.shouldCleanTaskDir(ctx, taskDir)
|
|
switch action {
|
|
case gcActionClean:
|
|
bytes := dirSize(taskDir)
|
|
d.cleanTaskDir(taskDir)
|
|
stats.cleaned++
|
|
stats.bytesReclaimed += bytes
|
|
cleanedHere++
|
|
case gcActionOrphan:
|
|
bytes := dirSize(taskDir)
|
|
d.cleanTaskDir(taskDir)
|
|
stats.orphaned++
|
|
stats.bytesReclaimed += bytes
|
|
cleanedHere++
|
|
case gcActionCleanArtifacts:
|
|
removed, bytes, perPattern := d.cleanTaskArtifacts(taskDir, d.cfg.GCArtifactPatterns)
|
|
if removed > 0 {
|
|
stats.artifactDirs++
|
|
stats.artifactRemoved += removed
|
|
stats.bytesReclaimed += bytes
|
|
for k, v := range perPattern {
|
|
stats.byPattern[k] += v
|
|
}
|
|
}
|
|
stats.skipped++ // task dir itself preserved
|
|
default:
|
|
stats.skipped++
|
|
}
|
|
}
|
|
|
|
// Remove the workspace directory itself if it's now empty.
|
|
if cleanedHere > 0 {
|
|
remaining, _ := os.ReadDir(wsDir)
|
|
if len(remaining) == 0 {
|
|
os.Remove(wsDir)
|
|
}
|
|
}
|
|
}
|
|
|
|
type gcAction int
|
|
|
|
const (
|
|
gcActionSkip gcAction = iota
|
|
gcActionClean // issue is done/cancelled and stale
|
|
gcActionOrphan // no meta or unknown issue and dir is old
|
|
gcActionCleanArtifacts // task completed long enough ago; drop regenerable artifacts only
|
|
)
|
|
|
|
// shouldCleanTaskDir decides whether a task directory should be removed.
|
|
// Dispatches on meta.Kind so chat / autopilot / quick-create tasks each
|
|
// follow the parent record that actually governs their lifecycle.
|
|
func (d *Daemon) shouldCleanTaskDir(ctx context.Context, taskDir string) gcAction {
|
|
// A task currently running on this env root must never be reclaimed —
|
|
// not even on the done/cancelled or orphan-404 paths. A new comment on
|
|
// an already-done issue can dispatch a follow-up task that reuses the
|
|
// prior workdir without bumping the issue's updated_at, so the regular
|
|
// TTL check alone wouldn't notice the resumed activity.
|
|
if d.isActiveEnvRoot(taskDir) {
|
|
return gcActionSkip
|
|
}
|
|
|
|
meta, err := execenv.ReadGCMeta(taskDir)
|
|
if err != nil {
|
|
return d.orphanByMTime(taskDir, "no meta")
|
|
}
|
|
|
|
action := d.shouldCleanTaskDirForKind(ctx, taskDir, meta)
|
|
if !meta.LocalDirectory {
|
|
return action
|
|
}
|
|
// local_directory tasks keep their envRoot indefinitely so the user
|
|
// can inspect output/ and logs/ for forensic context. The WorkDir is
|
|
// the user's own path and lives outside taskDir, so the envRoot
|
|
// itself is just the daemon's logbook for the run — never large, and
|
|
// safe to keep.
|
|
//
|
|
// gcActionClean → demote to artifact-pattern cleanup so envRoot
|
|
// (and especially the logbook) survives.
|
|
// gcActionOrphan → skip outright; we don't ever wipe a
|
|
// local_directory envRoot via the mtime path,
|
|
// since the parent issue / chat record going
|
|
// away should not collateral-delete the user's
|
|
// own audit trail.
|
|
//
|
|
// gcActionCleanArtifacts and gcActionSkip already obey the
|
|
// "no full envRoot RemoveAll" rule.
|
|
switch action {
|
|
case gcActionClean:
|
|
return gcActionCleanArtifacts
|
|
case gcActionOrphan:
|
|
return gcActionSkip
|
|
default:
|
|
return action
|
|
}
|
|
}
|
|
|
|
// shouldCleanTaskDirForKind runs the per-Kind dispatch without applying the
|
|
// local_directory override. Split out so shouldCleanTaskDir can intercept
|
|
// the result.
|
|
func (d *Daemon) shouldCleanTaskDirForKind(ctx context.Context, taskDir string, meta *execenv.GCMeta) gcAction {
|
|
switch meta.Kind {
|
|
case execenv.GCKindIssue:
|
|
return d.gcDecisionIssue(ctx, taskDir, meta)
|
|
case execenv.GCKindChat:
|
|
return d.gcDecisionChat(ctx, taskDir, meta)
|
|
case execenv.GCKindAutopilotRun:
|
|
return d.gcDecisionAutopilotRun(ctx, taskDir, meta)
|
|
case execenv.GCKindQuickCreate:
|
|
return d.gcDecisionQuickCreate(ctx, taskDir, meta)
|
|
default:
|
|
// Unknown kind: fall back to mtime-based orphan cleanup so a future
|
|
// daemon writing a kind we don't recognize doesn't get insta-wiped.
|
|
return d.orphanByMTime(taskDir, "unknown kind")
|
|
}
|
|
}
|
|
|
|
// orphanByMTime returns gcActionOrphan if the directory is older than
|
|
// GCOrphanTTL, gcActionSkip otherwise. Centralizes the "we have no parent
|
|
// record signal so just look at the disk" fallback used by every kind.
|
|
func (d *Daemon) orphanByMTime(taskDir, reason string) gcAction {
|
|
info, err := os.Stat(taskDir)
|
|
if err != nil {
|
|
return gcActionSkip
|
|
}
|
|
if time.Since(info.ModTime()) > d.cfg.GCOrphanTTL {
|
|
d.logger.Info("gc: orphan directory", "dir", taskDir, "reason", reason, "age", time.Since(info.ModTime()).Round(time.Hour))
|
|
return gcActionOrphan
|
|
}
|
|
return gcActionSkip
|
|
}
|
|
|
|
// isAccessNotFound detects the 404 returned by gc-check endpoints. The same
|
|
// status covers "row deleted" and "daemon token can't see this workspace"
|
|
// (the requireDaemonWorkspaceAccess anti-enumeration shape), so callers
|
|
// can't tell the two apart from the response alone.
|
|
func isAccessNotFound(err error) bool {
|
|
var reqErr *requestError
|
|
return errors.As(err, &reqErr) && reqErr.StatusCode == http.StatusNotFound
|
|
}
|
|
|
|
func (d *Daemon) gcDecisionIssue(ctx context.Context, taskDir string, meta *execenv.GCMeta) gcAction {
|
|
if strings.TrimSpace(meta.IssueID) == "" {
|
|
return d.orphanByMTime(taskDir, "empty issue id")
|
|
}
|
|
|
|
status, err := d.client.GetIssueGCCheck(ctx, meta.IssueID)
|
|
if err != nil {
|
|
if isAccessNotFound(err) {
|
|
// 404 is ambiguous: server returns it for both "issue deleted"
|
|
// and "daemon token has no access to the workspace". Fall back
|
|
// to the mtime-gated orphan cleanup so a scoped-down token
|
|
// can't instantly wipe dirs whose issues are still live.
|
|
return d.orphanByMTime(taskDir, "issue not accessible")
|
|
}
|
|
return gcActionSkip
|
|
}
|
|
|
|
if (status.Status == "done" || status.Status == "cancelled") &&
|
|
time.Since(status.UpdatedAt) > d.cfg.GCTTL {
|
|
d.logger.Info("gc: eligible for cleanup",
|
|
"dir", filepath.Base(taskDir),
|
|
"kind", "issue",
|
|
"issue", meta.IssueID,
|
|
"status", status.Status,
|
|
"updated_at", status.UpdatedAt.Format(time.RFC3339),
|
|
)
|
|
return gcActionClean
|
|
}
|
|
|
|
if d.cfg.GCArtifactTTL > 0 && len(d.cfg.GCArtifactPatterns) > 0 &&
|
|
!meta.CompletedAt.IsZero() && time.Since(meta.CompletedAt) > d.cfg.GCArtifactTTL {
|
|
d.logger.Info("gc: eligible for artifact cleanup",
|
|
"dir", filepath.Base(taskDir),
|
|
"kind", "issue",
|
|
"issue", meta.IssueID,
|
|
"status", status.Status,
|
|
"completed_at", meta.CompletedAt.Format(time.RFC3339),
|
|
)
|
|
return gcActionCleanArtifacts
|
|
}
|
|
|
|
return gcActionSkip
|
|
}
|
|
|
|
func (d *Daemon) gcDecisionChat(ctx context.Context, taskDir string, meta *execenv.GCMeta) gcAction {
|
|
if strings.TrimSpace(meta.ChatSessionID) == "" {
|
|
return d.orphanByMTime(taskDir, "empty chat session id")
|
|
}
|
|
|
|
status, err := d.client.GetChatSessionGCCheck(ctx, meta.ChatSessionID)
|
|
if err != nil {
|
|
if isAccessNotFound(err) {
|
|
// 404 means the chat_session row is gone — DeleteChatSession is
|
|
// a real DELETE, so a hard delete propagates here as soon as
|
|
// the user clicks the button. This is the strongest reclaim
|
|
// signal we get and it's exactly acceptance criterion #3:
|
|
// reclaim within one GC cycle (≤ GCInterval), not 72h.
|
|
//
|
|
// We don't gate on mtime: every chat_session_id in a meta file
|
|
// was written by this daemon under its current token, so there
|
|
// is no cross-workspace probe to defend against.
|
|
d.logger.Info("gc: eligible for cleanup",
|
|
"dir", filepath.Base(taskDir),
|
|
"kind", "chat",
|
|
"chat_session", meta.ChatSessionID,
|
|
"reason", "session not accessible (hard-deleted)",
|
|
)
|
|
return gcActionClean
|
|
}
|
|
return gcActionSkip
|
|
}
|
|
|
|
switch status.Status {
|
|
case "active":
|
|
// An active chat session must never be reclaimed by mtime — that
|
|
// would silently kill a user's idle session and break "PriorWorkDir"
|
|
// resume on their next message. This is the explicit short-circuit
|
|
// the issue body called out as verifyable behavior #2.
|
|
return gcActionSkip
|
|
case "archived":
|
|
if time.Since(status.UpdatedAt) > d.cfg.GCTTL {
|
|
d.logger.Info("gc: eligible for cleanup",
|
|
"dir", filepath.Base(taskDir),
|
|
"kind", "chat",
|
|
"chat_session", meta.ChatSessionID,
|
|
"status", status.Status,
|
|
"updated_at", status.UpdatedAt.Format(time.RFC3339),
|
|
)
|
|
return gcActionClean
|
|
}
|
|
}
|
|
return gcActionSkip
|
|
}
|
|
|
|
func (d *Daemon) gcDecisionAutopilotRun(ctx context.Context, taskDir string, meta *execenv.GCMeta) gcAction {
|
|
if strings.TrimSpace(meta.AutopilotRunID) == "" {
|
|
return d.orphanByMTime(taskDir, "empty autopilot run id")
|
|
}
|
|
|
|
status, err := d.client.GetAutopilotRunGCCheck(ctx, meta.AutopilotRunID)
|
|
if err != nil {
|
|
if isAccessNotFound(err) {
|
|
return d.orphanByMTime(taskDir, "autopilot run not accessible")
|
|
}
|
|
return gcActionSkip
|
|
}
|
|
|
|
// Terminal states per the autopilot_run CHECK constraint:
|
|
// completed, failed, skipped — the run finished its own work.
|
|
// issue_created — the run produced an issue task that owns
|
|
// its own workdir; this run's workdir is
|
|
// dead weight from here on.
|
|
// Non-terminal: pending, running. Skip until they reach a terminal state
|
|
// rather than trying to bound them by mtime — long autopilots are real.
|
|
//
|
|
// An autopilot run's workdir is never reused: unlike issue/chat tasks there
|
|
// is no PriorWorkDir path that hands a later run the same directory, so every
|
|
// run gets a fresh one. Whatever the run produced already lives server-side
|
|
// (and an issue_created run handed its work to an issue task that owns its own
|
|
// envRoot). So the moment the run reaches a terminal state the directory is
|
|
// dead weight and we reclaim it immediately, without waiting out GCTTL — the
|
|
// same reasoning gcDecisionQuickCreate applies to quick-create dirs. The
|
|
// active-env-root short-circuit in shouldCleanTaskDir still protects a run
|
|
// that is mid-flight, so this can't pull the rug from under live work.
|
|
if isAutopilotRunTerminal(status.Status) {
|
|
d.logger.Info("gc: eligible for cleanup",
|
|
"dir", filepath.Base(taskDir),
|
|
"kind", "autopilot_run",
|
|
"autopilot_run", meta.AutopilotRunID,
|
|
"status", status.Status,
|
|
)
|
|
return gcActionClean
|
|
}
|
|
return gcActionSkip
|
|
}
|
|
|
|
// isAutopilotRunTerminal mirrors the run.status CHECK in
|
|
// migrations/042_autopilot.up.sql. Non-terminal states are pending/running;
|
|
// every other value the schema allows is a final resting state from the
|
|
// daemon's POV (the run is no longer producing work in this workdir).
|
|
func isAutopilotRunTerminal(status string) bool {
|
|
switch status {
|
|
case "completed", "failed", "skipped", "issue_created":
|
|
return true
|
|
default:
|
|
return false
|
|
}
|
|
}
|
|
|
|
func (d *Daemon) gcDecisionQuickCreate(ctx context.Context, taskDir string, meta *execenv.GCMeta) gcAction {
|
|
if strings.TrimSpace(meta.TaskID) == "" {
|
|
return d.orphanByMTime(taskDir, "empty task id")
|
|
}
|
|
|
|
status, err := d.client.GetTaskGCCheck(ctx, meta.TaskID)
|
|
if err != nil {
|
|
if isAccessNotFound(err) {
|
|
// Task row was hard-deleted, or token can't see it. Either way,
|
|
// fall back to mtime-gated orphan to stay safe across scoped
|
|
// tokens — same reasoning as the issue path.
|
|
return d.orphanByMTime(taskDir, "task not accessible")
|
|
}
|
|
return gcActionSkip
|
|
}
|
|
|
|
// Quick-create workdirs are not reused by the issue task that
|
|
// LinkTaskToIssue eventually attaches — that issue gets its own
|
|
// envRoot. So as soon as the quick-create task itself reaches a
|
|
// terminal state we can reclaim the directory immediately, without
|
|
// waiting for GCTTL. If the user wants to revisit, the linked issue
|
|
// has the agent's output already.
|
|
if isAgentTaskTerminal(status.Status) {
|
|
d.logger.Info("gc: eligible for cleanup",
|
|
"dir", filepath.Base(taskDir),
|
|
"kind", "quick_create",
|
|
"task", meta.TaskID,
|
|
"status", status.Status,
|
|
)
|
|
return gcActionClean
|
|
}
|
|
return gcActionSkip
|
|
}
|
|
|
|
// isAgentTaskTerminal reports whether a value of agent_task_queue.status
|
|
// represents a final state. Mirrors the status enum used across the
|
|
// task service — see service/task.go for the canonical list.
|
|
func isAgentTaskTerminal(status string) bool {
|
|
switch status {
|
|
case "completed", "failed", "cancelled":
|
|
return true
|
|
default:
|
|
return false
|
|
}
|
|
}
|
|
|
|
// cleanTaskDir removes a task directory and logs the result.
|
|
func (d *Daemon) cleanTaskDir(taskDir string) {
|
|
if err := os.RemoveAll(taskDir); err != nil {
|
|
d.logger.Warn("gc: remove task dir failed", "dir", taskDir, "error", err)
|
|
} else {
|
|
d.logger.Info("gc: removed", "dir", taskDir)
|
|
}
|
|
}
|
|
|
|
// cleanTaskArtifacts walks taskDir and deletes every directory whose basename
|
|
// matches one of patterns. Returns (removedCount, bytesReclaimed, perPattern).
|
|
//
|
|
// Safety contract:
|
|
// - patterns are basename-only; entries with a path separator are dropped.
|
|
// - .git subtrees are never descended into, so the agent's git history stays
|
|
// intact even if a pattern would otherwise match.
|
|
// - symlinks are skipped entirely — neither the link nor its target is
|
|
// touched, so a malicious or stale link can't redirect the GC outside the
|
|
// workdir.
|
|
// - every removal target is verified to live inside taskDir, so a tampered
|
|
// .gc_meta.json can't trick the daemon into deleting outside its sandbox.
|
|
func (d *Daemon) cleanTaskArtifacts(taskDir string, patterns []string) (removed int, bytes int64, perPattern map[string]int) {
|
|
perPattern = map[string]int{}
|
|
if taskDir == "" || len(patterns) == 0 {
|
|
return
|
|
}
|
|
patternSet := make(map[string]struct{}, len(patterns))
|
|
for _, p := range patterns {
|
|
p = strings.TrimSpace(p)
|
|
if p == "" || strings.ContainsAny(p, "/\\") {
|
|
continue
|
|
}
|
|
patternSet[p] = struct{}{}
|
|
}
|
|
if len(patternSet) == 0 {
|
|
return
|
|
}
|
|
|
|
absRoot, err := filepath.Abs(taskDir)
|
|
if err != nil {
|
|
return
|
|
}
|
|
|
|
walkErr := filepath.WalkDir(absRoot, func(path string, entry os.DirEntry, err error) error {
|
|
if err != nil {
|
|
return nil // best-effort — keep walking
|
|
}
|
|
if path == absRoot {
|
|
return nil
|
|
}
|
|
if !entry.IsDir() {
|
|
return nil
|
|
}
|
|
// Never descend into .git — preserves agent commits even if a pattern
|
|
// like "objects" would otherwise match.
|
|
if entry.Name() == ".git" {
|
|
return filepath.SkipDir
|
|
}
|
|
// Refuse to follow symlinked directories. WalkDir reports them as type
|
|
// Dir on some platforms; lstat to be sure.
|
|
info, statErr := os.Lstat(path)
|
|
if statErr != nil {
|
|
return nil
|
|
}
|
|
if info.Mode()&os.ModeSymlink != 0 {
|
|
return filepath.SkipDir
|
|
}
|
|
if _, ok := patternSet[entry.Name()]; !ok {
|
|
return nil
|
|
}
|
|
// Containment check: target must remain inside taskDir.
|
|
rel, relErr := filepath.Rel(absRoot, path)
|
|
if relErr != nil || rel == "" || rel == "." || strings.HasPrefix(rel, "..") {
|
|
return filepath.SkipDir
|
|
}
|
|
size := dirSize(path)
|
|
if rmErr := os.RemoveAll(path); rmErr != nil {
|
|
d.logger.Warn("gc: artifact remove failed", "path", path, "error", rmErr)
|
|
return filepath.SkipDir
|
|
}
|
|
removed++
|
|
bytes += size
|
|
perPattern[entry.Name()]++
|
|
d.logger.Info("gc: artifact removed", "path", path, "bytes", size)
|
|
// Don't descend into the now-deleted subtree.
|
|
return filepath.SkipDir
|
|
})
|
|
if walkErr != nil {
|
|
d.logger.Warn("gc: artifact walk failed", "dir", taskDir, "error", walkErr)
|
|
}
|
|
return
|
|
}
|
|
|
|
// dirSize returns the total size of all regular files under root, in bytes.
|
|
// Non-fatal: errors during the walk are ignored so callers can report a
|
|
// best-effort byte count without aborting the whole GC cycle.
|
|
func dirSize(root string) int64 {
|
|
var total int64
|
|
_ = filepath.WalkDir(root, func(_ string, entry os.DirEntry, err error) error {
|
|
if err != nil {
|
|
return nil
|
|
}
|
|
if entry.IsDir() {
|
|
return nil
|
|
}
|
|
info, infoErr := entry.Info()
|
|
if infoErr != nil {
|
|
return nil
|
|
}
|
|
if info.Mode().IsRegular() {
|
|
total += info.Size()
|
|
}
|
|
return nil
|
|
})
|
|
return total
|
|
}
|
|
|
|
const (
|
|
gitCmdTimeout = 30 * time.Second
|
|
gitMaintenanceTimeout = 10 * time.Minute
|
|
)
|
|
|
|
// pruneRepoWorktrees runs `git worktree prune` on all bare repos in the cache.
|
|
func (d *Daemon) pruneRepoWorktrees(workspacesRoot string) {
|
|
reposRoot := filepath.Join(workspacesRoot, ".repos")
|
|
wsEntries, err := os.ReadDir(reposRoot)
|
|
if err != nil {
|
|
return
|
|
}
|
|
|
|
for _, wsEntry := range wsEntries {
|
|
if !wsEntry.IsDir() {
|
|
continue
|
|
}
|
|
wsRepoDir := filepath.Join(reposRoot, wsEntry.Name())
|
|
repoEntries, err := os.ReadDir(wsRepoDir)
|
|
if err != nil {
|
|
continue
|
|
}
|
|
for _, repoEntry := range repoEntries {
|
|
if !repoEntry.IsDir() {
|
|
continue
|
|
}
|
|
barePath := filepath.Join(wsRepoDir, repoEntry.Name())
|
|
if !isBareRepo(barePath) {
|
|
continue
|
|
}
|
|
d.pruneWorktree(barePath)
|
|
}
|
|
}
|
|
}
|
|
|
|
func (d *Daemon) pruneWorktree(barePath string) {
|
|
if d.repoCache != nil {
|
|
if err := d.repoCache.WithRepoLock(barePath, func() error {
|
|
d.pruneWorktreeLocked(barePath)
|
|
return nil
|
|
}); err != nil {
|
|
d.logger.Warn("gc: repo lock failed", "repo", barePath, "error", err)
|
|
return
|
|
}
|
|
return
|
|
}
|
|
|
|
d.pruneWorktreeLocked(barePath)
|
|
}
|
|
|
|
func (d *Daemon) pruneWorktreeLocked(barePath string) {
|
|
if out, err := runGitGCCommand(barePath, "worktree", "prune"); err != nil {
|
|
d.logger.Warn("gc: worktree prune failed",
|
|
"repo", barePath,
|
|
"output", out,
|
|
"error", err,
|
|
)
|
|
}
|
|
|
|
activeBranches, err := agentWorktreeBranches(barePath)
|
|
if err != nil {
|
|
d.logger.Warn("gc: worktree branch scan failed", "repo", barePath, "error", err)
|
|
return
|
|
}
|
|
|
|
agentBranches, err := listAgentBranches(barePath)
|
|
if err != nil {
|
|
d.logger.Warn("gc: agent branch scan failed", "repo", barePath, "error", err)
|
|
return
|
|
}
|
|
|
|
deleted := 0
|
|
for _, branch := range agentBranches {
|
|
if _, ok := activeBranches[branch]; ok {
|
|
continue
|
|
}
|
|
if out, err := runGitGCCommand(barePath, "branch", "-D", "--", branch); err != nil {
|
|
d.logger.Warn("gc: agent branch delete failed",
|
|
"repo", barePath,
|
|
"branch", branch,
|
|
"output", out,
|
|
"error", err,
|
|
)
|
|
continue
|
|
}
|
|
deleted++
|
|
}
|
|
if deleted == 0 {
|
|
return
|
|
}
|
|
d.logger.Info("gc: deleted stale agent branches", "repo", barePath, "count", deleted)
|
|
|
|
// Heavier maintenance only runs when we actually removed refs, so we don't
|
|
// turn every GC tick into a full `git gc --prune` on every cached repo. The
|
|
// prune step gets its own longer timeout because it can take minutes on a
|
|
// real bare cache; under the shared 30s budget it would be killed mid-run.
|
|
maintenance := []struct {
|
|
args []string
|
|
timeout time.Duration
|
|
}{
|
|
{args: []string{"reflog", "expire", "--expire=30.days", "--all"}, timeout: gitCmdTimeout},
|
|
{args: []string{"gc", "--prune=30.days"}, timeout: gitMaintenanceTimeout},
|
|
}
|
|
for _, step := range maintenance {
|
|
if out, err := runGitCommand(barePath, step.timeout, step.args...); err != nil {
|
|
d.logger.Warn("gc: git maintenance failed",
|
|
"repo", barePath,
|
|
"command", strings.Join(step.args, " "),
|
|
"output", out,
|
|
"error", err,
|
|
)
|
|
}
|
|
}
|
|
}
|
|
|
|
func runGitGCCommand(barePath string, args ...string) (string, error) {
|
|
return runGitCommand(barePath, gitCmdTimeout, args...)
|
|
}
|
|
|
|
func runGitCommand(barePath string, timeout time.Duration, args ...string) (string, error) {
|
|
ctx, cancel := context.WithTimeout(context.Background(), timeout)
|
|
defer cancel()
|
|
|
|
cmdArgs := append([]string{"-C", barePath}, args...)
|
|
cmd := exec.CommandContext(ctx, "git", cmdArgs...)
|
|
out, err := cmd.CombinedOutput()
|
|
return strings.TrimSpace(string(out)), err
|
|
}
|
|
|
|
func agentWorktreeBranches(barePath string) (map[string]struct{}, error) {
|
|
out, err := runGitGCCommand(barePath, "worktree", "list", "--porcelain")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
|
|
branches := make(map[string]struct{})
|
|
for _, line := range strings.Split(out, "\n") {
|
|
line = strings.TrimSpace(line)
|
|
if !strings.HasPrefix(line, "branch refs/heads/") {
|
|
continue
|
|
}
|
|
branch := strings.TrimPrefix(line, "branch refs/heads/")
|
|
if strings.HasPrefix(branch, "agent/") {
|
|
branches[branch] = struct{}{}
|
|
}
|
|
}
|
|
return branches, nil
|
|
}
|
|
|
|
func listAgentBranches(barePath string) ([]string, error) {
|
|
// Trailing slash narrows the pattern to the `agent/` namespace only. Without
|
|
// it, `for-each-ref` would also return a branch literally named `agent`,
|
|
// which `agentWorktreeBranches` ignores — that branch would then be deleted.
|
|
out, err := runGitGCCommand(barePath, "for-each-ref", "--format=%(refname:short)", "refs/heads/agent/")
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
if out == "" {
|
|
return nil, nil
|
|
}
|
|
|
|
var branches []string
|
|
for _, line := range strings.Split(out, "\n") {
|
|
branch := strings.TrimSpace(line)
|
|
if branch == "" {
|
|
continue
|
|
}
|
|
branches = append(branches, branch)
|
|
}
|
|
return branches, nil
|
|
}
|
|
|
|
// isBareRepo checks if a path looks like a bare git repository.
|
|
func isBareRepo(path string) bool {
|
|
if _, err := os.Stat(filepath.Join(path, "HEAD")); err != nil {
|
|
return false
|
|
}
|
|
if _, err := os.Stat(filepath.Join(path, "objects")); err != nil {
|
|
return false
|
|
}
|
|
return true
|
|
}
|