mirror of
https://github.com/multica-ai/multica.git
synced 2026-08-05 09:30:05 +02:00
* feat(daemon): add worktree_pool mode for local_directory (MUL-3483) ## What changed Squad workflows bound to the same `local_directory` resource used to serialise on a single path mutex — a documented pain point from GitHub issue #4377. This introduces an opt-in `worktree_pool` mode on the `local_directory` project resource. When enabled, each task gets its own `git worktree add` under a daemon-managed pool root, so sibling tasks on the same base repo now run truly in parallel while `git worktree add/remove/prune` stays serialised behind a per-repo mutex. ## Shape - `local_directory.resource_ref` gains three optional fields: `mode` ("in_place" default / "worktree_pool"), `pool_root` (defaults to `<parent>/.multica-worktrees/<base>`), `max_parallel` (defaults to 4). Legacy rows are byte-identical after round-trip: the server validator strips the pool fields on the default in_place path so older clients keep behaving exactly as before. - New `WorktreePoolManager` (`server/internal/daemon/worktree_pool.go`) owns pool allocation, per-repo git-metadata mutex, and cleanup. - `acquireLocalDirectoryLockIfNeeded` now branches on the ref's mode. in_place stays on `LocalPathLocker` and the shared tree; worktree_pool routes through the pool manager, publishes a lease keyed by task ID, and pins the agent to the freshly allocated worktree in `execenv.PrepareParams.LocalWorkDir`. - Pool saturation is a structured wait_reason (`worktree_pool saturated (N/M) on <path> (holders: ...)`), retrying on the existing cancel-poll interval — same UX as the historical path-mutex wait. ## Safety guardrails (also known footguns from prior art) - Repos with initialised submodules are refused up front. Multi-checkout of a superproject is explicitly unsupported by `git worktree(1)` BUGS and the per-worktree `modules/` directories bloat disk by pool size ×. - Dirty worktrees are NEVER `--force` removed on release. If the agent left uncommitted changes behind we keep the directory (and free the slot) so users can inspect. This is the failure mode claude-code#55724 documented and the pool must not regress into. - The per-repo mutex covers every `git worktree add/remove/prune` and `submodule status` invocation for a given base, matching the in-process-queue fix Anthropic settled on for claude-code#34645 (`.git/config.lock` races on concurrent add). - Task UUID is the source of truth for both branch (`multica/<uuid>`) and worktree path (`<pool_root>/<uuid>`) so a single agent running multiple worker tasks in parallel can never collide. - Non-empty leftover directories at the target path abort the allocation instead of silently starting the agent in an unknown state. ## Explicit MVP non-goals (deferred, tracked as follow-up work) - Windows worktree-remove retry (permission-denied on locked handles). - Detached-HEAD fast path for read-only exploration tasks. - `post-checkout` hook opt-out / serialisation. - Automatic `git lfs install`. - UI surfacing of the pool state / dirty worktree list. ## Tests - `worktree_pool_test.go` (new): full acquire→release lifecycle, parallel allocation, saturation with holder list, slot re-use after release, dirty-worktree preservation, concurrent-acquire serialisation (the config.lock guard), submodule refusal, missing base rejection, pool root auto-mkdir, non-empty leftover refusal, ctx cancel. - Handler validator gains three rejection cases (unknown mode, relative pool_root, negative max_parallel) and a round-trip test that pins the normalised JSON shape for both modes. - Daemon `localDirectoryRef` helpers get a defaults test and the pool root path derivation is pinned. ## Wire-compat and rollout - Default off. Existing rows keep the historical shape (no `mode`, `pool_root`, or `max_parallel` in the JSON) and behave exactly as before. - Opt-in via `--ref '{"local_path":"...","daemon_id":"...","mode":"worktree_pool"}'` today. CLI flag shortcuts (`--mode`, `--pool-root`, `--max-parallel`) can follow in a small tail PR — not blocking. - No DB migration. No UI change required. Co-authored-by: multica-agent <github@multica.ai> * feat(daemon): address worktree_pool review nits (MUL-3483) Follow-up to #4986. Three non-blocking review points from GPT-Boy: 1. **Daemon integration test for lease → runTask plumbing.** `TestAcquireLocalDirectory_WorktreePoolPublishesLease` (and its in_place counterpart) pin the exact contract runTask relies on when it reads `d.localLeases.Load(task.ID)` and feeds `lease.WorkDir` into `execenv.PrepareParams.LocalWorkDir`. A future refactor that drops the Store, mistypes the key, or swaps back to `assignment.AbsPath` on the pool branch will now fail here rather than silently defeat the whole point of worktree_pool mode. 2. **Untracked-only dirty case now classifies as dirty.** `worktreeIsDirty` used `--untracked-files=no`, which meant a worktree with only untracked files was reported "clean" and hit the `git worktree remove` branch — git itself would then refuse the removal because the file exists (so no data was lost), but the log path lied about what happened on disk. Switching to `--untracked-files=normal` routes agents' fresh drafts directly through the "leaving on disk for user inspection" branch, and `TestWorktreePool_UntrackedOnlyIsKept` pins the guarantee so nobody quietly reverts the flag later. 3. **Skill doc note on default `pool_root` location.** `multica-projects-and-resources/SKILL.md` now spells out the three new ref fields (`mode`, `pool_root`, `max_parallel`), the default `<parent>/.multica-worktrees/<repo>` location (next to the repo, not inside it), the write-permission requirement on the parent directory, and the submodule restriction — so agents advising self-host users hit the right doc line rather than reading source. Existing test suite still green: - `go vet ./...` clean - `go test ./internal/daemon/... ./internal/handler/... ./internal/service/...` all pass Co-authored-by: multica-agent <github@multica.ai> --------- Co-authored-by: Eve <eve@multica-ai.local> Co-authored-by: multica-agent <github@multica.ai>
867 lines
28 KiB
Go
867 lines
28 KiB
Go
package daemon
|
|
|
|
import (
|
|
"context"
|
|
"encoding/json"
|
|
"errors"
|
|
"log/slog"
|
|
"net/http"
|
|
"net/http/httptest"
|
|
"os"
|
|
"path/filepath"
|
|
"runtime"
|
|
"strings"
|
|
"sync"
|
|
"sync/atomic"
|
|
"testing"
|
|
"time"
|
|
)
|
|
|
|
func TestFindLocalDirectoryAssignment(t *testing.T) {
|
|
const thisDaemon = "d-mine"
|
|
otherDaemon := "d-other"
|
|
|
|
mkRef := func(t *testing.T, ref localDirectoryRef) json.RawMessage {
|
|
t.Helper()
|
|
raw, err := json.Marshal(ref)
|
|
if err != nil {
|
|
t.Fatalf("marshal: %v", err)
|
|
}
|
|
return raw
|
|
}
|
|
|
|
tmp := t.TempDir()
|
|
|
|
t.Run("no resources returns nil", func(t *testing.T) {
|
|
got, err := findLocalDirectoryAssignment(nil, thisDaemon)
|
|
if err != nil || got != nil {
|
|
t.Fatalf("expected (nil, nil), got (%+v, %v)", got, err)
|
|
}
|
|
})
|
|
|
|
t.Run("other daemon is skipped", func(t *testing.T) {
|
|
got, err := findLocalDirectoryAssignment([]ProjectResourceData{
|
|
{ID: "r1", ResourceType: localDirectoryResourceType, ResourceRef: mkRef(t, localDirectoryRef{LocalPath: tmp, DaemonID: otherDaemon})},
|
|
}, thisDaemon)
|
|
if err != nil || got != nil {
|
|
t.Fatalf("expected (nil, nil), got (%+v, %v)", got, err)
|
|
}
|
|
})
|
|
|
|
t.Run("non-matching type is skipped", func(t *testing.T) {
|
|
got, err := findLocalDirectoryAssignment([]ProjectResourceData{
|
|
{ID: "r1", ResourceType: "github_repo", ResourceRef: json.RawMessage(`{"url":"https://x"}`)},
|
|
}, thisDaemon)
|
|
if err != nil || got != nil {
|
|
t.Fatalf("expected (nil, nil), got (%+v, %v)", got, err)
|
|
}
|
|
})
|
|
|
|
t.Run("matching daemon returns assignment", func(t *testing.T) {
|
|
got, err := findLocalDirectoryAssignment([]ProjectResourceData{
|
|
{ID: "r1", ResourceType: localDirectoryResourceType, ResourceRef: mkRef(t, localDirectoryRef{LocalPath: tmp, DaemonID: thisDaemon})},
|
|
}, thisDaemon)
|
|
if err != nil {
|
|
t.Fatalf("err: %v", err)
|
|
}
|
|
if got == nil {
|
|
t.Fatalf("expected assignment, got nil")
|
|
}
|
|
if got.AbsPath != filepath.Clean(tmp) {
|
|
t.Errorf("AbsPath = %q, want %q", got.AbsPath, filepath.Clean(tmp))
|
|
}
|
|
if got.RealPath == "" {
|
|
t.Errorf("RealPath empty")
|
|
}
|
|
})
|
|
|
|
t.Run("missing daemon_id is rejected", func(t *testing.T) {
|
|
_, err := findLocalDirectoryAssignment([]ProjectResourceData{
|
|
{ID: "r1", ResourceType: localDirectoryResourceType, ResourceRef: mkRef(t, localDirectoryRef{LocalPath: tmp})},
|
|
}, thisDaemon)
|
|
if err == nil {
|
|
t.Fatalf("expected error for missing daemon_id")
|
|
}
|
|
})
|
|
|
|
t.Run("relative path is rejected", func(t *testing.T) {
|
|
_, err := findLocalDirectoryAssignment([]ProjectResourceData{
|
|
{ID: "r1", ResourceType: localDirectoryResourceType, ResourceRef: mkRef(t, localDirectoryRef{LocalPath: "relative/path", DaemonID: thisDaemon})},
|
|
}, thisDaemon)
|
|
if err == nil {
|
|
t.Fatalf("expected error for relative path")
|
|
}
|
|
})
|
|
|
|
t.Run("malformed ref json fails", func(t *testing.T) {
|
|
_, err := findLocalDirectoryAssignment([]ProjectResourceData{
|
|
{ID: "r1", ResourceType: localDirectoryResourceType, ResourceRef: json.RawMessage(`{not json`)},
|
|
}, thisDaemon)
|
|
if err == nil {
|
|
t.Fatalf("expected error for malformed json")
|
|
}
|
|
})
|
|
|
|
t.Run("two local_directory rows on this daemon fail fast", func(t *testing.T) {
|
|
// Server-side findLocalDirectoryConflict enforces one
|
|
// local_directory per (project, daemon). If two rows are
|
|
// somehow present (older API client, direct DB writes), the
|
|
// daemon must refuse to guess which directory to execute in.
|
|
tmp2 := t.TempDir()
|
|
_, err := findLocalDirectoryAssignment([]ProjectResourceData{
|
|
{ID: "r1", ResourceType: localDirectoryResourceType, ResourceRef: mkRef(t, localDirectoryRef{LocalPath: tmp, DaemonID: thisDaemon})},
|
|
{ID: "r2", ResourceType: localDirectoryResourceType, ResourceRef: mkRef(t, localDirectoryRef{LocalPath: tmp2, DaemonID: thisDaemon})},
|
|
}, thisDaemon)
|
|
if err == nil {
|
|
t.Fatalf("expected error for two local_directory rows pinned to this daemon")
|
|
}
|
|
if !strings.Contains(err.Error(), "multiple local_directory") {
|
|
t.Errorf("error %q did not mention multiple local_directory", err)
|
|
}
|
|
})
|
|
|
|
t.Run("local_directory rows on different daemons coexist", func(t *testing.T) {
|
|
// Different daemons MAY each carry one row — same path on
|
|
// different machines is allowed; this daemon only resolves
|
|
// its own row regardless of how many other-daemon rows are
|
|
// in the list.
|
|
got, err := findLocalDirectoryAssignment([]ProjectResourceData{
|
|
{ID: "r1", ResourceType: localDirectoryResourceType, ResourceRef: mkRef(t, localDirectoryRef{LocalPath: tmp, DaemonID: thisDaemon})},
|
|
{ID: "r2", ResourceType: localDirectoryResourceType, ResourceRef: mkRef(t, localDirectoryRef{LocalPath: tmp, DaemonID: otherDaemon})},
|
|
}, thisDaemon)
|
|
if err != nil {
|
|
t.Fatalf("err: %v", err)
|
|
}
|
|
if got == nil {
|
|
t.Fatalf("expected assignment, got nil")
|
|
}
|
|
})
|
|
}
|
|
|
|
func TestAcquireLocalDirectoryLockSkipsSquadLeaderTasks(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
const daemonID = "d-mine"
|
|
tmp := t.TempDir()
|
|
raw, err := json.Marshal(localDirectoryRef{LocalPath: tmp, DaemonID: daemonID})
|
|
if err != nil {
|
|
t.Fatalf("marshal: %v", err)
|
|
}
|
|
resources := []ProjectResourceData{
|
|
{ID: "r1", ResourceType: localDirectoryResourceType, ResourceRef: raw},
|
|
}
|
|
|
|
worker := Task{
|
|
ID: "worker-task",
|
|
ProjectResources: resources,
|
|
}
|
|
assignment, err := localDirectoryAssignmentForTask(worker, daemonID)
|
|
if err != nil {
|
|
t.Fatalf("worker assignment: %v", err)
|
|
}
|
|
if assignment == nil {
|
|
t.Fatal("worker assignment is nil")
|
|
}
|
|
|
|
d := &Daemon{
|
|
cfg: Config{DaemonID: daemonID},
|
|
localPathLocks: NewLocalPathLocker(),
|
|
logger: slog.Default(),
|
|
}
|
|
leader := Task{
|
|
ID: "leader-task",
|
|
IsLeaderTask: true,
|
|
ProjectResources: resources,
|
|
}
|
|
leaderAssignment, err := localDirectoryAssignmentForTask(leader, daemonID)
|
|
if err != nil {
|
|
t.Fatalf("leader assignment: %v", err)
|
|
}
|
|
if leaderAssignment != nil {
|
|
t.Fatalf("leader assignment = %+v, want nil", leaderAssignment)
|
|
}
|
|
leaderRelease, abort := d.acquireLocalDirectoryLockIfNeeded(context.Background(), leader, slog.Default())
|
|
if abort {
|
|
t.Fatal("leader lock acquisition aborted")
|
|
}
|
|
if leaderRelease != nil {
|
|
t.Fatal("leader lock acquisition returned a release callback")
|
|
}
|
|
if got := d.localPathLocks.Holder(assignment.RealPath); got != "" {
|
|
t.Fatalf("holder after leader skip = %q, want empty", got)
|
|
}
|
|
|
|
release, abort := d.acquireLocalDirectoryLockIfNeeded(context.Background(), worker, slog.Default())
|
|
if abort {
|
|
t.Fatal("worker lock acquisition aborted")
|
|
}
|
|
if release == nil {
|
|
t.Fatal("worker lock acquisition returned nil release")
|
|
}
|
|
defer release()
|
|
if got := d.localPathLocks.Holder(assignment.RealPath); got != worker.ID {
|
|
t.Fatalf("holder = %q, want %q", got, worker.ID)
|
|
}
|
|
|
|
leaderRelease, abort = d.acquireLocalDirectoryLockIfNeeded(context.Background(), leader, slog.Default())
|
|
if abort {
|
|
t.Fatal("leader lock acquisition aborted")
|
|
}
|
|
if leaderRelease != nil {
|
|
t.Fatal("leader lock acquisition returned a release callback")
|
|
}
|
|
if got := d.localPathLocks.Holder(assignment.RealPath); got != worker.ID {
|
|
t.Fatalf("holder after leader skip = %q, want %q", got, worker.ID)
|
|
}
|
|
}
|
|
|
|
func TestValidateLocalPath(t *testing.T) {
|
|
if runtime.GOOS == "windows" {
|
|
t.Skip("blacklist constants are POSIX-only in this test")
|
|
}
|
|
|
|
dir := t.TempDir()
|
|
|
|
t.Run("accepts a writable directory", func(t *testing.T) {
|
|
if err := validateLocalPath(dir); err != nil {
|
|
t.Errorf("unexpected: %v", err)
|
|
}
|
|
})
|
|
|
|
t.Run("rejects relative path", func(t *testing.T) {
|
|
if err := validateLocalPath("relative"); err == nil {
|
|
t.Errorf("expected error")
|
|
}
|
|
})
|
|
|
|
t.Run("rejects empty path", func(t *testing.T) {
|
|
if err := validateLocalPath(""); err == nil {
|
|
t.Errorf("expected error")
|
|
}
|
|
})
|
|
|
|
t.Run("rejects system roots", func(t *testing.T) {
|
|
for _, banned := range []string{"/", "/Users", "/home"} {
|
|
if err := validateLocalPath(banned); err == nil {
|
|
t.Errorf("expected error for %q", banned)
|
|
}
|
|
}
|
|
})
|
|
|
|
t.Run("rejects the user home directory", func(t *testing.T) {
|
|
home, err := os.UserHomeDir()
|
|
if err != nil || home == "" {
|
|
t.Skip("no home dir")
|
|
}
|
|
if err := validateLocalPath(home); err == nil {
|
|
t.Errorf("expected error for $HOME")
|
|
}
|
|
})
|
|
|
|
t.Run("rejects missing path", func(t *testing.T) {
|
|
missing := filepath.Join(dir, "does-not-exist")
|
|
if err := validateLocalPath(missing); err == nil {
|
|
t.Errorf("expected error")
|
|
}
|
|
})
|
|
|
|
t.Run("rejects a regular file", func(t *testing.T) {
|
|
f := filepath.Join(dir, "afile")
|
|
if err := os.WriteFile(f, []byte("hi"), 0o644); err != nil {
|
|
t.Fatalf("write: %v", err)
|
|
}
|
|
if err := validateLocalPath(f); err == nil {
|
|
t.Errorf("expected error")
|
|
}
|
|
})
|
|
|
|
t.Run("rejects an unwritable directory", func(t *testing.T) {
|
|
// chmod-based unwritable is unreliable as root; skip when uid==0.
|
|
if os.Getuid() == 0 {
|
|
t.Skip("test cannot run as root; chmod is a no-op")
|
|
}
|
|
ro := filepath.Join(dir, "ro")
|
|
if err := os.Mkdir(ro, 0o555); err != nil {
|
|
t.Fatalf("mkdir: %v", err)
|
|
}
|
|
t.Cleanup(func() { _ = os.Chmod(ro, 0o755) })
|
|
if err := validateLocalPath(ro); err == nil {
|
|
t.Errorf("expected error for read-only directory")
|
|
}
|
|
})
|
|
|
|
t.Run("rejects a symlink pointing at the user home", func(t *testing.T) {
|
|
home, err := os.UserHomeDir()
|
|
if err != nil || home == "" {
|
|
t.Skip("no home dir")
|
|
}
|
|
link := filepath.Join(dir, "home-link")
|
|
if err := os.Symlink(home, link); err != nil {
|
|
t.Fatalf("symlink: %v", err)
|
|
}
|
|
// The literal cleaned path is dir/home-link, which is NOT in the
|
|
// blacklist. Without the realpath check this used to pass.
|
|
err = validateLocalPath(link)
|
|
if err == nil {
|
|
t.Fatal("expected error for symlink pointing at $HOME")
|
|
}
|
|
if !strings.Contains(err.Error(), "user's home directory") {
|
|
t.Errorf("error %q did not flag the home-dir reason", err.Error())
|
|
}
|
|
})
|
|
|
|
t.Run("rejects a symlink pointing at a system root", func(t *testing.T) {
|
|
link := filepath.Join(dir, "root-link")
|
|
// Pick a banned system root that's predictably present on the
|
|
// host. /Users on macOS; /home on Linux. Fall back to /etc which
|
|
// is in the blacklist and exists on both.
|
|
target := "/etc"
|
|
if err := os.Symlink(target, link); err != nil {
|
|
t.Fatalf("symlink: %v", err)
|
|
}
|
|
err := validateLocalPath(link)
|
|
if err == nil {
|
|
t.Fatal("expected error for symlink pointing at a system root")
|
|
}
|
|
if !strings.Contains(err.Error(), "protected system root") {
|
|
t.Errorf("error %q did not flag the system-root reason", err.Error())
|
|
}
|
|
})
|
|
|
|
t.Run("accepts a symlink to a non-blacklisted directory", func(t *testing.T) {
|
|
target := filepath.Join(dir, "real-proj")
|
|
if err := os.Mkdir(target, 0o755); err != nil {
|
|
t.Fatalf("mkdir: %v", err)
|
|
}
|
|
link := filepath.Join(dir, "proj-link")
|
|
if err := os.Symlink(target, link); err != nil {
|
|
t.Fatalf("symlink: %v", err)
|
|
}
|
|
if err := validateLocalPath(link); err != nil {
|
|
t.Errorf("symlink to a regular directory should pass, got %v", err)
|
|
}
|
|
})
|
|
|
|
// macOS aliases /tmp, /etc, /var to /private/{tmp,etc,var} via OS-level
|
|
// symlinks. A user typing the canonical /private/... form in the picker
|
|
// would pass the literal blacklist (it doesn't contain /private/tmp)
|
|
// and EvalSymlinks would be a no-op (the input is already canonical),
|
|
// so the old "only re-check when realPath != absPath" gate skipped it.
|
|
// Cover the regression so the realpath blacklist always runs.
|
|
t.Run("rejects canonical macOS /private/{tmp,etc,var}", func(t *testing.T) {
|
|
if runtime.GOOS != "darwin" {
|
|
t.Skip("macOS-only: /private/* aliases don't exist elsewhere")
|
|
}
|
|
for _, p := range []string{"/private/tmp", "/private/etc", "/private/var"} {
|
|
if _, statErr := os.Stat(p); statErr != nil {
|
|
t.Logf("skipping %q: %v", p, statErr)
|
|
continue
|
|
}
|
|
err := validateLocalPath(p)
|
|
if err == nil {
|
|
t.Errorf("expected error for canonical %q", p)
|
|
continue
|
|
}
|
|
if !strings.Contains(err.Error(), "protected system root") {
|
|
t.Errorf("error %q for %q did not flag the system-root reason", err.Error(), p)
|
|
}
|
|
}
|
|
})
|
|
|
|
}
|
|
|
|
// TestIsDriveRoot covers the Windows drive-root generalisation. Static
|
|
// enumeration in the old blacklist (C..F) missed mounts at G:\ and up; the
|
|
// new check goes through filepath.VolumeName so any drive letter (and UNC
|
|
// roots) is rejected.
|
|
func TestIsDriveRoot(t *testing.T) {
|
|
if runtime.GOOS != "windows" {
|
|
// filepath.VolumeName returns "" on POSIX, so isDriveRoot always
|
|
// returns false off Windows. The semantic contract is enforced by
|
|
// the early `runtime.GOOS != "windows"` guard; the case table
|
|
// below is only meaningful on a Windows runner.
|
|
t.Skip("windows-only behaviour")
|
|
}
|
|
cases := []struct {
|
|
p string
|
|
want bool
|
|
}{
|
|
{`C:\`, true},
|
|
{`G:\`, true},
|
|
{`Z:\`, true},
|
|
{`C:/`, true},
|
|
{`C:`, true},
|
|
{`\\srv\share`, true},
|
|
{`\\srv\share\`, true},
|
|
{`C:\Users`, false},
|
|
{`D:\proj`, false},
|
|
{`C:\Users\me\code`, false},
|
|
}
|
|
for _, c := range cases {
|
|
if got := isDriveRoot(c.p); got != c.want {
|
|
t.Errorf("isDriveRoot(%q) = %v, want %v", c.p, got, c.want)
|
|
}
|
|
}
|
|
}
|
|
|
|
func TestLocalPathLockerSerializes(t *testing.T) {
|
|
locker := NewLocalPathLocker()
|
|
const path = "/some/path"
|
|
|
|
rel1, err := locker.Acquire(context.Background(), path, "task-1", nil)
|
|
if err != nil {
|
|
t.Fatalf("acquire 1: %v", err)
|
|
}
|
|
if got := locker.Holder(path); got != "task-1" {
|
|
t.Errorf("holder = %q, want task-1", got)
|
|
}
|
|
|
|
// task-2 must wait, with onWait fired and the holder reported.
|
|
var waitCalls atomic.Int32
|
|
var sawHolder atomic.Value
|
|
done := make(chan error, 1)
|
|
go func() {
|
|
rel, err := locker.Acquire(context.Background(), path, "task-2", func(holder string) {
|
|
waitCalls.Add(1)
|
|
sawHolder.Store(holder)
|
|
})
|
|
if err != nil {
|
|
done <- err
|
|
return
|
|
}
|
|
if got := locker.Holder(path); got != "task-2" {
|
|
done <- errorsNew("holder after handover = " + got)
|
|
return
|
|
}
|
|
rel()
|
|
done <- nil
|
|
}()
|
|
|
|
// give the goroutine time to enter the wait
|
|
deadline := time.Now().Add(time.Second)
|
|
for time.Now().Before(deadline) && waitCalls.Load() == 0 {
|
|
time.Sleep(5 * time.Millisecond)
|
|
}
|
|
if waitCalls.Load() != 1 {
|
|
t.Fatalf("onWait calls = %d, want 1", waitCalls.Load())
|
|
}
|
|
if got := sawHolder.Load(); got != "task-1" {
|
|
t.Errorf("onWait holder = %v, want task-1", got)
|
|
}
|
|
|
|
rel1()
|
|
select {
|
|
case err := <-done:
|
|
if err != nil {
|
|
t.Fatalf("waiter result: %v", err)
|
|
}
|
|
case <-time.After(2 * time.Second):
|
|
t.Fatal("waiter never woke")
|
|
}
|
|
if got := locker.Holder(path); got != "" {
|
|
t.Errorf("holder after release = %q, want empty", got)
|
|
}
|
|
}
|
|
|
|
func TestLocalPathLockerCtxCancel(t *testing.T) {
|
|
locker := NewLocalPathLocker()
|
|
const path = "/some/path"
|
|
|
|
rel1, err := locker.Acquire(context.Background(), path, "task-1", nil)
|
|
if err != nil {
|
|
t.Fatalf("acquire 1: %v", err)
|
|
}
|
|
defer rel1()
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
|
|
defer cancel()
|
|
|
|
_, err = locker.Acquire(ctx, path, "task-2", nil)
|
|
if err == nil {
|
|
t.Fatalf("expected ctx error, got nil")
|
|
}
|
|
if !errors.Is(err, context.DeadlineExceeded) {
|
|
t.Fatalf("err = %v, want DeadlineExceeded", err)
|
|
}
|
|
}
|
|
|
|
func TestLocalPathLockerDistinctPathsParallel(t *testing.T) {
|
|
locker := NewLocalPathLocker()
|
|
|
|
rel1, err := locker.Acquire(context.Background(), "/a", "task-1", nil)
|
|
if err != nil {
|
|
t.Fatalf("acquire 1: %v", err)
|
|
}
|
|
defer rel1()
|
|
|
|
// Different path must not block.
|
|
done := make(chan struct{})
|
|
var wg sync.WaitGroup
|
|
wg.Add(1)
|
|
go func() {
|
|
defer wg.Done()
|
|
rel2, err := locker.Acquire(context.Background(), "/b", "task-2", nil)
|
|
if err != nil {
|
|
t.Errorf("acquire 2: %v", err)
|
|
return
|
|
}
|
|
rel2()
|
|
close(done)
|
|
}()
|
|
select {
|
|
case <-done:
|
|
case <-time.After(time.Second):
|
|
t.Fatal("acquire on distinct path blocked")
|
|
}
|
|
wg.Wait()
|
|
}
|
|
|
|
// errorsNew is a tiny helper so the goroutine above can return a typed error
|
|
// without importing errors / fmt at the call site.
|
|
func errorsNew(msg string) error { return &waiterError{msg: msg} }
|
|
|
|
type waiterError struct{ msg string }
|
|
|
|
func (e *waiterError) Error() string { return e.msg }
|
|
|
|
// TestAcquireLocalDirectoryLock_CancelDuringWait covers the gap between
|
|
// dispatch and StartTask: while the path mutex is contended, the main
|
|
// per-task cancellation watcher hasn't started yet. If the issue is
|
|
// cancelled (or the task row is reassigned / deleted) during the wait,
|
|
// the daemon must notice promptly and bail — otherwise the slot stays
|
|
// pinned by a phantom waiter for the full lifetime of the holder.
|
|
func TestAcquireLocalDirectoryLock_CancelDuringWait(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
dir := t.TempDir() // valid, writable, non-blacklisted
|
|
// Pre-claim must use the same key the production path computes, which
|
|
// is the symlink-resolved realpath. On macOS, /tmp/... resolves to
|
|
// /private/tmp/..., so a literal preclaim with `dir` would miss the
|
|
// production key and the new acquire would win on the fast path.
|
|
realDir, err := filepath.EvalSymlinks(dir)
|
|
if err != nil {
|
|
t.Fatalf("evalsymlinks: %v", err)
|
|
}
|
|
|
|
// Server-side state for the fake. Mark the task cancelled only after
|
|
// we've seen the daemon call wait-local-directory, so the test can
|
|
// assert the watcher reacted to the post-park cancel rather than
|
|
// reading stale state on the very first poll.
|
|
var (
|
|
parked atomic.Bool
|
|
waitCall atomic.Int32
|
|
)
|
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
switch {
|
|
case strings.HasSuffix(r.URL.Path, "/wait-local-directory"):
|
|
waitCall.Add(1)
|
|
parked.Store(true)
|
|
w.WriteHeader(http.StatusOK)
|
|
case strings.HasSuffix(r.URL.Path, "/status"):
|
|
w.Header().Set("Content-Type", "application/json")
|
|
if parked.Load() {
|
|
_, _ = w.Write([]byte(`{"status":"cancelled"}`))
|
|
} else {
|
|
_, _ = w.Write([]byte(`{"status":"running"}`))
|
|
}
|
|
default:
|
|
// We don't expect /fail in the cancel path — assert that
|
|
// by failing loud if it gets called.
|
|
t.Errorf("unexpected daemon call: %s %s", r.Method, r.URL.Path)
|
|
w.WriteHeader(http.StatusOK)
|
|
}
|
|
}))
|
|
t.Cleanup(srv.Close)
|
|
|
|
const daemonID = "d-test"
|
|
const heldByTaskID = "task-holder"
|
|
const newTaskID = "task-waiter"
|
|
|
|
locker := NewLocalPathLocker()
|
|
// Pre-claim the lock so the new task has to wait. Use the resolved
|
|
// realpath as the key to match findLocalDirectoryAssignment.
|
|
release, err := locker.Acquire(context.Background(), realDir, heldByTaskID, nil)
|
|
if err != nil {
|
|
t.Fatalf("preclaim acquire: %v", err)
|
|
}
|
|
t.Cleanup(release)
|
|
|
|
d := &Daemon{
|
|
client: NewClient(srv.URL),
|
|
logger: slog.Default(),
|
|
localPathLocks: locker,
|
|
cancelPollInterval: 10 * time.Millisecond,
|
|
cfg: Config{DaemonID: daemonID},
|
|
}
|
|
|
|
ref, err := json.Marshal(localDirectoryRef{LocalPath: dir, DaemonID: daemonID})
|
|
if err != nil {
|
|
t.Fatalf("marshal ref: %v", err)
|
|
}
|
|
task := Task{
|
|
ID: newTaskID,
|
|
ProjectResources: []ProjectResourceData{
|
|
{ID: "r1", ResourceType: localDirectoryResourceType, ResourceRef: ref},
|
|
},
|
|
}
|
|
|
|
type result struct {
|
|
release func()
|
|
abort bool
|
|
}
|
|
done := make(chan result, 1)
|
|
go func() {
|
|
rel, abort := d.acquireLocalDirectoryLockIfNeeded(context.Background(), task, slog.Default())
|
|
done <- result{release: rel, abort: abort}
|
|
}()
|
|
|
|
select {
|
|
case got := <-done:
|
|
if !got.abort {
|
|
t.Fatal("expected abort=true after server-side cancel, got abort=false")
|
|
}
|
|
if got.release != nil {
|
|
t.Fatal("expected nil release on cancel, got a non-nil callback")
|
|
}
|
|
case <-time.After(2 * time.Second):
|
|
t.Fatal("acquireLocalDirectoryLockIfNeeded blocked past 2s — cancel was not observed during wait")
|
|
}
|
|
|
|
if got := waitCall.Load(); got != 1 {
|
|
t.Errorf("wait-local-directory calls = %d, want 1", got)
|
|
}
|
|
}
|
|
|
|
|
|
// TestLocalDirectoryRefModeDefaults pins the daemon-side helpers so a
|
|
// missing mode / max_parallel silently reads as in_place / the default,
|
|
// keeping backwards compatibility with rows written before MUL-3483.
|
|
func TestLocalDirectoryRefModeDefaults(t *testing.T) {
|
|
cases := []struct {
|
|
name string
|
|
ref localDirectoryRef
|
|
mode string
|
|
max int
|
|
}{
|
|
{"empty mode → in_place", localDirectoryRef{}, "in_place", defaultWorktreePoolMaxParallel},
|
|
{"explicit in_place", localDirectoryRef{Mode: "in_place"}, "in_place", defaultWorktreePoolMaxParallel},
|
|
{"worktree_pool", localDirectoryRef{Mode: "worktree_pool"}, "worktree_pool", defaultWorktreePoolMaxParallel},
|
|
{"custom max", localDirectoryRef{Mode: "worktree_pool", MaxParallel: 8}, "worktree_pool", 8},
|
|
{"negative max clamped", localDirectoryRef{Mode: "worktree_pool", MaxParallel: -1}, "worktree_pool", defaultWorktreePoolMaxParallel},
|
|
}
|
|
for _, tc := range cases {
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
if got := tc.ref.modeOrDefault(); got != tc.mode {
|
|
t.Errorf("modeOrDefault = %q, want %q", got, tc.mode)
|
|
}
|
|
if got := tc.ref.maxParallelOrDefault(); got != tc.max {
|
|
t.Errorf("maxParallelOrDefault = %d, want %d", got, tc.max)
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
// TestDefaultWorktreePoolRoot pins the fallback path we compose when the
|
|
// ref left pool_root blank. Placing the pool alongside the base repo
|
|
// (rather than inside it) prevents pool worktrees from ever showing up
|
|
// in the parent's `git status`.
|
|
func TestDefaultWorktreePoolRoot(t *testing.T) {
|
|
got := defaultWorktreePoolRoot("/home/u/code/proj")
|
|
want := filepath.Join("/home/u/code", ".multica-worktrees", "proj")
|
|
if got != want {
|
|
t.Errorf("defaultWorktreePoolRoot = %q, want %q", got, want)
|
|
}
|
|
}
|
|
|
|
|
|
// TestAcquireLocalDirectory_WorktreePoolPublishesLease is the plumbing
|
|
// regression guard flagged in the PR #4986 review: acquire the pool
|
|
// mode against a real git repo, verify the lease published to
|
|
// `d.localLeases` matches the freshly allocated worktree (so `runTask`
|
|
// feeds the pool path — NOT the base path — into
|
|
// `execenv.PrepareParams.LocalWorkDir`), and confirm release clears the
|
|
// lease along with the on-disk worktree.
|
|
//
|
|
// The subtle-but-important contract this test pins:
|
|
//
|
|
// 1. runTask reads the lease via `d.localLeases.Load(task.ID)`;
|
|
// 2. It uses `lease.WorkDir` as `prepParams.LocalWorkDir`;
|
|
// 3. A future refactor that drops the Store call, mistypes the key,
|
|
// or swaps in the base assignment.AbsPath would leave the agent
|
|
// running in the shared tree even though a worktree was created —
|
|
// silently defeating the whole point of worktree_pool mode. This
|
|
// test catches all three regressions in one shot.
|
|
func TestAcquireLocalDirectory_WorktreePoolPublishesLease(t *testing.T) {
|
|
// A real git repo is required — `git worktree add` refuses
|
|
// against a non-repo path, and validating the lease shape means
|
|
// we have to actually allocate. initGitRepo skips the test
|
|
// gracefully when git is missing from PATH.
|
|
base := t.TempDir()
|
|
initGitRepo(t, base)
|
|
|
|
const daemonID = "d-worktree-pool"
|
|
poolRoot := t.TempDir()
|
|
ref := localDirectoryRef{
|
|
LocalPath: base,
|
|
DaemonID: daemonID,
|
|
Mode: localDirectoryModeWorktreePool,
|
|
PoolRoot: poolRoot,
|
|
MaxParallel: 4,
|
|
}
|
|
raw, err := json.Marshal(ref)
|
|
if err != nil {
|
|
t.Fatalf("marshal ref: %v", err)
|
|
}
|
|
resources := []ProjectResourceData{{
|
|
ID: "r-wp",
|
|
ResourceType: localDirectoryResourceType,
|
|
ResourceRef: raw,
|
|
}}
|
|
const taskID = "task-lease-plumbing"
|
|
task := Task{ID: taskID, ProjectResources: resources}
|
|
|
|
d := &Daemon{
|
|
cfg: Config{DaemonID: daemonID},
|
|
client: newLeaseTestClient(),
|
|
localPathLocks: NewLocalPathLocker(),
|
|
worktreePool: NewWorktreePoolManager(discardLogger()),
|
|
logger: slog.Default(),
|
|
cancelPollInterval: 50 * time.Millisecond,
|
|
}
|
|
|
|
// Pre-condition: no lease yet.
|
|
if _, ok := d.localLeases.Load(taskID); ok {
|
|
t.Fatal("lease unexpectedly present before Acquire")
|
|
}
|
|
|
|
release, abort := d.acquireLocalDirectoryLockIfNeeded(context.Background(), task, slog.Default())
|
|
if abort {
|
|
t.Fatal("Acquire aborted unexpectedly")
|
|
}
|
|
if release == nil {
|
|
t.Fatal("Acquire returned nil release")
|
|
}
|
|
|
|
// Contract 1: lease is present under the task ID.
|
|
raw2, ok := d.localLeases.Load(taskID)
|
|
if !ok {
|
|
t.Fatal("lease missing after Acquire — runTask would fall back to base path")
|
|
}
|
|
lease, ok := raw2.(localDirectoryLease)
|
|
if !ok {
|
|
t.Fatalf("lease has wrong type: %T", raw2)
|
|
}
|
|
|
|
// Contract 2: mode is worktree_pool so downstream branching
|
|
// (result reporting / GC meta) can tell them apart.
|
|
if lease.Mode != localDirectoryModeWorktreePool {
|
|
t.Errorf("lease.Mode = %q, want %q", lease.Mode, localDirectoryModeWorktreePool)
|
|
}
|
|
|
|
// Contract 3: WorkDir points at a fresh directory UNDER pool_root
|
|
// (not at the base repo). This is the exact value runTask feeds
|
|
// into execenv.PrepareParams.LocalWorkDir — see daemon.go's
|
|
// `if localLease != nil && localLease.WorkDir != ""` branch.
|
|
if lease.WorkDir == "" {
|
|
t.Fatal("lease.WorkDir empty — runTask would fall back to base path")
|
|
}
|
|
if lease.WorkDir == base {
|
|
t.Fatalf("lease.WorkDir == base %q (pool mode collapsed to in_place)", base)
|
|
}
|
|
if rel, err := filepath.Rel(poolRoot, lease.WorkDir); err != nil || rel == "" || rel == "." || strings.HasPrefix(rel, "..") {
|
|
t.Fatalf("lease.WorkDir %q is not under pool_root %q (rel=%q, err=%v)", lease.WorkDir, poolRoot, rel, err)
|
|
}
|
|
if _, err := os.Stat(lease.WorkDir); err != nil {
|
|
t.Fatalf("lease.WorkDir %q does not exist on disk: %v", lease.WorkDir, err)
|
|
}
|
|
|
|
// Contract 4: branch matches multica/<task-uuid> — pinned here so
|
|
// a rename in the pool manager can't drift silently from what the
|
|
// server-side CompleteTask flow may want to inspect later.
|
|
if want := "multica/" + taskID; lease.Branch != want {
|
|
t.Errorf("lease.Branch = %q, want %q", lease.Branch, want)
|
|
}
|
|
|
|
release()
|
|
|
|
// Post-condition: lease cleared, worktree removed (clean tree,
|
|
// no dirty files means clean-remove path fires).
|
|
if _, ok := d.localLeases.Load(taskID); ok {
|
|
t.Fatal("lease unexpectedly still present after release")
|
|
}
|
|
if _, err := os.Stat(lease.WorkDir); !errors.Is(err, os.ErrNotExist) {
|
|
t.Fatalf("worktree %q not removed after release (stat err=%v)", lease.WorkDir, err)
|
|
}
|
|
}
|
|
|
|
// TestAcquireLocalDirectory_InPlaceAlsoPublishesLease covers the paired
|
|
// contract for the default mode: even though in_place doesn't change
|
|
// what runTask fed to LocalWorkDir historically, publishing a lease
|
|
// keyed by task ID unifies the runTask lookup path and prevents a
|
|
// future refactor from special-casing one mode over the other. Pinning
|
|
// it here means anyone editing acquireLocalDirectoryInPlace who forgets
|
|
// the Store call will fail this test rather than silently break the
|
|
// runTask branch that reads the map.
|
|
func TestAcquireLocalDirectory_InPlaceAlsoPublishesLease(t *testing.T) {
|
|
const daemonID = "d-in-place"
|
|
tmp := t.TempDir()
|
|
ref := localDirectoryRef{LocalPath: tmp, DaemonID: daemonID}
|
|
raw, err := json.Marshal(ref)
|
|
if err != nil {
|
|
t.Fatalf("marshal ref: %v", err)
|
|
}
|
|
resources := []ProjectResourceData{{
|
|
ID: "r-inplace",
|
|
ResourceType: localDirectoryResourceType,
|
|
ResourceRef: raw,
|
|
}}
|
|
const taskID = "task-in-place-plumbing"
|
|
task := Task{ID: taskID, ProjectResources: resources}
|
|
|
|
d := &Daemon{
|
|
cfg: Config{DaemonID: daemonID},
|
|
client: newLeaseTestClient(),
|
|
localPathLocks: NewLocalPathLocker(),
|
|
worktreePool: NewWorktreePoolManager(discardLogger()),
|
|
logger: slog.Default(),
|
|
cancelPollInterval: 50 * time.Millisecond,
|
|
}
|
|
release, abort := d.acquireLocalDirectoryLockIfNeeded(context.Background(), task, slog.Default())
|
|
if abort {
|
|
t.Fatal("Acquire aborted unexpectedly")
|
|
}
|
|
if release == nil {
|
|
t.Fatal("Acquire returned nil release")
|
|
}
|
|
defer release()
|
|
|
|
raw2, ok := d.localLeases.Load(taskID)
|
|
if !ok {
|
|
t.Fatal("lease missing after in_place Acquire")
|
|
}
|
|
lease := raw2.(localDirectoryLease)
|
|
if lease.Mode != localDirectoryModeInPlace {
|
|
t.Errorf("lease.Mode = %q, want %q", lease.Mode, localDirectoryModeInPlace)
|
|
}
|
|
if lease.WorkDir != filepath.Clean(tmp) {
|
|
t.Errorf("lease.WorkDir = %q, want %q (base path)", lease.WorkDir, filepath.Clean(tmp))
|
|
}
|
|
if lease.Branch != "" {
|
|
t.Errorf("lease.Branch = %q, want empty for in_place mode", lease.Branch)
|
|
}
|
|
}
|
|
|
|
// newLeaseTestClient stubs the daemon Client just enough for
|
|
// acquireLocalDirectory paths to run without a live server. Only the
|
|
// endpoints acquire may hit on the fast paths under test are wired up;
|
|
// anything else on the client remains zero-valued and will panic if the
|
|
// test ever expands into a code path that touches it (that's a good
|
|
// signal to add an explicit stub).
|
|
func newLeaseTestClient() *Client {
|
|
// The Acquire fast path in either mode does NOT need to talk to
|
|
// the server (no wait, no cancel poll, no MarkTaskWaitingLocalDirectory).
|
|
// Point the client at a placeholder URL so any accidental HTTP dial
|
|
// fails fast rather than blocking test cleanup.
|
|
return NewClient("http://127.0.0.1:0")
|
|
}
|