Files
multica/server/internal/daemon/runtime_profile_test.go
Multica Eve 6bb8cac9ea MUL-3332: daemon picks up new custom runtime profiles without restart (#4225)
* MUL-3332: daemon picks up new custom runtime profiles without restart

The workspaceSyncLoop's already-tracked branch refreshed only settings and
repos via refreshWorkspaceRepos and never re-fetched runtime profiles, so
a custom runtime profile created via the web UI / CLI did not become a
registered runtime row until the daemon restarted (or a runtimeGone
recovery happened to fire).

Detect server-side profile drift each sync tick by hashing the workspace's
profile list with profileSetSignature(), caching the digest on
workspaceState.profileSetSig, and triggering reregisterWorkspaceAfterRuntimeGone
when the live signature differs from the cached one. Steady-state syncs cost
exactly one extra GetRuntimeProfiles round trip; only real drift fans out to
a Register call.

The fetch is best-effort: a 404 / network blip preserves the cached signature
so a transient failure cannot loop the daemon into spurious re-registrations.

Tests in runtime_profile_drift_test.go cover digest stability under reorder,
field-by-field drift detection (add / enable-flip / command_name /
protocol_family / fixed_args / visibility), the no-drift hot path (no
re-register), the new-profile drift path (single re-register + index update +
sig converges), and best-effort fetch error handling.

Co-authored-by: multica-agent <github@multica.ai>

* MUL-3332: split orphan recovery from profile drift; converge to zero

Addresses two blocking review concerns on #4225 (raised by GPT-Boy):

1. Profile drift must not kill running tasks on existing runtimes.

   The first cut reused reregisterWorkspaceAfterRuntimeGone, which after
   re-register calls /recover-orphans for every returned runtime ID. The
   server's RecoverOrphanedTasksForRuntime hard-fails every
   dispatched/running/waiting_local_directory row on that runtime — the
   correct response when a runtime row was actually deleted server-side,
   but a catastrophic false positive on profile drift: a built-in runtime
   still actively executing the user's tasks would have its work killed
   just because the user added an unrelated sibling custom profile.

   Fix: extract applyRegisterResponseInPlace as the shared in-place state
   converger between the two paths, and stop calling /recover-orphans from
   the drift path. reregisterWorkspaceAfterRuntimeGone keeps the
   /recover-orphans call because in that path the rows really were gone.

2. Disabling the only profile on a custom-only daemon must converge.

   The first cut hit registerRuntimesForWorkspace's len(runtimes)==0 guard
   and bailed out, so the disabled profile's runtime stayed alive in
   local tracking and on the server (still polling, still heartbeating,
   still online for the full 150 s stale-heartbeat window).

   Fix: introduce ErrNoRuntimesToRegister as a sentinel, have
   registerRuntimesForWorkspace return profileSig even on the empty case
   (so the drift path can cache the converged-empty signature), and have
   the drift refresh's error handler take a convergeWorkspaceRuntimesToZero
   branch that clears local runtimeIDs / runtimeIndex entries and
   Deregisters the orphaned IDs so the server marks them offline
   immediately. The same Deregister step also runs on partial drift (a
   built-in survives, the disabled profile's runtime drops) so the user
   sees the dropped runtime go offline within the next sync tick instead
   of after the 150 s sweep.

Tests:

- TestRefreshWorkspaceRuntimeProfiles_DriftWithRunningRuntimeSkipsOrphanRecovery
  (mixed built-in + custom, add another profile, asserts zero
  /recover-orphans calls).
- TestRefreshWorkspaceRuntimeProfiles_DisableConvergesCustomOnlyDaemon
  (custom-only daemon, disable only profile, asserts local state
  cleared, signature converges to empty digest, Deregister called with
  the orphaned ID, no recover-orphans, follow-up tick is no-op).
- TestRefreshWorkspaceRuntimeProfiles_DisableOneOfManyDeregistersDroppedID
  (partial drift: only the dropped ID is Deregistered, surviving
  built-in is left alone and not orphan-recovered).
- TestRefreshWorkspaceRuntimeProfiles_NewProfileTriggersReregister
  extended to also assert no /recover-orphans calls.
- TestRegisterRuntimes_SkipsProfileNotOnPath strengthened to assert the
  ErrNoRuntimesToRegister sentinel and that profileSig is still returned
  on the empty path.

Co-authored-by: multica-agent <github@multica.ai>

---------

Co-authored-by: Eve <eve@multica-ai.local>
Co-authored-by: multica-agent <github@multica.ai>
2026-06-17 12:36:30 +08:00

360 lines
13 KiB
Go

package daemon
import (
"context"
"encoding/json"
"errors"
"net/http"
"net/http/httptest"
"strconv"
"strings"
"testing"
)
// stubLookPath swaps the package-level lookPath indirection used by
// registerRuntimesForWorkspace to resolve custom runtime-profile commands,
// so tests don't have to mutate the process PATH. resolved maps a command
// name to the absolute path it should resolve to; an absent name reports
// "not found".
func stubLookPath(t *testing.T, resolved map[string]string) {
t.Helper()
orig := lookPath
lookPath = func(cmd string) (string, error) {
if p, ok := resolved[cmd]; ok {
return p, nil
}
return "", &osExecNotFound{cmd: cmd}
}
t.Cleanup(func() { lookPath = orig })
}
type osExecNotFound struct{ cmd string }
func (e *osExecNotFound) Error() string { return "exec: " + e.cmd + ": not found in $PATH" }
// TestClient_GetRuntimeProfiles_RequestShape asserts the daemon GETs the
// documented path and parses the server's runtime_profiles payload.
func TestClient_GetRuntimeProfiles_RequestShape(t *testing.T) {
var gotMethod, gotPath string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotMethod = r.Method
gotPath = r.URL.Path
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{
"workspace_id":"ws-1",
"runtime_profiles":[{
"id":"prof-1",
"workspace_id":"ws-1",
"display_name":"Company Codex",
"protocol_family":"codex",
"command_name":"company-codex",
"description":null,
"fixed_args":["--foo"],
"visibility":"workspace",
"created_by":null,
"enabled":true,
"created_at":"2026-01-01T00:00:00Z",
"updated_at":"2026-01-01T00:00:00Z"
}]
}`))
}))
defer srv.Close()
c := NewClient(srv.URL)
c.SetToken("tok")
resp, err := c.GetRuntimeProfiles(context.Background(), "ws-1")
if err != nil {
t.Fatalf("GetRuntimeProfiles: %v", err)
}
if gotMethod != http.MethodGet {
t.Errorf("method = %q, want GET", gotMethod)
}
if gotPath != "/api/daemon/workspaces/ws-1/runtime-profiles" {
t.Errorf("path = %q, want /api/daemon/workspaces/ws-1/runtime-profiles", gotPath)
}
if resp.WorkspaceID != "ws-1" || len(resp.RuntimeProfiles) != 1 {
t.Fatalf("unexpected response: %+v", resp)
}
p := resp.RuntimeProfiles[0]
if p.ID != "prof-1" || p.ProtocolFamily != "codex" || p.CommandName != "company-codex" {
t.Errorf("profile fields wrong: %+v", p)
}
if !p.Enabled {
t.Errorf("profile should be enabled")
}
if len(p.FixedArgs) != 1 || p.FixedArgs[0] != "--foo" {
t.Errorf("fixed_args = %v, want [--foo]", p.FixedArgs)
}
}
// profileRegisterFixture wires a Daemon against a fake server that serves a
// configurable set of runtime profiles and captures the runtimes array sent
// to /api/daemon/register.
type profileRegisterFixture struct {
daemon *Daemon
server *httptest.Server
sentRuntimes []map[string]any
}
func newProfileRegisterFixture(t *testing.T, profiles []RuntimeProfile, profilesStatus int) *profileRegisterFixture {
t.Helper()
fx := &profileRegisterFixture{}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case r.URL.Path == "/api/daemon/register":
var body struct {
Runtimes []map[string]any `json:"runtimes"`
}
_ = json.NewDecoder(r.Body).Decode(&body)
fx.sentRuntimes = body.Runtimes
// Echo back a Runtime row per requested runtime, threading
// profile_id so the caller can populate runtimeIndex from it.
var resp RegisterResponse
for i, rt := range body.Runtimes {
id := "rt-" + strconv.Itoa(i)
profileID, _ := rt["profile_id"].(string)
typ, _ := rt["type"].(string)
resp.Runtimes = append(resp.Runtimes, Runtime{
ID: id,
Name: "n",
Provider: typ,
Status: "online",
ProfileID: profileID,
})
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(resp)
case len(r.URL.Path) > len("/runtime-profiles") && strings.HasSuffix(r.URL.Path, "/runtime-profiles"):
if profilesStatus != 0 && profilesStatus != http.StatusOK {
w.WriteHeader(profilesStatus)
return
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(RuntimeProfilesResponse{
WorkspaceID: "ws-1",
RuntimeProfiles: profiles,
})
default:
w.WriteHeader(http.StatusOK)
}
}))
t.Cleanup(srv.Close)
d := freshDaemon(srv.URL)
d.profileCommandPaths = make(map[string]string)
fx.daemon = d
fx.server = srv
return fx
}
// TestRegisterRuntimes_AppendsProfileRuntime verifies that a custom profile
// whose command resolves on PATH is appended as a runtime entry carrying
// profile_id, and that its resolved command path is recorded for runTask.
// Uses a custom-only host (no built-in agents) to also prove that path still
// registers.
func TestRegisterRuntimes_AppendsProfileRuntime(t *testing.T) {
t.Cleanup(stubAgentVersion(t))
stubLookPath(t, map[string]string{"company-codex": "/opt/bin/company-codex"})
profiles := []RuntimeProfile{{
ID: "prof-1",
WorkspaceID: "ws-1",
DisplayName: "Company Codex",
ProtocolFamily: "codex",
CommandName: "company-codex",
Visibility: "workspace",
Enabled: true,
}}
fx := newProfileRegisterFixture(t, profiles, http.StatusOK)
d := fx.daemon
// Custom-only host: no built-in agents configured.
d.cfg.Agents = map[string]AgentEntry{}
resp, _, err := d.registerRuntimesForWorkspace(context.Background(), "ws-1")
if err != nil {
t.Fatalf("registerRuntimesForWorkspace: %v", err)
}
// The register request must carry exactly one runtime: the profile.
if len(fx.sentRuntimes) != 1 {
t.Fatalf("sent runtimes = %d, want 1: %+v", len(fx.sentRuntimes), fx.sentRuntimes)
}
sent := fx.sentRuntimes[0]
if sent["type"] != "codex" {
t.Errorf("sent type = %v, want codex", sent["type"])
}
if sent["profile_id"] != "prof-1" {
t.Errorf("sent profile_id = %v, want prof-1", sent["profile_id"])
}
if sent["status"] != "online" {
t.Errorf("sent status = %v, want online", sent["status"])
}
// The resolved command path must be recorded keyed by profile_id.
if got := d.profileCommandPaths["prof-1"]; got != "/opt/bin/company-codex" {
t.Errorf("profileCommandPaths[prof-1] = %q, want /opt/bin/company-codex", got)
}
// The response runtime carries the profile_id back.
if len(resp.Runtimes) != 1 || resp.Runtimes[0].ProfileID != "prof-1" {
t.Fatalf("response runtimes wrong: %+v", resp.Runtimes)
}
}
// TestRegisterRuntimes_SkipsProfileNotOnPath verifies a profile whose command
// is missing on this host is skipped, and that a host with no built-in agents
// and no resolvable profiles fails registration with the documented sentinel
// (the drift-refresh path keys off ErrNoRuntimesToRegister to take the
// convergence-to-zero branch instead of treating it as a hard error).
func TestRegisterRuntimes_SkipsProfileNotOnPath(t *testing.T) {
t.Cleanup(stubAgentVersion(t))
stubLookPath(t, map[string]string{}) // nothing resolves
profiles := []RuntimeProfile{{
ID: "prof-1",
WorkspaceID: "ws-1",
DisplayName: "Company Codex",
ProtocolFamily: "codex",
CommandName: "company-codex",
Enabled: true,
}}
fx := newProfileRegisterFixture(t, profiles, http.StatusOK)
d := fx.daemon
d.cfg.Agents = map[string]AgentEntry{}
_, sig, err := d.registerRuntimesForWorkspace(context.Background(), "ws-1")
if !errors.Is(err, ErrNoRuntimesToRegister) {
t.Fatalf("expected ErrNoRuntimesToRegister, got %v", err)
}
if sig == "" {
t.Errorf("profileSig must still be returned even when registration short-circuits, so the drift path can cache the converged-empty signature")
}
if _, ok := d.profileCommandPaths["prof-1"]; ok {
t.Errorf("profileCommandPaths should not record an unresolved profile")
}
}
// TestRegisterRuntimes_ProfilesFetchErrorIsBestEffort verifies a 404 from the
// profiles endpoint does not fail registration when a built-in agent exists.
func TestRegisterRuntimes_ProfilesFetchErrorIsBestEffort(t *testing.T) {
t.Cleanup(stubAgentVersion(t))
stubLookPath(t, map[string]string{})
fx := newProfileRegisterFixture(t, nil, http.StatusNotFound)
d := fx.daemon
// Built-in agent present so registration has something to register.
d.cfg.Agents = map[string]AgentEntry{"claude": {Path: "/usr/bin/true"}}
resp, _, err := d.registerRuntimesForWorkspace(context.Background(), "ws-1")
if err != nil {
t.Fatalf("registration should succeed despite profiles 404: %v", err)
}
if len(fx.sentRuntimes) != 1 || fx.sentRuntimes[0]["type"] != "claude" {
t.Fatalf("expected only the built-in claude runtime, got %+v", fx.sentRuntimes)
}
if len(resp.Runtimes) != 1 {
t.Fatalf("response runtimes = %d, want 1", len(resp.Runtimes))
}
}
// TestRegisterRuntimes_PrefersCommandPathOverride verifies that a per-machine
// command path override (MUL-3284) is used in preference to the PATH lookup:
// the resolved/recorded path is the override, even when lookPath would resolve
// command_name to a different binary.
func TestRegisterRuntimes_PrefersCommandPathOverride(t *testing.T) {
t.Cleanup(stubAgentVersion(t))
// PATH would resolve to a *different* binary; the override must win.
stubLookPath(t, map[string]string{"company-codex": "/usr/bin/company-codex"})
stubProfilePathExecutable(t, map[string]bool{"/opt/custom/company-codex": true})
profiles := []RuntimeProfile{{
ID: "prof-1",
WorkspaceID: "ws-1",
DisplayName: "Company Codex",
ProtocolFamily: "codex",
CommandName: "company-codex",
Enabled: true,
}}
fx := newProfileRegisterFixture(t, profiles, http.StatusOK)
d := fx.daemon
d.cfg.Agents = map[string]AgentEntry{}
d.cfg.ProfileCommandOverrides = map[string]string{"prof-1": "/opt/custom/company-codex"}
if _, _, err := d.registerRuntimesForWorkspace(context.Background(), "ws-1"); err != nil {
t.Fatalf("registerRuntimesForWorkspace: %v", err)
}
if got := d.profileCommandPaths["prof-1"]; got != "/opt/custom/company-codex" {
t.Errorf("profileCommandPaths[prof-1] = %q, want the override /opt/custom/company-codex", got)
}
if len(fx.sentRuntimes) != 1 || fx.sentRuntimes[0]["profile_id"] != "prof-1" {
t.Fatalf("expected the profile runtime to register, got %+v", fx.sentRuntimes)
}
}
// TestRegisterRuntimes_OverrideNotExecutableFallsBackToPath verifies that an
// override pointing at a non-executable / missing path is ignored and the
// daemon falls back to resolving command_name on PATH.
func TestRegisterRuntimes_OverrideNotExecutableFallsBackToPath(t *testing.T) {
t.Cleanup(stubAgentVersion(t))
stubLookPath(t, map[string]string{"company-codex": "/usr/bin/company-codex"})
// Override path reports NOT executable -> must fall back to PATH.
stubProfilePathExecutable(t, map[string]bool{})
profiles := []RuntimeProfile{{
ID: "prof-1",
WorkspaceID: "ws-1",
DisplayName: "Company Codex",
ProtocolFamily: "codex",
CommandName: "company-codex",
Enabled: true,
}}
fx := newProfileRegisterFixture(t, profiles, http.StatusOK)
d := fx.daemon
d.cfg.Agents = map[string]AgentEntry{}
d.cfg.ProfileCommandOverrides = map[string]string{"prof-1": "/opt/stale/company-codex"}
if _, _, err := d.registerRuntimesForWorkspace(context.Background(), "ws-1"); err != nil {
t.Fatalf("registerRuntimesForWorkspace: %v", err)
}
if got := d.profileCommandPaths["prof-1"]; got != "/usr/bin/company-codex" {
t.Errorf("profileCommandPaths[prof-1] = %q, want the PATH fallback /usr/bin/company-codex", got)
}
}
// stubProfilePathExecutable swaps the package-level profilePathExecutable
// indirection so override-preference tests can decide which paths are
// "executable" without staging real files. An absent path reports false.
func stubProfilePathExecutable(t *testing.T, executable map[string]bool) {
t.Helper()
orig := profilePathExecutable
profilePathExecutable = func(path string) bool { return executable[path] }
t.Cleanup(func() { profilePathExecutable = orig })
}
// bookkeeping that runTask relies on to override the launch path.
func TestCustomCommandPathForRuntime(t *testing.T) {
d := freshDaemon("")
d.profileCommandPaths = map[string]string{"prof-1": "/opt/bin/company-codex"}
// rt-custom is a custom-profile runtime; rt-builtin is a normal one.
d.runtimeIndex["rt-custom"] = Runtime{ID: "rt-custom", Provider: "codex", ProfileID: "prof-1"}
d.runtimeIndex["rt-builtin"] = Runtime{ID: "rt-builtin", Provider: "claude"}
if path, ok := d.customCommandPathForRuntime("rt-custom"); !ok || path != "/opt/bin/company-codex" {
t.Errorf("custom runtime: got (%q, %v), want (/opt/bin/company-codex, true)", path, ok)
}
if path, ok := d.customCommandPathForRuntime("rt-builtin"); ok || path != "" {
t.Errorf("built-in runtime: got (%q, %v), want (\"\", false)", path, ok)
}
if path, ok := d.customCommandPathForRuntime("rt-unknown"); ok || path != "" {
t.Errorf("unknown runtime: got (%q, %v), want (\"\", false)", path, ok)
}
// A custom runtime whose profile path was never resolved on this host
// (profile_id not in profileCommandPaths) must report not-custom so
// runTask falls back to its normal provider lookup rather than launching
// an empty path.
d.runtimeIndex["rt-unresolved"] = Runtime{ID: "rt-unresolved", Provider: "codex", ProfileID: "prof-missing"}
if path, ok := d.customCommandPathForRuntime("rt-unresolved"); ok || path != "" {
t.Errorf("unresolved profile: got (%q, %v), want (\"\", false)", path, ok)
}
}