Files
multica/server/internal/daemon/agents_probe.go
Multica Eve 07a78d910d fix(daemon): discover agent CLIs installed after startup (MUL-5439) (#6086)
* fix(daemon): discover agent CLIs installed after startup (MUL-5439)

The built-in agent availability set was built exactly once, in LoadConfig, and
every later consumer read that static map. A CLI installed while the daemon was
running was therefore invisible until the daemon process restarted.

On Desktop that is worse than it sounds: the app defaults to autoStop=false and
auto-start only compares CLI versions, so quitting and reopening the app does
not restart the daemon. A user who installed a CLI, verified it in their shell,
and relaunched the app was left with a runtime that never appeared — which is
GH #6077, reported against Antigravity but not specific to it.

- Extract discovery into probeAgentCLIs (pure availability, no version gate).
- Add agentDiscoveryLoop: re-probe every 2 minutes and register providers that
  appeared, reusing applyRegisterResponseInPlace so nothing restarts and no
  in-flight task is interrupted. RecoverOrphans is deliberately not called here
  (MUL-3332): surviving runtimes may be executing tasks.
- Additive only. A provider that stops resolving is kept, because a narrower
  PATH or a version manager mid-upgrade would otherwise tear down a working
  runtime. Removal stays with an explicit restart.
- Hold the set in an atomic.Pointer copy-on-write: cfg.Agents was read unlocked
  from task-execution paths, so a mutable map would be a data race.
- Cache the login-shell PATH fallback process-wide with a 30m TTL keyed on
  PATH/SHELL/HOME, so the 2-minute loop stays a pure LookPath sweep instead of
  forking the user's rc files every round.
- Report skipped_agents on /health with the reason a discovered provider was
  dropped at registration, so "not installed" and "installed but rejected" stop
  looking identical (they were only distinguishable in the daemon log).
- Fix the onboarding template in all four locales: it told users that
  restarting the desktop app was enough, which is exactly the false lead the
  reporter followed.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon): retry discovery until registered, and never evict live runtimes

Addresses both P1s from review.

P1-1: a failed first attempt was never retried. refreshAgentAvailability
published a newly discovered provider into the availability set and only acted
on providers "gained" in that same round, so a version probe that timed out or a
register call that failed left the provider permanently unregistered — the user
was back to restarting the daemon, with skipped_agents able to explain the
problem but not fix it.

Registration is now driven by live state instead of by the round that discovered
the provider. providersMissingRuntimes derives, from runtimeIndex, which
discovered providers lack a built-in runtime in each tracked workspace;
convergeRuntimeRegistrations registers only those, for only the workspaces that
need them. Nothing records "already handled", so a version-probe failure, a
register failure, or a partial failure across workspaces all retry on the next
tick, and a provider rejected for being below the minimum version recovers on
its own after an in-place upgrade. A permanently stuck provider is bounded by
exponential backoff (one discovery interval up to 30m) on the expensive half
only; discovery itself stays on its 2-minute cadence, and the steady state
issues no version probes and no register calls at all.

P1-2: the "additive only" refresh could evict existing custom runtimes.
applyRegisterResponseInPlace treats the response as authoritative and drops
prior runtime IDs it does not mention — correct for the convergence paths that
re-derive a whole runtime set, wrong here. appendProfileRuntimes is best-effort,
so one failed GetRuntimeProfiles call yields a builtins-only response, which
would evict the workspace's custom profile runtimes from runtimeIndex and stop
their heartbeats, possibly mid-task.

Added mergeRegisterResponseInPlace: indexes and appends returned runtimes, never
deletes an unmentioned one, and keeps profileSetSig when the fetch failed. Safe
because the server's register endpoint is a pure per-entry upsert that prunes
nothing, so omitted runtimes still exist server-side. ID rotation is still
handled destructively for that one runtime, so a re-issued ID replaces its
predecessor instead of leaving two heartbeat goroutines.

New regression tests: first version probe fails then recovers; first register
call fails then recovers; partial failure across two workspaces retries only the
one that needs it and makes exactly one call; a failed profile fetch leaves the
custom runtime indexed, watched, and its signature intact; below-minimum
provider registers after an upgrade; steady state re-probes nothing; rotated
runtime ID is swapped not duplicated; stuck provider is retried but bounded.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon): keep CLI discovery out of custom-profile convergence (MUL-5439)

Third-round review P1: discovery could mask a concurrent profile disable.

The discovery path registered through registerRuntimesForWorkspaceBatch, which
fetches the workspace's custom runtime profiles and returns their content
signature, and the additive merge cached that signature. So if a user disabled a
custom profile at the same moment a newly installed CLI was discovered, the merge
correctly kept the disabled profile's runtime ID (it must not delete unmentioned
runtimes) while recording the POST-disable signature. refreshWorkspaceRuntimeProfiles
short-circuits on a matching signature, so the drift path then saw "already
converged" and the disabled runtime stayed tracked and heartbeating forever.

Discovery is now strictly built-ins only:

- registerBuiltinRuntimesForWorkspace posts a builtins-only register request. It
  never calls appendProfileRuntimes, so discovery cannot observe the profile set
  at all and there is no signature to cache.
- mergeRegisterResponseInPlace becomes mergeBuiltinRegisterResponse: it ignores
  any entry carrying a ProfileID (invariant guard — only the drift path may
  introduce a custom runtime), and neither reads nor writes profileSetSig.
- Custom profile add/edit/disable remains owned exclusively by
  refreshWorkspaceRuntimeProfiles.

Regression tests: a profile disabled concurrently with a new CLI install is
still converged away by the drift path (this test reproduces the reviewer's
exact failure — "disabled custom runtime rt-2 remained tracked" — when the old
signature write is replayed); the merge ignores profile-bearing entries; the
merge leaves profileSetSig untouched. The profile-fetch-failure test now also
asserts the signature is unchanged rather than merely non-empty.

Co-authored-by: multica-agent <github@multica.ai>

---------

Co-authored-by: Eve <eve@multica-ai.local>
Co-authored-by: multica-agent <github@multica.ai>
2026-07-29 15:08:56 +08:00

221 lines
9.0 KiB
Go

package daemon
import (
"os"
"strings"
"sync"
"time"
)
// shellResolveTTL bounds how long one login-shell PATH resolution is reused
// across probeAgentCLIs calls.
//
// This is deliberately much longer than agentDiscoveryInterval so the frequent
// discovery round stays a pure exec.LookPath sweep: resolveAgentsViaLoginShell
// forks the user's login shell and runs their rc files, and there is almost
// always at least one uninstalled provider to miss LookPath on, so a short TTL
// would turn discovery into a shell fork every few minutes for the life of the
// daemon.
//
// The practical effect: a CLI on the daemon's own PATH is discovered within
// agentDiscoveryInterval, while one reachable only through the login shell
// (nvm/fnm shims, a ~/.local/bin that only ~/.zshrc adds) takes up to this long
// — still without a restart, which is the part that was previously impossible.
var shellResolveTTL = 30 * time.Minute
var (
shellResolveMu sync.Mutex
shellResolveCache map[string]string
shellResolveKey string
shellResolvedAt time.Time
)
// shellResolveEnvKey fingerprints the environment that determines what a login
// shell resolves. A change to any of these invalidates the cache immediately,
// independent of the TTL — the cached answer was for a different environment.
func shellResolveEnvKey() string {
return strings.Join([]string{
os.Getenv("PATH"),
os.Getenv("SHELL"),
os.Getenv("HOME"),
}, "\x00")
}
// cachedShellResolvedAgents resolves every standard agent command name through
// the user's login shell, reusing the previous result for shellResolveTTL as
// long as the resolution-relevant environment is unchanged.
//
// resolveAgentsViaLoginShell forks the user's login shell, which runs their rc
// files, so this must stay a cache and not a per-probe call: probeAgentCLIs now
// runs periodically on a live daemon, and there is almost always at least one
// uninstalled provider to miss LookPath on.
func cachedShellResolvedAgents() map[string]string {
shellResolveMu.Lock()
defer shellResolveMu.Unlock()
key := shellResolveEnvKey()
if shellResolveCache != nil && shellResolveKey == key && time.Since(shellResolvedAt) < shellResolveTTL {
return shellResolveCache
}
resolved := resolveAgentsViaLoginShell(defaultAgentCommandNames)
if resolved == nil {
// Distinguish "resolved nothing" from "never resolved" so a failing
// shell doesn't get re-forked on every probe inside the TTL window.
resolved = map[string]string{}
}
shellResolveCache = resolved
shellResolveKey = key
shellResolvedAt = time.Now()
return shellResolveCache
}
// probeAgentCLIs discovers which built-in agent CLIs are installed on this
// machine and returns one AgentEntry per provider that resolved.
//
// This is pure discovery: no version detection and no minimum-version gate
// (detectBuiltinRuntimes owns those, per registration round). The result is
// therefore the machine's *availability* set, which is exactly what
// /health.agents reports and what `multica daemon probe-runtimes` prints.
//
// It is called once from LoadConfig at startup and again from the periodic
// workspace sync (refreshAgentAvailability), so a CLI the user installs while
// the daemon is already running gets picked up without a restart (MUL-5439).
// Everything it reads is process-external (PATH, MULTICA_*_PATH, MULTICA_*_MODEL),
// so re-running it is the only way to observe such an install.
//
// A var so tests can stub discovery without installing real CLIs.
var probeAgentCLIs = func() map[string]AgentEntry {
// Probe available agent CLIs. exec.LookPath is the primary path, but on
// macOS/Linux a GUI-launched daemon (Electron, Launchpad) does not
// inherit the user's interactive shell PATH — fnm/nvm/volta multishells,
// the Anthropic native installer prefix, and per-user npm prefixes all
// live in dirs that only get added to PATH by ~/.zshrc or ~/.bashrc.
// shellResolvedAgents asks the user's login shell, lazily on first miss,
// to resolve every standard agent name to its canonical absolute path,
// so we can find binaries the bare daemon process can't see. See
// resolveAgentsViaLoginShell for the details and constraints.
//
// Laziness matters: the happy path (every agent on the daemon's PATH or
// pinned to an explicit MULTICA_*_PATH) must not pay the cost of
// spawning the user's login shell — that touches their rc files and
// adds startup latency that scales with whatever they put in there. We
// only fork a shell when a bare command name actually missed LookPath.
//
// The resolution is cached process-wide with a TTL (not per call) because
// this function now also runs periodically on a live daemon: a per-call
// sync.Once would fork a login shell on every discovery round, since there
// is almost always at least one uninstalled provider to miss on. The TTL
// still lets a CLI installed into a login-shell-only PATH dir (nvm, fnm,
// ~/.local/bin via ~/.zshrc) be discovered without a restart (MUL-5439).
getShellResolved := cachedShellResolvedAgents
probe := func(envVar, defaultCmd, modelEnv string) (AgentEntry, bool) {
cmd := envOrDefault(envVar, defaultCmd)
if path, err := resolveAgentExecutablePath(cmd); err == nil {
return AgentEntry{
Path: path,
Command: cmd,
Model: strings.TrimSpace(os.Getenv(modelEnv)),
}, true
}
// The shell fallback only rescues bare command names. An operator
// who pinned MULTICA_*_PATH to an absolute or relative path that
// doesn't exist should hard-miss, not silently get a different
// binary.
if strings.ContainsAny(cmd, "/\\") {
return AgentEntry{}, false
}
if path, ok := getShellResolved()[cmd]; ok {
return AgentEntry{
Path: path,
Command: cmd,
Model: strings.TrimSpace(os.Getenv(modelEnv)),
}, true
}
if defaultCmd == "codex" && cmd == defaultCmd {
// Codex Desktop bundles its CLI inside the macOS app instead of
// installing it onto PATH.
for _, p := range codexDesktopAppBundlePaths() {
if _, err := os.Stat(p); err == nil {
return AgentEntry{
Path: p,
Command: cmd,
Model: strings.TrimSpace(os.Getenv(modelEnv)),
}, true
}
}
}
return AgentEntry{}, false
}
agents := map[string]AgentEntry{}
if e, ok := probe("MULTICA_CLAUDE_PATH", "claude", "MULTICA_CLAUDE_MODEL"); ok {
agents["claude"] = e
}
if e, ok := probe("MULTICA_CODEX_PATH", "codex", "MULTICA_CODEX_MODEL"); ok {
agents["codex"] = e
}
if e, ok := probe("MULTICA_OPENCODE_PATH", "opencode", "MULTICA_OPENCODE_MODEL"); ok {
agents["opencode"] = e
}
if e, ok := probe("MULTICA_DEVECO_PATH", "deveco", "MULTICA_DEVECO_MODEL"); ok {
agents["deveco"] = e
}
if e, ok := probe("MULTICA_OPENCLAW_PATH", "openclaw", "MULTICA_OPENCLAW_MODEL"); ok {
agents["openclaw"] = e
}
if e, ok := probe("MULTICA_HERMES_PATH", "hermes", "MULTICA_HERMES_MODEL"); ok {
agents["hermes"] = e
}
if e, ok := probe("MULTICA_PI_PATH", "pi", "MULTICA_PI_MODEL"); ok {
agents["pi"] = e
}
if e, ok := probe("MULTICA_CURSOR_PATH", "cursor-agent", "MULTICA_CURSOR_MODEL"); ok {
agents["cursor"] = e
}
if e, ok := probe("MULTICA_COPILOT_PATH", "copilot", "MULTICA_COPILOT_MODEL"); ok {
agents["copilot"] = e
}
if e, ok := probe("MULTICA_KIMI_PATH", "kimi", "MULTICA_KIMI_MODEL"); ok {
agents["kimi"] = e
}
if e, ok := probe("MULTICA_KIRO_PATH", "kiro-cli", "MULTICA_KIRO_MODEL"); ok {
agents["kiro"] = e
}
if e, ok := probe("MULTICA_CODEBUDDY_PATH", "codebuddy", "MULTICA_CODEBUDDY_MODEL"); ok {
agents["codebuddy"] = e
}
// agy 1.0.6 added a `--model` flag (MUL-3125), so Antigravity now takes a
// model env like every other backend. MULTICA_ANTIGRAVITY_MODEL seeds the
// daemon-wide default; its value is the exact `agy models` display string
// (e.g. "Claude Opus 4.6 (Thinking)"), not a provider/model slug.
if e, ok := probe("MULTICA_ANTIGRAVITY_PATH", "agy", "MULTICA_ANTIGRAVITY_MODEL"); ok {
agents["antigravity"] = e
}
qoderPath := envOrDefault("MULTICA_QODER_PATH", "qodercli")
if path, err := resolveAgentExecutablePath(qoderPath); err == nil {
agents["qoder"] = AgentEntry{
Path: path,
Command: qoderPath,
Model: strings.TrimSpace(os.Getenv("MULTICA_QODER_MODEL")),
}
}
// ByteDance official TRAE CLI (the `traecli` binary from https://docs.trae.cn/cli),
// driven over ACP via `traecli acp serve --yolo`. MULTICA_TRAECLI_MODEL seeds
// the daemon-wide default model (a model id from the user's logged-in traecli
// catalog).
if e, ok := probe("MULTICA_TRAECLI_PATH", "traecli", "MULTICA_TRAECLI_MODEL"); ok {
agents["traecli"] = e
}
// xAI Grok Build CLI (`grok`), driven over ACP via
// `grok agent --always-approve stdio`. MULTICA_GROK_MODEL seeds the
// daemon-wide default (e.g. grok-4.5).
if e, ok := probe("MULTICA_GROK_PATH", "grok", "MULTICA_GROK_MODEL"); ok {
agents["grok"] = e
}
// Qwen Code (`qwen`) runs headlessly with -p and stream-json. Its native
// QWEN.md and .qwen/skills task context is prepared by execenv.
if e, ok := probe("MULTICA_QWEN_PATH", "qwen", "MULTICA_QWEN_MODEL"); ok {
agents["qwen"] = e
}
return agents
}