Files
multica/server/internal/daemon/execenv/execenv.go
Bohan Jiang 6cc553e5a3 fix(daemon): isolate Codex sessions per task to unblock initialize (MUL-4424) (#5360)
* fix(daemon): isolate Codex sessions per task to unblock initialize (MUL-4424)

Codex 0.143+ backfills a per-home session-state DB by enumerating every
rollout visible under sessions/ during `initialize`. The per-task
CODEX_HOME symlinked the shared ~/.codex/sessions in, so a machine with a
large accumulated history (one reporter: ~2000 rollouts / ~22 GiB) stalled
`initialize` for tens of seconds — the app-server started but the task
produced no output before it was cancelled (github #5273).

Give each task its own local sessions/ directory instead:

- Fresh task: create an empty local sessions/ so backfill is trivial.
- Reused task with a real sessions/ dir: it is authoritative — leave it.
- Reused task still holding a legacy symlink (older build): migrate in
  place. Replace the symlink with a real dir; when resuming, symlink only
  the single rollout being resumed (never copy — a rollout can be GiB and
  this is on initialize's critical path); and drop the stale, rebuildable
  session-state DB (state_*.sqlite*, session_index.jsonl) so Codex
  re-indexes the task-local sessions. Unrelated per-task DBs (goals_*,
  logs_*, memories_*) are left intact.

Also point the token-usage fallback scan at the backend's per-task
CODEX_HOME instead of the daemon-global home, so usage isn't lost now that
sessions are isolated there.

Complements the #5319 handshake watchdog (which turns a silent stall into a
loud, phased timeout); this removes the underlying cause.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon): address Codex session isolation review (MUL-4424)

Resolves the three blockers from Elon's review of #5360:

1. local_directory context loss. local_directory tasks get a fresh
   codex-home per task ID (the daemon never reuses their workdir), so
   task-local isolation stranded every follow-up run with an empty
   sessions dir and silently restarted the conversation. Their only
   stable, GC-safe cross-task store is the user's own ~/.codex/sessions
   (a persistent store under WorkspacesRoot would be orphan-GC'd), so keep
   the shared-sessions symlink for them (IsLocalDirectory). Managed tasks
   stay isolated.

2. Migration resume robustness.
   - Rollout lookup now covers the flat layout and background-compressed
     .jsonl.zst rollouts, not just nested YYYY/MM/DD *.jsonl — both are
     legitimate Codex 0.144 history that were previously judged "not
     found", silently dropping resume.
   - Exposure hard-links first, then symlinks, never copies — hard links
     need no privilege and work on Windows within a volume, so the
     zero-copy path is exercised identically on CI.
   - The daemon now verifies the rollout is actually present in the task
     CODEX_HOME (execenv.CodexResumeRolloutPresent) before the brief is
     generated; if absent it clears the resume from both the backend and
     the brief instead of telling the agent it is continuing a lost thread.

3. session_index.jsonl is no longer deleted during migration — Codex uses
   it as the authoritative thread-id -> name store (not rebuildable from
   rollouts). Only the rebuildable state_*.sqlite* is reset.

Tests: 2-round local_directory resume across task IDs; compressed/flat
lookup; hard-link zero-copy (os.SameFile); session_index preserved;
CodexResumeRolloutPresent + the daemon gate helper (present keeps /
absent drops / non-codex + empty no-op).

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon): scope Codex sessions to a per-issue store; disclose lost resumes (MUL-4424)

Addresses the three blockers from Elon's second review of #5360.

1. local_directory still enumerated the whole machine history. The prior
   fix re-linked the entire ~/.codex/sessions into every fresh local_directory
   codex-home, so Codex still backfilled from thousands of unrelated rollouts on
   `initialize` (measured ~8.3s with 3450 rollouts; the reporter's 22 GiB could
   exceed the 30s watchdog). Point sessions/ at a persistent, per-(agent, issue)
   store under the shared Codex home (multica-sessions/<agent>/<issue>) that holds
   only this issue's rollouts. It is keyed stably across task IDs and lives
   outside the task-scoped envRoot the GC reclaims, so follow-up runs resume it
   while `initialize` only ever sees this issue's history.

2. Windows cross-volume resume was lost. Exposing a single rollout by hard-link
   (same-volume only) then file symlink (needs Windows privilege) can't cross a
   volume boundary. The store now lives on the shared Codex volume, so the resume
   rollout is hard-linked there zero-copy, and sessions/ is exposed to the task
   home via a directory link — a symlink on Unix, a junction on Windows — which
   crosses volumes without privilege and never copies a (possibly GiB) rollout on
   initialize's critical path. There is no remaining per-file cross-volume link.

3. An unavailable resume was a silent downgrade. Both resume gates
   (gateResumeToReusedWorkdir, gateCodexResumeToRolloutPresence) now set
   PriorSessionResumeUnavailable, and the runtime brief renders a Session
   Continuity Notice telling the agent to disclose to the user, up front in its
   reply, that the previous conversation context could not be restored and this
   run starts fresh — turning a silent restart into a user-visible one. The task
   is not failed: it can still do useful work without the prior context.

Managed fresh / reused-real-dir tasks keep their task-local, GC-collected
sessions dir unchanged; only the legacy-symlink migration with a resume routes
through the store (cross-volume-safe), and a home already linked to the store is
treated as authoritative on reuse.

Tests: local_directory per-issue store (only this issue's history, no whole-
machine leak); no-key fallback to an empty dir; two-round resume across task IDs
through the store; legacy migration routed through the store with a zero-copy
hard link; reused store link stays authoritative; both gates set the
resume-unavailable flag; brief renders the continuity notice only when a resume
was lost. execenv + daemon + pkg/agent packages, go vet, and gofmt all pass.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon): disclose live resume-RPC loss; bound Codex session store lifecycle (MUL-4424)

Addresses the two blockers from Elon's third review of #5360.

1. A real thread/resume failure was still a silent new session. The brief's
   Session Continuity Notice only covers losses the daemon detects before launch
   (workdir not reused, rollout absent). But when the rollout is present yet
   Codex rejects the live thread/resume (corrupt/incompatible rollout, server-side
   thread GC, schema drift), startOrResumeThread falls back to thread/start and
   the run succeeds on a fresh thread with no user-facing signal. Carry the
   original resume intent into the backend as ExecOptions.ResumeExpected (set from
   the post-gate PriorSessionID, so a pre-flight drop still routes through the
   brief and never double-notifies), and when a resume was expected but the
   backend landed on a fresh thread, prepend the same continuity notice to the
   first turn/start input. This also covers the daemon's transport-error
   fresh-session retry, which clears ResumeSessionID but not ResumeExpected.

2. The persistent per-issue store had no data lifecycle. multica-sessions stores
   live outside the task-scoped envRoot the GC reclaims (so resume survives across
   task IDs), which meant a done/abandoned issue's prompts and full rollouts (one
   reporter: a single 1.5 GiB rollout) accumulated forever and were never freed on
   issue/agent/workspace deletion. Add PruneCodexSessionStores: the daemon GC loop
   reclaims any store untouched for GCCodexSessionTTL (default 14 days, configurable
   via MULTICA_GC_CODEX_SESSION_TTL, 0 disables). A store's newest rollout mtime is
   its last activity, so an active or recently-resumed task keeps its store fresh
   and is never reclaimed, while a deleted issue's store ages out — an eventual
   reclamation guarantee without needing deletion events.

Tests: codexTurnInput discloses on resume fallback and stays silent on success /
fresh start (paired with the existing live-RPC fallback test); store pruning
reclaims aged stores, keeps recent ones, isolates issues, cleans empty agent
dirs, and is disable-able. execenv / daemon / pkg/agent, go vet, gofmt all pass.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon): protect a reopened Codex session store from GC mid-mount (MUL-4424)

Addresses Elon's fourth-review blocker: reopening an issue idle past
GCCodexSessionTTL could lose context, because mounting its per-issue session
store (MkdirAll + rollout lookup + task-home link) never refreshed the store's
mtime, so a GC cycle firing before the resumed turn wrote its first rollout saw
a >TTL-old store and reclaimed it — a stat->remove race with no in-use guard.

Two complementary defenses:

- Activity refresh: linkCodexSessionsToStore now os.Chtimes the store to now
  after linking, so codexStoreStat (which reads the newest mtime as last
  activity) sees a just-used store. This fixes the sequential repro — a mount
  immediately followed by a prune keeps the store.

- In-process active-store guard: the daemon marks the per-issue store in-use
  (execenv.CodexSessionStorePath) from before Prepare/Reuse mounts it until the
  task ends, and PruneCodexSessionStores now takes an isActive predicate and
  skips any store a live task holds. Because prepare and prune run in the same
  process, this closes the remaining concurrent stat->remove window the mtime
  refresh alone cannot. Reference-counted, mirroring the env-root guard.

Tests: a reopened >TTL store survives a GC cycle after remount and stays
resumable; an idle-on-disk store marked active is skipped, then reclaimed once
inactive; the existing idle-reclaim / isolation / disable / empty-agent-dir
cases still pass. execenv + daemon, go vet, gofmt all pass.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon): make Codex store delete atomic with mark-active (MUL-4424)

Addresses Elon's fifth-review blocker: the active-store guard's check and delete
were not one atomic step. PruneCodexSessionStores called isActive (which locked,
read, and unlocked) and only then RemoveAll'd, leaving a window where a task
could markActiveCodexStore between the check and the removal and still lose its
store — the exact mark-then-delete interleaving Elon reproduced.

Replace the point-in-time isActive predicate with a reserve-for-deletion
protocol that shares one lock with mark-active:

- reserveCodexStoreForDeletion(store) atomically refuses when a live task holds
  the store (or another delete already reserved it) and otherwise marks it
  reserved, all under one activeCodexStoresMu acquisition. PruneCodexSessionStores
  reserves before RemoveAll and commits after, so confirm-inactive and remove are
  effectively atomic against a concurrent mark.
- markActiveCodexStore now waits (on a sync.Cond) while a store is reserved, so a
  task never mounts a store mid-removal; committing the removal wakes it and the
  store is recreated fresh by Prepare (with the continuity notice).

So mark-before-reserve keeps the store (reserve refused); reserve-before-mark
removes it and blocks the late mark until the removal commits. The genuinely
idle case still reclaims.

Tests (daemon, run under -race): mark-then-reserve is refused; reserve blocks a
concurrent mark until commit then the store reads active; a second reserve is
refused mid-flight. The execenv prune tests move to the reserve seam; the
activity-refresh / reopen-then-prune / isolation / disable cases still pass.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon): namespace Codex session stores per profile for cross-daemon safety (MUL-4424)

Addresses Elon's sixth-review blocker: the in-process reservation guard cannot
span processes, but Multica supports multiple profile daemons on one machine
(e.g. production + staging) that share the same ~/.codex. Each daemon's GC
scanned the whole multica-sessions root, so a staging daemon could reclaim a
store a production task was actively resuming — its reservation lived only in the
other process's memory.

Isolate by profile instead of trying to lock across processes:

- Store path is now <shared>/multica-sessions/<namespace>/<agent>/<issue>, where
  namespace is the daemon's profile (empty -> "default"). PrepareParams/ReuseParams
  carry Profile; codexSessionStoreKey and CodexSessionStorePath fold it in.
- PruneCodexSessionStores takes the profile and scans ONLY that namespace, so a
  daemon never even sees another profile's stores, let alone deletes them. The
  per-profile trees are disjoint, so the in-process guard is sufficient within a
  namespace (profiles get separate daemon state, so no two daemons share one).

Test: a "staging"-owned idle store is untouched by a default-profile prune and
reclaimed only by staging's own prune. Existing prune/guard/reopen tests move
under the namespace. execenv + daemon under -race, go vet, gofmt all pass.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon): make the Codex store profile→namespace map injective (MUL-4424)

Addresses Elon's seventh-review blocker: the per-profile namespace was derived by
dropping unsafe characters, which is not injective. The CLI treats the empty
(default) profile and a profile literally named "default" as separate daemons,
yet both mapped to namespace "default"; likewise "staging.prod" and "stagingprod"
both mapped to "stagingprod". Two distinct daemons then shared one store tree, so
one could again reclaim the other's live session — the cross-process blocker
reopened for those profile names.

Make codexSessionStoreNamespace injective: the empty profile gets a reserved
bare literal "default", and every named profile is hex-encoded (bijective,
filesystem-safe) under a "p_" prefix a bare literal can never collide with. So
"" -> "default" while "default" -> "p_64656661756c74", and "staging.prod" /
"stagingprod" get distinct hex segments. sanitizeCodexPathSegment stays for the
UUID agent/issue segments (injective for real UUIDs); only the user-controlled
profile needed the encoding.

Tests: codexSessionStoreNamespace is distinct for "" vs "default", punctuation
variants, case variants, and an encoded-looking name; and end-to-end, pruning one
profile never reclaims the other's store for the "" vs "default" and
"staging.prod" vs "stagingprod" pairs. execenv + daemon under -race, go vet,
gofmt all pass.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon): fixed-length Codex store namespace so long profiles fit (MUL-4424)

Addresses Elon's eighth-review blocker: hex-encoding the full profile doubled the
namespace segment length. A profile can be as long as a filesystem segment allows
(~255 bytes) and the CLI persists it as its own config dir, but the store
namespace "p_" + hex(profile) reached 2 + 127*2 = 256 bytes at 127 chars,
overflowing the 255-byte single-segment limit — the profile's own dir created
fine, then the session store failed with "file name too long".

Derive the namespace from a fixed-length hash instead: a named profile is now
"p_" + hex(sha256(profile)) — a constant 64 hex chars (66 with the prefix),
filesystem-safe and collision-resistant. The empty (default) profile keeps its
reserved bare literal "default", which the "p_"-prefixed 66-char segment can
never equal. Still injective across the CLI's distinct-daemon cases; just no
longer length-expanding.

Test: the namespace stays <=255 bytes and creatable for profiles up to the
255-byte segment limit (127- and 255-char cases that overflowed under hex); the
prior injectivity and cross-profile prune-isolation tests still hold. execenv +
daemon under -race, go vet, gofmt all pass.

Co-authored-by: multica-agent <github@multica.ai>

---------

Co-authored-by: J <j@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
2026-07-15 00:48:30 +08:00

742 lines
36 KiB
Go

// Package execenv manages isolated per-task execution environments for the daemon.
// Each task gets its own directory with injected context files. Repositories are
// checked out on demand by the agent via `multica repo checkout`.
package execenv
import (
"encoding/json"
"fmt"
"log/slog"
"os"
"path/filepath"
"time"
"github.com/multica-ai/multica/server/internal/runtimeapps"
)
// RepoContextForEnv describes a workspace repo available for checkout.
type RepoContextForEnv struct {
URL string // remote URL
Description string // optional repo description
Ref string // optional default checkout ref for this task
}
// ProjectResourceForEnv describes a single resource attached to the issue's
// project. The resource_ref payload is type-specific JSON; the agent reads
// resources.json on disk for the full structure. This struct only carries
// fields the meta-skill template needs to render a human-readable summary
// (URL for github_repo, generic label otherwise).
type ProjectResourceForEnv struct {
ID string // server-assigned UUID
ResourceType string // e.g. "github_repo"
ResourceRef json.RawMessage // raw JSONB payload from the API
Label string // optional user-supplied label
}
// PrepareParams holds all inputs needed to set up an execution environment.
type PrepareParams struct {
WorkspacesRoot string // base path for all envs (e.g., ~/multica_workspaces)
WorkspaceID string // workspace UUID — tasks are grouped under this
TaskID string // task UUID — used for directory name
AgentName string // for git branch naming only
// Profile is the daemon's profile name (empty = default). It namespaces the
// per-issue Codex session store so a second profile-daemon sharing the same
// ~/.codex cannot see or GC this daemon's stores (MUL-4424).
Profile string
Provider string // agent provider (determines runtime config and skill injection paths)
CodexVersion string // detected Codex CLI version (only used when Provider == "codex")
OpenclawBin string // resolved openclaw CLI path (only used when Provider == "openclaw"); empty = look up on PATH
// McpConfig is the agent's saved `mcp_config` JSON, forwarded to the
// provider-specific config preparer when that provider materialises MCP
// via a per-task config file. Cursor and OpenClaw consume it here; other
// providers wire MCP via ExecOptions.McpConfig in the agent backend.
McpConfig json.RawMessage
// CursorMcpAuthSource is an explicit opt-in path to a Cursor mcp-auth.json
// file, or the Cursor project data directory containing it. Only Cursor's
// managed MCP path consumes it.
CursorMcpAuthSource string
// OpenclawGateway pins the OpenClaw Gateway endpoint inside the per-task
// wrapper. Only consulted when Provider == "openclaw" and the agent's
// runtime_config selected gateway mode (issue #3260). Zero means "inherit
// whatever the user's global openclaw.json already configures".
OpenclawGateway OpenclawGatewayPin
// LocalWorkDir, when non-empty, redirects the agent's working directory
// to a user-supplied absolute path instead of the synthesised envRoot/
// workdir. The path is NOT copied or mounted — the agent operates on
// the user's directory in place. The daemon still creates envRoot for
// output/, logs/, and .gc_meta.json; only the workdir slot is
// substituted. Used by the local_directory project_resource flow
// (MUL-2663). When set, the envRoot/workdir directory is not created.
LocalWorkDir string
// HermesSourceHome is the shared Hermes home the per-task overlay is seeded
// from — resolved by the daemon via execenv.ResolveHermesProfile so it honors
// the agent's custom_env HERMES_HOME and any -p/--profile or sticky selection.
// Only used for the hermes provider; empty falls back to the platform default.
HermesSourceHome string
// HermesSourceMustExist fails the overlay build closed when HermesSourceHome
// is absent — set when an explicit named profile was requested so a typo
// doesn't silently seed from an empty home and drop the user's auth/config.
HermesSourceMustExist bool
// HermesEnv is the sanitized effective env (agent custom_env minus the daemon
// blocklisted keys) used to expand ${VAR} in Hermes external_dirs so it
// matches what the Hermes child process actually sees. Only used for hermes.
HermesEnv map[string]string
Task TaskContextForEnv // context data for writing files
}
// TaskContextForEnv is the subset of task context used for writing context files.
type TaskContextForEnv struct {
IssueID string
TriggerCommentID string // comment that triggered this task (empty for on_assign)
TriggerThreadID string // root comment ID for the triggering thread; falls back to TriggerCommentID when empty
// CommentReplyTargets is set for a comment run that coalesced comments
// spanning MORE THAN ONE root thread (MUL-4348). When it has >=2 entries the
// workflow's reply step fans out — one reply per thread — instead of the
// single --parent=trigger cookbook, keeping this persistent brief in sync
// with the per-turn prompt so a cross-thread run cannot get one source
// telling it "one comment" and the other "one per thread". Same-thread
// follow-ups collapse to a single group upstream, so this stays empty and
// the single-parent path is used (no duplicate replies).
CommentReplyTargets []ThreadReplyTarget
NewCommentCount int // issue-wide comments since this agent's last run (excludes its own and the injected trigger)
NewCommentsSince string // RFC3339 anchor (last run's started_at) the count is measured from; empty on cold start
PriorSessionResumed bool // true when the daemon will resume an existing provider session for this task
// PriorSessionResumeUnavailable is true when this task carried a prior
// session the daemon expected to resume but could NOT (the reused workdir was
// gone, or the Codex rollout was not present in the task CODEX_HOME). The
// brief surfaces this so the agent tells the user its previous conversation
// context is gone and this run starts fresh — turning a silent context loss
// into a user-visible one (MUL-4424). Distinct from an ordinary cold start,
// which never had a prior session to lose.
PriorSessionResumeUnavailable bool
AgentID string // unique ID of the dispatched agent
AgentName string
AgentInstructions string // agent identity/persona instructions, injected into CLAUDE.md
AgentSkills []SkillContextForEnv
Repos []RepoContextForEnv // workspace repos available for checkout
ProjectID string // issue's project, when present
ProjectTitle string // human-readable project title
ProjectDescription string // durable project-level context, rendered into the brief's Project Context section
ProjectResources []ProjectResourceForEnv // resources attached to the project
ChatSessionID string // non-empty for chat tasks
AutopilotRunID string // non-empty for autopilot run_only tasks
AutopilotID string
AutopilotTitle string
AutopilotDescription string
AutopilotSource string
AutopilotTriggerPayload string
QuickCreatePrompt string // non-empty for quick-create tasks
HandoffNote string // assignment handoff instruction; rendered into issue_context.md (MUL-3375)
IsSquadLeader bool // true when the agent is acting as a squad leader (may exit silently on no_action)
// WorkspaceContext is the workspace-level system prompt (workspace.context
// in the DB). Rendered into the brief as `## Workspace Context` when
// non-empty so every agent in the workspace sees the same shared context,
// regardless of issue / chat / autopilot / quick-create.
WorkspaceContext string
// ConnectedApps lists per-run external app capabilities mounted through
// MCP overlays. Rendered briefly so the agent can map app names such as
// Notion to the actual MCP server name (`composio`).
ConnectedApps []runtimeapps.ConnectedApp
// RequestingUserName + RequestingUserProfileDescription describe the
// human the agent is acting on behalf of. v1 sources them from the
// runtime owner (the user who registered the daemon). Rendered into the
// brief as the `## Requesting User` section only when description is
// non-empty — empty means the user opted out of injecting profile
// context and the agent stays anonymous-user mode.
RequestingUserName string
RequestingUserProfileDescription string
// Initiator* identify the actor who triggered THIS task (the real
// requester) as distinct from the runtime owner. Rendered into the brief
// as `## Task Initiator` when a name is present; InitiatorEmail is shown
// only for member initiators. Empty for on-assign / autopilot /
// quick-create tasks, which have no attributable human initiator. See
// MUL-2645.
InitiatorType string
InitiatorID string
InitiatorName string
InitiatorEmail string
}
// SkillContextForEnv represents a skill to be written into the execution environment.
type SkillContextForEnv struct {
Name string
Description string
Content string
Files []SkillFileContextForEnv
}
// SkillFileContextForEnv represents a supporting file within a skill.
type SkillFileContextForEnv struct {
Path string
Content string
}
// Environment represents a prepared, isolated execution environment.
type Environment struct {
// RootDir is the top-level env directory ({workspacesRoot}/{task_id_short}/).
RootDir string
// WorkDir is the directory to pass as Cwd to the agent. Normally
// ({RootDir}/workdir/); when the task is bound to a local_directory
// project_resource, it is the user's path instead. See LocalDirectory.
WorkDir string
// LocalDirectory is true when WorkDir points at a user-supplied path
// outside RootDir (the local_directory flow). Callers that key behavior
// on "may I remove WorkDir as scratch?" must check this — for example
// the GC loop never deletes the user's directory.
LocalDirectory bool
// CodexHome is the path to the per-task CODEX_HOME directory (set only for codex provider).
CodexHome string
// OpenclawConfigPath is the path to the per-task synthesized OpenClaw
// config (set only for openclaw provider). The daemon exports this as
// OPENCLAW_CONFIG_PATH on the openclaw subprocess so its native skill
// scanner pins workspaceDir to WorkDir.
OpenclawConfigPath string
// OpenclawIncludeRoot is the directory of the user's active OpenClaw
// config (set only for openclaw provider with an on-disk user config).
// The daemon must prepend it to OPENCLAW_INCLUDE_ROOTS so OpenClaw is
// allowed to follow the wrapper's `$include` link out of envRoot into
// the user's config — by default OpenClaw confines `$include` to the
// directory holding the wrapper file. Empty when no $include is
// emitted (fresh install).
OpenclawIncludeRoot string
// CursorDataDir is the per-task Cursor data directory (set only for
// cursor provider when the agent has managed mcp_config). The daemon
// exports this as CURSOR_DATA_DIR so project-level MCP approvals are
// isolated from the user's persistent ~/.cursor/projects state.
CursorDataDir string
// HermesHome is the path to the per-task HERMES_HOME overlay (set only for
// the hermes provider, and only when the agent has skills bound — empty
// otherwise, leaving the user's real home in place). It mirrors ~/.hermes/
// via symlink, derives a config.yaml that references the user's real skills
// as an external root, and holds the bound skills in its skills/ subdir. The
// daemon exports it as HERMES_HOME so the hermes CLI discovers those skills
// natively — Hermes has no workspace-relative discovery, so the previous
// .agent_context/skills/ fallback was never read (issue #5242). See
// hermes_home.go.
HermesHome string
logger *slog.Logger // for cleanup logging
}
// PredictRootDir returns the env root path that Prepare would create for the
// given task, without performing any I/O. Callers use this to claim ownership
// of the directory (e.g. against the GC loop) before Prepare/Reuse runs.
func PredictRootDir(workspacesRoot, workspaceID, taskID string) string {
if workspacesRoot == "" || workspaceID == "" || taskID == "" {
return ""
}
return filepath.Join(workspacesRoot, workspaceID, shortID(taskID))
}
// Prepare creates an isolated execution environment for a task.
// The workdir starts empty (no repo checkouts). The agent checks out repos
// on demand via `multica repo checkout <url>`.
func Prepare(params PrepareParams, logger *slog.Logger) (*Environment, error) {
if params.WorkspacesRoot == "" {
return nil, fmt.Errorf("execenv: workspaces root is required")
}
if params.WorkspaceID == "" {
return nil, fmt.Errorf("execenv: workspace ID is required")
}
if params.TaskID == "" {
return nil, fmt.Errorf("execenv: task ID is required")
}
envRoot := filepath.Join(params.WorkspacesRoot, params.WorkspaceID, shortID(params.TaskID))
// Self-heal the root-level daemon marker on every task start so a marker
// removed while the daemon runs is restored before the agent spawns. The
// per-workdir marker written below only covers cwds inside the workdir;
// the root marker keeps the CLI fail-closed guard active for subprocesses
// that lose all MULTICA_* env vars AND escape above the workdir. Non-fatal:
// without it the workdir marker still protects the common case.
if err := EnsureWorkspacesRootMarker(params.WorkspacesRoot); err != nil && logger != nil {
logger.Warn("execenv: workspaces root marker not written; fail-closed guard limited to the task workdir", "error", err)
}
// Remove existing env if present (defensive — task IDs are unique).
if _, err := os.Stat(envRoot); err == nil {
if err := os.RemoveAll(envRoot); err != nil {
return nil, fmt.Errorf("execenv: remove existing env: %w", err)
}
}
// Create directory tree. For the standard flow the agent's workdir is
// envRoot/workdir; for local_directory tasks the user's path takes its
// place and we only need to create the scratch directories under
// envRoot.
workDir := filepath.Join(envRoot, "workdir")
scratchDirs := []string{filepath.Join(envRoot, "output"), filepath.Join(envRoot, "logs")}
if params.LocalWorkDir == "" {
scratchDirs = append(scratchDirs, workDir)
} else {
workDir = params.LocalWorkDir
}
for _, dir := range scratchDirs {
if err := os.MkdirAll(dir, 0o755); err != nil {
return nil, fmt.Errorf("execenv: create directory %s: %w", dir, err)
}
}
env := &Environment{
RootDir: envRoot,
WorkDir: workDir,
LocalDirectory: params.LocalWorkDir != "",
logger: logger,
}
// Write context files into workdir (skills go to provider-native paths).
// Track every file/dir we create in a manifest so CleanupSidecars can
// roll a local_directory workdir back to its pre-Prepare state. Cloud
// tasks don't need the manifest (the GC loop wipes envRoot wholesale),
// but we always write one — it's cheap, keeps Prepare/Reuse symmetric,
// and avoids a conditional that would silently disable cleanup if the
// local_directory detection logic ever drifts.
manifest := &sidecarManifest{}
if err := writeContextFiles(workDir, params.Provider, params.Task, manifest); err != nil {
return nil, fmt.Errorf("execenv: write context files: %w", err)
}
// For Codex, set up a per-task CODEX_HOME seeded from ~/.codex/ with skills.
if params.Provider == "codex" {
codexHome := filepath.Join(envRoot, "codex-home")
if err := prepareCodexHomeWithOpts(codexHome, CodexHomeOptions{CodexVersion: params.CodexVersion, IsLocalDirectory: params.LocalWorkDir != "", SessionStoreKey: codexSessionStoreKey(params.Profile, params.Task.AgentID, params.Task.IssueID)}, logger); err != nil {
return nil, fmt.Errorf("execenv: prepare codex-home: %w", err)
}
if err := hydrateCodexSkills(codexHome, params.Task.AgentSkills, logger); err != nil {
return nil, fmt.Errorf("execenv: hydrate codex skills: %w", err)
}
env.CodexHome = codexHome
}
// For Hermes, redirect HERMES_HOME to a per-task compatibility overlay ONLY
// when the agent has skills bound. A skill-less Hermes task keeps the user's
// real home and its original behavior untouched. The overlay makes the bound
// skills visible — Hermes discovers skills only from its home, so the old
// .agent_context/skills/ fallback was never read (issue #5242). See
// hermes_home.go.
if params.Provider == "hermes" && len(params.Task.AgentSkills) > 0 {
hermesHome := filepath.Join(envRoot, "hermes-home")
if err := prepareHermesHome(hermesHome, params.HermesSourceHome, params.HermesSourceMustExist, params.Task.AgentSkills, params.HermesEnv, logger); err != nil {
return nil, fmt.Errorf("execenv: prepare hermes-home: %w", err)
}
env.HermesHome = hermesHome
}
// For Cursor, materialize managed MCP into project-local config and use
// an isolated CURSOR_DATA_DIR for the per-workdir approval sidecar. Cursor
// still reads ~/.cursor/mcp.json, but only servers with approval entries in
// this per-task data dir can load, so user-global MCP servers do not leak
// into managed-MCP runs.
if params.Provider == "cursor" {
cursorDataDir, err := prepareCursorMcpConfig(envRoot, workDir, params.McpConfig, params.CursorMcpAuthSource, manifest)
if err != nil {
return nil, fmt.Errorf("execenv: prepare cursor mcp config: %w", err)
}
env.CursorDataDir = cursorDataDir
}
if err := writeSidecarManifest(envRoot, manifest); err != nil {
logger.Warn("execenv: write sidecar manifest failed (non-fatal)", "error", err)
}
// For OpenClaw, synthesize a per-task config that pins workspace to
// workDir. The skill scanner then reads {workDir}/skills/ (written by
// writeContextFiles above). Fail closed on errors: a malformed user
// config that the openclaw CLI can't read is a real problem and
// silently degrading to a minimal config would mask it by booting
// OpenClaw without the agents / providers / API keys it expects.
if params.Provider == "openclaw" {
result, err := prepareOpenclawConfig(envRoot, workDir, OpenclawConfigPrep{
OpenclawBin: params.OpenclawBin,
McpConfig: params.McpConfig,
Gateway: params.OpenclawGateway,
})
if err != nil {
return nil, fmt.Errorf("execenv: prepare openclaw config: %w", err)
}
env.OpenclawConfigPath = result.ConfigPath
env.OpenclawIncludeRoot = result.IncludeRoot
}
logger.Info("execenv: prepared env", "root", envRoot, "repos_available", len(params.Task.Repos))
return env, nil
}
// ReuseParams describes the inputs to Reuse. It mirrors PrepareParams for
// the per-provider knobs (CodexVersion, OpenclawBin) so callers can pass
// the same resolved binary path on both first-run and reuse paths.
type ReuseParams struct {
// WorkspacesRoot is the daemon-owned root under which all task envs live.
// Passed on reuse so the root-level fail-closed marker is self-healed here
// too — a marker removed while the daemon runs is restored before a reused
// task spawns, not only on the fresh-Prepare path.
WorkspacesRoot string
WorkDir string
Provider string
CodexVersion string // only used when Provider == "codex"
// ResumeSessionID is the prior Codex thread/session ID this reused task
// intends to resume, when any. Only consulted when Provider == "codex" and
// only used while migrating a legacy per-task home whose sessions/ still
// symlinks the shared ~/.codex/sessions — the single rollout for this ID is
// exposed into the new task-local sessions dir so thread/resume still finds
// it. Empty means a fresh thread. See prepareCodexSessionsDir (MUL-4424).
ResumeSessionID string
OpenclawBin string // only used when Provider == "openclaw"; empty = PATH lookup
// McpConfig is the agent's saved `mcp_config` JSON. Reused on reuse so a
// freshly-saved managed set re-materialises into the wrapper before the
// task starts — without this a stale wrapper from a prior run would keep
// the old MCP set in play.
McpConfig json.RawMessage
// CursorMcpAuthSource mirrors PrepareParams.CursorMcpAuthSource on reuse.
CursorMcpAuthSource string
// OpenclawGateway is the per-task Gateway pin re-applied on reuse so the
// agent picks up any runtime_config changes saved since the prior run.
OpenclawGateway OpenclawGatewayPin
// Profile is the daemon's profile name (empty = default), mirroring
// PrepareParams.Profile so a reused task keys its per-issue Codex session
// store into the same profile namespace (MUL-4424).
Profile string
// LocalDirectory is true when the reused WorkDir is a user-supplied
// directory (the local_directory flow). The flag is propagated into
// the returned Environment so downstream callers (notably the GC
// loop) keep the "never delete the user's directory" invariant on
// reuse paths.
LocalDirectory bool
// HermesSourceHome and HermesEnv mirror PrepareParams on reuse so the Hermes
// overlay re-derives against the agent's current source home / profile and
// external_dirs vars.
HermesSourceHome string
HermesSourceMustExist bool
HermesEnv map[string]string
Task TaskContextForEnv // refreshed context files / skills
}
// Reuse wraps an existing workdir into an Environment and refreshes context files.
// Returns nil if the workdir does not exist (caller should fall back to Prepare).
func Reuse(params ReuseParams, logger *slog.Logger) *Environment {
if _, err := os.Stat(params.WorkDir); err != nil {
return nil
}
// Self-heal the root-level daemon marker on the reuse path too, so a marker
// removed while the daemon runs is restored before a reused task spawns —
// otherwise reuse could run without the fail-closed guard until the next
// fresh Prepare. Non-fatal: the per-workdir marker still protects the common
// case, and an empty WorkspacesRoot (legacy callers) simply skips this.
if params.WorkspacesRoot != "" {
if err := EnsureWorkspacesRootMarker(params.WorkspacesRoot); err != nil && logger != nil {
logger.Warn("execenv: workspaces root marker not written on reuse; fail-closed guard limited to the task workdir", "error", err)
}
}
rootDir := filepath.Dir(params.WorkDir)
if params.LocalDirectory {
// For local_directory tasks the user's WorkDir is unrelated to
// envRoot (envRoot still lives under workspacesRoot/{wsID}/...),
// so reading it from filepath.Dir(WorkDir) would point at the
// parent of the user's directory. Callers that need a real
// RootDir on the reuse path should arrange to pass it in
// explicitly; for v1 the daemon only ever reuses local_directory
// workdirs after a fresh Prepare in the same task lifetime, so
// the empty RootDir on reuse is fine for the current callers
// (GC writes meta from Prepare's result, not Reuse's).
rootDir = ""
}
env := &Environment{
RootDir: rootDir,
WorkDir: params.WorkDir,
LocalDirectory: params.LocalDirectory,
logger: logger,
}
// Roll back the previous dispatch's sidecar writes before refreshing.
// On reuse the workdir still holds the prior run's issue_context.md and
// skill directories; without clearing them first, writeSkillFiles sees
// its own earlier output occupying the canonical slug and falls back to
// a collision-free sibling (issue-review, issue-review-multica,
// issue-review-multica-2, …), accumulating a fresh duplicate on every
// re-dispatch to the same issue. allocateCollisionFreeSkillDir exists to
// dodge *user*-owned skill dirs (the local_directory flow), not our own
// prior writes, so we undo them via the prior manifest first and let the
// refresh below re-create each skill at its natural slug. This also brings
// the standard providers in line with the Codex path, where
// hydrateCodexSkills already wipes its skills dir before re-hydrating.
//
// Two steps, in order:
// 1. removeReusedManagedSkillDirs reclaims the platform's own skill
// directories even when a prior-run agent left a file inside one.
// CleanupSidecars alone can't do this — it preserves any recorded dir
// the agent populated (correct on the local_directory teardown path),
// which would otherwise keep the canonical slug occupied and push the
// refresh back to issue-review-multica.
// 2. CleanupSidecars rolls back the remaining sidecar files
// (issue_context.md, project resources) and the manifest itself.
//
// No-op when RootDir is empty (legacy local_directory reuse, which the
// daemon skips anyway) or when no prior manifest exists (older build).
if env.RootDir != "" {
if err := removeReusedManagedSkillDirs(env.RootDir, skillsDirPath(params.WorkDir, params.Provider)); err != nil {
logger.Warn("execenv: reclaim managed skill dirs on reuse failed", "error", err)
}
if err := CleanupSidecars(env.RootDir); err != nil {
logger.Warn("execenv: roll back prior sidecars on reuse failed", "error", err)
}
}
// Refresh context files (issue_context.md, skills). Reuse tracks a
// fresh manifest under env.RootDir so a later CleanupSidecars sees
// the up-to-date list of writes (an old manifest from a prior run
// would otherwise reference files this Reuse no longer creates). For
// local_directory tasks the daemon skips Reuse entirely (see
// daemon.runTask), but writing the manifest unconditionally keeps
// Prepare/Reuse symmetric so a future caller can rely on the
// manifest being current after either path. RootDir is empty on the
// legacy local_directory Reuse fallback — skip the persist in that
// case to avoid creating a stray manifest at the filesystem root.
manifest := &sidecarManifest{}
if err := writeContextFiles(params.WorkDir, params.Provider, params.Task, manifest); err != nil {
logger.Warn("execenv: refresh context files failed", "error", err)
}
// Restore CodexHome for Codex provider — the per-task codex-home directory
// lives alongside the workdir. Re-run prepareCodexHomeWithOpts to ensure
// config (especially sandbox/network access) is up to date.
if params.Provider == "codex" {
codexHome := filepath.Join(env.RootDir, "codex-home")
if err := prepareCodexHomeWithOpts(codexHome, CodexHomeOptions{CodexVersion: params.CodexVersion, ResumeSessionID: params.ResumeSessionID, IsLocalDirectory: params.LocalDirectory, SessionStoreKey: codexSessionStoreKey(params.Profile, params.Task.AgentID, params.Task.IssueID)}, logger); err != nil {
logger.Warn("execenv: refresh codex-home failed", "error", err)
} else {
env.CodexHome = codexHome
if err := hydrateCodexSkills(codexHome, params.Task.AgentSkills, logger); err != nil {
logger.Warn("execenv: refresh codex skills failed", "error", err)
}
}
}
// Refresh (or tear down) the per-task HERMES_HOME on reuse. With skills
// bound, rebuild the overlay so an added/removed/edited skill and the
// mirrored home/config track the user's current ~/.hermes/ before the next
// hermes process starts. With no skills bound, drop the redirect entirely so
// the task reverts to the user's real home — matching a fresh Prepare for a
// skill-less agent.
if params.Provider == "hermes" && env.RootDir != "" {
hermesHome := filepath.Join(env.RootDir, "hermes-home")
if len(params.Task.AgentSkills) > 0 {
if err := prepareHermesHome(hermesHome, params.HermesSourceHome, params.HermesSourceMustExist, params.Task.AgentSkills, params.HermesEnv, logger); err != nil {
// Fail closed: a half-built overlay must not run. Returning nil
// makes the daemon fall back to a fresh Prepare, whose error
// then blocks dispatch rather than silently dropping the bound
// skill.
logger.Warn("execenv: refresh hermes-home failed; forcing fresh prepare", "error", err)
return nil
}
env.HermesHome = hermesHome
} else {
env.HermesHome = ""
if err := os.RemoveAll(hermesHome); err != nil {
logger.Warn("execenv: remove stale hermes-home failed", "error", err)
}
}
}
// Refresh Cursor's managed MCP sidecars on reuse. A newly saved agent
// mcp_config must replace the prior run's .cursor/mcp.json and isolated
// approvals before the next cursor-agent process starts.
if params.Provider == "cursor" && env.RootDir != "" {
cursorDataDir, err := prepareCursorMcpConfig(env.RootDir, params.WorkDir, params.McpConfig, params.CursorMcpAuthSource, manifest)
if err != nil {
logger.Warn("execenv: refresh cursor mcp config failed", "error", err)
return nil
}
env.CursorDataDir = cursorDataDir
}
if env.RootDir != "" {
if err := writeSidecarManifest(env.RootDir, manifest); err != nil {
logger.Warn("execenv: refresh sidecar manifest failed", "error", err)
}
}
// Refresh the per-task OpenClaw config on reuse — the user may have
// added/removed agents or rotated providers since the prior task ran,
// and the workspace override always re-targets the current workDir.
// Fail closed: a user config that can no longer be parsed should block
// reuse rather than degrade to a minimal config that boots OpenClaw
// without the registered agents.
if params.Provider == "openclaw" {
result, err := prepareOpenclawConfig(env.RootDir, params.WorkDir, OpenclawConfigPrep{
OpenclawBin: params.OpenclawBin,
McpConfig: params.McpConfig,
Gateway: params.OpenclawGateway,
})
if err != nil {
logger.Warn("execenv: refresh openclaw config failed", "error", err)
return nil
}
env.OpenclawConfigPath = result.ConfigPath
env.OpenclawIncludeRoot = result.IncludeRoot
}
logger.Info("execenv: reusing env", "workdir", params.WorkDir)
return env
}
// hydrateCodexSkills populates the per-task CODEX_HOME/skills directory with
// both user-installed skills (from the shared ~/.codex/skills/) and
// workspace-assigned skills. Workspace skills win on name conflict — they are
// written last and seedUserCodexSkills already pre-filters their names.
//
// The skills directory is wiped first so two stale-state classes that the
// Reuse path would otherwise leak are gone:
//
// - A name now claimed by a workspace skill that previously held only a
// user-seeded copy — support files from the user version would otherwise
// linger under the workspace skill's directory.
// - A user skill removed from the shared ~/.codex/skills/ since the last
// run — its old contents would otherwise remain visible to the codex
// CLI.
//
// Codex is the only runtime that needs this two-stage hydration because the
// daemon sets CODEX_HOME to a per-task directory, isolating the CLI from the
// user's real ~/.codex/. Other runtimes leave HOME untouched and discover
// user-level skills natively (see context.go for the workdir-local paths
// they use for workspace skills).
func hydrateCodexSkills(codexHome string, workspaceSkills []SkillContextForEnv, logger *slog.Logger) error {
skillsDir := filepath.Join(codexHome, "skills")
if err := os.RemoveAll(skillsDir); err != nil {
return fmt.Errorf("clear codex skills dir: %w", err)
}
if err := seedUserCodexSkills(codexHome, workspaceSkills, logger); err != nil {
logger.Warn("execenv: seed user codex skills failed", "error", err)
}
if len(workspaceSkills) == 0 {
return nil
}
// Codex skills live under env.RootDir/codex-home, which the GC loop
// (cloud) or env teardown (local_directory) wipes wholesale — they
// don't sit inside the user's workdir and don't need sidecar manifest
// tracking.
return writeSkillFiles(skillsDir, workspaceSkills, nil)
}
// GCMetaKind identifies which kind of parent record a task workdir belongs to.
// The GC loop dispatches its decision tree on this value so chat / autopilot /
// quick-create tasks are no longer forced through the issue-centric path.
type GCMetaKind string
const (
GCKindIssue GCMetaKind = "issue"
GCKindChat GCMetaKind = "chat"
GCKindAutopilotRun GCMetaKind = "autopilot_run"
GCKindQuickCreate GCMetaKind = "quick_create"
)
// GCMeta is persisted to .gc_meta.json inside the env root so the GC loop
// can decide whether the directory is reclaimable. It is a discriminated
// union keyed on Kind: only the ID field matching Kind is meaningful.
//
// Older meta files (pre-v2) lack the Kind field; readers must default empty
// Kind to GCKindIssue for backward compatibility — only IssueID was written
// before, and only issue-centric tasks ever produced a meta file.
type GCMeta struct {
Kind GCMetaKind `json:"kind,omitempty"`
IssueID string `json:"issue_id,omitempty"`
ChatSessionID string `json:"chat_session_id,omitempty"`
AutopilotRunID string `json:"autopilot_run_id,omitempty"`
TaskID string `json:"task_id,omitempty"`
WorkspaceID string `json:"workspace_id"`
CompletedAt time.Time `json:"completed_at"`
// LocalDirectory marks tasks whose WorkDir pointed at a user-owned
// path rather than the synthesised envRoot/workdir. The GC loop honours
// this by never falling into the gcActionClean branch (which would
// RemoveAll envRoot — safe by structure, but we still want to keep the
// envRoot's output/ and logs/ around longer so users can inspect what
// the agent did in their own tree). Pattern-based artifact cleanup is
// still allowed.
LocalDirectory bool `json:"local_directory,omitempty"`
}
const gcMetaFile = ".gc_meta.json"
// WriteGCMeta writes GC metadata into the given directory. The caller is
// responsible for choosing Kind and populating the matching ID field;
// CompletedAt is stamped here so callers don't have to think about clocks.
func WriteGCMeta(envRoot string, meta GCMeta, logger *slog.Logger) error {
if envRoot == "" {
return nil
}
if meta.Kind == "" {
// Defensive: a task that doesn't fit any known kind would write a
// meta file the GC loop can't dispatch on. Skip silently — the
// directory falls back to the orphan-by-mtime path.
logger.Debug("execenv: skipping .gc_meta.json write: kind is empty", "envRoot", envRoot)
return nil
}
meta.CompletedAt = time.Now().UTC()
data, err := json.Marshal(meta)
if err != nil {
return fmt.Errorf("marshal gc meta: %w", err)
}
return os.WriteFile(filepath.Join(envRoot, gcMetaFile), data, 0o644)
}
// ReadGCMeta reads GC metadata from a task directory root. Pre-v2 meta files
// (no kind field) are normalized to GCKindIssue so the legacy issue path
// keeps working without a migration.
func ReadGCMeta(envRoot string) (*GCMeta, error) {
data, err := os.ReadFile(filepath.Join(envRoot, gcMetaFile))
if err != nil {
return nil, err
}
var meta GCMeta
if err := json.Unmarshal(data, &meta); err != nil {
return nil, err
}
if meta.Kind == "" {
meta.Kind = GCKindIssue
}
return &meta, nil
}
// Cleanup tears down the execution environment.
// If removeAll is true, the entire env root is deleted. Otherwise, workdir is
// removed but output/ and logs/ are preserved for debugging.
//
// For local_directory tasks (env.LocalDirectory==true) WorkDir is the
// user's own path — Cleanup MUST NEVER delete it, regardless of removeAll.
// In that mode we only ever delete the envRoot scratch directory.
func (env *Environment) Cleanup(removeAll bool) error {
if env == nil {
return nil
}
if env.LocalDirectory {
// Never touch the user's directory. RootDir is the daemon's own
// scratch; safe to remove when the caller asked for a full
// teardown.
if removeAll && env.RootDir != "" {
if err := os.RemoveAll(env.RootDir); err != nil {
env.logger.Warn("execenv: cleanup local_directory envRoot failed", "error", err)
return err
}
}
return nil
}
if removeAll {
if err := os.RemoveAll(env.RootDir); err != nil {
env.logger.Warn("execenv: cleanup removeAll failed", "error", err)
return err
}
return nil
}
// Partial cleanup: remove workdir, keep output/ and logs/.
if err := os.RemoveAll(env.WorkDir); err != nil {
env.logger.Warn("execenv: cleanup workdir failed", "error", err)
return err
}
return nil
}