Files
multica/server/pkg/agent/codex.go
Bohan Jiang 6cc553e5a3 fix(daemon): isolate Codex sessions per task to unblock initialize (MUL-4424) (#5360)
* fix(daemon): isolate Codex sessions per task to unblock initialize (MUL-4424)

Codex 0.143+ backfills a per-home session-state DB by enumerating every
rollout visible under sessions/ during `initialize`. The per-task
CODEX_HOME symlinked the shared ~/.codex/sessions in, so a machine with a
large accumulated history (one reporter: ~2000 rollouts / ~22 GiB) stalled
`initialize` for tens of seconds — the app-server started but the task
produced no output before it was cancelled (github #5273).

Give each task its own local sessions/ directory instead:

- Fresh task: create an empty local sessions/ so backfill is trivial.
- Reused task with a real sessions/ dir: it is authoritative — leave it.
- Reused task still holding a legacy symlink (older build): migrate in
  place. Replace the symlink with a real dir; when resuming, symlink only
  the single rollout being resumed (never copy — a rollout can be GiB and
  this is on initialize's critical path); and drop the stale, rebuildable
  session-state DB (state_*.sqlite*, session_index.jsonl) so Codex
  re-indexes the task-local sessions. Unrelated per-task DBs (goals_*,
  logs_*, memories_*) are left intact.

Also point the token-usage fallback scan at the backend's per-task
CODEX_HOME instead of the daemon-global home, so usage isn't lost now that
sessions are isolated there.

Complements the #5319 handshake watchdog (which turns a silent stall into a
loud, phased timeout); this removes the underlying cause.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon): address Codex session isolation review (MUL-4424)

Resolves the three blockers from Elon's review of #5360:

1. local_directory context loss. local_directory tasks get a fresh
   codex-home per task ID (the daemon never reuses their workdir), so
   task-local isolation stranded every follow-up run with an empty
   sessions dir and silently restarted the conversation. Their only
   stable, GC-safe cross-task store is the user's own ~/.codex/sessions
   (a persistent store under WorkspacesRoot would be orphan-GC'd), so keep
   the shared-sessions symlink for them (IsLocalDirectory). Managed tasks
   stay isolated.

2. Migration resume robustness.
   - Rollout lookup now covers the flat layout and background-compressed
     .jsonl.zst rollouts, not just nested YYYY/MM/DD *.jsonl — both are
     legitimate Codex 0.144 history that were previously judged "not
     found", silently dropping resume.
   - Exposure hard-links first, then symlinks, never copies — hard links
     need no privilege and work on Windows within a volume, so the
     zero-copy path is exercised identically on CI.
   - The daemon now verifies the rollout is actually present in the task
     CODEX_HOME (execenv.CodexResumeRolloutPresent) before the brief is
     generated; if absent it clears the resume from both the backend and
     the brief instead of telling the agent it is continuing a lost thread.

3. session_index.jsonl is no longer deleted during migration — Codex uses
   it as the authoritative thread-id -> name store (not rebuildable from
   rollouts). Only the rebuildable state_*.sqlite* is reset.

Tests: 2-round local_directory resume across task IDs; compressed/flat
lookup; hard-link zero-copy (os.SameFile); session_index preserved;
CodexResumeRolloutPresent + the daemon gate helper (present keeps /
absent drops / non-codex + empty no-op).

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon): scope Codex sessions to a per-issue store; disclose lost resumes (MUL-4424)

Addresses the three blockers from Elon's second review of #5360.

1. local_directory still enumerated the whole machine history. The prior
   fix re-linked the entire ~/.codex/sessions into every fresh local_directory
   codex-home, so Codex still backfilled from thousands of unrelated rollouts on
   `initialize` (measured ~8.3s with 3450 rollouts; the reporter's 22 GiB could
   exceed the 30s watchdog). Point sessions/ at a persistent, per-(agent, issue)
   store under the shared Codex home (multica-sessions/<agent>/<issue>) that holds
   only this issue's rollouts. It is keyed stably across task IDs and lives
   outside the task-scoped envRoot the GC reclaims, so follow-up runs resume it
   while `initialize` only ever sees this issue's history.

2. Windows cross-volume resume was lost. Exposing a single rollout by hard-link
   (same-volume only) then file symlink (needs Windows privilege) can't cross a
   volume boundary. The store now lives on the shared Codex volume, so the resume
   rollout is hard-linked there zero-copy, and sessions/ is exposed to the task
   home via a directory link — a symlink on Unix, a junction on Windows — which
   crosses volumes without privilege and never copies a (possibly GiB) rollout on
   initialize's critical path. There is no remaining per-file cross-volume link.

3. An unavailable resume was a silent downgrade. Both resume gates
   (gateResumeToReusedWorkdir, gateCodexResumeToRolloutPresence) now set
   PriorSessionResumeUnavailable, and the runtime brief renders a Session
   Continuity Notice telling the agent to disclose to the user, up front in its
   reply, that the previous conversation context could not be restored and this
   run starts fresh — turning a silent restart into a user-visible one. The task
   is not failed: it can still do useful work without the prior context.

Managed fresh / reused-real-dir tasks keep their task-local, GC-collected
sessions dir unchanged; only the legacy-symlink migration with a resume routes
through the store (cross-volume-safe), and a home already linked to the store is
treated as authoritative on reuse.

Tests: local_directory per-issue store (only this issue's history, no whole-
machine leak); no-key fallback to an empty dir; two-round resume across task IDs
through the store; legacy migration routed through the store with a zero-copy
hard link; reused store link stays authoritative; both gates set the
resume-unavailable flag; brief renders the continuity notice only when a resume
was lost. execenv + daemon + pkg/agent packages, go vet, and gofmt all pass.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon): disclose live resume-RPC loss; bound Codex session store lifecycle (MUL-4424)

Addresses the two blockers from Elon's third review of #5360.

1. A real thread/resume failure was still a silent new session. The brief's
   Session Continuity Notice only covers losses the daemon detects before launch
   (workdir not reused, rollout absent). But when the rollout is present yet
   Codex rejects the live thread/resume (corrupt/incompatible rollout, server-side
   thread GC, schema drift), startOrResumeThread falls back to thread/start and
   the run succeeds on a fresh thread with no user-facing signal. Carry the
   original resume intent into the backend as ExecOptions.ResumeExpected (set from
   the post-gate PriorSessionID, so a pre-flight drop still routes through the
   brief and never double-notifies), and when a resume was expected but the
   backend landed on a fresh thread, prepend the same continuity notice to the
   first turn/start input. This also covers the daemon's transport-error
   fresh-session retry, which clears ResumeSessionID but not ResumeExpected.

2. The persistent per-issue store had no data lifecycle. multica-sessions stores
   live outside the task-scoped envRoot the GC reclaims (so resume survives across
   task IDs), which meant a done/abandoned issue's prompts and full rollouts (one
   reporter: a single 1.5 GiB rollout) accumulated forever and were never freed on
   issue/agent/workspace deletion. Add PruneCodexSessionStores: the daemon GC loop
   reclaims any store untouched for GCCodexSessionTTL (default 14 days, configurable
   via MULTICA_GC_CODEX_SESSION_TTL, 0 disables). A store's newest rollout mtime is
   its last activity, so an active or recently-resumed task keeps its store fresh
   and is never reclaimed, while a deleted issue's store ages out — an eventual
   reclamation guarantee without needing deletion events.

Tests: codexTurnInput discloses on resume fallback and stays silent on success /
fresh start (paired with the existing live-RPC fallback test); store pruning
reclaims aged stores, keeps recent ones, isolates issues, cleans empty agent
dirs, and is disable-able. execenv / daemon / pkg/agent, go vet, gofmt all pass.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon): protect a reopened Codex session store from GC mid-mount (MUL-4424)

Addresses Elon's fourth-review blocker: reopening an issue idle past
GCCodexSessionTTL could lose context, because mounting its per-issue session
store (MkdirAll + rollout lookup + task-home link) never refreshed the store's
mtime, so a GC cycle firing before the resumed turn wrote its first rollout saw
a >TTL-old store and reclaimed it — a stat->remove race with no in-use guard.

Two complementary defenses:

- Activity refresh: linkCodexSessionsToStore now os.Chtimes the store to now
  after linking, so codexStoreStat (which reads the newest mtime as last
  activity) sees a just-used store. This fixes the sequential repro — a mount
  immediately followed by a prune keeps the store.

- In-process active-store guard: the daemon marks the per-issue store in-use
  (execenv.CodexSessionStorePath) from before Prepare/Reuse mounts it until the
  task ends, and PruneCodexSessionStores now takes an isActive predicate and
  skips any store a live task holds. Because prepare and prune run in the same
  process, this closes the remaining concurrent stat->remove window the mtime
  refresh alone cannot. Reference-counted, mirroring the env-root guard.

Tests: a reopened >TTL store survives a GC cycle after remount and stays
resumable; an idle-on-disk store marked active is skipped, then reclaimed once
inactive; the existing idle-reclaim / isolation / disable / empty-agent-dir
cases still pass. execenv + daemon, go vet, gofmt all pass.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon): make Codex store delete atomic with mark-active (MUL-4424)

Addresses Elon's fifth-review blocker: the active-store guard's check and delete
were not one atomic step. PruneCodexSessionStores called isActive (which locked,
read, and unlocked) and only then RemoveAll'd, leaving a window where a task
could markActiveCodexStore between the check and the removal and still lose its
store — the exact mark-then-delete interleaving Elon reproduced.

Replace the point-in-time isActive predicate with a reserve-for-deletion
protocol that shares one lock with mark-active:

- reserveCodexStoreForDeletion(store) atomically refuses when a live task holds
  the store (or another delete already reserved it) and otherwise marks it
  reserved, all under one activeCodexStoresMu acquisition. PruneCodexSessionStores
  reserves before RemoveAll and commits after, so confirm-inactive and remove are
  effectively atomic against a concurrent mark.
- markActiveCodexStore now waits (on a sync.Cond) while a store is reserved, so a
  task never mounts a store mid-removal; committing the removal wakes it and the
  store is recreated fresh by Prepare (with the continuity notice).

So mark-before-reserve keeps the store (reserve refused); reserve-before-mark
removes it and blocks the late mark until the removal commits. The genuinely
idle case still reclaims.

Tests (daemon, run under -race): mark-then-reserve is refused; reserve blocks a
concurrent mark until commit then the store reads active; a second reserve is
refused mid-flight. The execenv prune tests move to the reserve seam; the
activity-refresh / reopen-then-prune / isolation / disable cases still pass.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon): namespace Codex session stores per profile for cross-daemon safety (MUL-4424)

Addresses Elon's sixth-review blocker: the in-process reservation guard cannot
span processes, but Multica supports multiple profile daemons on one machine
(e.g. production + staging) that share the same ~/.codex. Each daemon's GC
scanned the whole multica-sessions root, so a staging daemon could reclaim a
store a production task was actively resuming — its reservation lived only in the
other process's memory.

Isolate by profile instead of trying to lock across processes:

- Store path is now <shared>/multica-sessions/<namespace>/<agent>/<issue>, where
  namespace is the daemon's profile (empty -> "default"). PrepareParams/ReuseParams
  carry Profile; codexSessionStoreKey and CodexSessionStorePath fold it in.
- PruneCodexSessionStores takes the profile and scans ONLY that namespace, so a
  daemon never even sees another profile's stores, let alone deletes them. The
  per-profile trees are disjoint, so the in-process guard is sufficient within a
  namespace (profiles get separate daemon state, so no two daemons share one).

Test: a "staging"-owned idle store is untouched by a default-profile prune and
reclaimed only by staging's own prune. Existing prune/guard/reopen tests move
under the namespace. execenv + daemon under -race, go vet, gofmt all pass.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon): make the Codex store profile→namespace map injective (MUL-4424)

Addresses Elon's seventh-review blocker: the per-profile namespace was derived by
dropping unsafe characters, which is not injective. The CLI treats the empty
(default) profile and a profile literally named "default" as separate daemons,
yet both mapped to namespace "default"; likewise "staging.prod" and "stagingprod"
both mapped to "stagingprod". Two distinct daemons then shared one store tree, so
one could again reclaim the other's live session — the cross-process blocker
reopened for those profile names.

Make codexSessionStoreNamespace injective: the empty profile gets a reserved
bare literal "default", and every named profile is hex-encoded (bijective,
filesystem-safe) under a "p_" prefix a bare literal can never collide with. So
"" -> "default" while "default" -> "p_64656661756c74", and "staging.prod" /
"stagingprod" get distinct hex segments. sanitizeCodexPathSegment stays for the
UUID agent/issue segments (injective for real UUIDs); only the user-controlled
profile needed the encoding.

Tests: codexSessionStoreNamespace is distinct for "" vs "default", punctuation
variants, case variants, and an encoded-looking name; and end-to-end, pruning one
profile never reclaims the other's store for the "" vs "default" and
"staging.prod" vs "stagingprod" pairs. execenv + daemon under -race, go vet,
gofmt all pass.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon): fixed-length Codex store namespace so long profiles fit (MUL-4424)

Addresses Elon's eighth-review blocker: hex-encoding the full profile doubled the
namespace segment length. A profile can be as long as a filesystem segment allows
(~255 bytes) and the CLI persists it as its own config dir, but the store
namespace "p_" + hex(profile) reached 2 + 127*2 = 256 bytes at 127 chars,
overflowing the 255-byte single-segment limit — the profile's own dir created
fine, then the session store failed with "file name too long".

Derive the namespace from a fixed-length hash instead: a named profile is now
"p_" + hex(sha256(profile)) — a constant 64 hex chars (66 with the prefix),
filesystem-safe and collision-resistant. The empty (default) profile keeps its
reserved bare literal "default", which the "p_"-prefixed 66-char segment can
never equal. Still injective across the CLI's distinct-daemon cases; just no
longer length-expanding.

Test: the namespace stays <=255 bytes and creatable for profiles up to the
255-byte segment limit (127- and 255-char cases that overflowed under hex); the
prior injectivity and cross-profile prune-isolation tests still hold. execenv +
daemon under -race, go vet, gofmt all pass.

Co-authored-by: multica-agent <github@multica.ai>

---------

Co-authored-by: J <j@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
2026-07-15 00:48:30 +08:00

2304 lines
73 KiB
Go

package agent
import (
"bufio"
"context"
"encoding/json"
"errors"
"fmt"
"log/slog"
"os"
"os/exec"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"sync"
"sync/atomic"
"syscall"
"time"
)
// codexBlockedArgs are flags hardcoded by the daemon that must not be
// overridden by user-configured custom_args. The mcp_servers config keys
// live in the per-task `$CODEX_HOME/config.toml` (written by
// ensureCodexMcpConfig); user-supplied `-c mcp_servers.…` overrides are
// stripped separately by filterCodexCustomConfigOverrides because they
// share the `-c` flag with legitimate non-MCP overrides like `-c model=…`.
var codexBlockedArgs = map[string]blockedArgMode{
"--listen": blockedWithValue, // stdio:// transport for daemon communication
}
// codexStderrTailBytes bounds the stderr tail captured for inclusion in
// error messages when codex exits before the JSON-RPC handshake (e.g. the
// user supplied a custom_args flag that the `app-server` subcommand
// rejects). Kept as its own constant so bumping codex independently of
// other agents stays easy if codex starts shipping longer failure traces.
const (
codexStderrTailBytes = 2048
defaultCodexSemanticInactivityTimeout = 10 * time.Minute
defaultCodexFirstTurnNoProgressTimeout = 30 * time.Second
defaultCodexHandshakeTimeout = 30 * time.Second
codexVersionDiagnosticTimeout = 2 * time.Second
// codexGracefulShutdownTimeout bounds how long the lifecycle goroutine
// waits for codex to exit on its own after stdin is closed, before forcing
// a context-cancel kill. A clean exit lets codex run its shutdown path and
// flush buffered telemetry — OTEL batch exporters only force-flush on
// graceful shutdown, so killing it immediately (the prior behavior) drops
// the task's spans/metrics/logs.
codexGracefulShutdownTimeout = 10 * time.Second
)
// codexGracefulShutdownTimeoutNanos optionally overrides
// codexGracefulShutdownTimeout for tests, in nanoseconds. Zero or negative
// values keep the production default. Tests for the cleanup-on-scanner-
// overflow path (#4520) use it to shrink the grace window from 10 s to a
// few hundred ms so the regression runs in a normal `go test` budget
// instead of burning two full grace windows per cleanup phase. Mirrors
// the opencodeTerminateGraceNanos hook.
var codexGracefulShutdownTimeoutNanos atomic.Int64
func codexGracefulShutdown() time.Duration {
if n := codexGracefulShutdownTimeoutNanos.Load(); n > 0 {
return time.Duration(n)
}
return codexGracefulShutdownTimeout
}
// CodexSemanticInactivityMarker prefixes timeout errors emitted when Codex
// stops making semantic progress while the process is still alive.
const CodexSemanticInactivityMarker = "codex semantic inactivity timeout"
// CodexFirstTurnNoProgressMarker identifies the app-server failure mode where
// Codex accepts a turn and then never emits any item, completion, or error.
const CodexFirstTurnNoProgressMarker = "codex app-server no progress timeout"
// CodexHandshakeTimeoutMarker identifies a Codex app-server startup RPC that
// did not answer within the bounded handshake window.
const CodexHandshakeTimeoutMarker = "codex app-server handshake timeout"
const codexModelCatalogRefreshTimeoutSignal = "failed to refresh available models: timeout waiting for child process to exit"
var errCodexProcessExited = errors.New("codex process exited")
type codexTimeoutKind int
const (
codexTimeoutNone codexTimeoutKind = iota
codexTimeoutSemanticInactivity
codexTimeoutFirstTurnNoProgress
)
type codexTimeoutDiagnostic struct {
Kind codexTimeoutKind
Timeout time.Duration
LastActivity string
ThreadID string
TurnID string
Model string
CodexVersion string
}
// codexBackend implements Backend by spawning `codex app-server --listen stdio://`
// and communicating via JSON-RPC 2.0 over stdin/stdout.
type codexBackend struct {
cfg Config
}
func buildCodexArgs(opts ExecOptions, logger *slog.Logger) []string {
args := []string{"app-server", "--listen", "stdio://"}
extra := filterCustomArgs(opts.ExtraArgs, codexBlockedArgs, logger)
custom := filterCustomArgs(opts.CustomArgs, codexBlockedArgs, logger)
// Only claim ownership of the `mcp_servers` namespace when the agent
// actually has a managed mcp_config in the MCP Tab. Otherwise existing
// users who configure MCP via `custom_args: ["-c", "mcp_servers.…"]`
// would silently lose those entries after this PR ships. With managed
// mcp_config present, daemon-written `$CODEX_HOME/config.toml` is the
// authoritative source and stray `-c mcp_servers.*` overrides are
// dropped to keep last-wins from re-shadowing it.
if hasManagedCodexMcpConfig(opts.McpConfig) {
extra = filterCodexCustomConfigOverrides(extra, logger)
custom = filterCodexCustomConfigOverrides(custom, logger)
}
args = append(args, extra...)
args = append(args, custom...)
return args
}
// hasManagedCodexMcpConfig reports whether the agent's mcp_config field is
// "present" in the API three-state sense: a non-null JSON value. Both
// `{}` and `{"mcpServers":{}}` count as present (the admin saved an empty
// managed set — strict mode, no global fallback); only SQL NULL or the
// literal JSON `null` count as absent (CLI default).
func hasManagedCodexMcpConfig(raw json.RawMessage) bool {
return hasManagedMcpConfig(raw)
}
// codexManagedMcpConfigKeyRe matches the daemon-managed config namespace
// (`mcp_servers.…`) when it appears as the value of a Codex `-c` /
// `--config` flag. Used by filterCodexCustomConfigOverrides to drop user
// overrides that would otherwise shadow what the MCP Tab writes into
// `$CODEX_HOME/config.toml`.
var codexManagedMcpConfigKeyRe = regexp.MustCompile(`^\s*mcp_servers(?:\s*\.|\s*=|\s*$)`)
// filterCodexCustomConfigOverrides drops `-c mcp_servers.…=` and
// `--config mcp_servers.…=` entries from custom args. Codex's `-c` is
// last-wins (verified against codex-cli 0.132.0), so without this filter a
// user-written `-c mcp_servers.fetch=…` in custom_args would silently
// override whatever the MCP Tab saved into the per-task config.toml. We
// own the `mcp_servers` namespace via the managed block, so user attempts
// to write into it are dropped with a warning rather than allowed to win.
// Other `-c`/`--config` keys (e.g. `-c model="o3"`) pass through unchanged.
func filterCodexCustomConfigOverrides(args []string, logger *slog.Logger) []string {
if len(args) == 0 {
return args
}
filtered := make([]string, 0, len(args))
for i := 0; i < len(args); i++ {
arg := args[i]
flag := arg
inlineValue := ""
hasInlineValue := false
if idx := strings.Index(arg, "="); idx > 0 {
flag = arg[:idx]
inlineValue = arg[idx+1:]
hasInlineValue = true
}
if flag == "-c" || flag == "--config" {
value := inlineValue
if !hasInlineValue && i+1 < len(args) {
value = args[i+1]
}
if codexManagedMcpConfigKeyRe.MatchString(value) {
if logger != nil {
// Log the key only, never the value — mcp_servers.<name>.env
// is allowed to carry secrets and the whole point of moving
// this to config.toml is to keep raw values out of logs/argv.
key := value
if eqIdx := strings.Index(value, "="); eqIdx >= 0 {
key = value[:eqIdx]
}
logger.Warn("custom_args: blocked mcp_servers override; daemon manages this via CODEX_HOME/config.toml",
"flag", flag, "key", strings.TrimSpace(key))
}
if !hasInlineValue && i+1 < len(args) {
i++ // skip the value arg
}
continue
}
}
filtered = append(filtered, arg)
}
return filtered
}
// Markers delimiting the daemon-managed `[mcp_servers.*]` block in
// `$CODEX_HOME/config.toml`. Match the existing sandbox / multi-agent /
// memory marker pattern so ops can grep all managed blocks consistently.
const (
multicaCodexMcpBeginMarker = "# BEGIN multica-managed mcp_servers (do not edit; regenerated by daemon)"
multicaCodexMcpEndMarker = "# END multica-managed mcp_servers"
)
var codexMcpBlockRe = regexp.MustCompile(
`(?ms)^` + regexp.QuoteMeta(multicaCodexMcpBeginMarker) +
`.*?^` + regexp.QuoteMeta(multicaCodexMcpEndMarker) + `\n*`)
// userCodexMcpServersTableHeaderRe matches `[mcp_servers.<name>]` (and its
// quoted-key form `[mcp_servers."<name>"]`) at the start of a line. Used
// to strip user-provided mcp_servers tables from the per-task config when
// the agent has its own mcp_config — mirrors Claude's `--strict-mcp-config`
// model where the daemon's set is authoritative.
var userCodexMcpServersTableHeaderRe = regexp.MustCompile(
`^\s*\[\s*mcp_servers\s*\.\s*(?:"[^"]*"|[^\]\s]+)\s*\]\s*(?:#.*)?$`)
// ensureCodexMcpConfig writes (or clears) the daemon-managed
// `[mcp_servers.*]` block in `$CODEX_HOME/config.toml`. The block is the
// authoritative source of MCP servers for this run: with mcp_config set
// in the agent UI the daemon also strips any inherited
// `[mcp_servers.*]` tables from the per-task config so the user's global
// `~/.codex/config.toml` doesn't shadow or collide with the managed set.
//
// The file mode is 0o600 because `mcp_servers.<id>.env` values may carry
// secrets (API keys, bearer tokens); the per-task home is owned by the
// daemon's user, so 0o600 keeps secrets out of any world-readable copy
// while still letting the codex child read them.
//
// A malformed mcp_config is returned as an error and the caller decides
// whether to surface or warn — same fail-soft contract the prior argv
// path had.
func ensureCodexMcpConfig(configPath string, mcpConfig json.RawMessage, logger *slog.Logger) error {
data, err := os.ReadFile(configPath)
if err != nil && !os.IsNotExist(err) {
return fmt.Errorf("read config.toml: %w", err)
}
existing := string(data)
// Always strip a prior managed block so reruns and clear-config flows
// converge on a clean state.
stripped := codexMcpBlockRe.ReplaceAllString(existing, "")
managed := hasManagedCodexMcpConfig(mcpConfig)
block, _, renderErr := renderCodexMcpServersBlock(mcpConfig)
if renderErr != nil {
return renderErr
}
var updated string
if managed {
// Agent has a managed MCP set (possibly empty — `{}` /
// `{"mcpServers":{}}` count as "saved an empty set" in the API's
// three-state semantics, distinct from nil/null which means
// "fall back to CLI default"). Strip any user-defined
// `[mcp_servers.*]` tables inherited from `~/.codex/config.toml`
// so the managed set is strict — mirrors Claude's
// `--strict-mcp-config`. Two reasons we cannot mix:
// 1. TOML rejects redefining the same table; a user table
// named `[mcp_servers.fetch]` would crash codex if the
// agent also defined `fetch`.
// 2. An admin saving an explicit list in the MCP Tab would
// otherwise see user-global servers silently joined in,
// which contradicts the UI affordance.
stripped = stripCodexUserMcpServerTables(stripped)
stripped = strings.TrimRight(stripped, "\n")
// When the managed set is empty we still write the marker
// block (with no tables between). This pins "managed but
// empty" on disk so the next run can find and strip the
// markers, and so the file's intent is grep-able by ops.
if block == "" {
block = multicaCodexMcpBeginMarker + "\n" + multicaCodexMcpEndMarker + "\n"
}
if stripped == "" {
updated = block
} else {
updated = stripped + "\n\n" + block
}
} else {
// No managed config: just remove any prior managed block and
// leave inherited user tables alone (CLI default fallback).
updated = stripped
}
if updated == existing {
return nil
}
if err := os.WriteFile(configPath, []byte(updated), 0o600); err != nil {
return fmt.Errorf("write config.toml: %w", err)
}
// os.WriteFile applies the mode only when creating a new file; if the
// per-task config.toml was already on disk at 0o644 (the default mode
// used by execenv.copyFile when seeding from ~/.codex/config.toml),
// the secret-bearing values we just wrote would inherit that wider
// mode. Chmod unconditionally to keep the secret in the daemon
// owner's lane regardless of the prior mode.
if err := os.Chmod(configPath, 0o600); err != nil {
return fmt.Errorf("chmod config.toml to 0600: %w", err)
}
if logger != nil {
logger.Debug("codex: wrote managed mcp_servers block to config.toml",
"config_path", configPath, "managed", managed)
}
return nil
}
// renderCodexMcpServersBlock renders the agent's mcp_config JSON
// (Claude-style `{"mcpServers": {...}}`) as a TOML block of
// `[mcp_servers.<name>]` tables wrapped in BEGIN/END markers. Returns
// (block, hasServers, err); hasServers=false means the input had no
// servers to render (empty/null mcp_config) and the caller should only
// strip the prior managed block.
//
// Stdio server keys (`args`, `env`, `command`) pass through verbatim —
// Codex's config schema happens to use the same names today. Remote HTTP
// servers use Codex-specific keys, so they are normalised here rather than
// leaking provider details into the UI/dispatch layer.
func renderCodexMcpServersBlock(raw json.RawMessage) (string, bool, error) {
if len(raw) == 0 {
return "", false, nil
}
var parsed struct {
McpServers map[string]json.RawMessage `json:"mcpServers"`
}
if err := json.Unmarshal(raw, &parsed); err != nil {
return "", false, fmt.Errorf("parse mcp_config json: %w", err)
}
if len(parsed.McpServers) == 0 {
return "", false, nil
}
names := make([]string, 0, len(parsed.McpServers))
for name := range parsed.McpServers {
names = append(names, name)
}
sort.Strings(names)
var sb strings.Builder
sb.WriteString(multicaCodexMcpBeginMarker)
sb.WriteString("\n")
for i, name := range names {
if !isCodexBareTomlKey(name) {
return "", false, fmt.Errorf("mcp server name %q must be ASCII alphanumeric / _ / - to fit Codex's bare-key requirement", name)
}
var serverVal map[string]any
if err := json.Unmarshal(parsed.McpServers[name], &serverVal); err != nil {
return "", false, fmt.Errorf("mcp_servers.%s: %w", name, err)
}
if serverVal == nil {
return "", false, fmt.Errorf("mcp_servers.%s must be a JSON object", name)
}
serverVal = normalizeCodexMcpServerConfig(serverVal)
if i > 0 {
sb.WriteString("\n")
}
sb.WriteString("[mcp_servers.")
sb.WriteString(name)
sb.WriteString("]\n")
keys := make([]string, 0, len(serverVal))
for k := range serverVal {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
tomlValue, err := jsonValueToCodexTOMLInline(serverVal[k])
if err != nil {
return "", false, fmt.Errorf("mcp_servers.%s.%s: %w", name, k, err)
}
sb.WriteString(codexTOMLKey(k))
sb.WriteString(" = ")
sb.WriteString(tomlValue)
sb.WriteString("\n")
}
}
sb.WriteString(multicaCodexMcpEndMarker)
sb.WriteString("\n")
return sb.String(), true, nil
}
func normalizeCodexMcpServerConfig(server map[string]any) map[string]any {
if !isCodexRemoteMcpServer(server) {
normalized := make(map[string]any, len(server))
for k, v := range server {
if isMulticaMcpSelectorKey(k) {
continue
}
normalized[k] = v
}
return normalized
}
normalized := make(map[string]any, len(server)+1)
for k, v := range server {
switch {
case isMulticaMcpSelectorKey(k):
continue
case k == "type":
continue
case k == "headers":
if _, ok := server["http_headers"]; !ok {
normalized["http_headers"] = v
}
default:
normalized[k] = v
}
}
normalized["experimental_use_rmcp_client"] = true
return normalized
}
func isMulticaMcpSelectorKey(k string) bool {
switch k {
case "tools", "prompts", "resources":
return true
default:
return false
}
}
func isCodexRemoteMcpServer(server map[string]any) bool {
if typ, ok := server["type"].(string); ok && strings.EqualFold(typ, "http") {
return true
}
_, hasURL := server["url"]
_, hasCommand := server["command"]
return hasURL && !hasCommand
}
// stripCodexUserMcpServerTables removes every `[mcp_servers.*]` table
// (header + body lines until the next top-level table header or EOF) from
// a TOML config string. Sub-tables like `[mcp_servers.fetch.env]` count
// as part of the parent table and are dropped along with it.
func stripCodexUserMcpServerTables(content string) string {
lines := strings.Split(content, "\n")
out := make([]string, 0, len(lines))
skipping := false
for _, line := range lines {
if userCodexMcpServersTableHeaderRe.MatchString(line) {
skipping = true
continue
}
if skipping {
trimmed := strings.TrimSpace(line)
if strings.HasPrefix(trimmed, "[") {
// Next table header. If it's still an `mcp_servers.*`
// table (including a sub-table), keep skipping; otherwise
// stop and emit this line.
if userCodexMcpServersTableHeaderRe.MatchString(line) ||
strings.HasPrefix(trimmed, "[mcp_servers.") ||
strings.HasPrefix(trimmed, "[ mcp_servers.") {
continue
}
skipping = false
out = append(out, line)
continue
}
continue
}
out = append(out, line)
}
return strings.Join(out, "\n")
}
// jsonValueToCodexTOMLInline serialises a JSON value as a TOML inline
// value. Only the subset Codex's `-c` accepts is supported: strings,
// numbers, booleans, arrays, and inline tables. JSON nulls are rejected
// because TOML has no null and silently dropping them would be confusing.
func jsonValueToCodexTOMLInline(v any) (string, error) {
switch x := v.(type) {
case nil:
return "", fmt.Errorf("null is not a valid TOML value")
case bool:
if x {
return "true", nil
}
return "false", nil
case float64:
if x == float64(int64(x)) {
return strconv.FormatInt(int64(x), 10), nil
}
return strconv.FormatFloat(x, 'f', -1, 64), nil
case string:
return codexTOMLBasicString(x), nil
case []any:
parts := make([]string, len(x))
for i, e := range x {
p, err := jsonValueToCodexTOMLInline(e)
if err != nil {
return "", err
}
parts[i] = p
}
return "[" + strings.Join(parts, ", ") + "]", nil
case map[string]any:
keys := make([]string, 0, len(x))
for k := range x {
keys = append(keys, k)
}
sort.Strings(keys)
parts := make([]string, len(keys))
for i, k := range keys {
p, err := jsonValueToCodexTOMLInline(x[k])
if err != nil {
return "", err
}
parts[i] = codexTOMLKey(k) + " = " + p
}
return "{ " + strings.Join(parts, ", ") + " }", nil
default:
return "", fmt.Errorf("unsupported value type %T", v)
}
}
func codexTOMLBasicString(s string) string {
var sb strings.Builder
sb.Grow(len(s) + 2)
sb.WriteByte('"')
for _, r := range s {
switch r {
case '\\':
sb.WriteString(`\\`)
case '"':
sb.WriteString(`\"`)
case '\b':
sb.WriteString(`\b`)
case '\t':
sb.WriteString(`\t`)
case '\n':
sb.WriteString(`\n`)
case '\f':
sb.WriteString(`\f`)
case '\r':
sb.WriteString(`\r`)
default:
if r < 0x20 || r == 0x7f {
sb.WriteString(fmt.Sprintf(`\u%04x`, r))
} else {
sb.WriteRune(r)
}
}
}
sb.WriteByte('"')
return sb.String()
}
func codexTOMLKey(s string) string {
if isCodexBareTomlKey(s) {
return s
}
return codexTOMLBasicString(s)
}
func isCodexBareTomlKey(s string) bool {
if s == "" {
return false
}
for _, r := range s {
switch {
case r >= 'a' && r <= 'z':
case r >= 'A' && r <= 'Z':
case r >= '0' && r <= '9':
case r == '_' || r == '-':
default:
return false
}
}
return true
}
func (b *codexBackend) Execute(ctx context.Context, prompt string, opts ExecOptions) (*Session, error) {
execPath := b.cfg.ExecutablePath
if execPath == "" {
execPath = "codex"
}
if _, err := exec.LookPath(execPath); err != nil {
return nil, fmt.Errorf("codex executable not found at %q: %w", execPath, err)
}
timeout := opts.Timeout
semanticInactivityTimeout := opts.SemanticInactivityTimeout
if semanticInactivityTimeout == 0 {
semanticInactivityTimeout = defaultCodexSemanticInactivityTimeout
}
handshakeTimeout := opts.HandshakeTimeout
if handshakeTimeout <= 0 {
handshakeTimeout = defaultCodexHandshakeTimeout
}
runCtx, cancel := runContext(ctx, timeout)
// Materialise the agent's MCP config into the per-task
// `$CODEX_HOME/config.toml`. Argv would be the simpler path, but
// `mcp_servers.<id>.env` is allowed to carry secrets (Codex docs:
// https://developers.openai.com/codex/mcp#configure-with-configtoml)
// and our UI already treats mcp_config as a redacted-for-non-admins
// field. Process argv ends up in OS-level `ps` listings and is also
// echoed into the daemon's `agent command` log line below, so any
// inline env-bearing TOML would defeat the redaction. Writing through
// config.toml at 0o600 keeps the secret values out of argv and logs.
if codexHome := strings.TrimSpace(b.cfg.Env["CODEX_HOME"]); codexHome != "" {
if err := ensureCodexMcpConfig(filepath.Join(codexHome, "config.toml"), opts.McpConfig, b.cfg.Logger); err != nil {
// Fail closed when we can't materialise the managed config.
// Warning-and-launching would silently fall back to the
// user's global `~/.codex/config.toml` MCP servers and
// look indistinguishable from "the saved config was
// applied", which is exactly the surprise the MCP Tab is
// supposed to remove.
cancel()
return nil, fmt.Errorf("apply codex mcp_config: %w", err)
}
} else if hasManagedCodexMcpConfig(opts.McpConfig) {
// Managed mcp_config saved but no CODEX_HOME to anchor it.
// Same reasoning as above: silently launching would inherit
// whatever MCP setup the host user has, which is the wrong
// shape of failure.
cancel()
return nil, fmt.Errorf("codex: mcp_config is set but CODEX_HOME env var is not configured; cannot apply managed MCP")
}
codexArgs := buildCodexArgs(opts, b.cfg.Logger)
cmd := exec.CommandContext(runCtx, execPath, codexArgs...)
hideAgentWindow(cmd)
// Run codex in its own process group so a cancel-on-stuck cleanup
// reaches the whole tree — the codex Node wrapper plus the native
// Rust app-server it spawns — not just the direct child. Without
// this, killing the leader leaves grandchildren as orphans that
// keep consuming memory until the OS reaps them; see #4520, where a
// scanner overflow during thread/resume otherwise leaked Codex
// processes indefinitely. configureProcessGroup is a no-op on
// Windows.
configureProcessGroup(cmd)
// Override the default exec.CommandContext cancel behaviour. The
// default sends SIGKILL only to cmd.Process (the leader); we instead
// signal the whole process group so descendants die too. Returning
// nil keeps exec from logging a spurious error; cmd.WaitDelay below
// still backstops cmd.Wait() if the kill leaves an open pipe.
cmd.Cancel = func() error {
if cmd.Process != nil {
signalProcessGroup(cmd.Process, syscall.SIGKILL)
}
return nil
}
// Bound the wait after the context is cancelled so a stuck child (or an
// open pipe held by a grandchild) can't hang cmd.Wait() forever. Matches
// the other long-lived backends (claude, copilot, cursor, …).
cmd.WaitDelay = 10 * time.Second
b.cfg.Logger.Info("agent command", "exec", execPath, "args", codexArgs)
if opts.Cwd != "" {
cmd.Dir = opts.Cwd
}
cmd.Env = buildEnv(b.cfg.Env)
stdout, err := cmd.StdoutPipe()
if err != nil {
cancel()
return nil, fmt.Errorf("codex stdout pipe: %w", err)
}
stdin, err := cmd.StdinPipe()
if err != nil {
cancel()
return nil, fmt.Errorf("codex stdin pipe: %w", err)
}
stderrBuf := newStderrTail(newLogWriter(b.cfg.Logger, "[codex:stderr] "), codexStderrTailBytes)
cmd.Stderr = stderrBuf
if err := cmd.Start(); err != nil {
cancel()
return nil, fmt.Errorf("start codex: %w", err)
}
b.cfg.Logger.Info("codex started app-server", "pid", cmd.Process.Pid, "cwd", opts.Cwd)
msgCh := make(chan Message, 256)
resCh := make(chan Result, 1)
semanticActivityCh := make(chan string, 256)
var outputMu sync.Mutex
var output strings.Builder
// turnDone is set before starting the reader goroutine so there is no
// race between the lifecycle goroutine writing and the reader reading.
turnDone := make(chan bool, 1) // true = aborted
c := &codexClient{
cfg: b.cfg,
stdin: stdin,
pending: make(map[int]*pendingRPC),
processDone: make(chan struct{}),
handshakeTimeout: handshakeTimeout,
notificationProtocol: "unknown",
onMessage: func(msg Message) {
logCodexAgentMessage(b.cfg.Logger, msg)
if msg.Type == MessageText {
outputMu.Lock()
output.WriteString(msg.Content)
outputMu.Unlock()
}
trySend(msgCh, msg)
trySendString(semanticActivityCh, describeCodexSemanticActivity(msg))
},
onSemanticActivity: func(description string) {
b.cfg.Logger.Debug("codex semantic activity observed", "activity", description)
trySendString(semanticActivityCh, description)
},
onTurnDone: func(aborted bool) {
select {
case turnDone <- aborted:
default:
}
},
}
// Start reading stdout in background
readerDone := make(chan struct{})
go func() {
defer close(readerDone)
scanner := bufio.NewScanner(stdout)
scanner.Buffer(make([]byte, 0, 1024*1024), 10*1024*1024)
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
if line == "" {
continue
}
c.handleLine(line)
}
if err := scanner.Err(); err != nil {
c.markProcessExited(fmt.Errorf("%w: %v", errCodexProcessExited, err))
return
}
c.markProcessExited(errCodexProcessExited)
}()
// drainAndWait closes stdin so codex shuts down, then joins cmd.Wait().
// cmd.Wait() is the only Go-stdlib-documented synchronization point for
// os/exec's internal stderr/stdout copy goroutines — until it returns,
// stderrBuf may not have observed every byte codex wrote before it
// exited, and stderrBuf.Tail() can come back empty or truncated. Any
// code that reads stderrBuf.Tail() must call drainAndWait() first.
// sync.Once makes it safe to call from both error paths and the deferred
// cleanup.
//
// drainAndWait is also the cleanup safety net for the scanner-overflow
// path (#4520). When codex emits a single stdout line larger than the
// scanner's MaxScanTokenSize, the reader goroutine returns with
// scanner.Err() set, fails all in-flight RPCs via markProcessExited, and
// closes readerDone — but the codex child process is still alive and is
// now blocked trying to write the rest of the oversized line into a
// stdout pipe nobody is reading. A naive stdin.Close()+cmd.Wait() then
// hangs forever: codex never reaches its stdin-read syscall, so it never
// sees EOF, never exits, and cmd.Wait() never returns. The lifecycle
// goroutine therefore never sends a failed Result, the outer daemon
// blocks on its result channel, and the higher-level fresh-session
// fallback never fires.
//
// To stay correct under both clean shutdown and the stuck-child case,
// drainAndWait runs in two bounded phases:
//
// 1. Close stdin and wait for the reader goroutine to finish, capped by
// codexGracefulShutdownTimeout. The reader exits when codex closes
// stdout on its own (clean shutdown — gives OTEL batch exporters a
// chance to flush) OR when the scanner errors out (overflow case —
// readerDone is already closed and the select returns immediately).
// Per os/exec docs, calling cmd.Wait() while reads are still
// in-flight on a StdoutPipe-returned pipe is incorrect because Wait
// closes the pipe and turns pending reads into spurious errors, so
// we must wait for the reader first.
//
// 2. Wait for cmd.Wait() to return, capped by another
// codexGracefulShutdownTimeout. Normally this returns immediately
// because the process has already exited. In the stuck-child case
// the process is still alive — we cancel the runCtx, which fires
// cmd.Cancel (the group-SIGKILL helper installed above), and
// cmd.WaitDelay then guarantees cmd.Wait() returns even if pipes
// stay open.
var waitOnce sync.Once
drainAndWait := func() {
waitOnce.Do(func() {
stdin.Close()
grace := codexGracefulShutdown()
// Phase 1: let the reader finish before invoking cmd.Wait().
select {
case <-readerDone:
// reader drained cleanly (codex shutdown closed stdout)
// or aborted early (e.g. scanner overflow). Either way it
// is now safe to call cmd.Wait().
case <-time.After(grace):
// codex did not close stdout within the grace window. Force
// the shutdown via context cancellation — cmd.Cancel
// group-kills the tree, the reader unblocks when stdout
// EOFs, and we proceed to phase 2.
b.cfg.Logger.Warn("codex did not close stdout after stdin EOF; forcing shutdown",
"pid", cmd.Process.Pid,
"grace", grace.String(),
)
cancel()
<-readerDone
}
// Phase 2: bound cmd.Wait() in case the process is still alive
// (scanner-overflow case: reader exited early on its own while
// codex stayed blocked writing into a full stdout pipe).
waitCh := make(chan struct{})
go func() {
_ = cmd.Wait()
close(waitCh)
}()
select {
case <-waitCh:
// reaped cleanly.
case <-time.After(grace):
b.cfg.Logger.Warn("codex process still alive after reader exited; forcing shutdown",
"pid", cmd.Process.Pid,
"grace", grace.String(),
)
cancel()
// WaitDelay (10s) is the final backstop: even if the
// group-kill races with an open pipe held by a
// descendant, cmd.Wait() returns within WaitDelay of the
// cancel.
<-waitCh
}
})
}
// Drive the session lifecycle in a goroutine.
// Shutdown sequence: lifecycle goroutine closes stdin + cancels context →
// codex process exits → reader goroutine's scanner.Scan() returns false →
// readerDone closes → lifecycle goroutine collects final output and sends Result.
go func() {
defer cancel()
defer close(msgCh)
defer close(resCh)
defer drainAndWait()
startTime := time.Now()
finalStatus := "completed"
var finalError string
// 1. Initialize handshake
_, err := c.request(runCtx, "initialize", map[string]any{
"clientInfo": map[string]any{
"name": "multica-agent-sdk",
"title": "Multica Agent SDK",
"version": "0.2.0",
},
"capabilities": map[string]any{
"experimentalApi": true,
},
})
if err != nil {
drainAndWait() // flush os/exec stderr goroutine before sampling Tail
finalStatus = "failed"
finalError = withAgentStderr(fmt.Sprintf("codex initialize failed: %v", err), "codex", stderrBuf.Tail())
resCh <- Result{Status: finalStatus, Error: finalError, DurationMs: time.Since(startTime).Milliseconds()}
return
}
c.notify("initialized")
// 2. Start a new thread, or resume the prior one for this issue. When
// resume fails (thread GCed on the server, schema drift, etc.) we fall
// back to a fresh thread so the task still makes progress.
threadID, resumed, err := c.startOrResumeThread(runCtx, opts, b.cfg.Logger)
if err != nil {
drainAndWait() // flush os/exec stderr goroutine before sampling Tail
finalStatus = "failed"
finalError = withAgentStderr(err.Error(), "codex", stderrBuf.Tail())
resCh <- Result{Status: finalStatus, Error: finalError, DurationMs: time.Since(startTime).Milliseconds()}
return
}
c.threadID = threadID
if resumed {
b.cfg.Logger.Info("codex thread resumed", "thread_id", threadID)
} else {
b.cfg.Logger.Info("codex thread started", "thread_id", threadID)
}
// 3. Send turn and wait for completion. When a resume was expected but we
// ended up on a fresh thread (the live thread/resume RPC was rejected — a
// corrupt/incompatible rollout, server-side thread GC, schema drift — or a
// transport failure forced a fresh retry), prepend a continuity notice so
// the agent tells the user the prior conversation could not be restored.
// The daemon's pre-flight gates only catch cases detectable before launch;
// this covers the ones only the live resume reveals (MUL-4424).
turnParams := map[string]any{
"threadId": threadID,
"input": codexTurnInput(prompt, opts.ResumeExpected, resumed),
}
// Per-turn reasoning override. Mirrors the per-thread injection in
// startOrResumeThread; keeping both in sync is enforced by the
// shared `codexReasoningInjection` fixture in codex_test.go (see
// MUL-2339 — Trump's constraint that the three injection points
// must not drift independently).
applyCodexReasoningEffort(turnParams, opts.ThinkingLevel)
waitingForTurn := true
var timeoutDiagnostic codexTimeoutDiagnostic
var processExitErr error
finishTurn := func(aborted bool) {
waitingForTurn = false
switch {
case aborted:
finalStatus = "aborted"
if errMsg := c.getTurnError(); errMsg != "" {
finalError = errMsg
} else {
finalError = "turn was aborted"
}
default:
if errMsg := c.getTurnError(); errMsg != "" {
finalStatus = "failed"
finalError = errMsg
}
}
}
_, err = c.request(runCtx, "turn/start", turnParams)
if err != nil {
select {
case aborted := <-turnDone:
finishTurn(aborted)
default:
drainAndWait() // flush os/exec stderr goroutine before sampling Tail
finalStatus = "failed"
finalError = withAgentStderr(fmt.Sprintf("codex turn/start failed: %v", err), "codex", stderrBuf.Tail())
resCh <- Result{Status: finalStatus, Error: finalError, DurationMs: time.Since(startTime).Milliseconds()}
return
}
}
lastSemanticActivity := time.Now()
lastSemanticActivityDescription := "turn/start"
semanticTimer := time.NewTimer(semanticInactivityTimeout)
defer semanticTimer.Stop()
firstTurnNoProgressTimeout := codexFirstTurnNoProgressTimeout(semanticInactivityTimeout)
var firstTurnNoProgressTimer *time.Timer
var firstTurnNoProgressTimerC <-chan time.Time
firstTurnStarted := false
firstTurnProgressObserved := false
stopFirstTurnNoProgressTimer := func() {
if firstTurnNoProgressTimer == nil {
return
}
stopTimer(firstTurnNoProgressTimer)
firstTurnNoProgressTimerC = nil
}
defer stopFirstTurnNoProgressTimer()
finishRunContextDone := func() {
waitingForTurn = false
if runCtx.Err() == context.DeadlineExceeded {
finalStatus = "timeout"
finalError = fmt.Sprintf("codex timed out after %s", timeout)
} else {
finalStatus = "aborted"
finalError = "execution cancelled"
}
}
for waitingForTurn {
select {
case aborted := <-turnDone:
finishTurn(aborted)
case activity := <-semanticActivityCh:
lastSemanticActivity = time.Now()
lastSemanticActivityDescription = activity
resetTimer(semanticTimer, semanticInactivityTimeout)
if activity == "status:running" && !firstTurnStarted {
firstTurnStarted = true
firstTurnNoProgressTimer = time.NewTimer(firstTurnNoProgressTimeout)
firstTurnNoProgressTimerC = firstTurnNoProgressTimer.C
} else if firstTurnStarted && !firstTurnProgressObserved && isCodexFirstTurnProgressActivity(activity) {
firstTurnProgressObserved = true
stopFirstTurnNoProgressTimer()
}
case <-firstTurnNoProgressTimerC:
waitingForTurn = false
finalStatus = "timeout"
timeoutDiagnostic = codexTimeoutDiagnostic{
Kind: codexTimeoutFirstTurnNoProgress,
Timeout: firstTurnNoProgressTimeout,
LastActivity: lastSemanticActivityDescription,
ThreadID: threadID,
TurnID: c.turnID,
Model: opts.Model,
}
b.cfg.Logger.Warn(CodexFirstTurnNoProgressMarker,
"pid", cmd.Process.Pid,
"thread_id", threadID,
"turn_id", c.turnID,
"timeout", firstTurnNoProgressTimeout.String(),
"last_activity", lastSemanticActivityDescription,
)
case <-semanticTimer.C:
waitingForTurn = false
finalStatus = "timeout"
timeoutDiagnostic = codexTimeoutDiagnostic{
Kind: codexTimeoutSemanticInactivity,
Timeout: semanticInactivityTimeout,
LastActivity: lastSemanticActivityDescription,
ThreadID: threadID,
TurnID: c.turnID,
Model: opts.Model,
}
b.cfg.Logger.Warn(CodexSemanticInactivityMarker,
"pid", cmd.Process.Pid,
"thread_id", threadID,
"turn_id", c.turnID,
"timeout", semanticInactivityTimeout.String(),
"last_activity", lastSemanticActivityDescription,
"idle_for", time.Since(lastSemanticActivity).Round(time.Millisecond).String(),
)
case <-runCtx.Done():
finishRunContextDone()
case <-c.processDone:
select {
case aborted := <-turnDone:
finishTurn(aborted)
default:
if runCtx.Err() != nil {
finishRunContextDone()
} else {
waitingForTurn = false
finalStatus = "failed"
processExitErr = c.getProcessErr()
if processExitErr == nil {
processExitErr = errCodexProcessExited
}
finalError = processExitErr.Error()
}
}
}
}
duration := time.Since(startTime)
b.cfg.Logger.Info("codex finished", "pid", cmd.Process.Pid, "status", finalStatus, "duration", duration.Round(time.Millisecond).String())
// Run cleanup. drainAndWait handles the graceful-then-cancel pattern
// in two bounded phases (see its declaration): wait for the reader,
// then wait for cmd.Wait(), force-cancelling either if the grace
// window expires. A clean shutdown lets codex flush OTEL telemetry;
// a stuck process is killed via the process-group SIGKILL.
drainAndWait()
if processExitErr != nil {
finalError = withAgentStderr(processExitErr.Error(), "codex", stderrBuf.Tail())
}
if timeoutDiagnostic.Kind != codexTimeoutNone {
timeoutDiagnostic.CodexVersion = detectCodexVersionForDiagnostics(context.Background(), execPath, cmd.Env, b.cfg.Logger)
finalError = buildCodexTimeoutDiagnosticError(timeoutDiagnostic, stderrBuf.Tail())
}
outputMu.Lock()
finalOutput := output.String()
outputMu.Unlock()
// Build usage map from accumulated codex usage.
// First check JSON-RPC notifications (often empty for Codex).
var usageMap map[string]TokenUsage
c.usageMu.Lock()
u := c.usage
c.usageMu.Unlock()
// Fallback: if no usage from JSON-RPC, scan Codex session JSONL logs.
// Codex writes token_count events to $CODEX_HOME/sessions/YYYY/MM/DD/*.jsonl;
// scan this backend's per-task CODEX_HOME, since sessions are isolated
// there rather than in the shared ~/.codex/sessions (MUL-4424).
if u.InputTokens == 0 && u.OutputTokens == 0 {
taskCodexHome := strings.TrimSpace(b.cfg.Env["CODEX_HOME"])
if scanned := scanCodexSessionUsage(startTime, taskCodexHome); scanned != nil {
u = scanned.usage
if scanned.model != "" && opts.Model == "" {
opts.Model = scanned.model
}
}
}
if u.InputTokens > 0 || u.OutputTokens > 0 || u.CacheReadTokens > 0 || u.CacheWriteTokens > 0 {
model := opts.Model
if model == "" {
model = "unknown"
}
usageMap = map[string]TokenUsage{model: u}
}
resCh <- Result{
Status: finalStatus,
Output: finalOutput,
Error: finalError,
SessionID: threadID,
DurationMs: duration.Milliseconds(),
Usage: usageMap,
}
}()
return &Session{Messages: msgCh, Result: resCh}, nil
}
// codexResumeUnavailableNotice is prepended to the first turn's input when a
// resume was expected but Codex ended up on a fresh thread. It mirrors the
// daemon brief's Session Continuity Notice so the disclosure is identical
// whether the loss is detected pre-launch (daemon gate) or only by the live
// thread/resume RPC (MUL-4424).
const codexResumeUnavailableNotice = "[System notice] You were expected to continue an earlier conversation, but restoring that session failed and this is a fresh thread with no memory of the previous turns. Rebuild context from the issue/thread, and when you reply, tell the user up front (one short sentence) that the previous conversation context could not be restored and this is a new session.\n\n"
// codexTurnInput builds the input content for the first turn/start. When a
// resume was expected (resumeExpected) but the backend landed on a fresh thread
// (!resumed), it prepends codexResumeUnavailableNotice so the user learns the
// prior context was lost instead of the run silently continuing as new. The
// notice is folded into the same text block as the prompt to stay within the
// single-text-block turn input Codex already accepts.
func codexTurnInput(prompt string, resumeExpected, resumed bool) []map[string]any {
text := prompt
if resumeExpected && !resumed {
text = codexResumeUnavailableNotice + prompt
}
return []map[string]any{{"type": "text", "text": text}}
}
// startOrResumeThread picks between Codex's thread/resume and thread/start
// based on opts.ResumeSessionID. When a prior thread ID is provided it first
// tries thread/resume; recoverable protocol errors (unknown thread, schema
// mismatch) fall back to thread/start so the task still executes, while
// transport/process failures fail fast because the app-server can no longer
// answer a fresh start request. The returned threadID is what subsequent
// turn/start calls must reference, and resumed indicates whether the prior
// thread was picked up (only useful for logging).
func (c *codexClient) startOrResumeThread(ctx context.Context, opts ExecOptions, logger *slog.Logger) (string, bool, error) {
if priorThreadID := opts.ResumeSessionID; priorThreadID != "" {
// thread/resume reuses the thread's persisted model and reasoning
// effort; only override fields the daemon actually cares about.
resumeParams := map[string]any{
"threadId": priorThreadID,
"cwd": opts.Cwd,
"model": nilIfEmpty(opts.Model),
"developerInstructions": nilIfEmpty(opts.SystemPrompt),
}
// Explicit override of the persisted reasoning effort: without
// this, a Codex resume silently reuses whatever level the prior
// session was created with, even when the user has flipped the
// agent's thinking_level since. See MUL-2339 — Elon flagged that
// resume must honour the live config, not the stored one.
applyCodexReasoningEffort(resumeParams, opts.ThinkingLevel)
resumeResult, err := c.request(ctx, "thread/resume", resumeParams)
if err == nil {
if threadID := extractThreadID(resumeResult); threadID != "" {
return threadID, true, nil
}
logger.Warn("codex thread/resume returned no thread ID; falling back to thread/start", "prior_thread_id", priorThreadID)
} else {
if isCodexTransportError(err) {
logger.Warn("codex thread/resume failed due to transport error; not falling back to thread/start", "prior_thread_id", priorThreadID, "error", err)
return "", false, fmt.Errorf("codex thread/resume failed: %w", err)
}
logger.Warn("codex thread/resume failed; falling back to thread/start", "prior_thread_id", priorThreadID, "error", err)
}
}
startParams := map[string]any{
"model": nilIfEmpty(opts.Model),
"modelProvider": nil,
"profile": nil,
"cwd": opts.Cwd,
"approvalPolicy": nil,
"sandbox": nil,
"config": nil,
"baseInstructions": nil,
"developerInstructions": nilIfEmpty(opts.SystemPrompt),
"compactPrompt": nil,
"includeApplyPatchTool": nil,
"experimentalRawEvents": false,
"persistExtendedHistory": true,
}
applyCodexReasoningEffort(startParams, opts.ThinkingLevel)
startResult, err := c.request(ctx, "thread/start", startParams)
if err != nil {
return "", false, fmt.Errorf("codex thread/start failed: %w", err)
}
threadID := extractThreadID(startResult)
if threadID == "" {
return "", false, fmt.Errorf("codex thread/start returned no thread ID")
}
c.trySetThreadName(ctx, threadID, opts.ThreadName, logger)
return threadID, false, nil
}
func (c *codexClient) trySetThreadName(ctx context.Context, threadID, name string, logger *slog.Logger) {
name = strings.TrimSpace(name)
if name == "" {
return
}
if err := c.setThreadName(ctx, threadID, name); err != nil {
logger.Warn("codex thread/name/set failed; continuing without provider-native thread title",
"thread_id", threadID, "error", err)
}
}
func (c *codexClient) setThreadName(ctx context.Context, threadID, name string) error {
_, err := c.request(ctx, "thread/name/set", map[string]any{
"threadId": threadID,
"name": name,
})
return err
}
// applyCodexReasoningEffort writes the per-agent thinking_level into a
// Codex app-server request. The three points — thread/start.config,
// thread/resume.config, turn/start.effort — all flow through this helper
// so any future protocol/key change touches one site rather than three
// (per Trump's MUL-2339 review constraint).
//
// The shape is detected from the params keys:
// - turn/start always carries `input`, and the schema exposes the
// reasoning override as the top-level `effort` field.
// - thread/start and thread/resume nest it under
// `config.model_reasoning_effort`.
//
// Empty `level` is a no-op: we deliberately do NOT emit a key when the
// caller didn't request an override, so the upstream defaults (config
// file, account-scoped model preference) stay in charge. This also
// guarantees `effort: ""` never reaches the CLI — Codex rejects empty
// strings on this field.
func applyCodexReasoningEffort(params map[string]any, level string) {
if params == nil || level == "" {
return
}
if _, isTurnStart := params["input"]; isTurnStart {
params["effort"] = level
return
}
cfg, _ := params["config"].(map[string]any)
if cfg == nil {
cfg = map[string]any{}
}
cfg["model_reasoning_effort"] = level
params["config"] = cfg
}
func resetTimer(timer *time.Timer, d time.Duration) {
if !timer.Stop() {
select {
case <-timer.C:
default:
}
}
timer.Reset(d)
}
func stopTimer(timer *time.Timer) {
if timer == nil {
return
}
if !timer.Stop() {
select {
case <-timer.C:
default:
}
}
}
func codexFirstTurnNoProgressTimeout(semanticInactivityTimeout time.Duration) time.Duration {
if semanticInactivityTimeout <= 0 || semanticInactivityTimeout > defaultCodexFirstTurnNoProgressTimeout {
return defaultCodexFirstTurnNoProgressTimeout
}
scaled := semanticInactivityTimeout * 4 / 5
if scaled <= 0 {
return semanticInactivityTimeout
}
return scaled
}
func isCodexFirstTurnProgressActivity(activity string) bool {
return activity != "" && activity != "status:running" && activity != "error:retry"
}
func buildCodexTimeoutDiagnosticError(diag codexTimeoutDiagnostic, stderrTail string) string {
var msg string
switch diag.Kind {
case codexTimeoutFirstTurnNoProgress:
msg = fmt.Sprintf("%s after %s: received turn start but no item, message, tool, turn/completed, or error event (%s)",
CodexFirstTurnNoProgressMarker,
diag.Timeout,
formatCodexDiagnosticFields(diag),
)
case codexTimeoutSemanticInactivity:
msg = fmt.Sprintf("%s after %s without agent progress (last activity: %s; %s)",
CodexSemanticInactivityMarker,
diag.Timeout,
nonEmptyCodexDiagnosticValue(diag.LastActivity),
formatCodexDiagnosticFields(diag),
)
default:
msg = "codex timed out"
}
msg = appendCodexKnownStderrHint(msg, stderrTail)
return withAgentStderr(msg, "codex", stderrTail)
}
func formatCodexDiagnosticFields(diag codexTimeoutDiagnostic) string {
return fmt.Sprintf("codex_version=%q thread_id=%q turn_id=%q model=%q",
nonEmptyCodexDiagnosticValue(diag.CodexVersion),
nonEmptyCodexDiagnosticValue(diag.ThreadID),
nonEmptyCodexDiagnosticValue(diag.TurnID),
formatCodexDiagnosticModel(diag.Model),
)
}
func nonEmptyCodexDiagnosticValue(value string) string {
if strings.TrimSpace(value) == "" {
return "unknown"
}
return value
}
func formatCodexDiagnosticModel(model string) string {
if strings.TrimSpace(model) == "" {
return "default(empty)"
}
return model
}
func appendCodexKnownStderrHint(msg, stderrTail string) string {
if strings.Contains(stderrTail, codexModelCatalogRefreshTimeoutSignal) {
return msg + "; diagnosis: Codex stderr shows the model catalog refresh timed out. Try setting an explicit model, switching Codex CLI versions, or using another runtime while Codex app-server recovers"
}
return msg
}
func detectCodexVersionForDiagnostics(ctx context.Context, execPath string, env []string, logger *slog.Logger) string {
versionCtx, cancel := context.WithTimeout(ctx, codexVersionDiagnosticTimeout)
defer cancel()
cmd := exec.CommandContext(versionCtx, execPath, "--version")
cmd.Env = env
data, err := cmd.Output()
if err != nil {
if logger != nil {
logger.Debug("codex version diagnostic failed", "error", err)
}
return "unknown"
}
version := extractVersionLine(string(data))
if strings.TrimSpace(version) == "" {
return "unknown"
}
return version
}
func trySendString(ch chan<- string, value string) {
select {
case ch <- value:
default:
}
}
func logCodexAgentMessage(logger *slog.Logger, msg Message) {
if logger == nil {
return
}
attrs := []any{
"type", string(msg.Type),
"tool", msg.Tool,
"call_id", msg.CallID,
"status", msg.Status,
"content_len", len(msg.Content),
"output_len", len(msg.Output),
}
logger.Info("codex agent message received", attrs...)
if msg.Type == MessageToolResult {
logger.Info("codex tool_result observed", "tool", msg.Tool, "call_id", msg.CallID, "output_len", len(msg.Output))
}
}
func describeCodexSemanticActivity(msg Message) string {
switch msg.Type {
case MessageToolUse, MessageToolResult:
if msg.Tool != "" {
return fmt.Sprintf("%s:%s", msg.Type, msg.Tool)
}
case MessageStatus:
if msg.Status != "" {
return fmt.Sprintf("%s:%s", msg.Type, msg.Status)
}
}
return string(msg.Type)
}
// ── codexClient: JSON-RPC 2.0 transport ──
type codexClient struct {
cfg Config
stdin interface{ Write([]byte) (int, error) }
mu sync.Mutex
nextID int
pending map[int]*pendingRPC
processDone chan struct{}
processErr error
handshakeTimeout time.Duration
threadID string
turnID string
onMessage func(Message)
onSemanticActivity func(description string)
onTurnDone func(aborted bool)
notificationProtocol string // "unknown", "legacy", "raw"
turnStarted bool
completedTurnIDs map[string]bool
usageMu sync.Mutex
usage TokenUsage // accumulated from turn events
turnErrorMu sync.Mutex
turnError string // captured from turn/completed status=failed or terminal error notifications
}
func (c *codexClient) setTurnError(msg string) {
if msg == "" {
return
}
c.turnErrorMu.Lock()
defer c.turnErrorMu.Unlock()
if c.turnError == "" {
c.turnError = msg
}
}
func (c *codexClient) getTurnError() string {
c.turnErrorMu.Lock()
defer c.turnErrorMu.Unlock()
return c.turnError
}
type pendingRPC struct {
ch chan rpcResult
method string
}
type rpcResult struct {
result json.RawMessage
err error
}
type codexHandshakeTimeoutError struct {
Method string
Timeout time.Duration
}
func (e *codexHandshakeTimeoutError) Error() string {
return fmt.Sprintf("%s: %s did not respond after %s", CodexHandshakeTimeoutMarker, e.Method, e.Timeout)
}
func (e *codexHandshakeTimeoutError) Unwrap() error {
return context.DeadlineExceeded
}
func isCodexHandshakeRPC(method string) bool {
switch method {
case "initialize", "thread/start", "thread/resume", "thread/name/set", "turn/start":
return true
default:
return false
}
}
func codexRequestContextError(ctx context.Context) error {
var handshakeErr *codexHandshakeTimeoutError
if errors.As(context.Cause(ctx), &handshakeErr) {
return handshakeErr
}
return ctx.Err()
}
func (c *codexClient) request(ctx context.Context, method string, params any) (json.RawMessage, error) {
if err := ctx.Err(); err != nil {
return nil, err
}
requestCtx := ctx
cancelRequest := func() {}
if c.handshakeTimeout > 0 && isCodexHandshakeRPC(method) {
timeoutErr := &codexHandshakeTimeoutError{Method: method, Timeout: c.handshakeTimeout}
requestCtx, cancelRequest = context.WithTimeoutCause(ctx, c.handshakeTimeout, timeoutErr)
}
defer cancelRequest()
c.mu.Lock()
if c.processErr != nil {
err := c.processErr
c.mu.Unlock()
return nil, err
}
if c.processDone == nil {
c.processDone = make(chan struct{})
}
processDone := c.processDone
c.nextID++
id := c.nextID
pr := &pendingRPC{ch: make(chan rpcResult, 1), method: method}
c.pending[id] = pr
c.mu.Unlock()
msg := map[string]any{
"jsonrpc": "2.0",
"id": id,
"method": method,
"params": params,
}
data, err := json.Marshal(msg)
if err != nil {
c.mu.Lock()
delete(c.pending, id)
c.mu.Unlock()
return nil, err
}
data = append(data, '\n')
if _, err := c.stdin.Write(data); err != nil {
c.mu.Lock()
delete(c.pending, id)
c.mu.Unlock()
return nil, fmt.Errorf("write %s: %w", method, err)
}
if method == "turn/start" {
threadID := ""
if paramMap, ok := params.(map[string]any); ok {
threadID, _ = paramMap["threadId"].(string)
}
c.cfg.Logger.Info("codex turn/start sent", "request_id", id, "thread_id", threadID)
}
select {
case res := <-pr.ch:
return res.result, res.err
case <-processDone:
select {
case res := <-pr.ch:
return res.result, res.err
default:
}
c.mu.Lock()
delete(c.pending, id)
err := c.processErr
c.mu.Unlock()
if requestCtx.Err() != nil {
return nil, codexRequestContextError(requestCtx)
}
if err == nil {
err = errCodexProcessExited
}
return nil, err
case <-requestCtx.Done():
c.mu.Lock()
delete(c.pending, id)
c.mu.Unlock()
return nil, codexRequestContextError(requestCtx)
}
}
func (c *codexClient) notify(method string) {
msg := map[string]any{
"jsonrpc": "2.0",
"method": method,
}
data, _ := json.Marshal(msg)
data = append(data, '\n')
_, _ = c.stdin.Write(data)
}
func (c *codexClient) respond(id int, result any) {
msg := map[string]any{
"jsonrpc": "2.0",
"id": id,
"result": result,
}
data, _ := json.Marshal(msg)
data = append(data, '\n')
_, _ = c.stdin.Write(data)
}
func (c *codexClient) respondError(id int, code int, message string) {
msg := map[string]any{
"jsonrpc": "2.0",
"id": id,
"error": map[string]any{
"code": code,
"message": message,
},
}
data, _ := json.Marshal(msg)
data = append(data, '\n')
_, _ = c.stdin.Write(data)
}
func (c *codexClient) closeAllPending(err error) {
c.mu.Lock()
defer c.mu.Unlock()
for id, pr := range c.pending {
pr.ch <- rpcResult{err: err}
delete(c.pending, id)
}
}
func (c *codexClient) markProcessExited(err error) {
if err == nil {
err = errCodexProcessExited
}
c.mu.Lock()
defer c.mu.Unlock()
if c.processErr == nil {
c.processErr = err
if c.processDone != nil {
close(c.processDone)
}
}
for id, pr := range c.pending {
pr.ch <- rpcResult{err: err}
delete(c.pending, id)
}
}
func (c *codexClient) getProcessErr() error {
c.mu.Lock()
defer c.mu.Unlock()
return c.processErr
}
func isCodexTransportError(err error) bool {
if err == nil {
return false
}
if errors.Is(err, errCodexProcessExited) {
return true
}
var handshakeErr *codexHandshakeTimeoutError
if errors.As(err, &handshakeErr) {
return true
}
return strings.HasPrefix(err.Error(), "write ")
}
func (c *codexClient) handleLine(line string) {
var raw map[string]json.RawMessage
if err := json.Unmarshal([]byte(line), &raw); err != nil {
return
}
// Check if it's a response to our request
if _, hasID := raw["id"]; hasID {
if _, hasResult := raw["result"]; hasResult {
c.handleResponse(raw)
return
}
if _, hasError := raw["error"]; hasError {
c.handleResponse(raw)
return
}
// Server request (has id + method)
if _, hasMethod := raw["method"]; hasMethod {
c.handleServerRequest(raw)
return
}
}
// Notification (no id, has method)
if _, hasMethod := raw["method"]; hasMethod {
c.handleNotification(raw)
}
}
func (c *codexClient) handleResponse(raw map[string]json.RawMessage) {
var id int
if err := json.Unmarshal(raw["id"], &id); err != nil {
return
}
c.mu.Lock()
pr, ok := c.pending[id]
if ok {
delete(c.pending, id)
}
c.mu.Unlock()
if !ok {
return
}
if errData, hasErr := raw["error"]; hasErr {
var rpcErr struct {
Code int `json:"code"`
Message string `json:"message"`
}
_ = json.Unmarshal(errData, &rpcErr)
pr.ch <- rpcResult{err: fmt.Errorf("%s: %s (code=%d)", pr.method, rpcErr.Message, rpcErr.Code)}
} else {
pr.ch <- rpcResult{result: raw["result"]}
}
}
func (c *codexClient) handleServerRequest(raw map[string]json.RawMessage) {
var id int
_ = json.Unmarshal(raw["id"], &id)
var method string
_ = json.Unmarshal(raw["method"], &method)
// Auto-approve all exec/patch requests in daemon mode
switch method {
case "item/commandExecution/requestApproval", "execCommandApproval":
c.respond(id, map[string]any{"decision": "accept"})
case "item/fileChange/requestApproval", "applyPatchApproval":
c.respond(id, map[string]any{"decision": "accept"})
case "item/permissions/requestApproval":
c.respond(id, codexPermissionsApprovalResponse(raw["params"], c.cfg.Logger))
case "mcpServer/elicitation/request":
c.respond(id, map[string]any{"action": "accept", "content": nil, "_meta": nil})
default:
msg := fmt.Sprintf("unsupported codex app-server request: %s", method)
c.cfg.Logger.Warn("codex: unhandled server request", "method", method, "id", id)
c.setTurnError(msg)
c.respondError(id, -32601, msg)
}
}
// codexPermissionsApprovalResponse builds the auto-grant reply for a Codex
// item/permissions/requestApproval server request. In daemon mode there is no
// human to approve, so we echo back the requested network / fileSystem profile
// and scope it to the current turn, mirroring the other auto-accept branches in
// handleServerRequest.
//
// The grant is intentionally limited to the network / fileSystem keys we
// understand. A parse failure and any dropped key are logged so that a future
// app-server protocol that adds a new permission shape is visible in daemon
// logs instead of being silently narrowed away.
func codexPermissionsApprovalResponse(params json.RawMessage, logger *slog.Logger) map[string]any {
var payload struct {
Permissions map[string]any `json:"permissions"`
}
if err := json.Unmarshal(params, &payload); err != nil && logger != nil {
logger.Warn("codex: failed to parse permission approval request; granting empty turn-scoped profile", "error", err)
}
granted := map[string]any{}
var dropped []string
for key, value := range payload.Permissions {
switch key {
case "network", "fileSystem":
if value != nil {
granted[key] = value
}
default:
dropped = append(dropped, key)
}
}
if len(dropped) > 0 && logger != nil {
sort.Strings(dropped)
logger.Warn("codex: dropping unrecognized permission keys from approval request; add explicit handling if the app-server protocol expanded", "keys", dropped)
}
return map[string]any{
"permissions": granted,
"scope": "turn",
}
}
func (c *codexClient) handleNotification(raw map[string]json.RawMessage) {
var method string
_ = json.Unmarshal(raw["method"], &method)
var params map[string]any
if p, ok := raw["params"]; ok {
_ = json.Unmarshal(p, &params)
}
// Legacy codex/event notifications
if method == "codex/event" || strings.HasPrefix(method, "codex/event/") {
c.notificationProtocol = "legacy"
msgData, ok := params["msg"]
if !ok {
return
}
msgMap, ok := msgData.(map[string]any)
if !ok {
return
}
c.handleEvent(msgMap)
return
}
// Raw v2 notifications
if c.notificationProtocol != "legacy" {
if c.notificationProtocol == "unknown" &&
(method == "turn/started" || method == "turn/completed" ||
method == "thread/started" || strings.HasPrefix(method, "item/")) {
c.notificationProtocol = "raw"
}
if c.notificationProtocol == "raw" {
c.handleRawNotification(method, params)
}
}
}
func (c *codexClient) handleEvent(msg map[string]any) {
msgType, _ := msg["type"].(string)
switch msgType {
case "task_started":
c.turnStarted = true
if c.onMessage != nil {
c.onMessage(Message{Type: MessageStatus, Status: "running", SessionID: c.threadID})
}
case "agent_message":
text, _ := msg["message"].(string)
if text != "" && c.onMessage != nil {
c.onMessage(Message{Type: MessageText, Content: text})
}
case "exec_command_begin":
callID, _ := msg["call_id"].(string)
command, _ := msg["command"].(string)
if c.onMessage != nil {
c.onMessage(Message{
Type: MessageToolUse,
Tool: "exec_command",
CallID: callID,
Input: map[string]any{"command": command},
})
}
case "exec_command_end":
callID, _ := msg["call_id"].(string)
output, _ := msg["output"].(string)
if c.onMessage != nil {
c.onMessage(Message{
Type: MessageToolResult,
Tool: "exec_command",
CallID: callID,
Output: output,
})
}
case "patch_apply_begin":
callID, _ := msg["call_id"].(string)
if c.onMessage != nil {
c.onMessage(Message{
Type: MessageToolUse,
Tool: "patch_apply",
CallID: callID,
})
}
case "patch_apply_end":
callID, _ := msg["call_id"].(string)
if c.onMessage != nil {
c.onMessage(Message{
Type: MessageToolResult,
Tool: "patch_apply",
CallID: callID,
})
}
case "task_complete":
// Extract usage from legacy task_complete if present.
c.extractUsageFromMap(msg)
if c.onTurnDone != nil {
c.onTurnDone(false)
}
case "turn_aborted":
if c.onTurnDone != nil {
c.onTurnDone(true)
}
}
}
func (c *codexClient) handleRawNotification(method string, params map[string]any) {
// Ignore notifications from threads other than the one we are tracking.
// Codex multiplexes subagent threads (e.g. memory consolidation) on the
// same stdio pipe; only our thread should drive turn lifecycle and output.
//
// The v2 app-server-protocol schema guarantees a top-level threadId on
// every notification, so this dispatch-level guard transparently covers
// every handler below. If a future codex revision introduces notifications
// without threadId, they fall through (ok=false) — re-audit this guard
// when bumping codex.
if threadID, ok := params["threadId"].(string); ok && c.threadID != "" && threadID != c.threadID {
return
}
switch method {
case "turn/started":
c.turnStarted = true
if turnID := extractNestedString(params, "turn", "id"); turnID != "" {
c.turnID = turnID
}
if c.onMessage != nil {
c.onMessage(Message{Type: MessageStatus, Status: "running", SessionID: c.threadID})
}
case "turn/completed":
turnID := extractNestedString(params, "turn", "id")
status := extractNestedString(params, "turn", "status")
threadID, _ := params["threadId"].(string)
c.cfg.Logger.Info("codex turn/completed received", "thread_id", threadID, "turn_id", turnID, "status", status)
aborted := status == "cancelled" || status == "canceled" ||
status == "aborted" || status == "interrupted"
// Capture the error message from failed turns so callers can surface
// a real reason instead of falling back to "empty output".
if status == "failed" {
errMsg := extractNestedString(params, "turn", "error", "message")
if errMsg == "" {
errMsg = "codex turn failed"
}
c.setTurnError(errMsg)
}
if c.completedTurnIDs == nil {
c.completedTurnIDs = map[string]bool{}
}
if turnID != "" {
if c.completedTurnIDs[turnID] {
return
}
c.completedTurnIDs[turnID] = true
}
// Extract usage from turn/completed if present (e.g. params.turn.usage).
if turn, ok := params["turn"].(map[string]any); ok {
c.extractUsageFromMap(turn)
}
if c.onTurnDone != nil {
c.onTurnDone(aborted)
}
case "error":
// Top-level protocol error. Retrying notifications (willRetry=true) are
// transient reconnect attempts; only capture terminal errors so we
// don't stomp on a real failure later with a retry placeholder.
willRetry, _ := params["willRetry"].(bool)
errMsg := extractNestedString(params, "error", "message")
if errMsg == "" {
errMsg = extractNestedString(params, "message")
}
if errMsg != "" {
c.cfg.Logger.Warn("codex error notification", "message", errMsg, "will_retry", willRetry)
if c.onSemanticActivity != nil {
if willRetry {
c.onSemanticActivity("error:retry")
} else {
c.onSemanticActivity("error:terminal")
}
}
if !willRetry {
c.setTurnError(errMsg)
if c.onTurnDone != nil {
c.onTurnDone(false)
}
}
}
case "thread/status/changed":
statusType := extractNestedString(params, "status", "type")
if statusType == "idle" && c.turnStarted {
if c.onTurnDone != nil {
c.onTurnDone(false)
}
}
default:
if strings.HasPrefix(method, "item/") {
c.handleItemNotification(method, params)
}
}
}
func (c *codexClient) handleItemNotification(method string, params map[string]any) {
item, _ := params["item"].(map[string]any)
itemType, _ := item["type"].(string)
itemID, _ := item["id"].(string)
if isCodexItemProgressActivity(method) && c.onSemanticActivity != nil {
c.onSemanticActivity(describeCodexItemProgressActivity(method, itemType, itemID))
}
if item == nil {
return
}
switch {
case method == "item/started" && itemType == "commandExecution":
command, _ := item["command"].(string)
if c.onMessage != nil {
c.onMessage(Message{
Type: MessageToolUse,
Tool: "exec_command",
CallID: itemID,
Input: map[string]any{"command": command},
})
}
case method == "item/completed" && itemType == "commandExecution":
output, _ := item["aggregatedOutput"].(string)
if c.onMessage != nil {
c.onMessage(Message{
Type: MessageToolResult,
Tool: "exec_command",
CallID: itemID,
Output: output,
})
}
case method == "item/started" && itemType == "fileChange":
if c.onMessage != nil {
c.onMessage(Message{
Type: MessageToolUse,
Tool: "patch_apply",
CallID: itemID,
})
}
case method == "item/completed" && itemType == "fileChange":
if c.onMessage != nil {
c.onMessage(Message{
Type: MessageToolResult,
Tool: "patch_apply",
CallID: itemID,
})
}
case method == "item/completed" && itemType == "agentMessage":
text, _ := item["text"].(string)
if text != "" && c.onMessage != nil {
c.onMessage(Message{Type: MessageText, Content: text})
}
phase, _ := item["phase"].(string)
if phase == "final_answer" && c.turnStarted {
if c.onTurnDone != nil {
c.onTurnDone(false)
}
}
}
}
func isCodexItemProgressActivity(method string) bool {
return strings.HasPrefix(method, "item/")
}
func describeCodexItemProgressActivity(method, itemType, itemID string) string {
if itemType == "" {
itemType = "unknown"
}
if itemID == "" {
return fmt.Sprintf("%s:%s", method, itemType)
}
return fmt.Sprintf("%s:%s:%s", method, itemType, itemID)
}
// extractUsageFromMap extracts token usage from a map that may contain
// "usage", "token_usage", or "tokens" fields. Handles various Codex formats.
func (c *codexClient) extractUsageFromMap(data map[string]any) {
// Try common field names for usage data.
var usageMap map[string]any
for _, key := range []string{"usage", "token_usage", "tokens"} {
if v, ok := data[key].(map[string]any); ok {
usageMap = v
break
}
}
if usageMap == nil {
return
}
c.usageMu.Lock()
defer c.usageMu.Unlock()
// Codex reports cached input as a prompt-token detail: cached_input_tokens
// are included in input_tokens. Persist mutually-exclusive buckets so
// dashboard cost math does not charge cached input twice.
inputTokens := codexInt64(usageMap, "input_tokens", "input", "prompt_tokens")
cacheReadTokens := codexInt64(usageMap, "cached_input_tokens", "cache_read_tokens", "cache_read_input_tokens")
c.usage.InputTokens += codexUncachedInputTokens(inputTokens, cacheReadTokens)
c.usage.OutputTokens += codexInt64(usageMap, "output_tokens", "output", "completion_tokens")
c.usage.CacheReadTokens += cacheReadTokens
c.usage.CacheWriteTokens += codexInt64(usageMap, "cache_write_tokens", "cache_creation_input_tokens")
}
func codexUncachedInputTokens(inputTokens, cachedInputTokens int64) int64 {
uncached := inputTokens - cachedInputTokens
if uncached < 0 {
return 0
}
return uncached
}
// codexInt64 returns the first non-zero int64 value from the map for the given keys.
func codexInt64(m map[string]any, keys ...string) int64 {
for _, key := range keys {
switch v := m[key].(type) {
case float64:
if v != 0 {
return int64(v)
}
case int64:
if v != 0 {
return v
}
}
}
return 0
}
// ── Codex session log scanner ──
// codexSessionUsage holds usage extracted from a Codex session JSONL file.
type codexSessionUsage struct {
usage TokenUsage
model string
}
// scanCodexSessionUsage scans Codex session JSONL files written after startTime
// to extract token usage. Codex writes token_count events to
// $CODEX_HOME/sessions/YYYY/MM/DD/*.jsonl. codexHome is the backend's per-task
// CODEX_HOME; sessions are isolated there rather than in the shared
// ~/.codex/sessions (MUL-4424), so usage must be read from it.
func scanCodexSessionUsage(startTime time.Time, codexHome string) *codexSessionUsage {
root := codexSessionRoot(codexHome)
if root == "" {
return nil
}
// Look in today's session directory.
dateDir := filepath.Join(root,
fmt.Sprintf("%04d", startTime.Year()),
fmt.Sprintf("%02d", int(startTime.Month())),
fmt.Sprintf("%02d", startTime.Day()),
)
files, err := filepath.Glob(filepath.Join(dateDir, "*.jsonl"))
if err != nil || len(files) == 0 {
return nil
}
// Only scan files modified after startTime (this task's session).
var result codexSessionUsage
for _, f := range files {
info, err := os.Stat(f)
if err != nil || info.ModTime().Before(startTime) {
continue
}
if u := parseCodexSessionFile(f); u != nil {
// Take the last matching file's data (usually there's only one per task).
result = *u
}
}
if result.usage.InputTokens == 0 && result.usage.OutputTokens == 0 {
return nil
}
return &result
}
// codexSessionRoot returns the Codex sessions directory. It prefers the
// explicit per-task codexHome the backend is running with (so usage is read
// from the same task-local sessions Codex actually wrote to), then the ambient
// CODEX_HOME, then ~/.codex.
func codexSessionRoot(codexHome string) string {
if codexHome = strings.TrimSpace(codexHome); codexHome == "" {
codexHome = os.Getenv("CODEX_HOME")
}
if codexHome != "" {
dir := filepath.Join(codexHome, "sessions")
if info, err := os.Stat(dir); err == nil && info.IsDir() {
return dir
}
}
home, err := os.UserHomeDir()
if err != nil {
return ""
}
dir := filepath.Join(home, ".codex", "sessions")
if info, err := os.Stat(dir); err == nil && info.IsDir() {
return dir
}
return ""
}
// codexSessionTokenCount represents a token_count event in Codex JSONL.
type codexSessionTokenCount struct {
Type string `json:"type"`
Payload *struct {
Type string `json:"type"`
Info *struct {
TotalTokenUsage *struct {
InputTokens int64 `json:"input_tokens"`
OutputTokens int64 `json:"output_tokens"`
CachedInputTokens int64 `json:"cached_input_tokens"`
CacheReadInputTokens int64 `json:"cache_read_input_tokens"`
ReasoningOutputTokens int64 `json:"reasoning_output_tokens"`
} `json:"total_token_usage"`
LastTokenUsage *struct {
InputTokens int64 `json:"input_tokens"`
OutputTokens int64 `json:"output_tokens"`
CachedInputTokens int64 `json:"cached_input_tokens"`
CacheReadInputTokens int64 `json:"cache_read_input_tokens"`
ReasoningOutputTokens int64 `json:"reasoning_output_tokens"`
} `json:"last_token_usage"`
Model string `json:"model"`
} `json:"info"`
Model string `json:"model"`
} `json:"payload"`
}
// parseCodexSessionFile extracts the final token_count from a Codex session file.
func parseCodexSessionFile(path string) *codexSessionUsage {
f, err := os.Open(path)
if err != nil {
return nil
}
defer f.Close()
var result codexSessionUsage
found := false
scanner := bufio.NewScanner(f)
scanner.Buffer(make([]byte, 0, 256*1024), 1024*1024)
for scanner.Scan() {
line := scanner.Bytes()
// Fast pre-filter.
if !bytesContainsStr(line, "token_count") && !bytesContainsStr(line, "turn_context") {
continue
}
var evt codexSessionTokenCount
if err := json.Unmarshal(line, &evt); err != nil || evt.Payload == nil {
continue
}
// Track model from turn_context events.
if evt.Type == "turn_context" && evt.Payload.Model != "" {
result.model = evt.Payload.Model
continue
}
// Extract token usage from token_count events.
if evt.Payload.Type == "token_count" && evt.Payload.Info != nil {
usage := evt.Payload.Info.TotalTokenUsage
if usage == nil {
usage = evt.Payload.Info.LastTokenUsage
}
if usage != nil {
cachedTokens := usage.CachedInputTokens
if cachedTokens == 0 {
cachedTokens = usage.CacheReadInputTokens
}
result.usage = TokenUsage{
InputTokens: codexUncachedInputTokens(usage.InputTokens, cachedTokens),
OutputTokens: usage.OutputTokens + usage.ReasoningOutputTokens,
CacheReadTokens: cachedTokens,
}
if evt.Payload.Info.Model != "" {
result.model = evt.Payload.Info.Model
}
found = true
}
}
}
if !found {
return nil
}
return &result
}
// bytesContainsStr checks if b contains the string s (without allocating).
func bytesContainsStr(b []byte, s string) bool {
return strings.Contains(string(b), s)
}
// ── Helpers ──
func extractThreadID(result json.RawMessage) string {
var r struct {
Thread struct {
ID string `json:"id"`
} `json:"thread"`
}
if err := json.Unmarshal(result, &r); err != nil {
return ""
}
return r.Thread.ID
}
func extractNestedString(m map[string]any, keys ...string) string {
current := any(m)
for _, key := range keys {
obj, ok := current.(map[string]any)
if !ok {
return ""
}
current = obj[key]
}
s, _ := current.(string)
return s
}
func nilIfEmpty(s string) any {
if s == "" {
return nil
}
return s
}