mirror of
https://github.com/multica-ai/multica.git
synced 2026-07-31 00:40:46 +02:00
* fix(daemon): isolate Codex sessions per task to unblock initialize (MUL-4424) Codex 0.143+ backfills a per-home session-state DB by enumerating every rollout visible under sessions/ during `initialize`. The per-task CODEX_HOME symlinked the shared ~/.codex/sessions in, so a machine with a large accumulated history (one reporter: ~2000 rollouts / ~22 GiB) stalled `initialize` for tens of seconds — the app-server started but the task produced no output before it was cancelled (github #5273). Give each task its own local sessions/ directory instead: - Fresh task: create an empty local sessions/ so backfill is trivial. - Reused task with a real sessions/ dir: it is authoritative — leave it. - Reused task still holding a legacy symlink (older build): migrate in place. Replace the symlink with a real dir; when resuming, symlink only the single rollout being resumed (never copy — a rollout can be GiB and this is on initialize's critical path); and drop the stale, rebuildable session-state DB (state_*.sqlite*, session_index.jsonl) so Codex re-indexes the task-local sessions. Unrelated per-task DBs (goals_*, logs_*, memories_*) are left intact. Also point the token-usage fallback scan at the backend's per-task CODEX_HOME instead of the daemon-global home, so usage isn't lost now that sessions are isolated there. Complements the #5319 handshake watchdog (which turns a silent stall into a loud, phased timeout); this removes the underlying cause. Co-authored-by: multica-agent <github@multica.ai> * fix(daemon): address Codex session isolation review (MUL-4424) Resolves the three blockers from Elon's review of #5360: 1. local_directory context loss. local_directory tasks get a fresh codex-home per task ID (the daemon never reuses their workdir), so task-local isolation stranded every follow-up run with an empty sessions dir and silently restarted the conversation. Their only stable, GC-safe cross-task store is the user's own ~/.codex/sessions (a persistent store under WorkspacesRoot would be orphan-GC'd), so keep the shared-sessions symlink for them (IsLocalDirectory). Managed tasks stay isolated. 2. Migration resume robustness. - Rollout lookup now covers the flat layout and background-compressed .jsonl.zst rollouts, not just nested YYYY/MM/DD *.jsonl — both are legitimate Codex 0.144 history that were previously judged "not found", silently dropping resume. - Exposure hard-links first, then symlinks, never copies — hard links need no privilege and work on Windows within a volume, so the zero-copy path is exercised identically on CI. - The daemon now verifies the rollout is actually present in the task CODEX_HOME (execenv.CodexResumeRolloutPresent) before the brief is generated; if absent it clears the resume from both the backend and the brief instead of telling the agent it is continuing a lost thread. 3. session_index.jsonl is no longer deleted during migration — Codex uses it as the authoritative thread-id -> name store (not rebuildable from rollouts). Only the rebuildable state_*.sqlite* is reset. Tests: 2-round local_directory resume across task IDs; compressed/flat lookup; hard-link zero-copy (os.SameFile); session_index preserved; CodexResumeRolloutPresent + the daemon gate helper (present keeps / absent drops / non-codex + empty no-op). Co-authored-by: multica-agent <github@multica.ai> * fix(daemon): scope Codex sessions to a per-issue store; disclose lost resumes (MUL-4424) Addresses the three blockers from Elon's second review of #5360. 1. local_directory still enumerated the whole machine history. The prior fix re-linked the entire ~/.codex/sessions into every fresh local_directory codex-home, so Codex still backfilled from thousands of unrelated rollouts on `initialize` (measured ~8.3s with 3450 rollouts; the reporter's 22 GiB could exceed the 30s watchdog). Point sessions/ at a persistent, per-(agent, issue) store under the shared Codex home (multica-sessions/<agent>/<issue>) that holds only this issue's rollouts. It is keyed stably across task IDs and lives outside the task-scoped envRoot the GC reclaims, so follow-up runs resume it while `initialize` only ever sees this issue's history. 2. Windows cross-volume resume was lost. Exposing a single rollout by hard-link (same-volume only) then file symlink (needs Windows privilege) can't cross a volume boundary. The store now lives on the shared Codex volume, so the resume rollout is hard-linked there zero-copy, and sessions/ is exposed to the task home via a directory link — a symlink on Unix, a junction on Windows — which crosses volumes without privilege and never copies a (possibly GiB) rollout on initialize's critical path. There is no remaining per-file cross-volume link. 3. An unavailable resume was a silent downgrade. Both resume gates (gateResumeToReusedWorkdir, gateCodexResumeToRolloutPresence) now set PriorSessionResumeUnavailable, and the runtime brief renders a Session Continuity Notice telling the agent to disclose to the user, up front in its reply, that the previous conversation context could not be restored and this run starts fresh — turning a silent restart into a user-visible one. The task is not failed: it can still do useful work without the prior context. Managed fresh / reused-real-dir tasks keep their task-local, GC-collected sessions dir unchanged; only the legacy-symlink migration with a resume routes through the store (cross-volume-safe), and a home already linked to the store is treated as authoritative on reuse. Tests: local_directory per-issue store (only this issue's history, no whole- machine leak); no-key fallback to an empty dir; two-round resume across task IDs through the store; legacy migration routed through the store with a zero-copy hard link; reused store link stays authoritative; both gates set the resume-unavailable flag; brief renders the continuity notice only when a resume was lost. execenv + daemon + pkg/agent packages, go vet, and gofmt all pass. Co-authored-by: multica-agent <github@multica.ai> * fix(daemon): disclose live resume-RPC loss; bound Codex session store lifecycle (MUL-4424) Addresses the two blockers from Elon's third review of #5360. 1. A real thread/resume failure was still a silent new session. The brief's Session Continuity Notice only covers losses the daemon detects before launch (workdir not reused, rollout absent). But when the rollout is present yet Codex rejects the live thread/resume (corrupt/incompatible rollout, server-side thread GC, schema drift), startOrResumeThread falls back to thread/start and the run succeeds on a fresh thread with no user-facing signal. Carry the original resume intent into the backend as ExecOptions.ResumeExpected (set from the post-gate PriorSessionID, so a pre-flight drop still routes through the brief and never double-notifies), and when a resume was expected but the backend landed on a fresh thread, prepend the same continuity notice to the first turn/start input. This also covers the daemon's transport-error fresh-session retry, which clears ResumeSessionID but not ResumeExpected. 2. The persistent per-issue store had no data lifecycle. multica-sessions stores live outside the task-scoped envRoot the GC reclaims (so resume survives across task IDs), which meant a done/abandoned issue's prompts and full rollouts (one reporter: a single 1.5 GiB rollout) accumulated forever and were never freed on issue/agent/workspace deletion. Add PruneCodexSessionStores: the daemon GC loop reclaims any store untouched for GCCodexSessionTTL (default 14 days, configurable via MULTICA_GC_CODEX_SESSION_TTL, 0 disables). A store's newest rollout mtime is its last activity, so an active or recently-resumed task keeps its store fresh and is never reclaimed, while a deleted issue's store ages out — an eventual reclamation guarantee without needing deletion events. Tests: codexTurnInput discloses on resume fallback and stays silent on success / fresh start (paired with the existing live-RPC fallback test); store pruning reclaims aged stores, keeps recent ones, isolates issues, cleans empty agent dirs, and is disable-able. execenv / daemon / pkg/agent, go vet, gofmt all pass. Co-authored-by: multica-agent <github@multica.ai> * fix(daemon): protect a reopened Codex session store from GC mid-mount (MUL-4424) Addresses Elon's fourth-review blocker: reopening an issue idle past GCCodexSessionTTL could lose context, because mounting its per-issue session store (MkdirAll + rollout lookup + task-home link) never refreshed the store's mtime, so a GC cycle firing before the resumed turn wrote its first rollout saw a >TTL-old store and reclaimed it — a stat->remove race with no in-use guard. Two complementary defenses: - Activity refresh: linkCodexSessionsToStore now os.Chtimes the store to now after linking, so codexStoreStat (which reads the newest mtime as last activity) sees a just-used store. This fixes the sequential repro — a mount immediately followed by a prune keeps the store. - In-process active-store guard: the daemon marks the per-issue store in-use (execenv.CodexSessionStorePath) from before Prepare/Reuse mounts it until the task ends, and PruneCodexSessionStores now takes an isActive predicate and skips any store a live task holds. Because prepare and prune run in the same process, this closes the remaining concurrent stat->remove window the mtime refresh alone cannot. Reference-counted, mirroring the env-root guard. Tests: a reopened >TTL store survives a GC cycle after remount and stays resumable; an idle-on-disk store marked active is skipped, then reclaimed once inactive; the existing idle-reclaim / isolation / disable / empty-agent-dir cases still pass. execenv + daemon, go vet, gofmt all pass. Co-authored-by: multica-agent <github@multica.ai> * fix(daemon): make Codex store delete atomic with mark-active (MUL-4424) Addresses Elon's fifth-review blocker: the active-store guard's check and delete were not one atomic step. PruneCodexSessionStores called isActive (which locked, read, and unlocked) and only then RemoveAll'd, leaving a window where a task could markActiveCodexStore between the check and the removal and still lose its store — the exact mark-then-delete interleaving Elon reproduced. Replace the point-in-time isActive predicate with a reserve-for-deletion protocol that shares one lock with mark-active: - reserveCodexStoreForDeletion(store) atomically refuses when a live task holds the store (or another delete already reserved it) and otherwise marks it reserved, all under one activeCodexStoresMu acquisition. PruneCodexSessionStores reserves before RemoveAll and commits after, so confirm-inactive and remove are effectively atomic against a concurrent mark. - markActiveCodexStore now waits (on a sync.Cond) while a store is reserved, so a task never mounts a store mid-removal; committing the removal wakes it and the store is recreated fresh by Prepare (with the continuity notice). So mark-before-reserve keeps the store (reserve refused); reserve-before-mark removes it and blocks the late mark until the removal commits. The genuinely idle case still reclaims. Tests (daemon, run under -race): mark-then-reserve is refused; reserve blocks a concurrent mark until commit then the store reads active; a second reserve is refused mid-flight. The execenv prune tests move to the reserve seam; the activity-refresh / reopen-then-prune / isolation / disable cases still pass. Co-authored-by: multica-agent <github@multica.ai> * fix(daemon): namespace Codex session stores per profile for cross-daemon safety (MUL-4424) Addresses Elon's sixth-review blocker: the in-process reservation guard cannot span processes, but Multica supports multiple profile daemons on one machine (e.g. production + staging) that share the same ~/.codex. Each daemon's GC scanned the whole multica-sessions root, so a staging daemon could reclaim a store a production task was actively resuming — its reservation lived only in the other process's memory. Isolate by profile instead of trying to lock across processes: - Store path is now <shared>/multica-sessions/<namespace>/<agent>/<issue>, where namespace is the daemon's profile (empty -> "default"). PrepareParams/ReuseParams carry Profile; codexSessionStoreKey and CodexSessionStorePath fold it in. - PruneCodexSessionStores takes the profile and scans ONLY that namespace, so a daemon never even sees another profile's stores, let alone deletes them. The per-profile trees are disjoint, so the in-process guard is sufficient within a namespace (profiles get separate daemon state, so no two daemons share one). Test: a "staging"-owned idle store is untouched by a default-profile prune and reclaimed only by staging's own prune. Existing prune/guard/reopen tests move under the namespace. execenv + daemon under -race, go vet, gofmt all pass. Co-authored-by: multica-agent <github@multica.ai> * fix(daemon): make the Codex store profile→namespace map injective (MUL-4424) Addresses Elon's seventh-review blocker: the per-profile namespace was derived by dropping unsafe characters, which is not injective. The CLI treats the empty (default) profile and a profile literally named "default" as separate daemons, yet both mapped to namespace "default"; likewise "staging.prod" and "stagingprod" both mapped to "stagingprod". Two distinct daemons then shared one store tree, so one could again reclaim the other's live session — the cross-process blocker reopened for those profile names. Make codexSessionStoreNamespace injective: the empty profile gets a reserved bare literal "default", and every named profile is hex-encoded (bijective, filesystem-safe) under a "p_" prefix a bare literal can never collide with. So "" -> "default" while "default" -> "p_64656661756c74", and "staging.prod" / "stagingprod" get distinct hex segments. sanitizeCodexPathSegment stays for the UUID agent/issue segments (injective for real UUIDs); only the user-controlled profile needed the encoding. Tests: codexSessionStoreNamespace is distinct for "" vs "default", punctuation variants, case variants, and an encoded-looking name; and end-to-end, pruning one profile never reclaims the other's store for the "" vs "default" and "staging.prod" vs "stagingprod" pairs. execenv + daemon under -race, go vet, gofmt all pass. Co-authored-by: multica-agent <github@multica.ai> * fix(daemon): fixed-length Codex store namespace so long profiles fit (MUL-4424) Addresses Elon's eighth-review blocker: hex-encoding the full profile doubled the namespace segment length. A profile can be as long as a filesystem segment allows (~255 bytes) and the CLI persists it as its own config dir, but the store namespace "p_" + hex(profile) reached 2 + 127*2 = 256 bytes at 127 chars, overflowing the 255-byte single-segment limit — the profile's own dir created fine, then the session store failed with "file name too long". Derive the namespace from a fixed-length hash instead: a named profile is now "p_" + hex(sha256(profile)) — a constant 64 hex chars (66 with the prefix), filesystem-safe and collision-resistant. The empty (default) profile keeps its reserved bare literal "default", which the "p_"-prefixed 66-char segment can never equal. Still injective across the CLI's distinct-daemon cases; just no longer length-expanding. Test: the namespace stays <=255 bytes and creatable for profiles up to the 255-byte segment limit (127- and 255-char cases that overflowed under hex); the prior injectivity and cross-profile prune-isolation tests still hold. execenv + daemon under -race, go vet, gofmt all pass. Co-authored-by: multica-agent <github@multica.ai> --------- Co-authored-by: J <j@multica.ai> Co-authored-by: multica-agent <github@multica.ai>
2304 lines
73 KiB
Go
2304 lines
73 KiB
Go
package agent
|
|
|
|
import (
|
|
"bufio"
|
|
"context"
|
|
"encoding/json"
|
|
"errors"
|
|
"fmt"
|
|
"log/slog"
|
|
"os"
|
|
"os/exec"
|
|
"path/filepath"
|
|
"regexp"
|
|
"sort"
|
|
"strconv"
|
|
"strings"
|
|
"sync"
|
|
"sync/atomic"
|
|
"syscall"
|
|
"time"
|
|
)
|
|
|
|
// codexBlockedArgs are flags hardcoded by the daemon that must not be
|
|
// overridden by user-configured custom_args. The mcp_servers config keys
|
|
// live in the per-task `$CODEX_HOME/config.toml` (written by
|
|
// ensureCodexMcpConfig); user-supplied `-c mcp_servers.…` overrides are
|
|
// stripped separately by filterCodexCustomConfigOverrides because they
|
|
// share the `-c` flag with legitimate non-MCP overrides like `-c model=…`.
|
|
var codexBlockedArgs = map[string]blockedArgMode{
|
|
"--listen": blockedWithValue, // stdio:// transport for daemon communication
|
|
}
|
|
|
|
// codexStderrTailBytes bounds the stderr tail captured for inclusion in
|
|
// error messages when codex exits before the JSON-RPC handshake (e.g. the
|
|
// user supplied a custom_args flag that the `app-server` subcommand
|
|
// rejects). Kept as its own constant so bumping codex independently of
|
|
// other agents stays easy if codex starts shipping longer failure traces.
|
|
const (
|
|
codexStderrTailBytes = 2048
|
|
defaultCodexSemanticInactivityTimeout = 10 * time.Minute
|
|
defaultCodexFirstTurnNoProgressTimeout = 30 * time.Second
|
|
defaultCodexHandshakeTimeout = 30 * time.Second
|
|
codexVersionDiagnosticTimeout = 2 * time.Second
|
|
// codexGracefulShutdownTimeout bounds how long the lifecycle goroutine
|
|
// waits for codex to exit on its own after stdin is closed, before forcing
|
|
// a context-cancel kill. A clean exit lets codex run its shutdown path and
|
|
// flush buffered telemetry — OTEL batch exporters only force-flush on
|
|
// graceful shutdown, so killing it immediately (the prior behavior) drops
|
|
// the task's spans/metrics/logs.
|
|
codexGracefulShutdownTimeout = 10 * time.Second
|
|
)
|
|
|
|
// codexGracefulShutdownTimeoutNanos optionally overrides
|
|
// codexGracefulShutdownTimeout for tests, in nanoseconds. Zero or negative
|
|
// values keep the production default. Tests for the cleanup-on-scanner-
|
|
// overflow path (#4520) use it to shrink the grace window from 10 s to a
|
|
// few hundred ms so the regression runs in a normal `go test` budget
|
|
// instead of burning two full grace windows per cleanup phase. Mirrors
|
|
// the opencodeTerminateGraceNanos hook.
|
|
var codexGracefulShutdownTimeoutNanos atomic.Int64
|
|
|
|
func codexGracefulShutdown() time.Duration {
|
|
if n := codexGracefulShutdownTimeoutNanos.Load(); n > 0 {
|
|
return time.Duration(n)
|
|
}
|
|
return codexGracefulShutdownTimeout
|
|
}
|
|
|
|
// CodexSemanticInactivityMarker prefixes timeout errors emitted when Codex
|
|
// stops making semantic progress while the process is still alive.
|
|
const CodexSemanticInactivityMarker = "codex semantic inactivity timeout"
|
|
|
|
// CodexFirstTurnNoProgressMarker identifies the app-server failure mode where
|
|
// Codex accepts a turn and then never emits any item, completion, or error.
|
|
const CodexFirstTurnNoProgressMarker = "codex app-server no progress timeout"
|
|
|
|
// CodexHandshakeTimeoutMarker identifies a Codex app-server startup RPC that
|
|
// did not answer within the bounded handshake window.
|
|
const CodexHandshakeTimeoutMarker = "codex app-server handshake timeout"
|
|
|
|
const codexModelCatalogRefreshTimeoutSignal = "failed to refresh available models: timeout waiting for child process to exit"
|
|
|
|
var errCodexProcessExited = errors.New("codex process exited")
|
|
|
|
type codexTimeoutKind int
|
|
|
|
const (
|
|
codexTimeoutNone codexTimeoutKind = iota
|
|
codexTimeoutSemanticInactivity
|
|
codexTimeoutFirstTurnNoProgress
|
|
)
|
|
|
|
type codexTimeoutDiagnostic struct {
|
|
Kind codexTimeoutKind
|
|
Timeout time.Duration
|
|
LastActivity string
|
|
ThreadID string
|
|
TurnID string
|
|
Model string
|
|
CodexVersion string
|
|
}
|
|
|
|
// codexBackend implements Backend by spawning `codex app-server --listen stdio://`
|
|
// and communicating via JSON-RPC 2.0 over stdin/stdout.
|
|
type codexBackend struct {
|
|
cfg Config
|
|
}
|
|
|
|
func buildCodexArgs(opts ExecOptions, logger *slog.Logger) []string {
|
|
args := []string{"app-server", "--listen", "stdio://"}
|
|
extra := filterCustomArgs(opts.ExtraArgs, codexBlockedArgs, logger)
|
|
custom := filterCustomArgs(opts.CustomArgs, codexBlockedArgs, logger)
|
|
// Only claim ownership of the `mcp_servers` namespace when the agent
|
|
// actually has a managed mcp_config in the MCP Tab. Otherwise existing
|
|
// users who configure MCP via `custom_args: ["-c", "mcp_servers.…"]`
|
|
// would silently lose those entries after this PR ships. With managed
|
|
// mcp_config present, daemon-written `$CODEX_HOME/config.toml` is the
|
|
// authoritative source and stray `-c mcp_servers.*` overrides are
|
|
// dropped to keep last-wins from re-shadowing it.
|
|
if hasManagedCodexMcpConfig(opts.McpConfig) {
|
|
extra = filterCodexCustomConfigOverrides(extra, logger)
|
|
custom = filterCodexCustomConfigOverrides(custom, logger)
|
|
}
|
|
args = append(args, extra...)
|
|
args = append(args, custom...)
|
|
return args
|
|
}
|
|
|
|
// hasManagedCodexMcpConfig reports whether the agent's mcp_config field is
|
|
// "present" in the API three-state sense: a non-null JSON value. Both
|
|
// `{}` and `{"mcpServers":{}}` count as present (the admin saved an empty
|
|
// managed set — strict mode, no global fallback); only SQL NULL or the
|
|
// literal JSON `null` count as absent (CLI default).
|
|
func hasManagedCodexMcpConfig(raw json.RawMessage) bool {
|
|
return hasManagedMcpConfig(raw)
|
|
}
|
|
|
|
// codexManagedMcpConfigKeyRe matches the daemon-managed config namespace
|
|
// (`mcp_servers.…`) when it appears as the value of a Codex `-c` /
|
|
// `--config` flag. Used by filterCodexCustomConfigOverrides to drop user
|
|
// overrides that would otherwise shadow what the MCP Tab writes into
|
|
// `$CODEX_HOME/config.toml`.
|
|
var codexManagedMcpConfigKeyRe = regexp.MustCompile(`^\s*mcp_servers(?:\s*\.|\s*=|\s*$)`)
|
|
|
|
// filterCodexCustomConfigOverrides drops `-c mcp_servers.…=` and
|
|
// `--config mcp_servers.…=` entries from custom args. Codex's `-c` is
|
|
// last-wins (verified against codex-cli 0.132.0), so without this filter a
|
|
// user-written `-c mcp_servers.fetch=…` in custom_args would silently
|
|
// override whatever the MCP Tab saved into the per-task config.toml. We
|
|
// own the `mcp_servers` namespace via the managed block, so user attempts
|
|
// to write into it are dropped with a warning rather than allowed to win.
|
|
// Other `-c`/`--config` keys (e.g. `-c model="o3"`) pass through unchanged.
|
|
func filterCodexCustomConfigOverrides(args []string, logger *slog.Logger) []string {
|
|
if len(args) == 0 {
|
|
return args
|
|
}
|
|
filtered := make([]string, 0, len(args))
|
|
for i := 0; i < len(args); i++ {
|
|
arg := args[i]
|
|
flag := arg
|
|
inlineValue := ""
|
|
hasInlineValue := false
|
|
if idx := strings.Index(arg, "="); idx > 0 {
|
|
flag = arg[:idx]
|
|
inlineValue = arg[idx+1:]
|
|
hasInlineValue = true
|
|
}
|
|
if flag == "-c" || flag == "--config" {
|
|
value := inlineValue
|
|
if !hasInlineValue && i+1 < len(args) {
|
|
value = args[i+1]
|
|
}
|
|
if codexManagedMcpConfigKeyRe.MatchString(value) {
|
|
if logger != nil {
|
|
// Log the key only, never the value — mcp_servers.<name>.env
|
|
// is allowed to carry secrets and the whole point of moving
|
|
// this to config.toml is to keep raw values out of logs/argv.
|
|
key := value
|
|
if eqIdx := strings.Index(value, "="); eqIdx >= 0 {
|
|
key = value[:eqIdx]
|
|
}
|
|
logger.Warn("custom_args: blocked mcp_servers override; daemon manages this via CODEX_HOME/config.toml",
|
|
"flag", flag, "key", strings.TrimSpace(key))
|
|
}
|
|
if !hasInlineValue && i+1 < len(args) {
|
|
i++ // skip the value arg
|
|
}
|
|
continue
|
|
}
|
|
}
|
|
filtered = append(filtered, arg)
|
|
}
|
|
return filtered
|
|
}
|
|
|
|
// Markers delimiting the daemon-managed `[mcp_servers.*]` block in
|
|
// `$CODEX_HOME/config.toml`. Match the existing sandbox / multi-agent /
|
|
// memory marker pattern so ops can grep all managed blocks consistently.
|
|
const (
|
|
multicaCodexMcpBeginMarker = "# BEGIN multica-managed mcp_servers (do not edit; regenerated by daemon)"
|
|
multicaCodexMcpEndMarker = "# END multica-managed mcp_servers"
|
|
)
|
|
|
|
var codexMcpBlockRe = regexp.MustCompile(
|
|
`(?ms)^` + regexp.QuoteMeta(multicaCodexMcpBeginMarker) +
|
|
`.*?^` + regexp.QuoteMeta(multicaCodexMcpEndMarker) + `\n*`)
|
|
|
|
// userCodexMcpServersTableHeaderRe matches `[mcp_servers.<name>]` (and its
|
|
// quoted-key form `[mcp_servers."<name>"]`) at the start of a line. Used
|
|
// to strip user-provided mcp_servers tables from the per-task config when
|
|
// the agent has its own mcp_config — mirrors Claude's `--strict-mcp-config`
|
|
// model where the daemon's set is authoritative.
|
|
var userCodexMcpServersTableHeaderRe = regexp.MustCompile(
|
|
`^\s*\[\s*mcp_servers\s*\.\s*(?:"[^"]*"|[^\]\s]+)\s*\]\s*(?:#.*)?$`)
|
|
|
|
// ensureCodexMcpConfig writes (or clears) the daemon-managed
|
|
// `[mcp_servers.*]` block in `$CODEX_HOME/config.toml`. The block is the
|
|
// authoritative source of MCP servers for this run: with mcp_config set
|
|
// in the agent UI the daemon also strips any inherited
|
|
// `[mcp_servers.*]` tables from the per-task config so the user's global
|
|
// `~/.codex/config.toml` doesn't shadow or collide with the managed set.
|
|
//
|
|
// The file mode is 0o600 because `mcp_servers.<id>.env` values may carry
|
|
// secrets (API keys, bearer tokens); the per-task home is owned by the
|
|
// daemon's user, so 0o600 keeps secrets out of any world-readable copy
|
|
// while still letting the codex child read them.
|
|
//
|
|
// A malformed mcp_config is returned as an error and the caller decides
|
|
// whether to surface or warn — same fail-soft contract the prior argv
|
|
// path had.
|
|
func ensureCodexMcpConfig(configPath string, mcpConfig json.RawMessage, logger *slog.Logger) error {
|
|
data, err := os.ReadFile(configPath)
|
|
if err != nil && !os.IsNotExist(err) {
|
|
return fmt.Errorf("read config.toml: %w", err)
|
|
}
|
|
existing := string(data)
|
|
|
|
// Always strip a prior managed block so reruns and clear-config flows
|
|
// converge on a clean state.
|
|
stripped := codexMcpBlockRe.ReplaceAllString(existing, "")
|
|
|
|
managed := hasManagedCodexMcpConfig(mcpConfig)
|
|
block, _, renderErr := renderCodexMcpServersBlock(mcpConfig)
|
|
if renderErr != nil {
|
|
return renderErr
|
|
}
|
|
|
|
var updated string
|
|
if managed {
|
|
// Agent has a managed MCP set (possibly empty — `{}` /
|
|
// `{"mcpServers":{}}` count as "saved an empty set" in the API's
|
|
// three-state semantics, distinct from nil/null which means
|
|
// "fall back to CLI default"). Strip any user-defined
|
|
// `[mcp_servers.*]` tables inherited from `~/.codex/config.toml`
|
|
// so the managed set is strict — mirrors Claude's
|
|
// `--strict-mcp-config`. Two reasons we cannot mix:
|
|
// 1. TOML rejects redefining the same table; a user table
|
|
// named `[mcp_servers.fetch]` would crash codex if the
|
|
// agent also defined `fetch`.
|
|
// 2. An admin saving an explicit list in the MCP Tab would
|
|
// otherwise see user-global servers silently joined in,
|
|
// which contradicts the UI affordance.
|
|
stripped = stripCodexUserMcpServerTables(stripped)
|
|
stripped = strings.TrimRight(stripped, "\n")
|
|
// When the managed set is empty we still write the marker
|
|
// block (with no tables between). This pins "managed but
|
|
// empty" on disk so the next run can find and strip the
|
|
// markers, and so the file's intent is grep-able by ops.
|
|
if block == "" {
|
|
block = multicaCodexMcpBeginMarker + "\n" + multicaCodexMcpEndMarker + "\n"
|
|
}
|
|
if stripped == "" {
|
|
updated = block
|
|
} else {
|
|
updated = stripped + "\n\n" + block
|
|
}
|
|
} else {
|
|
// No managed config: just remove any prior managed block and
|
|
// leave inherited user tables alone (CLI default fallback).
|
|
updated = stripped
|
|
}
|
|
|
|
if updated == existing {
|
|
return nil
|
|
}
|
|
if err := os.WriteFile(configPath, []byte(updated), 0o600); err != nil {
|
|
return fmt.Errorf("write config.toml: %w", err)
|
|
}
|
|
// os.WriteFile applies the mode only when creating a new file; if the
|
|
// per-task config.toml was already on disk at 0o644 (the default mode
|
|
// used by execenv.copyFile when seeding from ~/.codex/config.toml),
|
|
// the secret-bearing values we just wrote would inherit that wider
|
|
// mode. Chmod unconditionally to keep the secret in the daemon
|
|
// owner's lane regardless of the prior mode.
|
|
if err := os.Chmod(configPath, 0o600); err != nil {
|
|
return fmt.Errorf("chmod config.toml to 0600: %w", err)
|
|
}
|
|
if logger != nil {
|
|
logger.Debug("codex: wrote managed mcp_servers block to config.toml",
|
|
"config_path", configPath, "managed", managed)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// renderCodexMcpServersBlock renders the agent's mcp_config JSON
|
|
// (Claude-style `{"mcpServers": {...}}`) as a TOML block of
|
|
// `[mcp_servers.<name>]` tables wrapped in BEGIN/END markers. Returns
|
|
// (block, hasServers, err); hasServers=false means the input had no
|
|
// servers to render (empty/null mcp_config) and the caller should only
|
|
// strip the prior managed block.
|
|
//
|
|
// Stdio server keys (`args`, `env`, `command`) pass through verbatim —
|
|
// Codex's config schema happens to use the same names today. Remote HTTP
|
|
// servers use Codex-specific keys, so they are normalised here rather than
|
|
// leaking provider details into the UI/dispatch layer.
|
|
func renderCodexMcpServersBlock(raw json.RawMessage) (string, bool, error) {
|
|
if len(raw) == 0 {
|
|
return "", false, nil
|
|
}
|
|
var parsed struct {
|
|
McpServers map[string]json.RawMessage `json:"mcpServers"`
|
|
}
|
|
if err := json.Unmarshal(raw, &parsed); err != nil {
|
|
return "", false, fmt.Errorf("parse mcp_config json: %w", err)
|
|
}
|
|
if len(parsed.McpServers) == 0 {
|
|
return "", false, nil
|
|
}
|
|
|
|
names := make([]string, 0, len(parsed.McpServers))
|
|
for name := range parsed.McpServers {
|
|
names = append(names, name)
|
|
}
|
|
sort.Strings(names)
|
|
|
|
var sb strings.Builder
|
|
sb.WriteString(multicaCodexMcpBeginMarker)
|
|
sb.WriteString("\n")
|
|
for i, name := range names {
|
|
if !isCodexBareTomlKey(name) {
|
|
return "", false, fmt.Errorf("mcp server name %q must be ASCII alphanumeric / _ / - to fit Codex's bare-key requirement", name)
|
|
}
|
|
var serverVal map[string]any
|
|
if err := json.Unmarshal(parsed.McpServers[name], &serverVal); err != nil {
|
|
return "", false, fmt.Errorf("mcp_servers.%s: %w", name, err)
|
|
}
|
|
if serverVal == nil {
|
|
return "", false, fmt.Errorf("mcp_servers.%s must be a JSON object", name)
|
|
}
|
|
serverVal = normalizeCodexMcpServerConfig(serverVal)
|
|
if i > 0 {
|
|
sb.WriteString("\n")
|
|
}
|
|
sb.WriteString("[mcp_servers.")
|
|
sb.WriteString(name)
|
|
sb.WriteString("]\n")
|
|
keys := make([]string, 0, len(serverVal))
|
|
for k := range serverVal {
|
|
keys = append(keys, k)
|
|
}
|
|
sort.Strings(keys)
|
|
for _, k := range keys {
|
|
tomlValue, err := jsonValueToCodexTOMLInline(serverVal[k])
|
|
if err != nil {
|
|
return "", false, fmt.Errorf("mcp_servers.%s.%s: %w", name, k, err)
|
|
}
|
|
sb.WriteString(codexTOMLKey(k))
|
|
sb.WriteString(" = ")
|
|
sb.WriteString(tomlValue)
|
|
sb.WriteString("\n")
|
|
}
|
|
}
|
|
sb.WriteString(multicaCodexMcpEndMarker)
|
|
sb.WriteString("\n")
|
|
return sb.String(), true, nil
|
|
}
|
|
|
|
func normalizeCodexMcpServerConfig(server map[string]any) map[string]any {
|
|
if !isCodexRemoteMcpServer(server) {
|
|
normalized := make(map[string]any, len(server))
|
|
for k, v := range server {
|
|
if isMulticaMcpSelectorKey(k) {
|
|
continue
|
|
}
|
|
normalized[k] = v
|
|
}
|
|
return normalized
|
|
}
|
|
|
|
normalized := make(map[string]any, len(server)+1)
|
|
for k, v := range server {
|
|
switch {
|
|
case isMulticaMcpSelectorKey(k):
|
|
continue
|
|
case k == "type":
|
|
continue
|
|
case k == "headers":
|
|
if _, ok := server["http_headers"]; !ok {
|
|
normalized["http_headers"] = v
|
|
}
|
|
default:
|
|
normalized[k] = v
|
|
}
|
|
}
|
|
normalized["experimental_use_rmcp_client"] = true
|
|
return normalized
|
|
}
|
|
|
|
func isMulticaMcpSelectorKey(k string) bool {
|
|
switch k {
|
|
case "tools", "prompts", "resources":
|
|
return true
|
|
default:
|
|
return false
|
|
}
|
|
}
|
|
|
|
func isCodexRemoteMcpServer(server map[string]any) bool {
|
|
if typ, ok := server["type"].(string); ok && strings.EqualFold(typ, "http") {
|
|
return true
|
|
}
|
|
_, hasURL := server["url"]
|
|
_, hasCommand := server["command"]
|
|
return hasURL && !hasCommand
|
|
}
|
|
|
|
// stripCodexUserMcpServerTables removes every `[mcp_servers.*]` table
|
|
// (header + body lines until the next top-level table header or EOF) from
|
|
// a TOML config string. Sub-tables like `[mcp_servers.fetch.env]` count
|
|
// as part of the parent table and are dropped along with it.
|
|
func stripCodexUserMcpServerTables(content string) string {
|
|
lines := strings.Split(content, "\n")
|
|
out := make([]string, 0, len(lines))
|
|
skipping := false
|
|
for _, line := range lines {
|
|
if userCodexMcpServersTableHeaderRe.MatchString(line) {
|
|
skipping = true
|
|
continue
|
|
}
|
|
if skipping {
|
|
trimmed := strings.TrimSpace(line)
|
|
if strings.HasPrefix(trimmed, "[") {
|
|
// Next table header. If it's still an `mcp_servers.*`
|
|
// table (including a sub-table), keep skipping; otherwise
|
|
// stop and emit this line.
|
|
if userCodexMcpServersTableHeaderRe.MatchString(line) ||
|
|
strings.HasPrefix(trimmed, "[mcp_servers.") ||
|
|
strings.HasPrefix(trimmed, "[ mcp_servers.") {
|
|
continue
|
|
}
|
|
skipping = false
|
|
out = append(out, line)
|
|
continue
|
|
}
|
|
continue
|
|
}
|
|
out = append(out, line)
|
|
}
|
|
return strings.Join(out, "\n")
|
|
}
|
|
|
|
// jsonValueToCodexTOMLInline serialises a JSON value as a TOML inline
|
|
// value. Only the subset Codex's `-c` accepts is supported: strings,
|
|
// numbers, booleans, arrays, and inline tables. JSON nulls are rejected
|
|
// because TOML has no null and silently dropping them would be confusing.
|
|
func jsonValueToCodexTOMLInline(v any) (string, error) {
|
|
switch x := v.(type) {
|
|
case nil:
|
|
return "", fmt.Errorf("null is not a valid TOML value")
|
|
case bool:
|
|
if x {
|
|
return "true", nil
|
|
}
|
|
return "false", nil
|
|
case float64:
|
|
if x == float64(int64(x)) {
|
|
return strconv.FormatInt(int64(x), 10), nil
|
|
}
|
|
return strconv.FormatFloat(x, 'f', -1, 64), nil
|
|
case string:
|
|
return codexTOMLBasicString(x), nil
|
|
case []any:
|
|
parts := make([]string, len(x))
|
|
for i, e := range x {
|
|
p, err := jsonValueToCodexTOMLInline(e)
|
|
if err != nil {
|
|
return "", err
|
|
}
|
|
parts[i] = p
|
|
}
|
|
return "[" + strings.Join(parts, ", ") + "]", nil
|
|
case map[string]any:
|
|
keys := make([]string, 0, len(x))
|
|
for k := range x {
|
|
keys = append(keys, k)
|
|
}
|
|
sort.Strings(keys)
|
|
parts := make([]string, len(keys))
|
|
for i, k := range keys {
|
|
p, err := jsonValueToCodexTOMLInline(x[k])
|
|
if err != nil {
|
|
return "", err
|
|
}
|
|
parts[i] = codexTOMLKey(k) + " = " + p
|
|
}
|
|
return "{ " + strings.Join(parts, ", ") + " }", nil
|
|
default:
|
|
return "", fmt.Errorf("unsupported value type %T", v)
|
|
}
|
|
}
|
|
|
|
func codexTOMLBasicString(s string) string {
|
|
var sb strings.Builder
|
|
sb.Grow(len(s) + 2)
|
|
sb.WriteByte('"')
|
|
for _, r := range s {
|
|
switch r {
|
|
case '\\':
|
|
sb.WriteString(`\\`)
|
|
case '"':
|
|
sb.WriteString(`\"`)
|
|
case '\b':
|
|
sb.WriteString(`\b`)
|
|
case '\t':
|
|
sb.WriteString(`\t`)
|
|
case '\n':
|
|
sb.WriteString(`\n`)
|
|
case '\f':
|
|
sb.WriteString(`\f`)
|
|
case '\r':
|
|
sb.WriteString(`\r`)
|
|
default:
|
|
if r < 0x20 || r == 0x7f {
|
|
sb.WriteString(fmt.Sprintf(`\u%04x`, r))
|
|
} else {
|
|
sb.WriteRune(r)
|
|
}
|
|
}
|
|
}
|
|
sb.WriteByte('"')
|
|
return sb.String()
|
|
}
|
|
|
|
func codexTOMLKey(s string) string {
|
|
if isCodexBareTomlKey(s) {
|
|
return s
|
|
}
|
|
return codexTOMLBasicString(s)
|
|
}
|
|
|
|
func isCodexBareTomlKey(s string) bool {
|
|
if s == "" {
|
|
return false
|
|
}
|
|
for _, r := range s {
|
|
switch {
|
|
case r >= 'a' && r <= 'z':
|
|
case r >= 'A' && r <= 'Z':
|
|
case r >= '0' && r <= '9':
|
|
case r == '_' || r == '-':
|
|
default:
|
|
return false
|
|
}
|
|
}
|
|
return true
|
|
}
|
|
|
|
func (b *codexBackend) Execute(ctx context.Context, prompt string, opts ExecOptions) (*Session, error) {
|
|
execPath := b.cfg.ExecutablePath
|
|
if execPath == "" {
|
|
execPath = "codex"
|
|
}
|
|
if _, err := exec.LookPath(execPath); err != nil {
|
|
return nil, fmt.Errorf("codex executable not found at %q: %w", execPath, err)
|
|
}
|
|
|
|
timeout := opts.Timeout
|
|
semanticInactivityTimeout := opts.SemanticInactivityTimeout
|
|
if semanticInactivityTimeout == 0 {
|
|
semanticInactivityTimeout = defaultCodexSemanticInactivityTimeout
|
|
}
|
|
handshakeTimeout := opts.HandshakeTimeout
|
|
if handshakeTimeout <= 0 {
|
|
handshakeTimeout = defaultCodexHandshakeTimeout
|
|
}
|
|
runCtx, cancel := runContext(ctx, timeout)
|
|
|
|
// Materialise the agent's MCP config into the per-task
|
|
// `$CODEX_HOME/config.toml`. Argv would be the simpler path, but
|
|
// `mcp_servers.<id>.env` is allowed to carry secrets (Codex docs:
|
|
// https://developers.openai.com/codex/mcp#configure-with-configtoml)
|
|
// and our UI already treats mcp_config as a redacted-for-non-admins
|
|
// field. Process argv ends up in OS-level `ps` listings and is also
|
|
// echoed into the daemon's `agent command` log line below, so any
|
|
// inline env-bearing TOML would defeat the redaction. Writing through
|
|
// config.toml at 0o600 keeps the secret values out of argv and logs.
|
|
if codexHome := strings.TrimSpace(b.cfg.Env["CODEX_HOME"]); codexHome != "" {
|
|
if err := ensureCodexMcpConfig(filepath.Join(codexHome, "config.toml"), opts.McpConfig, b.cfg.Logger); err != nil {
|
|
// Fail closed when we can't materialise the managed config.
|
|
// Warning-and-launching would silently fall back to the
|
|
// user's global `~/.codex/config.toml` MCP servers and
|
|
// look indistinguishable from "the saved config was
|
|
// applied", which is exactly the surprise the MCP Tab is
|
|
// supposed to remove.
|
|
cancel()
|
|
return nil, fmt.Errorf("apply codex mcp_config: %w", err)
|
|
}
|
|
} else if hasManagedCodexMcpConfig(opts.McpConfig) {
|
|
// Managed mcp_config saved but no CODEX_HOME to anchor it.
|
|
// Same reasoning as above: silently launching would inherit
|
|
// whatever MCP setup the host user has, which is the wrong
|
|
// shape of failure.
|
|
cancel()
|
|
return nil, fmt.Errorf("codex: mcp_config is set but CODEX_HOME env var is not configured; cannot apply managed MCP")
|
|
}
|
|
|
|
codexArgs := buildCodexArgs(opts, b.cfg.Logger)
|
|
cmd := exec.CommandContext(runCtx, execPath, codexArgs...)
|
|
hideAgentWindow(cmd)
|
|
// Run codex in its own process group so a cancel-on-stuck cleanup
|
|
// reaches the whole tree — the codex Node wrapper plus the native
|
|
// Rust app-server it spawns — not just the direct child. Without
|
|
// this, killing the leader leaves grandchildren as orphans that
|
|
// keep consuming memory until the OS reaps them; see #4520, where a
|
|
// scanner overflow during thread/resume otherwise leaked Codex
|
|
// processes indefinitely. configureProcessGroup is a no-op on
|
|
// Windows.
|
|
configureProcessGroup(cmd)
|
|
// Override the default exec.CommandContext cancel behaviour. The
|
|
// default sends SIGKILL only to cmd.Process (the leader); we instead
|
|
// signal the whole process group so descendants die too. Returning
|
|
// nil keeps exec from logging a spurious error; cmd.WaitDelay below
|
|
// still backstops cmd.Wait() if the kill leaves an open pipe.
|
|
cmd.Cancel = func() error {
|
|
if cmd.Process != nil {
|
|
signalProcessGroup(cmd.Process, syscall.SIGKILL)
|
|
}
|
|
return nil
|
|
}
|
|
// Bound the wait after the context is cancelled so a stuck child (or an
|
|
// open pipe held by a grandchild) can't hang cmd.Wait() forever. Matches
|
|
// the other long-lived backends (claude, copilot, cursor, …).
|
|
cmd.WaitDelay = 10 * time.Second
|
|
b.cfg.Logger.Info("agent command", "exec", execPath, "args", codexArgs)
|
|
if opts.Cwd != "" {
|
|
cmd.Dir = opts.Cwd
|
|
}
|
|
cmd.Env = buildEnv(b.cfg.Env)
|
|
|
|
stdout, err := cmd.StdoutPipe()
|
|
if err != nil {
|
|
cancel()
|
|
return nil, fmt.Errorf("codex stdout pipe: %w", err)
|
|
}
|
|
stdin, err := cmd.StdinPipe()
|
|
if err != nil {
|
|
cancel()
|
|
return nil, fmt.Errorf("codex stdin pipe: %w", err)
|
|
}
|
|
stderrBuf := newStderrTail(newLogWriter(b.cfg.Logger, "[codex:stderr] "), codexStderrTailBytes)
|
|
cmd.Stderr = stderrBuf
|
|
|
|
if err := cmd.Start(); err != nil {
|
|
cancel()
|
|
return nil, fmt.Errorf("start codex: %w", err)
|
|
}
|
|
|
|
b.cfg.Logger.Info("codex started app-server", "pid", cmd.Process.Pid, "cwd", opts.Cwd)
|
|
|
|
msgCh := make(chan Message, 256)
|
|
resCh := make(chan Result, 1)
|
|
semanticActivityCh := make(chan string, 256)
|
|
|
|
var outputMu sync.Mutex
|
|
var output strings.Builder
|
|
|
|
// turnDone is set before starting the reader goroutine so there is no
|
|
// race between the lifecycle goroutine writing and the reader reading.
|
|
turnDone := make(chan bool, 1) // true = aborted
|
|
|
|
c := &codexClient{
|
|
cfg: b.cfg,
|
|
stdin: stdin,
|
|
pending: make(map[int]*pendingRPC),
|
|
processDone: make(chan struct{}),
|
|
handshakeTimeout: handshakeTimeout,
|
|
notificationProtocol: "unknown",
|
|
onMessage: func(msg Message) {
|
|
logCodexAgentMessage(b.cfg.Logger, msg)
|
|
if msg.Type == MessageText {
|
|
outputMu.Lock()
|
|
output.WriteString(msg.Content)
|
|
outputMu.Unlock()
|
|
}
|
|
trySend(msgCh, msg)
|
|
trySendString(semanticActivityCh, describeCodexSemanticActivity(msg))
|
|
},
|
|
onSemanticActivity: func(description string) {
|
|
b.cfg.Logger.Debug("codex semantic activity observed", "activity", description)
|
|
trySendString(semanticActivityCh, description)
|
|
},
|
|
onTurnDone: func(aborted bool) {
|
|
select {
|
|
case turnDone <- aborted:
|
|
default:
|
|
}
|
|
},
|
|
}
|
|
|
|
// Start reading stdout in background
|
|
readerDone := make(chan struct{})
|
|
go func() {
|
|
defer close(readerDone)
|
|
scanner := bufio.NewScanner(stdout)
|
|
scanner.Buffer(make([]byte, 0, 1024*1024), 10*1024*1024)
|
|
for scanner.Scan() {
|
|
line := strings.TrimSpace(scanner.Text())
|
|
if line == "" {
|
|
continue
|
|
}
|
|
c.handleLine(line)
|
|
}
|
|
if err := scanner.Err(); err != nil {
|
|
c.markProcessExited(fmt.Errorf("%w: %v", errCodexProcessExited, err))
|
|
return
|
|
}
|
|
c.markProcessExited(errCodexProcessExited)
|
|
}()
|
|
|
|
// drainAndWait closes stdin so codex shuts down, then joins cmd.Wait().
|
|
// cmd.Wait() is the only Go-stdlib-documented synchronization point for
|
|
// os/exec's internal stderr/stdout copy goroutines — until it returns,
|
|
// stderrBuf may not have observed every byte codex wrote before it
|
|
// exited, and stderrBuf.Tail() can come back empty or truncated. Any
|
|
// code that reads stderrBuf.Tail() must call drainAndWait() first.
|
|
// sync.Once makes it safe to call from both error paths and the deferred
|
|
// cleanup.
|
|
//
|
|
// drainAndWait is also the cleanup safety net for the scanner-overflow
|
|
// path (#4520). When codex emits a single stdout line larger than the
|
|
// scanner's MaxScanTokenSize, the reader goroutine returns with
|
|
// scanner.Err() set, fails all in-flight RPCs via markProcessExited, and
|
|
// closes readerDone — but the codex child process is still alive and is
|
|
// now blocked trying to write the rest of the oversized line into a
|
|
// stdout pipe nobody is reading. A naive stdin.Close()+cmd.Wait() then
|
|
// hangs forever: codex never reaches its stdin-read syscall, so it never
|
|
// sees EOF, never exits, and cmd.Wait() never returns. The lifecycle
|
|
// goroutine therefore never sends a failed Result, the outer daemon
|
|
// blocks on its result channel, and the higher-level fresh-session
|
|
// fallback never fires.
|
|
//
|
|
// To stay correct under both clean shutdown and the stuck-child case,
|
|
// drainAndWait runs in two bounded phases:
|
|
//
|
|
// 1. Close stdin and wait for the reader goroutine to finish, capped by
|
|
// codexGracefulShutdownTimeout. The reader exits when codex closes
|
|
// stdout on its own (clean shutdown — gives OTEL batch exporters a
|
|
// chance to flush) OR when the scanner errors out (overflow case —
|
|
// readerDone is already closed and the select returns immediately).
|
|
// Per os/exec docs, calling cmd.Wait() while reads are still
|
|
// in-flight on a StdoutPipe-returned pipe is incorrect because Wait
|
|
// closes the pipe and turns pending reads into spurious errors, so
|
|
// we must wait for the reader first.
|
|
//
|
|
// 2. Wait for cmd.Wait() to return, capped by another
|
|
// codexGracefulShutdownTimeout. Normally this returns immediately
|
|
// because the process has already exited. In the stuck-child case
|
|
// the process is still alive — we cancel the runCtx, which fires
|
|
// cmd.Cancel (the group-SIGKILL helper installed above), and
|
|
// cmd.WaitDelay then guarantees cmd.Wait() returns even if pipes
|
|
// stay open.
|
|
var waitOnce sync.Once
|
|
drainAndWait := func() {
|
|
waitOnce.Do(func() {
|
|
stdin.Close()
|
|
|
|
grace := codexGracefulShutdown()
|
|
|
|
// Phase 1: let the reader finish before invoking cmd.Wait().
|
|
select {
|
|
case <-readerDone:
|
|
// reader drained cleanly (codex shutdown closed stdout)
|
|
// or aborted early (e.g. scanner overflow). Either way it
|
|
// is now safe to call cmd.Wait().
|
|
case <-time.After(grace):
|
|
// codex did not close stdout within the grace window. Force
|
|
// the shutdown via context cancellation — cmd.Cancel
|
|
// group-kills the tree, the reader unblocks when stdout
|
|
// EOFs, and we proceed to phase 2.
|
|
b.cfg.Logger.Warn("codex did not close stdout after stdin EOF; forcing shutdown",
|
|
"pid", cmd.Process.Pid,
|
|
"grace", grace.String(),
|
|
)
|
|
cancel()
|
|
<-readerDone
|
|
}
|
|
|
|
// Phase 2: bound cmd.Wait() in case the process is still alive
|
|
// (scanner-overflow case: reader exited early on its own while
|
|
// codex stayed blocked writing into a full stdout pipe).
|
|
waitCh := make(chan struct{})
|
|
go func() {
|
|
_ = cmd.Wait()
|
|
close(waitCh)
|
|
}()
|
|
select {
|
|
case <-waitCh:
|
|
// reaped cleanly.
|
|
case <-time.After(grace):
|
|
b.cfg.Logger.Warn("codex process still alive after reader exited; forcing shutdown",
|
|
"pid", cmd.Process.Pid,
|
|
"grace", grace.String(),
|
|
)
|
|
cancel()
|
|
// WaitDelay (10s) is the final backstop: even if the
|
|
// group-kill races with an open pipe held by a
|
|
// descendant, cmd.Wait() returns within WaitDelay of the
|
|
// cancel.
|
|
<-waitCh
|
|
}
|
|
})
|
|
}
|
|
|
|
// Drive the session lifecycle in a goroutine.
|
|
// Shutdown sequence: lifecycle goroutine closes stdin + cancels context →
|
|
// codex process exits → reader goroutine's scanner.Scan() returns false →
|
|
// readerDone closes → lifecycle goroutine collects final output and sends Result.
|
|
go func() {
|
|
defer cancel()
|
|
defer close(msgCh)
|
|
defer close(resCh)
|
|
defer drainAndWait()
|
|
|
|
startTime := time.Now()
|
|
finalStatus := "completed"
|
|
var finalError string
|
|
|
|
// 1. Initialize handshake
|
|
_, err := c.request(runCtx, "initialize", map[string]any{
|
|
"clientInfo": map[string]any{
|
|
"name": "multica-agent-sdk",
|
|
"title": "Multica Agent SDK",
|
|
"version": "0.2.0",
|
|
},
|
|
"capabilities": map[string]any{
|
|
"experimentalApi": true,
|
|
},
|
|
})
|
|
if err != nil {
|
|
drainAndWait() // flush os/exec stderr goroutine before sampling Tail
|
|
finalStatus = "failed"
|
|
finalError = withAgentStderr(fmt.Sprintf("codex initialize failed: %v", err), "codex", stderrBuf.Tail())
|
|
resCh <- Result{Status: finalStatus, Error: finalError, DurationMs: time.Since(startTime).Milliseconds()}
|
|
return
|
|
}
|
|
c.notify("initialized")
|
|
|
|
// 2. Start a new thread, or resume the prior one for this issue. When
|
|
// resume fails (thread GCed on the server, schema drift, etc.) we fall
|
|
// back to a fresh thread so the task still makes progress.
|
|
threadID, resumed, err := c.startOrResumeThread(runCtx, opts, b.cfg.Logger)
|
|
if err != nil {
|
|
drainAndWait() // flush os/exec stderr goroutine before sampling Tail
|
|
finalStatus = "failed"
|
|
finalError = withAgentStderr(err.Error(), "codex", stderrBuf.Tail())
|
|
resCh <- Result{Status: finalStatus, Error: finalError, DurationMs: time.Since(startTime).Milliseconds()}
|
|
return
|
|
}
|
|
c.threadID = threadID
|
|
if resumed {
|
|
b.cfg.Logger.Info("codex thread resumed", "thread_id", threadID)
|
|
} else {
|
|
b.cfg.Logger.Info("codex thread started", "thread_id", threadID)
|
|
}
|
|
|
|
// 3. Send turn and wait for completion. When a resume was expected but we
|
|
// ended up on a fresh thread (the live thread/resume RPC was rejected — a
|
|
// corrupt/incompatible rollout, server-side thread GC, schema drift — or a
|
|
// transport failure forced a fresh retry), prepend a continuity notice so
|
|
// the agent tells the user the prior conversation could not be restored.
|
|
// The daemon's pre-flight gates only catch cases detectable before launch;
|
|
// this covers the ones only the live resume reveals (MUL-4424).
|
|
turnParams := map[string]any{
|
|
"threadId": threadID,
|
|
"input": codexTurnInput(prompt, opts.ResumeExpected, resumed),
|
|
}
|
|
// Per-turn reasoning override. Mirrors the per-thread injection in
|
|
// startOrResumeThread; keeping both in sync is enforced by the
|
|
// shared `codexReasoningInjection` fixture in codex_test.go (see
|
|
// MUL-2339 — Trump's constraint that the three injection points
|
|
// must not drift independently).
|
|
applyCodexReasoningEffort(turnParams, opts.ThinkingLevel)
|
|
waitingForTurn := true
|
|
var timeoutDiagnostic codexTimeoutDiagnostic
|
|
var processExitErr error
|
|
finishTurn := func(aborted bool) {
|
|
waitingForTurn = false
|
|
switch {
|
|
case aborted:
|
|
finalStatus = "aborted"
|
|
if errMsg := c.getTurnError(); errMsg != "" {
|
|
finalError = errMsg
|
|
} else {
|
|
finalError = "turn was aborted"
|
|
}
|
|
default:
|
|
if errMsg := c.getTurnError(); errMsg != "" {
|
|
finalStatus = "failed"
|
|
finalError = errMsg
|
|
}
|
|
}
|
|
}
|
|
_, err = c.request(runCtx, "turn/start", turnParams)
|
|
if err != nil {
|
|
select {
|
|
case aborted := <-turnDone:
|
|
finishTurn(aborted)
|
|
default:
|
|
drainAndWait() // flush os/exec stderr goroutine before sampling Tail
|
|
finalStatus = "failed"
|
|
finalError = withAgentStderr(fmt.Sprintf("codex turn/start failed: %v", err), "codex", stderrBuf.Tail())
|
|
resCh <- Result{Status: finalStatus, Error: finalError, DurationMs: time.Since(startTime).Milliseconds()}
|
|
return
|
|
}
|
|
}
|
|
|
|
lastSemanticActivity := time.Now()
|
|
lastSemanticActivityDescription := "turn/start"
|
|
semanticTimer := time.NewTimer(semanticInactivityTimeout)
|
|
defer semanticTimer.Stop()
|
|
|
|
firstTurnNoProgressTimeout := codexFirstTurnNoProgressTimeout(semanticInactivityTimeout)
|
|
var firstTurnNoProgressTimer *time.Timer
|
|
var firstTurnNoProgressTimerC <-chan time.Time
|
|
firstTurnStarted := false
|
|
firstTurnProgressObserved := false
|
|
stopFirstTurnNoProgressTimer := func() {
|
|
if firstTurnNoProgressTimer == nil {
|
|
return
|
|
}
|
|
stopTimer(firstTurnNoProgressTimer)
|
|
firstTurnNoProgressTimerC = nil
|
|
}
|
|
defer stopFirstTurnNoProgressTimer()
|
|
|
|
finishRunContextDone := func() {
|
|
waitingForTurn = false
|
|
if runCtx.Err() == context.DeadlineExceeded {
|
|
finalStatus = "timeout"
|
|
finalError = fmt.Sprintf("codex timed out after %s", timeout)
|
|
} else {
|
|
finalStatus = "aborted"
|
|
finalError = "execution cancelled"
|
|
}
|
|
}
|
|
for waitingForTurn {
|
|
select {
|
|
case aborted := <-turnDone:
|
|
finishTurn(aborted)
|
|
case activity := <-semanticActivityCh:
|
|
lastSemanticActivity = time.Now()
|
|
lastSemanticActivityDescription = activity
|
|
resetTimer(semanticTimer, semanticInactivityTimeout)
|
|
if activity == "status:running" && !firstTurnStarted {
|
|
firstTurnStarted = true
|
|
firstTurnNoProgressTimer = time.NewTimer(firstTurnNoProgressTimeout)
|
|
firstTurnNoProgressTimerC = firstTurnNoProgressTimer.C
|
|
} else if firstTurnStarted && !firstTurnProgressObserved && isCodexFirstTurnProgressActivity(activity) {
|
|
firstTurnProgressObserved = true
|
|
stopFirstTurnNoProgressTimer()
|
|
}
|
|
case <-firstTurnNoProgressTimerC:
|
|
waitingForTurn = false
|
|
finalStatus = "timeout"
|
|
timeoutDiagnostic = codexTimeoutDiagnostic{
|
|
Kind: codexTimeoutFirstTurnNoProgress,
|
|
Timeout: firstTurnNoProgressTimeout,
|
|
LastActivity: lastSemanticActivityDescription,
|
|
ThreadID: threadID,
|
|
TurnID: c.turnID,
|
|
Model: opts.Model,
|
|
}
|
|
b.cfg.Logger.Warn(CodexFirstTurnNoProgressMarker,
|
|
"pid", cmd.Process.Pid,
|
|
"thread_id", threadID,
|
|
"turn_id", c.turnID,
|
|
"timeout", firstTurnNoProgressTimeout.String(),
|
|
"last_activity", lastSemanticActivityDescription,
|
|
)
|
|
case <-semanticTimer.C:
|
|
waitingForTurn = false
|
|
finalStatus = "timeout"
|
|
timeoutDiagnostic = codexTimeoutDiagnostic{
|
|
Kind: codexTimeoutSemanticInactivity,
|
|
Timeout: semanticInactivityTimeout,
|
|
LastActivity: lastSemanticActivityDescription,
|
|
ThreadID: threadID,
|
|
TurnID: c.turnID,
|
|
Model: opts.Model,
|
|
}
|
|
b.cfg.Logger.Warn(CodexSemanticInactivityMarker,
|
|
"pid", cmd.Process.Pid,
|
|
"thread_id", threadID,
|
|
"turn_id", c.turnID,
|
|
"timeout", semanticInactivityTimeout.String(),
|
|
"last_activity", lastSemanticActivityDescription,
|
|
"idle_for", time.Since(lastSemanticActivity).Round(time.Millisecond).String(),
|
|
)
|
|
case <-runCtx.Done():
|
|
finishRunContextDone()
|
|
case <-c.processDone:
|
|
select {
|
|
case aborted := <-turnDone:
|
|
finishTurn(aborted)
|
|
default:
|
|
if runCtx.Err() != nil {
|
|
finishRunContextDone()
|
|
} else {
|
|
waitingForTurn = false
|
|
finalStatus = "failed"
|
|
processExitErr = c.getProcessErr()
|
|
if processExitErr == nil {
|
|
processExitErr = errCodexProcessExited
|
|
}
|
|
finalError = processExitErr.Error()
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
duration := time.Since(startTime)
|
|
b.cfg.Logger.Info("codex finished", "pid", cmd.Process.Pid, "status", finalStatus, "duration", duration.Round(time.Millisecond).String())
|
|
|
|
// Run cleanup. drainAndWait handles the graceful-then-cancel pattern
|
|
// in two bounded phases (see its declaration): wait for the reader,
|
|
// then wait for cmd.Wait(), force-cancelling either if the grace
|
|
// window expires. A clean shutdown lets codex flush OTEL telemetry;
|
|
// a stuck process is killed via the process-group SIGKILL.
|
|
drainAndWait()
|
|
|
|
if processExitErr != nil {
|
|
finalError = withAgentStderr(processExitErr.Error(), "codex", stderrBuf.Tail())
|
|
}
|
|
if timeoutDiagnostic.Kind != codexTimeoutNone {
|
|
timeoutDiagnostic.CodexVersion = detectCodexVersionForDiagnostics(context.Background(), execPath, cmd.Env, b.cfg.Logger)
|
|
finalError = buildCodexTimeoutDiagnosticError(timeoutDiagnostic, stderrBuf.Tail())
|
|
}
|
|
|
|
outputMu.Lock()
|
|
finalOutput := output.String()
|
|
outputMu.Unlock()
|
|
|
|
// Build usage map from accumulated codex usage.
|
|
// First check JSON-RPC notifications (often empty for Codex).
|
|
var usageMap map[string]TokenUsage
|
|
c.usageMu.Lock()
|
|
u := c.usage
|
|
c.usageMu.Unlock()
|
|
|
|
// Fallback: if no usage from JSON-RPC, scan Codex session JSONL logs.
|
|
// Codex writes token_count events to $CODEX_HOME/sessions/YYYY/MM/DD/*.jsonl;
|
|
// scan this backend's per-task CODEX_HOME, since sessions are isolated
|
|
// there rather than in the shared ~/.codex/sessions (MUL-4424).
|
|
if u.InputTokens == 0 && u.OutputTokens == 0 {
|
|
taskCodexHome := strings.TrimSpace(b.cfg.Env["CODEX_HOME"])
|
|
if scanned := scanCodexSessionUsage(startTime, taskCodexHome); scanned != nil {
|
|
u = scanned.usage
|
|
if scanned.model != "" && opts.Model == "" {
|
|
opts.Model = scanned.model
|
|
}
|
|
}
|
|
}
|
|
|
|
if u.InputTokens > 0 || u.OutputTokens > 0 || u.CacheReadTokens > 0 || u.CacheWriteTokens > 0 {
|
|
model := opts.Model
|
|
if model == "" {
|
|
model = "unknown"
|
|
}
|
|
usageMap = map[string]TokenUsage{model: u}
|
|
}
|
|
|
|
resCh <- Result{
|
|
Status: finalStatus,
|
|
Output: finalOutput,
|
|
Error: finalError,
|
|
SessionID: threadID,
|
|
DurationMs: duration.Milliseconds(),
|
|
Usage: usageMap,
|
|
}
|
|
}()
|
|
|
|
return &Session{Messages: msgCh, Result: resCh}, nil
|
|
}
|
|
|
|
// codexResumeUnavailableNotice is prepended to the first turn's input when a
|
|
// resume was expected but Codex ended up on a fresh thread. It mirrors the
|
|
// daemon brief's Session Continuity Notice so the disclosure is identical
|
|
// whether the loss is detected pre-launch (daemon gate) or only by the live
|
|
// thread/resume RPC (MUL-4424).
|
|
const codexResumeUnavailableNotice = "[System notice] You were expected to continue an earlier conversation, but restoring that session failed and this is a fresh thread with no memory of the previous turns. Rebuild context from the issue/thread, and when you reply, tell the user up front (one short sentence) that the previous conversation context could not be restored and this is a new session.\n\n"
|
|
|
|
// codexTurnInput builds the input content for the first turn/start. When a
|
|
// resume was expected (resumeExpected) but the backend landed on a fresh thread
|
|
// (!resumed), it prepends codexResumeUnavailableNotice so the user learns the
|
|
// prior context was lost instead of the run silently continuing as new. The
|
|
// notice is folded into the same text block as the prompt to stay within the
|
|
// single-text-block turn input Codex already accepts.
|
|
func codexTurnInput(prompt string, resumeExpected, resumed bool) []map[string]any {
|
|
text := prompt
|
|
if resumeExpected && !resumed {
|
|
text = codexResumeUnavailableNotice + prompt
|
|
}
|
|
return []map[string]any{{"type": "text", "text": text}}
|
|
}
|
|
|
|
// startOrResumeThread picks between Codex's thread/resume and thread/start
|
|
// based on opts.ResumeSessionID. When a prior thread ID is provided it first
|
|
// tries thread/resume; recoverable protocol errors (unknown thread, schema
|
|
// mismatch) fall back to thread/start so the task still executes, while
|
|
// transport/process failures fail fast because the app-server can no longer
|
|
// answer a fresh start request. The returned threadID is what subsequent
|
|
// turn/start calls must reference, and resumed indicates whether the prior
|
|
// thread was picked up (only useful for logging).
|
|
func (c *codexClient) startOrResumeThread(ctx context.Context, opts ExecOptions, logger *slog.Logger) (string, bool, error) {
|
|
if priorThreadID := opts.ResumeSessionID; priorThreadID != "" {
|
|
// thread/resume reuses the thread's persisted model and reasoning
|
|
// effort; only override fields the daemon actually cares about.
|
|
resumeParams := map[string]any{
|
|
"threadId": priorThreadID,
|
|
"cwd": opts.Cwd,
|
|
"model": nilIfEmpty(opts.Model),
|
|
"developerInstructions": nilIfEmpty(opts.SystemPrompt),
|
|
}
|
|
// Explicit override of the persisted reasoning effort: without
|
|
// this, a Codex resume silently reuses whatever level the prior
|
|
// session was created with, even when the user has flipped the
|
|
// agent's thinking_level since. See MUL-2339 — Elon flagged that
|
|
// resume must honour the live config, not the stored one.
|
|
applyCodexReasoningEffort(resumeParams, opts.ThinkingLevel)
|
|
resumeResult, err := c.request(ctx, "thread/resume", resumeParams)
|
|
if err == nil {
|
|
if threadID := extractThreadID(resumeResult); threadID != "" {
|
|
return threadID, true, nil
|
|
}
|
|
logger.Warn("codex thread/resume returned no thread ID; falling back to thread/start", "prior_thread_id", priorThreadID)
|
|
} else {
|
|
if isCodexTransportError(err) {
|
|
logger.Warn("codex thread/resume failed due to transport error; not falling back to thread/start", "prior_thread_id", priorThreadID, "error", err)
|
|
return "", false, fmt.Errorf("codex thread/resume failed: %w", err)
|
|
}
|
|
logger.Warn("codex thread/resume failed; falling back to thread/start", "prior_thread_id", priorThreadID, "error", err)
|
|
}
|
|
}
|
|
|
|
startParams := map[string]any{
|
|
"model": nilIfEmpty(opts.Model),
|
|
"modelProvider": nil,
|
|
"profile": nil,
|
|
"cwd": opts.Cwd,
|
|
"approvalPolicy": nil,
|
|
"sandbox": nil,
|
|
"config": nil,
|
|
"baseInstructions": nil,
|
|
"developerInstructions": nilIfEmpty(opts.SystemPrompt),
|
|
"compactPrompt": nil,
|
|
"includeApplyPatchTool": nil,
|
|
"experimentalRawEvents": false,
|
|
"persistExtendedHistory": true,
|
|
}
|
|
applyCodexReasoningEffort(startParams, opts.ThinkingLevel)
|
|
startResult, err := c.request(ctx, "thread/start", startParams)
|
|
if err != nil {
|
|
return "", false, fmt.Errorf("codex thread/start failed: %w", err)
|
|
}
|
|
threadID := extractThreadID(startResult)
|
|
if threadID == "" {
|
|
return "", false, fmt.Errorf("codex thread/start returned no thread ID")
|
|
}
|
|
c.trySetThreadName(ctx, threadID, opts.ThreadName, logger)
|
|
return threadID, false, nil
|
|
}
|
|
|
|
func (c *codexClient) trySetThreadName(ctx context.Context, threadID, name string, logger *slog.Logger) {
|
|
name = strings.TrimSpace(name)
|
|
if name == "" {
|
|
return
|
|
}
|
|
if err := c.setThreadName(ctx, threadID, name); err != nil {
|
|
logger.Warn("codex thread/name/set failed; continuing without provider-native thread title",
|
|
"thread_id", threadID, "error", err)
|
|
}
|
|
}
|
|
|
|
func (c *codexClient) setThreadName(ctx context.Context, threadID, name string) error {
|
|
_, err := c.request(ctx, "thread/name/set", map[string]any{
|
|
"threadId": threadID,
|
|
"name": name,
|
|
})
|
|
return err
|
|
}
|
|
|
|
// applyCodexReasoningEffort writes the per-agent thinking_level into a
|
|
// Codex app-server request. The three points — thread/start.config,
|
|
// thread/resume.config, turn/start.effort — all flow through this helper
|
|
// so any future protocol/key change touches one site rather than three
|
|
// (per Trump's MUL-2339 review constraint).
|
|
//
|
|
// The shape is detected from the params keys:
|
|
// - turn/start always carries `input`, and the schema exposes the
|
|
// reasoning override as the top-level `effort` field.
|
|
// - thread/start and thread/resume nest it under
|
|
// `config.model_reasoning_effort`.
|
|
//
|
|
// Empty `level` is a no-op: we deliberately do NOT emit a key when the
|
|
// caller didn't request an override, so the upstream defaults (config
|
|
// file, account-scoped model preference) stay in charge. This also
|
|
// guarantees `effort: ""` never reaches the CLI — Codex rejects empty
|
|
// strings on this field.
|
|
func applyCodexReasoningEffort(params map[string]any, level string) {
|
|
if params == nil || level == "" {
|
|
return
|
|
}
|
|
if _, isTurnStart := params["input"]; isTurnStart {
|
|
params["effort"] = level
|
|
return
|
|
}
|
|
cfg, _ := params["config"].(map[string]any)
|
|
if cfg == nil {
|
|
cfg = map[string]any{}
|
|
}
|
|
cfg["model_reasoning_effort"] = level
|
|
params["config"] = cfg
|
|
}
|
|
|
|
func resetTimer(timer *time.Timer, d time.Duration) {
|
|
if !timer.Stop() {
|
|
select {
|
|
case <-timer.C:
|
|
default:
|
|
}
|
|
}
|
|
timer.Reset(d)
|
|
}
|
|
|
|
func stopTimer(timer *time.Timer) {
|
|
if timer == nil {
|
|
return
|
|
}
|
|
if !timer.Stop() {
|
|
select {
|
|
case <-timer.C:
|
|
default:
|
|
}
|
|
}
|
|
}
|
|
|
|
func codexFirstTurnNoProgressTimeout(semanticInactivityTimeout time.Duration) time.Duration {
|
|
if semanticInactivityTimeout <= 0 || semanticInactivityTimeout > defaultCodexFirstTurnNoProgressTimeout {
|
|
return defaultCodexFirstTurnNoProgressTimeout
|
|
}
|
|
scaled := semanticInactivityTimeout * 4 / 5
|
|
if scaled <= 0 {
|
|
return semanticInactivityTimeout
|
|
}
|
|
return scaled
|
|
}
|
|
|
|
func isCodexFirstTurnProgressActivity(activity string) bool {
|
|
return activity != "" && activity != "status:running" && activity != "error:retry"
|
|
}
|
|
|
|
func buildCodexTimeoutDiagnosticError(diag codexTimeoutDiagnostic, stderrTail string) string {
|
|
var msg string
|
|
switch diag.Kind {
|
|
case codexTimeoutFirstTurnNoProgress:
|
|
msg = fmt.Sprintf("%s after %s: received turn start but no item, message, tool, turn/completed, or error event (%s)",
|
|
CodexFirstTurnNoProgressMarker,
|
|
diag.Timeout,
|
|
formatCodexDiagnosticFields(diag),
|
|
)
|
|
case codexTimeoutSemanticInactivity:
|
|
msg = fmt.Sprintf("%s after %s without agent progress (last activity: %s; %s)",
|
|
CodexSemanticInactivityMarker,
|
|
diag.Timeout,
|
|
nonEmptyCodexDiagnosticValue(diag.LastActivity),
|
|
formatCodexDiagnosticFields(diag),
|
|
)
|
|
default:
|
|
msg = "codex timed out"
|
|
}
|
|
msg = appendCodexKnownStderrHint(msg, stderrTail)
|
|
return withAgentStderr(msg, "codex", stderrTail)
|
|
}
|
|
|
|
func formatCodexDiagnosticFields(diag codexTimeoutDiagnostic) string {
|
|
return fmt.Sprintf("codex_version=%q thread_id=%q turn_id=%q model=%q",
|
|
nonEmptyCodexDiagnosticValue(diag.CodexVersion),
|
|
nonEmptyCodexDiagnosticValue(diag.ThreadID),
|
|
nonEmptyCodexDiagnosticValue(diag.TurnID),
|
|
formatCodexDiagnosticModel(diag.Model),
|
|
)
|
|
}
|
|
|
|
func nonEmptyCodexDiagnosticValue(value string) string {
|
|
if strings.TrimSpace(value) == "" {
|
|
return "unknown"
|
|
}
|
|
return value
|
|
}
|
|
|
|
func formatCodexDiagnosticModel(model string) string {
|
|
if strings.TrimSpace(model) == "" {
|
|
return "default(empty)"
|
|
}
|
|
return model
|
|
}
|
|
|
|
func appendCodexKnownStderrHint(msg, stderrTail string) string {
|
|
if strings.Contains(stderrTail, codexModelCatalogRefreshTimeoutSignal) {
|
|
return msg + "; diagnosis: Codex stderr shows the model catalog refresh timed out. Try setting an explicit model, switching Codex CLI versions, or using another runtime while Codex app-server recovers"
|
|
}
|
|
return msg
|
|
}
|
|
|
|
func detectCodexVersionForDiagnostics(ctx context.Context, execPath string, env []string, logger *slog.Logger) string {
|
|
versionCtx, cancel := context.WithTimeout(ctx, codexVersionDiagnosticTimeout)
|
|
defer cancel()
|
|
|
|
cmd := exec.CommandContext(versionCtx, execPath, "--version")
|
|
cmd.Env = env
|
|
data, err := cmd.Output()
|
|
if err != nil {
|
|
if logger != nil {
|
|
logger.Debug("codex version diagnostic failed", "error", err)
|
|
}
|
|
return "unknown"
|
|
}
|
|
version := extractVersionLine(string(data))
|
|
if strings.TrimSpace(version) == "" {
|
|
return "unknown"
|
|
}
|
|
return version
|
|
}
|
|
|
|
func trySendString(ch chan<- string, value string) {
|
|
select {
|
|
case ch <- value:
|
|
default:
|
|
}
|
|
}
|
|
|
|
func logCodexAgentMessage(logger *slog.Logger, msg Message) {
|
|
if logger == nil {
|
|
return
|
|
}
|
|
attrs := []any{
|
|
"type", string(msg.Type),
|
|
"tool", msg.Tool,
|
|
"call_id", msg.CallID,
|
|
"status", msg.Status,
|
|
"content_len", len(msg.Content),
|
|
"output_len", len(msg.Output),
|
|
}
|
|
logger.Info("codex agent message received", attrs...)
|
|
if msg.Type == MessageToolResult {
|
|
logger.Info("codex tool_result observed", "tool", msg.Tool, "call_id", msg.CallID, "output_len", len(msg.Output))
|
|
}
|
|
}
|
|
|
|
func describeCodexSemanticActivity(msg Message) string {
|
|
switch msg.Type {
|
|
case MessageToolUse, MessageToolResult:
|
|
if msg.Tool != "" {
|
|
return fmt.Sprintf("%s:%s", msg.Type, msg.Tool)
|
|
}
|
|
case MessageStatus:
|
|
if msg.Status != "" {
|
|
return fmt.Sprintf("%s:%s", msg.Type, msg.Status)
|
|
}
|
|
}
|
|
return string(msg.Type)
|
|
}
|
|
|
|
// ── codexClient: JSON-RPC 2.0 transport ──
|
|
|
|
type codexClient struct {
|
|
cfg Config
|
|
stdin interface{ Write([]byte) (int, error) }
|
|
mu sync.Mutex
|
|
nextID int
|
|
pending map[int]*pendingRPC
|
|
processDone chan struct{}
|
|
processErr error
|
|
handshakeTimeout time.Duration
|
|
threadID string
|
|
turnID string
|
|
onMessage func(Message)
|
|
onSemanticActivity func(description string)
|
|
onTurnDone func(aborted bool)
|
|
|
|
notificationProtocol string // "unknown", "legacy", "raw"
|
|
turnStarted bool
|
|
completedTurnIDs map[string]bool
|
|
|
|
usageMu sync.Mutex
|
|
usage TokenUsage // accumulated from turn events
|
|
|
|
turnErrorMu sync.Mutex
|
|
turnError string // captured from turn/completed status=failed or terminal error notifications
|
|
}
|
|
|
|
func (c *codexClient) setTurnError(msg string) {
|
|
if msg == "" {
|
|
return
|
|
}
|
|
c.turnErrorMu.Lock()
|
|
defer c.turnErrorMu.Unlock()
|
|
if c.turnError == "" {
|
|
c.turnError = msg
|
|
}
|
|
}
|
|
|
|
func (c *codexClient) getTurnError() string {
|
|
c.turnErrorMu.Lock()
|
|
defer c.turnErrorMu.Unlock()
|
|
return c.turnError
|
|
}
|
|
|
|
type pendingRPC struct {
|
|
ch chan rpcResult
|
|
method string
|
|
}
|
|
|
|
type rpcResult struct {
|
|
result json.RawMessage
|
|
err error
|
|
}
|
|
|
|
type codexHandshakeTimeoutError struct {
|
|
Method string
|
|
Timeout time.Duration
|
|
}
|
|
|
|
func (e *codexHandshakeTimeoutError) Error() string {
|
|
return fmt.Sprintf("%s: %s did not respond after %s", CodexHandshakeTimeoutMarker, e.Method, e.Timeout)
|
|
}
|
|
|
|
func (e *codexHandshakeTimeoutError) Unwrap() error {
|
|
return context.DeadlineExceeded
|
|
}
|
|
|
|
func isCodexHandshakeRPC(method string) bool {
|
|
switch method {
|
|
case "initialize", "thread/start", "thread/resume", "thread/name/set", "turn/start":
|
|
return true
|
|
default:
|
|
return false
|
|
}
|
|
}
|
|
|
|
func codexRequestContextError(ctx context.Context) error {
|
|
var handshakeErr *codexHandshakeTimeoutError
|
|
if errors.As(context.Cause(ctx), &handshakeErr) {
|
|
return handshakeErr
|
|
}
|
|
return ctx.Err()
|
|
}
|
|
|
|
func (c *codexClient) request(ctx context.Context, method string, params any) (json.RawMessage, error) {
|
|
if err := ctx.Err(); err != nil {
|
|
return nil, err
|
|
}
|
|
requestCtx := ctx
|
|
cancelRequest := func() {}
|
|
if c.handshakeTimeout > 0 && isCodexHandshakeRPC(method) {
|
|
timeoutErr := &codexHandshakeTimeoutError{Method: method, Timeout: c.handshakeTimeout}
|
|
requestCtx, cancelRequest = context.WithTimeoutCause(ctx, c.handshakeTimeout, timeoutErr)
|
|
}
|
|
defer cancelRequest()
|
|
|
|
c.mu.Lock()
|
|
if c.processErr != nil {
|
|
err := c.processErr
|
|
c.mu.Unlock()
|
|
return nil, err
|
|
}
|
|
if c.processDone == nil {
|
|
c.processDone = make(chan struct{})
|
|
}
|
|
processDone := c.processDone
|
|
c.nextID++
|
|
id := c.nextID
|
|
pr := &pendingRPC{ch: make(chan rpcResult, 1), method: method}
|
|
c.pending[id] = pr
|
|
c.mu.Unlock()
|
|
|
|
msg := map[string]any{
|
|
"jsonrpc": "2.0",
|
|
"id": id,
|
|
"method": method,
|
|
"params": params,
|
|
}
|
|
data, err := json.Marshal(msg)
|
|
if err != nil {
|
|
c.mu.Lock()
|
|
delete(c.pending, id)
|
|
c.mu.Unlock()
|
|
return nil, err
|
|
}
|
|
data = append(data, '\n')
|
|
if _, err := c.stdin.Write(data); err != nil {
|
|
c.mu.Lock()
|
|
delete(c.pending, id)
|
|
c.mu.Unlock()
|
|
return nil, fmt.Errorf("write %s: %w", method, err)
|
|
}
|
|
if method == "turn/start" {
|
|
threadID := ""
|
|
if paramMap, ok := params.(map[string]any); ok {
|
|
threadID, _ = paramMap["threadId"].(string)
|
|
}
|
|
c.cfg.Logger.Info("codex turn/start sent", "request_id", id, "thread_id", threadID)
|
|
}
|
|
|
|
select {
|
|
case res := <-pr.ch:
|
|
return res.result, res.err
|
|
case <-processDone:
|
|
select {
|
|
case res := <-pr.ch:
|
|
return res.result, res.err
|
|
default:
|
|
}
|
|
c.mu.Lock()
|
|
delete(c.pending, id)
|
|
err := c.processErr
|
|
c.mu.Unlock()
|
|
if requestCtx.Err() != nil {
|
|
return nil, codexRequestContextError(requestCtx)
|
|
}
|
|
if err == nil {
|
|
err = errCodexProcessExited
|
|
}
|
|
return nil, err
|
|
case <-requestCtx.Done():
|
|
c.mu.Lock()
|
|
delete(c.pending, id)
|
|
c.mu.Unlock()
|
|
return nil, codexRequestContextError(requestCtx)
|
|
}
|
|
}
|
|
|
|
func (c *codexClient) notify(method string) {
|
|
msg := map[string]any{
|
|
"jsonrpc": "2.0",
|
|
"method": method,
|
|
}
|
|
data, _ := json.Marshal(msg)
|
|
data = append(data, '\n')
|
|
_, _ = c.stdin.Write(data)
|
|
}
|
|
|
|
func (c *codexClient) respond(id int, result any) {
|
|
msg := map[string]any{
|
|
"jsonrpc": "2.0",
|
|
"id": id,
|
|
"result": result,
|
|
}
|
|
data, _ := json.Marshal(msg)
|
|
data = append(data, '\n')
|
|
_, _ = c.stdin.Write(data)
|
|
}
|
|
|
|
func (c *codexClient) respondError(id int, code int, message string) {
|
|
msg := map[string]any{
|
|
"jsonrpc": "2.0",
|
|
"id": id,
|
|
"error": map[string]any{
|
|
"code": code,
|
|
"message": message,
|
|
},
|
|
}
|
|
data, _ := json.Marshal(msg)
|
|
data = append(data, '\n')
|
|
_, _ = c.stdin.Write(data)
|
|
}
|
|
|
|
func (c *codexClient) closeAllPending(err error) {
|
|
c.mu.Lock()
|
|
defer c.mu.Unlock()
|
|
for id, pr := range c.pending {
|
|
pr.ch <- rpcResult{err: err}
|
|
delete(c.pending, id)
|
|
}
|
|
}
|
|
|
|
func (c *codexClient) markProcessExited(err error) {
|
|
if err == nil {
|
|
err = errCodexProcessExited
|
|
}
|
|
c.mu.Lock()
|
|
defer c.mu.Unlock()
|
|
if c.processErr == nil {
|
|
c.processErr = err
|
|
if c.processDone != nil {
|
|
close(c.processDone)
|
|
}
|
|
}
|
|
for id, pr := range c.pending {
|
|
pr.ch <- rpcResult{err: err}
|
|
delete(c.pending, id)
|
|
}
|
|
}
|
|
|
|
func (c *codexClient) getProcessErr() error {
|
|
c.mu.Lock()
|
|
defer c.mu.Unlock()
|
|
return c.processErr
|
|
}
|
|
|
|
func isCodexTransportError(err error) bool {
|
|
if err == nil {
|
|
return false
|
|
}
|
|
if errors.Is(err, errCodexProcessExited) {
|
|
return true
|
|
}
|
|
var handshakeErr *codexHandshakeTimeoutError
|
|
if errors.As(err, &handshakeErr) {
|
|
return true
|
|
}
|
|
return strings.HasPrefix(err.Error(), "write ")
|
|
}
|
|
|
|
func (c *codexClient) handleLine(line string) {
|
|
var raw map[string]json.RawMessage
|
|
if err := json.Unmarshal([]byte(line), &raw); err != nil {
|
|
return
|
|
}
|
|
|
|
// Check if it's a response to our request
|
|
if _, hasID := raw["id"]; hasID {
|
|
if _, hasResult := raw["result"]; hasResult {
|
|
c.handleResponse(raw)
|
|
return
|
|
}
|
|
if _, hasError := raw["error"]; hasError {
|
|
c.handleResponse(raw)
|
|
return
|
|
}
|
|
// Server request (has id + method)
|
|
if _, hasMethod := raw["method"]; hasMethod {
|
|
c.handleServerRequest(raw)
|
|
return
|
|
}
|
|
}
|
|
|
|
// Notification (no id, has method)
|
|
if _, hasMethod := raw["method"]; hasMethod {
|
|
c.handleNotification(raw)
|
|
}
|
|
}
|
|
|
|
func (c *codexClient) handleResponse(raw map[string]json.RawMessage) {
|
|
var id int
|
|
if err := json.Unmarshal(raw["id"], &id); err != nil {
|
|
return
|
|
}
|
|
|
|
c.mu.Lock()
|
|
pr, ok := c.pending[id]
|
|
if ok {
|
|
delete(c.pending, id)
|
|
}
|
|
c.mu.Unlock()
|
|
|
|
if !ok {
|
|
return
|
|
}
|
|
|
|
if errData, hasErr := raw["error"]; hasErr {
|
|
var rpcErr struct {
|
|
Code int `json:"code"`
|
|
Message string `json:"message"`
|
|
}
|
|
_ = json.Unmarshal(errData, &rpcErr)
|
|
pr.ch <- rpcResult{err: fmt.Errorf("%s: %s (code=%d)", pr.method, rpcErr.Message, rpcErr.Code)}
|
|
} else {
|
|
pr.ch <- rpcResult{result: raw["result"]}
|
|
}
|
|
}
|
|
|
|
func (c *codexClient) handleServerRequest(raw map[string]json.RawMessage) {
|
|
var id int
|
|
_ = json.Unmarshal(raw["id"], &id)
|
|
|
|
var method string
|
|
_ = json.Unmarshal(raw["method"], &method)
|
|
|
|
// Auto-approve all exec/patch requests in daemon mode
|
|
switch method {
|
|
case "item/commandExecution/requestApproval", "execCommandApproval":
|
|
c.respond(id, map[string]any{"decision": "accept"})
|
|
case "item/fileChange/requestApproval", "applyPatchApproval":
|
|
c.respond(id, map[string]any{"decision": "accept"})
|
|
case "item/permissions/requestApproval":
|
|
c.respond(id, codexPermissionsApprovalResponse(raw["params"], c.cfg.Logger))
|
|
case "mcpServer/elicitation/request":
|
|
c.respond(id, map[string]any{"action": "accept", "content": nil, "_meta": nil})
|
|
default:
|
|
msg := fmt.Sprintf("unsupported codex app-server request: %s", method)
|
|
c.cfg.Logger.Warn("codex: unhandled server request", "method", method, "id", id)
|
|
c.setTurnError(msg)
|
|
c.respondError(id, -32601, msg)
|
|
}
|
|
}
|
|
|
|
// codexPermissionsApprovalResponse builds the auto-grant reply for a Codex
|
|
// item/permissions/requestApproval server request. In daemon mode there is no
|
|
// human to approve, so we echo back the requested network / fileSystem profile
|
|
// and scope it to the current turn, mirroring the other auto-accept branches in
|
|
// handleServerRequest.
|
|
//
|
|
// The grant is intentionally limited to the network / fileSystem keys we
|
|
// understand. A parse failure and any dropped key are logged so that a future
|
|
// app-server protocol that adds a new permission shape is visible in daemon
|
|
// logs instead of being silently narrowed away.
|
|
func codexPermissionsApprovalResponse(params json.RawMessage, logger *slog.Logger) map[string]any {
|
|
var payload struct {
|
|
Permissions map[string]any `json:"permissions"`
|
|
}
|
|
if err := json.Unmarshal(params, &payload); err != nil && logger != nil {
|
|
logger.Warn("codex: failed to parse permission approval request; granting empty turn-scoped profile", "error", err)
|
|
}
|
|
|
|
granted := map[string]any{}
|
|
var dropped []string
|
|
for key, value := range payload.Permissions {
|
|
switch key {
|
|
case "network", "fileSystem":
|
|
if value != nil {
|
|
granted[key] = value
|
|
}
|
|
default:
|
|
dropped = append(dropped, key)
|
|
}
|
|
}
|
|
if len(dropped) > 0 && logger != nil {
|
|
sort.Strings(dropped)
|
|
logger.Warn("codex: dropping unrecognized permission keys from approval request; add explicit handling if the app-server protocol expanded", "keys", dropped)
|
|
}
|
|
|
|
return map[string]any{
|
|
"permissions": granted,
|
|
"scope": "turn",
|
|
}
|
|
}
|
|
|
|
func (c *codexClient) handleNotification(raw map[string]json.RawMessage) {
|
|
var method string
|
|
_ = json.Unmarshal(raw["method"], &method)
|
|
|
|
var params map[string]any
|
|
if p, ok := raw["params"]; ok {
|
|
_ = json.Unmarshal(p, ¶ms)
|
|
}
|
|
|
|
// Legacy codex/event notifications
|
|
if method == "codex/event" || strings.HasPrefix(method, "codex/event/") {
|
|
c.notificationProtocol = "legacy"
|
|
msgData, ok := params["msg"]
|
|
if !ok {
|
|
return
|
|
}
|
|
msgMap, ok := msgData.(map[string]any)
|
|
if !ok {
|
|
return
|
|
}
|
|
c.handleEvent(msgMap)
|
|
return
|
|
}
|
|
|
|
// Raw v2 notifications
|
|
if c.notificationProtocol != "legacy" {
|
|
if c.notificationProtocol == "unknown" &&
|
|
(method == "turn/started" || method == "turn/completed" ||
|
|
method == "thread/started" || strings.HasPrefix(method, "item/")) {
|
|
c.notificationProtocol = "raw"
|
|
}
|
|
|
|
if c.notificationProtocol == "raw" {
|
|
c.handleRawNotification(method, params)
|
|
}
|
|
}
|
|
}
|
|
|
|
func (c *codexClient) handleEvent(msg map[string]any) {
|
|
msgType, _ := msg["type"].(string)
|
|
|
|
switch msgType {
|
|
case "task_started":
|
|
c.turnStarted = true
|
|
if c.onMessage != nil {
|
|
c.onMessage(Message{Type: MessageStatus, Status: "running", SessionID: c.threadID})
|
|
}
|
|
case "agent_message":
|
|
text, _ := msg["message"].(string)
|
|
if text != "" && c.onMessage != nil {
|
|
c.onMessage(Message{Type: MessageText, Content: text})
|
|
}
|
|
case "exec_command_begin":
|
|
callID, _ := msg["call_id"].(string)
|
|
command, _ := msg["command"].(string)
|
|
if c.onMessage != nil {
|
|
c.onMessage(Message{
|
|
Type: MessageToolUse,
|
|
Tool: "exec_command",
|
|
CallID: callID,
|
|
Input: map[string]any{"command": command},
|
|
})
|
|
}
|
|
case "exec_command_end":
|
|
callID, _ := msg["call_id"].(string)
|
|
output, _ := msg["output"].(string)
|
|
if c.onMessage != nil {
|
|
c.onMessage(Message{
|
|
Type: MessageToolResult,
|
|
Tool: "exec_command",
|
|
CallID: callID,
|
|
Output: output,
|
|
})
|
|
}
|
|
case "patch_apply_begin":
|
|
callID, _ := msg["call_id"].(string)
|
|
if c.onMessage != nil {
|
|
c.onMessage(Message{
|
|
Type: MessageToolUse,
|
|
Tool: "patch_apply",
|
|
CallID: callID,
|
|
})
|
|
}
|
|
case "patch_apply_end":
|
|
callID, _ := msg["call_id"].(string)
|
|
if c.onMessage != nil {
|
|
c.onMessage(Message{
|
|
Type: MessageToolResult,
|
|
Tool: "patch_apply",
|
|
CallID: callID,
|
|
})
|
|
}
|
|
case "task_complete":
|
|
// Extract usage from legacy task_complete if present.
|
|
c.extractUsageFromMap(msg)
|
|
if c.onTurnDone != nil {
|
|
c.onTurnDone(false)
|
|
}
|
|
case "turn_aborted":
|
|
if c.onTurnDone != nil {
|
|
c.onTurnDone(true)
|
|
}
|
|
}
|
|
}
|
|
|
|
func (c *codexClient) handleRawNotification(method string, params map[string]any) {
|
|
// Ignore notifications from threads other than the one we are tracking.
|
|
// Codex multiplexes subagent threads (e.g. memory consolidation) on the
|
|
// same stdio pipe; only our thread should drive turn lifecycle and output.
|
|
//
|
|
// The v2 app-server-protocol schema guarantees a top-level threadId on
|
|
// every notification, so this dispatch-level guard transparently covers
|
|
// every handler below. If a future codex revision introduces notifications
|
|
// without threadId, they fall through (ok=false) — re-audit this guard
|
|
// when bumping codex.
|
|
if threadID, ok := params["threadId"].(string); ok && c.threadID != "" && threadID != c.threadID {
|
|
return
|
|
}
|
|
|
|
switch method {
|
|
case "turn/started":
|
|
c.turnStarted = true
|
|
if turnID := extractNestedString(params, "turn", "id"); turnID != "" {
|
|
c.turnID = turnID
|
|
}
|
|
if c.onMessage != nil {
|
|
c.onMessage(Message{Type: MessageStatus, Status: "running", SessionID: c.threadID})
|
|
}
|
|
|
|
case "turn/completed":
|
|
turnID := extractNestedString(params, "turn", "id")
|
|
status := extractNestedString(params, "turn", "status")
|
|
threadID, _ := params["threadId"].(string)
|
|
c.cfg.Logger.Info("codex turn/completed received", "thread_id", threadID, "turn_id", turnID, "status", status)
|
|
aborted := status == "cancelled" || status == "canceled" ||
|
|
status == "aborted" || status == "interrupted"
|
|
|
|
// Capture the error message from failed turns so callers can surface
|
|
// a real reason instead of falling back to "empty output".
|
|
if status == "failed" {
|
|
errMsg := extractNestedString(params, "turn", "error", "message")
|
|
if errMsg == "" {
|
|
errMsg = "codex turn failed"
|
|
}
|
|
c.setTurnError(errMsg)
|
|
}
|
|
|
|
if c.completedTurnIDs == nil {
|
|
c.completedTurnIDs = map[string]bool{}
|
|
}
|
|
if turnID != "" {
|
|
if c.completedTurnIDs[turnID] {
|
|
return
|
|
}
|
|
c.completedTurnIDs[turnID] = true
|
|
}
|
|
|
|
// Extract usage from turn/completed if present (e.g. params.turn.usage).
|
|
if turn, ok := params["turn"].(map[string]any); ok {
|
|
c.extractUsageFromMap(turn)
|
|
}
|
|
|
|
if c.onTurnDone != nil {
|
|
c.onTurnDone(aborted)
|
|
}
|
|
|
|
case "error":
|
|
// Top-level protocol error. Retrying notifications (willRetry=true) are
|
|
// transient reconnect attempts; only capture terminal errors so we
|
|
// don't stomp on a real failure later with a retry placeholder.
|
|
willRetry, _ := params["willRetry"].(bool)
|
|
errMsg := extractNestedString(params, "error", "message")
|
|
if errMsg == "" {
|
|
errMsg = extractNestedString(params, "message")
|
|
}
|
|
if errMsg != "" {
|
|
c.cfg.Logger.Warn("codex error notification", "message", errMsg, "will_retry", willRetry)
|
|
if c.onSemanticActivity != nil {
|
|
if willRetry {
|
|
c.onSemanticActivity("error:retry")
|
|
} else {
|
|
c.onSemanticActivity("error:terminal")
|
|
}
|
|
}
|
|
if !willRetry {
|
|
c.setTurnError(errMsg)
|
|
if c.onTurnDone != nil {
|
|
c.onTurnDone(false)
|
|
}
|
|
}
|
|
}
|
|
|
|
case "thread/status/changed":
|
|
statusType := extractNestedString(params, "status", "type")
|
|
if statusType == "idle" && c.turnStarted {
|
|
if c.onTurnDone != nil {
|
|
c.onTurnDone(false)
|
|
}
|
|
}
|
|
|
|
default:
|
|
if strings.HasPrefix(method, "item/") {
|
|
c.handleItemNotification(method, params)
|
|
}
|
|
}
|
|
}
|
|
|
|
func (c *codexClient) handleItemNotification(method string, params map[string]any) {
|
|
item, _ := params["item"].(map[string]any)
|
|
itemType, _ := item["type"].(string)
|
|
itemID, _ := item["id"].(string)
|
|
if isCodexItemProgressActivity(method) && c.onSemanticActivity != nil {
|
|
c.onSemanticActivity(describeCodexItemProgressActivity(method, itemType, itemID))
|
|
}
|
|
if item == nil {
|
|
return
|
|
}
|
|
|
|
switch {
|
|
case method == "item/started" && itemType == "commandExecution":
|
|
command, _ := item["command"].(string)
|
|
if c.onMessage != nil {
|
|
c.onMessage(Message{
|
|
Type: MessageToolUse,
|
|
Tool: "exec_command",
|
|
CallID: itemID,
|
|
Input: map[string]any{"command": command},
|
|
})
|
|
}
|
|
|
|
case method == "item/completed" && itemType == "commandExecution":
|
|
output, _ := item["aggregatedOutput"].(string)
|
|
if c.onMessage != nil {
|
|
c.onMessage(Message{
|
|
Type: MessageToolResult,
|
|
Tool: "exec_command",
|
|
CallID: itemID,
|
|
Output: output,
|
|
})
|
|
}
|
|
|
|
case method == "item/started" && itemType == "fileChange":
|
|
if c.onMessage != nil {
|
|
c.onMessage(Message{
|
|
Type: MessageToolUse,
|
|
Tool: "patch_apply",
|
|
CallID: itemID,
|
|
})
|
|
}
|
|
|
|
case method == "item/completed" && itemType == "fileChange":
|
|
if c.onMessage != nil {
|
|
c.onMessage(Message{
|
|
Type: MessageToolResult,
|
|
Tool: "patch_apply",
|
|
CallID: itemID,
|
|
})
|
|
}
|
|
|
|
case method == "item/completed" && itemType == "agentMessage":
|
|
text, _ := item["text"].(string)
|
|
if text != "" && c.onMessage != nil {
|
|
c.onMessage(Message{Type: MessageText, Content: text})
|
|
}
|
|
phase, _ := item["phase"].(string)
|
|
if phase == "final_answer" && c.turnStarted {
|
|
if c.onTurnDone != nil {
|
|
c.onTurnDone(false)
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
func isCodexItemProgressActivity(method string) bool {
|
|
return strings.HasPrefix(method, "item/")
|
|
}
|
|
|
|
func describeCodexItemProgressActivity(method, itemType, itemID string) string {
|
|
if itemType == "" {
|
|
itemType = "unknown"
|
|
}
|
|
if itemID == "" {
|
|
return fmt.Sprintf("%s:%s", method, itemType)
|
|
}
|
|
return fmt.Sprintf("%s:%s:%s", method, itemType, itemID)
|
|
}
|
|
|
|
// extractUsageFromMap extracts token usage from a map that may contain
|
|
// "usage", "token_usage", or "tokens" fields. Handles various Codex formats.
|
|
func (c *codexClient) extractUsageFromMap(data map[string]any) {
|
|
// Try common field names for usage data.
|
|
var usageMap map[string]any
|
|
for _, key := range []string{"usage", "token_usage", "tokens"} {
|
|
if v, ok := data[key].(map[string]any); ok {
|
|
usageMap = v
|
|
break
|
|
}
|
|
}
|
|
if usageMap == nil {
|
|
return
|
|
}
|
|
|
|
c.usageMu.Lock()
|
|
defer c.usageMu.Unlock()
|
|
|
|
// Codex reports cached input as a prompt-token detail: cached_input_tokens
|
|
// are included in input_tokens. Persist mutually-exclusive buckets so
|
|
// dashboard cost math does not charge cached input twice.
|
|
inputTokens := codexInt64(usageMap, "input_tokens", "input", "prompt_tokens")
|
|
cacheReadTokens := codexInt64(usageMap, "cached_input_tokens", "cache_read_tokens", "cache_read_input_tokens")
|
|
c.usage.InputTokens += codexUncachedInputTokens(inputTokens, cacheReadTokens)
|
|
c.usage.OutputTokens += codexInt64(usageMap, "output_tokens", "output", "completion_tokens")
|
|
c.usage.CacheReadTokens += cacheReadTokens
|
|
c.usage.CacheWriteTokens += codexInt64(usageMap, "cache_write_tokens", "cache_creation_input_tokens")
|
|
}
|
|
|
|
func codexUncachedInputTokens(inputTokens, cachedInputTokens int64) int64 {
|
|
uncached := inputTokens - cachedInputTokens
|
|
if uncached < 0 {
|
|
return 0
|
|
}
|
|
return uncached
|
|
}
|
|
|
|
// codexInt64 returns the first non-zero int64 value from the map for the given keys.
|
|
func codexInt64(m map[string]any, keys ...string) int64 {
|
|
for _, key := range keys {
|
|
switch v := m[key].(type) {
|
|
case float64:
|
|
if v != 0 {
|
|
return int64(v)
|
|
}
|
|
case int64:
|
|
if v != 0 {
|
|
return v
|
|
}
|
|
}
|
|
}
|
|
return 0
|
|
}
|
|
|
|
// ── Codex session log scanner ──
|
|
|
|
// codexSessionUsage holds usage extracted from a Codex session JSONL file.
|
|
type codexSessionUsage struct {
|
|
usage TokenUsage
|
|
model string
|
|
}
|
|
|
|
// scanCodexSessionUsage scans Codex session JSONL files written after startTime
|
|
// to extract token usage. Codex writes token_count events to
|
|
// $CODEX_HOME/sessions/YYYY/MM/DD/*.jsonl. codexHome is the backend's per-task
|
|
// CODEX_HOME; sessions are isolated there rather than in the shared
|
|
// ~/.codex/sessions (MUL-4424), so usage must be read from it.
|
|
func scanCodexSessionUsage(startTime time.Time, codexHome string) *codexSessionUsage {
|
|
root := codexSessionRoot(codexHome)
|
|
if root == "" {
|
|
return nil
|
|
}
|
|
|
|
// Look in today's session directory.
|
|
dateDir := filepath.Join(root,
|
|
fmt.Sprintf("%04d", startTime.Year()),
|
|
fmt.Sprintf("%02d", int(startTime.Month())),
|
|
fmt.Sprintf("%02d", startTime.Day()),
|
|
)
|
|
|
|
files, err := filepath.Glob(filepath.Join(dateDir, "*.jsonl"))
|
|
if err != nil || len(files) == 0 {
|
|
return nil
|
|
}
|
|
|
|
// Only scan files modified after startTime (this task's session).
|
|
var result codexSessionUsage
|
|
for _, f := range files {
|
|
info, err := os.Stat(f)
|
|
if err != nil || info.ModTime().Before(startTime) {
|
|
continue
|
|
}
|
|
if u := parseCodexSessionFile(f); u != nil {
|
|
// Take the last matching file's data (usually there's only one per task).
|
|
result = *u
|
|
}
|
|
}
|
|
|
|
if result.usage.InputTokens == 0 && result.usage.OutputTokens == 0 {
|
|
return nil
|
|
}
|
|
return &result
|
|
}
|
|
|
|
// codexSessionRoot returns the Codex sessions directory. It prefers the
|
|
// explicit per-task codexHome the backend is running with (so usage is read
|
|
// from the same task-local sessions Codex actually wrote to), then the ambient
|
|
// CODEX_HOME, then ~/.codex.
|
|
func codexSessionRoot(codexHome string) string {
|
|
if codexHome = strings.TrimSpace(codexHome); codexHome == "" {
|
|
codexHome = os.Getenv("CODEX_HOME")
|
|
}
|
|
if codexHome != "" {
|
|
dir := filepath.Join(codexHome, "sessions")
|
|
if info, err := os.Stat(dir); err == nil && info.IsDir() {
|
|
return dir
|
|
}
|
|
}
|
|
|
|
home, err := os.UserHomeDir()
|
|
if err != nil {
|
|
return ""
|
|
}
|
|
|
|
dir := filepath.Join(home, ".codex", "sessions")
|
|
if info, err := os.Stat(dir); err == nil && info.IsDir() {
|
|
return dir
|
|
}
|
|
return ""
|
|
}
|
|
|
|
// codexSessionTokenCount represents a token_count event in Codex JSONL.
|
|
type codexSessionTokenCount struct {
|
|
Type string `json:"type"`
|
|
Payload *struct {
|
|
Type string `json:"type"`
|
|
Info *struct {
|
|
TotalTokenUsage *struct {
|
|
InputTokens int64 `json:"input_tokens"`
|
|
OutputTokens int64 `json:"output_tokens"`
|
|
CachedInputTokens int64 `json:"cached_input_tokens"`
|
|
CacheReadInputTokens int64 `json:"cache_read_input_tokens"`
|
|
ReasoningOutputTokens int64 `json:"reasoning_output_tokens"`
|
|
} `json:"total_token_usage"`
|
|
LastTokenUsage *struct {
|
|
InputTokens int64 `json:"input_tokens"`
|
|
OutputTokens int64 `json:"output_tokens"`
|
|
CachedInputTokens int64 `json:"cached_input_tokens"`
|
|
CacheReadInputTokens int64 `json:"cache_read_input_tokens"`
|
|
ReasoningOutputTokens int64 `json:"reasoning_output_tokens"`
|
|
} `json:"last_token_usage"`
|
|
Model string `json:"model"`
|
|
} `json:"info"`
|
|
Model string `json:"model"`
|
|
} `json:"payload"`
|
|
}
|
|
|
|
// parseCodexSessionFile extracts the final token_count from a Codex session file.
|
|
func parseCodexSessionFile(path string) *codexSessionUsage {
|
|
f, err := os.Open(path)
|
|
if err != nil {
|
|
return nil
|
|
}
|
|
defer f.Close()
|
|
|
|
var result codexSessionUsage
|
|
found := false
|
|
|
|
scanner := bufio.NewScanner(f)
|
|
scanner.Buffer(make([]byte, 0, 256*1024), 1024*1024)
|
|
|
|
for scanner.Scan() {
|
|
line := scanner.Bytes()
|
|
|
|
// Fast pre-filter.
|
|
if !bytesContainsStr(line, "token_count") && !bytesContainsStr(line, "turn_context") {
|
|
continue
|
|
}
|
|
|
|
var evt codexSessionTokenCount
|
|
if err := json.Unmarshal(line, &evt); err != nil || evt.Payload == nil {
|
|
continue
|
|
}
|
|
|
|
// Track model from turn_context events.
|
|
if evt.Type == "turn_context" && evt.Payload.Model != "" {
|
|
result.model = evt.Payload.Model
|
|
continue
|
|
}
|
|
|
|
// Extract token usage from token_count events.
|
|
if evt.Payload.Type == "token_count" && evt.Payload.Info != nil {
|
|
usage := evt.Payload.Info.TotalTokenUsage
|
|
if usage == nil {
|
|
usage = evt.Payload.Info.LastTokenUsage
|
|
}
|
|
if usage != nil {
|
|
cachedTokens := usage.CachedInputTokens
|
|
if cachedTokens == 0 {
|
|
cachedTokens = usage.CacheReadInputTokens
|
|
}
|
|
result.usage = TokenUsage{
|
|
InputTokens: codexUncachedInputTokens(usage.InputTokens, cachedTokens),
|
|
OutputTokens: usage.OutputTokens + usage.ReasoningOutputTokens,
|
|
CacheReadTokens: cachedTokens,
|
|
}
|
|
if evt.Payload.Info.Model != "" {
|
|
result.model = evt.Payload.Info.Model
|
|
}
|
|
found = true
|
|
}
|
|
}
|
|
}
|
|
|
|
if !found {
|
|
return nil
|
|
}
|
|
return &result
|
|
}
|
|
|
|
// bytesContainsStr checks if b contains the string s (without allocating).
|
|
func bytesContainsStr(b []byte, s string) bool {
|
|
return strings.Contains(string(b), s)
|
|
}
|
|
|
|
// ── Helpers ──
|
|
|
|
func extractThreadID(result json.RawMessage) string {
|
|
var r struct {
|
|
Thread struct {
|
|
ID string `json:"id"`
|
|
} `json:"thread"`
|
|
}
|
|
if err := json.Unmarshal(result, &r); err != nil {
|
|
return ""
|
|
}
|
|
return r.Thread.ID
|
|
}
|
|
|
|
func extractNestedString(m map[string]any, keys ...string) string {
|
|
current := any(m)
|
|
for _, key := range keys {
|
|
obj, ok := current.(map[string]any)
|
|
if !ok {
|
|
return ""
|
|
}
|
|
current = obj[key]
|
|
}
|
|
s, _ := current.(string)
|
|
return s
|
|
}
|
|
|
|
func nilIfEmpty(s string) any {
|
|
if s == "" {
|
|
return nil
|
|
}
|
|
return s
|
|
}
|