Files
multica/server/pkg/agent/codex.go
LinYushen cb68669c73 feat(composio): gate MCP apps behind feature flag (#4876)
* feat(composio): server-side connect flow + connections REST (Notion MVP) (MUL-3720) (#4608)

* feat(composio): server-side connect flow + connections REST (Notion MVP) (MUL-3720)

Compose the merged server/pkg/composio SDK into a user-facing connection
manager: signed-state connect handshake, local user_composio_connection
mirror, idempotent disconnect, and a per-user MCP session helper (not yet
wired into task dispatch).

- migration 127_user_composio_connection (no FK/cascade, per DB rules)
- sqlc queries: upsert (idempotent on user_id+connected_account_id), list
  active, owner-scoped get, mark revoked
- internal/integrations/composio: signed HMAC-SHA256 state, BeginConnect,
  CompleteCallback (idempotent upsert), ListConnections, Disconnect
  (upstream 404 = idempotent success), CreateMCPSession (no-op when empty,
  pins connected_accounts per toolkit), CallbackRedirect
- REST handlers under /api/integrations/composio (user-scoped, 503 when
  COMPOSIO_API_KEY unset): connect/init, callback (302), connections list,
  delete
- router wiring gated by COMPOSIO_API_KEY; COMPOSIO_AUTH_CONFIGS_JSON maps
  toolkit->auth_config (MVP: notion); state secret from COMPOSIO_STATE_SECRET
  or derived from JWT_SECRET; callback base from COMPOSIO_CALLBACK_BASE_URL
  or MULTICA_PUBLIC_URL
- tests: state (expire/tamper/wrong-secret), service (mapping, callback
  idempotency, non-success, disconnect owner/404 idempotency, MCP pin),
  handlers (httptest), redact regression for Bearer mcp_ tokens

MVP scope: Notion only; no task-dispatch overlay, sharing, or webhook
event handling (later stages).

Co-authored-by: multica-agent <github@multica.ai>

* fix(composio): bind callback account to user + idempotent revoked disconnect (MUL-3720)

Address PR 4608 review (CHANGES_REQUESTED):

- callback: verify connected_account_id with Composio before mirroring it.
  The signed state only proved user/toolkit/exp, so a valid state paired with
  a tampered connected_account_id would be written verbatim. CompleteCallback
  now calls ListConnectedAccounts and fails closed (ErrAccountVerification)
  unless the account belongs to the state's user (composio_user_id == multica
  user id) and was created under the toolkit's auth config. No row is written
  on mismatch / unknown account / upstream error.

- disconnect: short-circuit to a no-op when the local row is already revoked,
  before touching upstream. Previously a second DELETE re-hit Composio and a
  non-404 upstream error surfaced as a 502, breaking the 204-idempotent
  contract.

- CreateMCPSession: document the v1 single-active-connection-per-(user,toolkit)
  constraint and make duplicate selection deterministic (newest-wins, rows are
  connected_at DESC) instead of order-dependent map overwrite. Stage 3 owns the
  real single-account-enforcement vs multi-account-shape decision.

Tests: tampered/wrong-auth-config/unknown-account callback rejection, revoked-row
disconnect no-op (asserts upstream not re-hit). composio pkg 85% coverage; all
green.

Co-authored-by: multica-agent <github@multica.ai>

* feat(composio): list all toolkits + dynamic auth-config resolution (MUL-3720)

Yushen's follow-up to the Notion MVP: surface the full Composio toolkit
catalog, render it in Settings, and drop the static env mapping in favor of
dynamic auth-config discovery.

Config correctness (per Composio docs):
- Remove COMPOSIO_AUTH_CONFIGS_JSON entirely. The toolkit→auth_config mapping
  is now resolved at request time from the project's /auth_configs (cached,
  5-min TTL), so enabling a toolkit is a dashboard action, not a redeploy.
- Do NOT add COMPOSIO_PROJECT_ID. The project API key (x-api-key) authenticates
  to exactly one project; the project is resolved from the key. Only org-level
  endpoints use x-org-api-key, which this integration never calls.

Backend:
- SDK: server/pkg/composio/auth_configs.go — ListAuthConfigs (toolkit_slug,
  is_composio_managed, show_disabled, limit, cursor).
- service: dynamic resolver (authConfigMap cache; betterAuthConfig prefers a
  custom/white-label config over Composio-managed, newest wins); BeginConnect
  and CompleteCallback resolve via it; ListToolkits fetches the full catalog
  (paginated, capped) annotated with connectable = has an enabled auth config,
  connectable-first ordering.
- handler + route: GET /api/integrations/composio/toolkits (user-scoped, 503
  when COMPOSIO_API_KEY unset) returning slug/name/logo/category/connectable.

Frontend:
- core: ComposioToolkit/ComposioConnection types, api client methods, and
  composio query options (@multica/core/composio).
- views: Settings → Integrations now has a Composio section rendering every
  toolkit as a card with search. Connect is gated on `connectable`;
  non-connectable toolkits show a muted "not configured" hint instead of a
  dead button. Connected toolkits show a badge + Disconnect (with confirm).
- i18n: composio block added to en/zh-Hans/ja/ko settings.

Tests: SDK + service (dynamic resolution, custom-over-managed preference,
connectable flag, resolver-error soft-degrade) and handler toolkits endpoint;
composio pkg 85.7% coverage. go build/vet/gofmt clean; core+views typecheck,
core+views lint, and core tests (691) all green.

Co-authored-by: multica-agent <github@multica.ai>

* fix(composio): close cross-toolkit callback fail-open by signing auth_config_id into state (MUL-3720)

Re-review blocker: CompleteCallback resolved the toolkit's auth config at
callback time and ignored a resolve error/empty result, while
verifyAccountOwnership skipped the auth-config comparison when the expected
value was empty. A user could then pass another toolkit's connected_account_id
into this toolkit's callback — the owner check passed and it was written under
the wrong toolkit_slug/account binding.

Fix: the auth_config_id is already resolved in BeginConnect (before the state
is signed), so sign it into the state and compare it exactly at callback. No
re-resolve, no fail-open. verifyAccountOwnership now fails closed when the
expected auth config is empty (rejects instead of skipping) and requires an
exact match — closing the cross-toolkit binding gap.

Tests: state round-trips auth_config_id; BeginConnect signs it; callback
rejects wrong/cross-toolkit auth config and an empty (no-mapping) auth config
fails closed. composio pkg 85.2% coverage, all green.

Frontend (non-blocking): the Composio settings tab now surfaces an error when
the connections query fails instead of silently rendering everything as
unconnected.

Co-authored-by: multica-agent <github@multica.ai>

* fix(composio): hide Settings section entirely when integration unconfigured (MUL-3720)

Decision (option 2, hide-then-merge): don't show a card that leaks the internal
COMPOSIO_API_KEY env-var name to every end user. IntegrationsTab now gates the
whole Composio section (heading + body) on the toolkits query — a 503 means the
key is unset, so the section is withheld instead of rendering the not-configured
card. Admin-only setup guidance is a later, role-gated affordance.

Removed the notConfigured card (and now-unused ApiError import) from
ComposioTab; it only mounts when configured. views typecheck + lint clean.

Co-authored-by: multica-agent <github@multica.ai>

---------

Co-authored-by: multica-agent <github@multica.ai>

* feat(composio): Stage 2 frontend polish — callback toast, last_used & expired UI, e2e (MUL-3718) (#4688)

* feat(composio): callback toast + refresh, last_used & expired UI, e2e (MUL-3718)

Co-authored-by: multica-agent <github@multica.ai>

* fix(composio): real callback redirect route + StrictMode-safe toast dedup (MUL-3718 review)

Co-authored-by: multica-agent <github@multica.ai>

---------

Co-authored-by: multica-agent <github@multica.ai>

* fix(composio): callback endpoint should not require Multica auth (MUL-3843) (#4709)

* fix(composio): move OAuth callback out of the Auth group (MUL-3843)

Composio 302-redirects the browser to /api/integrations/composio/callback
at the end of the OAuth flow, but PR #4608 mounted it inside the cookie-auth
middleware group. When the session cookie is absent (expired session,
SameSite=Strict / Safari ITP, private window, self-hosted callback subdomain)
the Auth middleware returned a hard 401 and a JSON blob instead of the
settings redirect, breaking the flow.

Identity never came from the cookie anyway: it is carried by the HMAC-signed
state param that CompleteCallback verifies (signature, expiry, replay) and
cross-checked by verifyAccountOwnership; h.Composio == nil still 503s. So the
callback is registered alongside the other public OAuth/webhook routes; the
other four composio endpoints stay session-gated.

Refs MUL-3843, MUL-3715.

Co-authored-by: multica-agent <github@multica.ai>

* fix(composio): correct stale callback routing comments (MUL-3843)

The package header and ComposioCallback doc comments still described the
callback as sitting under the Auth middleware group. After the route was
moved out (this PR), update both to state it is a public route whose identity
comes from the signed state — addressing review nit from 张大彪.

Refs MUL-3843.

Co-authored-by: multica-agent <github@multica.ai>

---------

Co-authored-by: multica-agent <github@multica.ai>

* feat(composio): inject MCP overlay into agent runtime at task dispatch (MUL-3721) (#4704)

Stage 3 of the Composio epic. Wires the per-user Composio MCP session into
every agent task so the agent process sees the initiator's connected tools
without any prompt-time plumbing.

Server side
  - Migration 128 adds agent_task_queue.runtime_mcp_overlay JSONB plus a
    BEFORE-UPDATE trigger that wipes the column on any transition into a
    terminal status (completed / failed / cancelled). A trigger is the single
    source of truth — future queries that flip status cannot bypass it.
  - composio.Service.BuildTaskOverlay(userID) reuses CreateMCPSession and
    emits the Claude-style { mcpServers: { composio: { type: http, url,
    headers } } } shape the daemon's existing sidecar generators consume.
    Returns (nil, nil) on zero active connections so we never burn a
    Composio session for a user with nothing to call.
  - TaskService grows a Composio ComposioOverlayBuilder seam, wired in
    router.go after composiointeg.NewService succeeds. Five enqueue paths
    (issue / mention / quick-create / chat / auto-retry) attach the overlay
    after CreateAgentTask returns and before the daemon is notified — so
    every claim reads a settled row, with no second daemon hop. Best-effort:
    a builder failure logs and proceeds with no overlay.
  - resolveInitiatorFromTriggerComment derives the initiator user from the
    trigger comment when it was authored by a member. Agent-authored
    triggers are not treated as initiators (their connected-apps view is
    empty by construction).

Daemon side
  - handler/daemon.go claim path merges task.runtime_mcp_overlay onto
    agent.mcp_config via mergeMCPOverlay before populating
    TaskAgentData.McpConfig. Overlay wins on server-name collisions
    because it carries the live user-scoped session URL. Errors fall back
    to the agent config unchanged — a bad overlay must not surprise-disable
    saved MCP tools. The existing execenv sidecar generators (cursor /
    codex / openclaw / opencode / hermes-kiro) need no changes: they keep
    consuming the merged result through TaskAgentData.McpConfig.

Tests
  - 9 merge cases (mcp_overlay_test): both-nil short-circuit, agent-only
    pass-through, overlay-only canonicalization, two-side merge, name
    collision (overlay wins), top-level key preservation, malformed agent
    fallback, malformed overlay fallback, non-object server rejection.
  - 4 dispatch cases (composio): zero-connections returns nil without
    CreateSession, happy-path emits the right shape with the right user
    id, empty-URL defensive branch, SDK error surfacing.
  - 4 TaskService helper cases: nil Composio is a no-op (Queries-safe),
    invalid initiator does not call the builder, nil overlay skips the
    UPDATE, builder error swallowed without panic.
  - Migration 128 verified to roll up + down + up cleanly against the test
    database.

Out of scope (deferred): assignment-triggered enqueue paths with no
trigger comment get no overlay attached today (no initiator UUID flows
through enqueueIssueTask in that case). Retry paths recompute the overlay
fresh from the parent's initiator_user_id instead of inheriting the bearer
from the parent row, so a stale token can never resurface on a retry.

Co-authored-by: Eve <eve@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>

* feat(composio): per-agent allowlist + originator-scoped MCP overlay (MUL-3869) (#4736)

* feat(composio): per-agent allowlist + originator-scoped MCP overlay (MUL-3869)

Stage 3.1 of the Composio epic (MUL-3721 parent). PR #4704 wired in the
runtime_mcp_overlay column and a per-task dispatch hook; this change
inverts the default from "all-on" to opt-in and locks the overlay to the
agent owner's own connected apps:

- Agents carry composio_toolkit_allowlist TEXT[]. NULL or [] => no MCP.
  Owner-only read/write; non-owner GET/PUT silently redacts/drops the
  field (same shape as mcp_config).
- agent_task_queue carries originator_user_id UUID. Set from the
  top-of-chain HUMAN at every enqueue path:
    * issue/mention comment by member  -> author_id
    * issue/mention comment by agent   -> inherit via comment.source_task_id
                                          -> parent task originator_user_id
    * quick-create                     -> requester_id
    * chat                             -> initiator_user_id
    * retry                            -> SQL-inherited from parent row
    * autopilot                        -> NULL (system-driven)
- BuildTaskOverlay (composio dispatch) now takes (ctx, originatorUserID,
  agent) and short-circuits on five gates: invalid originator,
  originator != agent.owner_id, empty allowlist, empty intersection of
  allowlist ∩ active connections, defensive empty session URL. Composio
  CreateSession is called with BOTH `toolkits.slugs` (the intersection)
  AND `connected_accounts` (the pinned account ids), narrowing the
  tool-router twice.
- The originator-vs-owner gate closes the agent-fanout privacy hole: any
  workspace member who can @-mention a public agent used to project the
  owner's connected apps into their run. Now the overlay only mounts
  when the human at the top of the chain IS the agent owner.

Tests:
- dispatch_test.go covers all 5 gates plus uppercase/whitespace slug
  normalisation.
- task_runtime_mcp_overlay_test.go covers the no-op gates of the new
  applyRuntimeMCPOverlay signature.
- agent_composio_allowlist_test.go (handler): owner roundtrip
  (list/empty/null), workspace-admin silent-drop, owner-only GET
  visibility, pure normaliseComposioToolkitAllowlist.
- resolve_originator_test.go (service, DB-backed): member-authored,
  agent-authored inherits via comment.source_task_id, invalid id.

Migration 129 up/down/up verified against docker postgres.

Co-authored-by: multica-agent <github@multica.ai>

* chore(composio): gofmt + regenerate sqlc with v1.31.1 (MUL-3869 review nits)

Co-authored-by: multica-agent <github@multica.ai>

---------

Co-authored-by: Eve <eve@multica-ai.local>
Co-authored-by: multica-agent <github@multica.ai>

* fix(composio): accept nested connected account auth config

* feat(views): creator-only MCP tab for per-agent Composio allowlist (MUL-3870) (#4743)

Stage 3.2 frontend on top of the Stage 3.1 backend (MUL-3869, 4708dba97).
Adds an agent-detail tab that lets the agent owner pick which of their own
active Composio connections this agent may mount as MCP servers, writing the
selection to agent.composio_toolkit_allowlist via the existing PUT /api/agents.

- core/types: composio_toolkit_allowlist (+ _redacted) on Agent; tri-state
  composio_toolkit_allowlist on UpdateAgentRequest (omit/no-change, null/clear,
  array/replace), matching the backend contract.
- core/agents: useUpdateAgentAllowlist - optimistic mutation hook (patches the
  cached workspace agent list, rolls back on error, invalidates on settle).
- views: AgentMcpTab renders the owner's active connections as checkboxes;
  empty state links to Settings -> Integrations; defensive redacted state.
- views: wired into AgentOverviewPane as tab "composio_mcp", labeled "MCP Apps"
  to disambiguate from the existing raw-JSON "MCP" (mcp_config) tab. The entry
  is gated to the creator (currentUserId === agent.owner_id), matching the
  backend's owner-only read/write of the allowlist.
- i18n: tabs.composio_mcp + tab_body.composio_mcp.* in en/ja/ko/zh-Hans.
- tests: agent-mcp-tab.test.tsx (gating, toggle->allowlist body, active-only,
  empty, redacted); e2e/agent-mcp.spec.ts (creator sees tab + PUT body,
  non-creator hidden) with Composio + agent endpoints mocked at the boundary.

Note: the product spec says "creator"; the schema has no creator_id - the
backend gate and redaction are keyed on owner_id, so the tab uses owner_id.

Co-authored-by: multica-agent <github@multica.ai>

* fix(composio): mount remote MCP for codex

* feat(agents): agent invocation permission system (MUL-3963) (#4844)

* feat(agents): agent invocation permission system (permission_mode + invocation targets)

MUL-3963: split who may INVOKE an agent out of the overloaded visibility
column into an explicit, extensible model on feature/composio-integration.

- DB: agent.permission_mode (private|public_to) + agent_invocation_target
  table (workspace/member/team targets) + lossless backfill from visibility
  (migration 130).
- canInvokeAgent: owner-only for private (NO admin bypass, NO A2A bypass);
  public_to honours the allow-list; A2A judged by the top-of-chain originator.
- All trigger paths rewired: issue assign, comment @agent/@squad, chat,
  quick-create, autopilot, squad leader, child-done.
- Agent API: permission_mode + invocation_targets on responses and
  create/update (owner-only writes); legacy visibility kept as a derived field
  so old clients never see a permission widening.
- Composio: BuildTaskOverlay now FOLLOWS invocation permission and uses the
  agent OWNER connection (removed the originator==owner gate); front-end warns
  when a shared agent enables Composio apps.
- CLI: --permission-mode / --public-to-workspace / --public-to-member (legacy
  --visibility still mapped).
- Frontend: AccessPicker (Private / workspace / specific people / team soon),
  permission rules mirror canInvokeAgent, Composio warning banner.
- Tests: migration backfill, admin cannot invoke others private, public_to
  workspace/member whitelist, A2A by originator, Composio overlay uses owner
  connection.

Co-authored-by: multica-agent <github@multica.ai>

* feat(agents): stackable, mixed public_to invocation targets (MUL-3963)

Follow-up on PR #4844: public_to now supports selecting MULTIPLE, MIXED
targets on one agent (e.g. Public to workspace + specific people + team),
with canInvokeAgent admitting on ANY matching target (OR).

- Frontend AccessPicker: reworked from a single exclusive kind into a
  stackable multi-select — an "Everyone in workspace" toggle, a member
  multi-select checklist, and a (disabled, v1) team placeholder can be
  combined freely. Emits the full union of selected targets; empty union
  collapses to Private. Existing team targets are preserved across saves.
  Added the access.public_group locale string (en/zh-Hans/ja/ko).
- Backend already supported this (agent_invocation_target is multi-row per
  agent; create/update take a target ARRAY and batch-replace the whole
  allow-list; canInvokeAgent OR-matches). Added tests to lock it in:
  mixed member+team targets, overlapping-member batch replace, and
  workspace+member stacking then narrowing.

Refs MUL-3963.

Co-authored-by: multica-agent <github@multica.ai>

* fix(agents): address review on invocation permission (MUL-3963)

张大彪 review on PR #4844 — three blockers + product ruling + nits:

1. Migration 130: drop the FK/cascade on agent_invocation_target
   (agent_id, created_by) per the Multica no-FK rule; relationships are now
   maintained in the app layer (matching MUL-3515 §4). Added
   DeleteAgentInvocationTargetsByArchivedRuntimeAgents and call it before
   DeleteArchivedAgentsByRuntime in all three runtime-delete paths
   (runtime.go x2, runtime_profile.go) so hard-deleting agents can't orphan
   target rows.
2. revokeAndRemoveMember: prune the leaving member's member-target grants
   (DeleteAgentInvocationTargetsByMember) in the same tx as the member-row
   delete, so a re-invited user can't reclaim a stale invocation grant.
3. Empty public_to is a phantom — parsePermissionInput now normalises a
   public_to with no resolvable targets to a single workspace target, so
   `--permission-mode public_to` alone (and any empty target array) means
   "public to workspace" instead of "shared but nobody can run it".

Product ruling: the system/no-human-originator → workspace-target path in
canInvokeAgent is a deliberate, documented exception (webhook/system/
workspace-wide automation); member/team targets still fail closed without a
resolved originator. Documented in code + locked with a test.

Nits: refreshed the stale "originator must be owner" comments — models.go
(via migration 130 COMMENT ON COLUMN + sqlc regen for composio_toolkit_allowlist
and originator_user_id) and agent-mcp-tab.tsx — to the owner-connection +
invocation-permission rules.

Tests: member remove/re-add regression, system workspace exception + member
fail-closed, empty public_to → workspace (plus the earlier mixed/overlap/
batch-replace suite). Migration 130 applied to the test DB; Go handler/service/
composio suites green; views typecheck clean.

Refs MUL-3963.

Co-authored-by: multica-agent <github@multica.ai>

* fix(agents): scope member invocation-target cleanup to one workspace (MUL-3963)

张大彪 3rd review — cross-workspace permission bug + comment nits:

- DeleteAgentInvocationTargetsByMember was a GLOBAL delete by user id, so
  removing a user from workspace A also wiped their member-target grants on
  agents in workspace B. Scoped it to a single workspace by joining through
  agent.workspace_id; revokeAndRemoveMember now passes (workspaceID, userID).
- Regression test TestRevokeMember_InvocationTargetCleanupIsWorkspaceScoped:
  same user allow-listed by agents in two workspaces; removal from one leaves
  the other workspace's target intact.
- Nits: refreshed the remaining stale "originator == agent.owner_id" /
  "owner-vs-originator" comments — CreateRetryTask (agent.sql, regenerated),
  and the AgentResponse allowlist doc + ListAgents/UpdateAgent redaction
  rationale in agent.go — to the owner-connection + invocation-permission rule.

Migration 130 applied to the test DB; Go handler/service/composio suites green;
go vet clean.

Refs MUL-3963.

Co-authored-by: multica-agent <github@multica.ai>

---------

Co-authored-by: multica-agent <github@multica.ai>

* fix(agents): agent access owner-only editable, read-only for others (MUL-3963) (#4853)

* fix(agents): make agent access owner-only editable, read-only for others (MUL-3963)

Interaction bug: a non-owner (incl. workspace admin) could open the AccessPicker
and set an agent public — the backend silently ignored it and the UI bounced
back to private. Access is owner-only, so non-owners must see a read-only state
and the backend must reject real changes explicitly.

Frontend:
- AccessPicker renders a static, non-interactive read-only state when the
  viewer is not the owner: the current access value + a lock affordance + a
  tooltip "Only the agent owner can change who can run this agent." No clickable
  trigger is rendered, so a non-owner can never open a control the backend would
  reject (the GitHub/Notion pattern for permission settings you can see but not
  edit). The editable multi-select picker is unchanged for the owner.
- agent-detail-inspector gates the picker on ownership specifically
  (currentUserId === agent.owner_id), NOT the general canEdit (which also admits
  admins, who may edit other fields but not access).
- New locale key access.owner_only_readonly (en/zh-Hans/ja/ko).

Backend:
- UpdateAgent now returns an explicit 403 when a non-owner submits a REAL
  permission change (permissionInputChangesAgent compares requested mode +
  target set against the persisted state); a no-op resubmit (admin PATCH-as-PUT
  echoing unchanged permission) is still tolerated so admin edits of other
  fields keep working. Replaces the previous silent-drop that caused the bounce.

Tests:
- access-picker.test.tsx: non-owner gets a non-interactive read-only display
  with the owner-only tooltip; owner gets an interactive picker; owner can pick
  a member and stack workspace + member.
- TestUpdateAgent_AccessChangeIsOwnerOnly: admin real change → 403; admin no-op
  resubmit → 200; admin editing other fields → 200; owner change → 200.

Incidental: fixed a pre-existing base typecheck break in
slash-command-suggestion.test.tsx (stray `signal` arg not in the suggestion
items type) that otherwise fails the whole @multica/views typecheck.

Refs MUL-3963.

Co-authored-by: multica-agent <github@multica.ai>

* fix(agents): compare legacy visibility, not expanded permission, for no-op detection (MUL-3963)

PR #4853 review: permissionInputChangesAgent expanded a legacy-only
visibility:"private" into a real private permission and compared it against the
agent's actual permission. A member-only public_to agent derives legacy
visibility "private", so an admin PATCH-as-PUT echoing visibility:"private"
while editing another field was misread as a public_to→private downgrade and
rejected with 403 — contradicting the "unchanged permission no-op is allowed"
contract.

Fix (per review): when a request carries ONLY legacy `visibility` (no
permission_mode / invocation_targets), derive the agent's CURRENT legacy
visibility from its real targets and compare the legacy string values. Equal =
no-op (allowed); a real legacy change (e.g. "workspace") still returns 403.
Requests that carry permission_mode / invocation_targets keep the precise
mode+target comparison.

Regression test TestUpdateAgent_LegacyVisibilityNoOpForMemberOnlyPublicTo:
member-only public_to agent — admin submitting visibility:"private" + a
non-permission field → 200 with targets unchanged; admin submitting
visibility:"workspace" → 403.

Go handler/composio suites green; migration 130 applied; go vet clean.

Refs MUL-3963.

Co-authored-by: multica-agent <github@multica.ai>

---------

Co-authored-by: multica-agent <github@multica.ai>

* feat(composio): brief agents on connected apps

* feat(composio): gate MCP apps behind feature flag

* fix(mobile): parse agent invocation permissions

* fix(tests): update agent fixtures for access fields

---------

Co-authored-by: multica-agent <github@multica.ai>
Co-authored-by: Multica Eve <eve@devv.ai>
Co-authored-by: Eve <eve@multica.ai>
Co-authored-by: Eve <eve@multica-ai.local>
2026-07-03 14:18:43 +08:00

2195 lines
68 KiB
Go

package agent
import (
"bufio"
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"log/slog"
"os"
"os/exec"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"sync"
"sync/atomic"
"syscall"
"time"
)
// codexBlockedArgs are flags hardcoded by the daemon that must not be
// overridden by user-configured custom_args. The mcp_servers config keys
// live in the per-task `$CODEX_HOME/config.toml` (written by
// ensureCodexMcpConfig); user-supplied `-c mcp_servers.…` overrides are
// stripped separately by filterCodexCustomConfigOverrides because they
// share the `-c` flag with legitimate non-MCP overrides like `-c model=…`.
var codexBlockedArgs = map[string]blockedArgMode{
"--listen": blockedWithValue, // stdio:// transport for daemon communication
}
// codexStderrTailBytes bounds the stderr tail captured for inclusion in
// error messages when codex exits before the JSON-RPC handshake (e.g. the
// user supplied a custom_args flag that the `app-server` subcommand
// rejects). Kept as its own constant so bumping codex independently of
// other agents stays easy if codex starts shipping longer failure traces.
const (
codexStderrTailBytes = 2048
defaultCodexSemanticInactivityTimeout = 10 * time.Minute
defaultCodexFirstTurnNoProgressTimeout = 30 * time.Second
codexVersionDiagnosticTimeout = 2 * time.Second
// codexGracefulShutdownTimeout bounds how long the lifecycle goroutine
// waits for codex to exit on its own after stdin is closed, before forcing
// a context-cancel kill. A clean exit lets codex run its shutdown path and
// flush buffered telemetry — OTEL batch exporters only force-flush on
// graceful shutdown, so killing it immediately (the prior behavior) drops
// the task's spans/metrics/logs.
codexGracefulShutdownTimeout = 10 * time.Second
)
// codexGracefulShutdownTimeoutNanos optionally overrides
// codexGracefulShutdownTimeout for tests, in nanoseconds. Zero or negative
// values keep the production default. Tests for the cleanup-on-scanner-
// overflow path (#4520) use it to shrink the grace window from 10 s to a
// few hundred ms so the regression runs in a normal `go test` budget
// instead of burning two full grace windows per cleanup phase. Mirrors
// the opencodeTerminateGraceNanos hook.
var codexGracefulShutdownTimeoutNanos atomic.Int64
func codexGracefulShutdown() time.Duration {
if n := codexGracefulShutdownTimeoutNanos.Load(); n > 0 {
return time.Duration(n)
}
return codexGracefulShutdownTimeout
}
// CodexSemanticInactivityMarker prefixes timeout errors emitted when Codex
// stops making semantic progress while the process is still alive.
const CodexSemanticInactivityMarker = "codex semantic inactivity timeout"
// CodexFirstTurnNoProgressMarker identifies the app-server failure mode where
// Codex accepts a turn and then never emits any item, completion, or error.
const CodexFirstTurnNoProgressMarker = "codex app-server no progress timeout"
const codexModelCatalogRefreshTimeoutSignal = "failed to refresh available models: timeout waiting for child process to exit"
var errCodexProcessExited = errors.New("codex process exited")
type codexTimeoutKind int
const (
codexTimeoutNone codexTimeoutKind = iota
codexTimeoutSemanticInactivity
codexTimeoutFirstTurnNoProgress
)
type codexTimeoutDiagnostic struct {
Kind codexTimeoutKind
Timeout time.Duration
LastActivity string
ThreadID string
TurnID string
Model string
CodexVersion string
}
// codexBackend implements Backend by spawning `codex app-server --listen stdio://`
// and communicating via JSON-RPC 2.0 over stdin/stdout.
type codexBackend struct {
cfg Config
}
func buildCodexArgs(opts ExecOptions, logger *slog.Logger) []string {
args := []string{"app-server", "--listen", "stdio://"}
extra := filterCustomArgs(opts.ExtraArgs, codexBlockedArgs, logger)
custom := filterCustomArgs(opts.CustomArgs, codexBlockedArgs, logger)
// Only claim ownership of the `mcp_servers` namespace when the agent
// actually has a managed mcp_config in the MCP Tab. Otherwise existing
// users who configure MCP via `custom_args: ["-c", "mcp_servers.…"]`
// would silently lose those entries after this PR ships. With managed
// mcp_config present, daemon-written `$CODEX_HOME/config.toml` is the
// authoritative source and stray `-c mcp_servers.*` overrides are
// dropped to keep last-wins from re-shadowing it.
if hasManagedCodexMcpConfig(opts.McpConfig) {
extra = filterCodexCustomConfigOverrides(extra, logger)
custom = filterCodexCustomConfigOverrides(custom, logger)
}
args = append(args, extra...)
args = append(args, custom...)
return args
}
// hasManagedCodexMcpConfig reports whether the agent's mcp_config field is
// "present" in the API three-state sense: a non-null JSON value. Both
// `{}` and `{"mcpServers":{}}` count as present (the admin saved an empty
// managed set — strict mode, no global fallback); only SQL NULL or the
// literal JSON `null` count as absent (CLI default).
func hasManagedCodexMcpConfig(raw json.RawMessage) bool {
trimmed := bytes.TrimSpace(raw)
if len(trimmed) == 0 {
return false
}
if bytes.Equal(trimmed, []byte("null")) {
return false
}
return true
}
// codexManagedMcpConfigKeyRe matches the daemon-managed config namespace
// (`mcp_servers.…`) when it appears as the value of a Codex `-c` /
// `--config` flag. Used by filterCodexCustomConfigOverrides to drop user
// overrides that would otherwise shadow what the MCP Tab writes into
// `$CODEX_HOME/config.toml`.
var codexManagedMcpConfigKeyRe = regexp.MustCompile(`^\s*mcp_servers(?:\s*\.|\s*=|\s*$)`)
// filterCodexCustomConfigOverrides drops `-c mcp_servers.…=` and
// `--config mcp_servers.…=` entries from custom args. Codex's `-c` is
// last-wins (verified against codex-cli 0.132.0), so without this filter a
// user-written `-c mcp_servers.fetch=…` in custom_args would silently
// override whatever the MCP Tab saved into the per-task config.toml. We
// own the `mcp_servers` namespace via the managed block, so user attempts
// to write into it are dropped with a warning rather than allowed to win.
// Other `-c`/`--config` keys (e.g. `-c model="o3"`) pass through unchanged.
func filterCodexCustomConfigOverrides(args []string, logger *slog.Logger) []string {
if len(args) == 0 {
return args
}
filtered := make([]string, 0, len(args))
for i := 0; i < len(args); i++ {
arg := args[i]
flag := arg
inlineValue := ""
hasInlineValue := false
if idx := strings.Index(arg, "="); idx > 0 {
flag = arg[:idx]
inlineValue = arg[idx+1:]
hasInlineValue = true
}
if flag == "-c" || flag == "--config" {
value := inlineValue
if !hasInlineValue && i+1 < len(args) {
value = args[i+1]
}
if codexManagedMcpConfigKeyRe.MatchString(value) {
if logger != nil {
// Log the key only, never the value — mcp_servers.<name>.env
// is allowed to carry secrets and the whole point of moving
// this to config.toml is to keep raw values out of logs/argv.
key := value
if eqIdx := strings.Index(value, "="); eqIdx >= 0 {
key = value[:eqIdx]
}
logger.Warn("custom_args: blocked mcp_servers override; daemon manages this via CODEX_HOME/config.toml",
"flag", flag, "key", strings.TrimSpace(key))
}
if !hasInlineValue && i+1 < len(args) {
i++ // skip the value arg
}
continue
}
}
filtered = append(filtered, arg)
}
return filtered
}
// Markers delimiting the daemon-managed `[mcp_servers.*]` block in
// `$CODEX_HOME/config.toml`. Match the existing sandbox / multi-agent /
// memory marker pattern so ops can grep all managed blocks consistently.
const (
multicaCodexMcpBeginMarker = "# BEGIN multica-managed mcp_servers (do not edit; regenerated by daemon)"
multicaCodexMcpEndMarker = "# END multica-managed mcp_servers"
)
var codexMcpBlockRe = regexp.MustCompile(
`(?ms)^` + regexp.QuoteMeta(multicaCodexMcpBeginMarker) +
`.*?^` + regexp.QuoteMeta(multicaCodexMcpEndMarker) + `\n*`)
// userCodexMcpServersTableHeaderRe matches `[mcp_servers.<name>]` (and its
// quoted-key form `[mcp_servers."<name>"]`) at the start of a line. Used
// to strip user-provided mcp_servers tables from the per-task config when
// the agent has its own mcp_config — mirrors Claude's `--strict-mcp-config`
// model where the daemon's set is authoritative.
var userCodexMcpServersTableHeaderRe = regexp.MustCompile(
`^\s*\[\s*mcp_servers\s*\.\s*(?:"[^"]*"|[^\]\s]+)\s*\]\s*(?:#.*)?$`)
// ensureCodexMcpConfig writes (or clears) the daemon-managed
// `[mcp_servers.*]` block in `$CODEX_HOME/config.toml`. The block is the
// authoritative source of MCP servers for this run: with mcp_config set
// in the agent UI the daemon also strips any inherited
// `[mcp_servers.*]` tables from the per-task config so the user's global
// `~/.codex/config.toml` doesn't shadow or collide with the managed set.
//
// The file mode is 0o600 because `mcp_servers.<id>.env` values may carry
// secrets (API keys, bearer tokens); the per-task home is owned by the
// daemon's user, so 0o600 keeps secrets out of any world-readable copy
// while still letting the codex child read them.
//
// A malformed mcp_config is returned as an error and the caller decides
// whether to surface or warn — same fail-soft contract the prior argv
// path had.
func ensureCodexMcpConfig(configPath string, mcpConfig json.RawMessage, logger *slog.Logger) error {
data, err := os.ReadFile(configPath)
if err != nil && !os.IsNotExist(err) {
return fmt.Errorf("read config.toml: %w", err)
}
existing := string(data)
// Always strip a prior managed block so reruns and clear-config flows
// converge on a clean state.
stripped := codexMcpBlockRe.ReplaceAllString(existing, "")
managed := hasManagedCodexMcpConfig(mcpConfig)
block, _, renderErr := renderCodexMcpServersBlock(mcpConfig)
if renderErr != nil {
return renderErr
}
var updated string
if managed {
// Agent has a managed MCP set (possibly empty — `{}` /
// `{"mcpServers":{}}` count as "saved an empty set" in the API's
// three-state semantics, distinct from nil/null which means
// "fall back to CLI default"). Strip any user-defined
// `[mcp_servers.*]` tables inherited from `~/.codex/config.toml`
// so the managed set is strict — mirrors Claude's
// `--strict-mcp-config`. Two reasons we cannot mix:
// 1. TOML rejects redefining the same table; a user table
// named `[mcp_servers.fetch]` would crash codex if the
// agent also defined `fetch`.
// 2. An admin saving an explicit list in the MCP Tab would
// otherwise see user-global servers silently joined in,
// which contradicts the UI affordance.
stripped = stripCodexUserMcpServerTables(stripped)
stripped = strings.TrimRight(stripped, "\n")
// When the managed set is empty we still write the marker
// block (with no tables between). This pins "managed but
// empty" on disk so the next run can find and strip the
// markers, and so the file's intent is grep-able by ops.
if block == "" {
block = multicaCodexMcpBeginMarker + "\n" + multicaCodexMcpEndMarker + "\n"
}
if stripped == "" {
updated = block
} else {
updated = stripped + "\n\n" + block
}
} else {
// No managed config: just remove any prior managed block and
// leave inherited user tables alone (CLI default fallback).
updated = stripped
}
if updated == existing {
return nil
}
if err := os.WriteFile(configPath, []byte(updated), 0o600); err != nil {
return fmt.Errorf("write config.toml: %w", err)
}
// os.WriteFile applies the mode only when creating a new file; if the
// per-task config.toml was already on disk at 0o644 (the default mode
// used by execenv.copyFile when seeding from ~/.codex/config.toml),
// the secret-bearing values we just wrote would inherit that wider
// mode. Chmod unconditionally to keep the secret in the daemon
// owner's lane regardless of the prior mode.
if err := os.Chmod(configPath, 0o600); err != nil {
return fmt.Errorf("chmod config.toml to 0600: %w", err)
}
if logger != nil {
logger.Debug("codex: wrote managed mcp_servers block to config.toml",
"config_path", configPath, "managed", managed)
}
return nil
}
// renderCodexMcpServersBlock renders the agent's mcp_config JSON
// (Claude-style `{"mcpServers": {...}}`) as a TOML block of
// `[mcp_servers.<name>]` tables wrapped in BEGIN/END markers. Returns
// (block, hasServers, err); hasServers=false means the input had no
// servers to render (empty/null mcp_config) and the caller should only
// strip the prior managed block.
//
// Stdio server keys (`args`, `env`, `command`) pass through verbatim —
// Codex's config schema happens to use the same names today. Remote HTTP
// servers use Codex-specific keys, so they are normalised here rather than
// leaking provider details into the UI/dispatch layer.
func renderCodexMcpServersBlock(raw json.RawMessage) (string, bool, error) {
if len(raw) == 0 {
return "", false, nil
}
var parsed struct {
McpServers map[string]json.RawMessage `json:"mcpServers"`
}
if err := json.Unmarshal(raw, &parsed); err != nil {
return "", false, fmt.Errorf("parse mcp_config json: %w", err)
}
if len(parsed.McpServers) == 0 {
return "", false, nil
}
names := make([]string, 0, len(parsed.McpServers))
for name := range parsed.McpServers {
names = append(names, name)
}
sort.Strings(names)
var sb strings.Builder
sb.WriteString(multicaCodexMcpBeginMarker)
sb.WriteString("\n")
for i, name := range names {
if !isCodexBareTomlKey(name) {
return "", false, fmt.Errorf("mcp server name %q must be ASCII alphanumeric / _ / - to fit Codex's bare-key requirement", name)
}
var serverVal map[string]any
if err := json.Unmarshal(parsed.McpServers[name], &serverVal); err != nil {
return "", false, fmt.Errorf("mcp_servers.%s: %w", name, err)
}
if serverVal == nil {
return "", false, fmt.Errorf("mcp_servers.%s must be a JSON object", name)
}
serverVal = normalizeCodexMcpServerConfig(serverVal)
if i > 0 {
sb.WriteString("\n")
}
sb.WriteString("[mcp_servers.")
sb.WriteString(name)
sb.WriteString("]\n")
keys := make([]string, 0, len(serverVal))
for k := range serverVal {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
tomlValue, err := jsonValueToCodexTOMLInline(serverVal[k])
if err != nil {
return "", false, fmt.Errorf("mcp_servers.%s.%s: %w", name, k, err)
}
sb.WriteString(codexTOMLKey(k))
sb.WriteString(" = ")
sb.WriteString(tomlValue)
sb.WriteString("\n")
}
}
sb.WriteString(multicaCodexMcpEndMarker)
sb.WriteString("\n")
return sb.String(), true, nil
}
func normalizeCodexMcpServerConfig(server map[string]any) map[string]any {
if !isCodexRemoteMcpServer(server) {
return server
}
normalized := make(map[string]any, len(server)+1)
for k, v := range server {
switch k {
case "type":
continue
case "headers":
if _, ok := server["http_headers"]; !ok {
normalized["http_headers"] = v
}
default:
normalized[k] = v
}
}
normalized["experimental_use_rmcp_client"] = true
return normalized
}
func isCodexRemoteMcpServer(server map[string]any) bool {
if typ, ok := server["type"].(string); ok && strings.EqualFold(typ, "http") {
return true
}
_, hasURL := server["url"]
_, hasCommand := server["command"]
return hasURL && !hasCommand
}
// stripCodexUserMcpServerTables removes every `[mcp_servers.*]` table
// (header + body lines until the next top-level table header or EOF) from
// a TOML config string. Sub-tables like `[mcp_servers.fetch.env]` count
// as part of the parent table and are dropped along with it.
func stripCodexUserMcpServerTables(content string) string {
lines := strings.Split(content, "\n")
out := make([]string, 0, len(lines))
skipping := false
for _, line := range lines {
if userCodexMcpServersTableHeaderRe.MatchString(line) {
skipping = true
continue
}
if skipping {
trimmed := strings.TrimSpace(line)
if strings.HasPrefix(trimmed, "[") {
// Next table header. If it's still an `mcp_servers.*`
// table (including a sub-table), keep skipping; otherwise
// stop and emit this line.
if userCodexMcpServersTableHeaderRe.MatchString(line) ||
strings.HasPrefix(trimmed, "[mcp_servers.") ||
strings.HasPrefix(trimmed, "[ mcp_servers.") {
continue
}
skipping = false
out = append(out, line)
continue
}
continue
}
out = append(out, line)
}
return strings.Join(out, "\n")
}
// jsonValueToCodexTOMLInline serialises a JSON value as a TOML inline
// value. Only the subset Codex's `-c` accepts is supported: strings,
// numbers, booleans, arrays, and inline tables. JSON nulls are rejected
// because TOML has no null and silently dropping them would be confusing.
func jsonValueToCodexTOMLInline(v any) (string, error) {
switch x := v.(type) {
case nil:
return "", fmt.Errorf("null is not a valid TOML value")
case bool:
if x {
return "true", nil
}
return "false", nil
case float64:
if x == float64(int64(x)) {
return strconv.FormatInt(int64(x), 10), nil
}
return strconv.FormatFloat(x, 'f', -1, 64), nil
case string:
return codexTOMLBasicString(x), nil
case []any:
parts := make([]string, len(x))
for i, e := range x {
p, err := jsonValueToCodexTOMLInline(e)
if err != nil {
return "", err
}
parts[i] = p
}
return "[" + strings.Join(parts, ", ") + "]", nil
case map[string]any:
keys := make([]string, 0, len(x))
for k := range x {
keys = append(keys, k)
}
sort.Strings(keys)
parts := make([]string, len(keys))
for i, k := range keys {
p, err := jsonValueToCodexTOMLInline(x[k])
if err != nil {
return "", err
}
parts[i] = codexTOMLKey(k) + " = " + p
}
return "{ " + strings.Join(parts, ", ") + " }", nil
default:
return "", fmt.Errorf("unsupported value type %T", v)
}
}
func codexTOMLBasicString(s string) string {
var sb strings.Builder
sb.Grow(len(s) + 2)
sb.WriteByte('"')
for _, r := range s {
switch r {
case '\\':
sb.WriteString(`\\`)
case '"':
sb.WriteString(`\"`)
case '\b':
sb.WriteString(`\b`)
case '\t':
sb.WriteString(`\t`)
case '\n':
sb.WriteString(`\n`)
case '\f':
sb.WriteString(`\f`)
case '\r':
sb.WriteString(`\r`)
default:
if r < 0x20 || r == 0x7f {
sb.WriteString(fmt.Sprintf(`\u%04x`, r))
} else {
sb.WriteRune(r)
}
}
}
sb.WriteByte('"')
return sb.String()
}
func codexTOMLKey(s string) string {
if isCodexBareTomlKey(s) {
return s
}
return codexTOMLBasicString(s)
}
func isCodexBareTomlKey(s string) bool {
if s == "" {
return false
}
for _, r := range s {
switch {
case r >= 'a' && r <= 'z':
case r >= 'A' && r <= 'Z':
case r >= '0' && r <= '9':
case r == '_' || r == '-':
default:
return false
}
}
return true
}
func (b *codexBackend) Execute(ctx context.Context, prompt string, opts ExecOptions) (*Session, error) {
execPath := b.cfg.ExecutablePath
if execPath == "" {
execPath = "codex"
}
if _, err := exec.LookPath(execPath); err != nil {
return nil, fmt.Errorf("codex executable not found at %q: %w", execPath, err)
}
timeout := opts.Timeout
semanticInactivityTimeout := opts.SemanticInactivityTimeout
if semanticInactivityTimeout == 0 {
semanticInactivityTimeout = defaultCodexSemanticInactivityTimeout
}
runCtx, cancel := runContext(ctx, timeout)
// Materialise the agent's MCP config into the per-task
// `$CODEX_HOME/config.toml`. Argv would be the simpler path, but
// `mcp_servers.<id>.env` is allowed to carry secrets (Codex docs:
// https://developers.openai.com/codex/mcp#configure-with-configtoml)
// and our UI already treats mcp_config as a redacted-for-non-admins
// field. Process argv ends up in OS-level `ps` listings and is also
// echoed into the daemon's `agent command` log line below, so any
// inline env-bearing TOML would defeat the redaction. Writing through
// config.toml at 0o600 keeps the secret values out of argv and logs.
if codexHome := strings.TrimSpace(b.cfg.Env["CODEX_HOME"]); codexHome != "" {
if err := ensureCodexMcpConfig(filepath.Join(codexHome, "config.toml"), opts.McpConfig, b.cfg.Logger); err != nil {
// Fail closed when we can't materialise the managed config.
// Warning-and-launching would silently fall back to the
// user's global `~/.codex/config.toml` MCP servers and
// look indistinguishable from "the saved config was
// applied", which is exactly the surprise the MCP Tab is
// supposed to remove.
cancel()
return nil, fmt.Errorf("apply codex mcp_config: %w", err)
}
} else if hasManagedCodexMcpConfig(opts.McpConfig) {
// Managed mcp_config saved but no CODEX_HOME to anchor it.
// Same reasoning as above: silently launching would inherit
// whatever MCP setup the host user has, which is the wrong
// shape of failure.
cancel()
return nil, fmt.Errorf("codex: mcp_config is set but CODEX_HOME env var is not configured; cannot apply managed MCP")
}
codexArgs := buildCodexArgs(opts, b.cfg.Logger)
cmd := exec.CommandContext(runCtx, execPath, codexArgs...)
hideAgentWindow(cmd)
// Run codex in its own process group so a cancel-on-stuck cleanup
// reaches the whole tree — the codex Node wrapper plus the native
// Rust app-server it spawns — not just the direct child. Without
// this, killing the leader leaves grandchildren as orphans that
// keep consuming memory until the OS reaps them; see #4520, where a
// scanner overflow during thread/resume otherwise leaked Codex
// processes indefinitely. configureProcessGroup is a no-op on
// Windows.
configureProcessGroup(cmd)
// Override the default exec.CommandContext cancel behaviour. The
// default sends SIGKILL only to cmd.Process (the leader); we instead
// signal the whole process group so descendants die too. Returning
// nil keeps exec from logging a spurious error; cmd.WaitDelay below
// still backstops cmd.Wait() if the kill leaves an open pipe.
cmd.Cancel = func() error {
if cmd.Process != nil {
signalProcessGroup(cmd.Process, syscall.SIGKILL)
}
return nil
}
// Bound the wait after the context is cancelled so a stuck child (or an
// open pipe held by a grandchild) can't hang cmd.Wait() forever. Matches
// the other long-lived backends (claude, copilot, cursor, …).
cmd.WaitDelay = 10 * time.Second
b.cfg.Logger.Info("agent command", "exec", execPath, "args", codexArgs)
if opts.Cwd != "" {
cmd.Dir = opts.Cwd
}
cmd.Env = buildEnv(b.cfg.Env)
stdout, err := cmd.StdoutPipe()
if err != nil {
cancel()
return nil, fmt.Errorf("codex stdout pipe: %w", err)
}
stdin, err := cmd.StdinPipe()
if err != nil {
cancel()
return nil, fmt.Errorf("codex stdin pipe: %w", err)
}
stderrBuf := newStderrTail(newLogWriter(b.cfg.Logger, "[codex:stderr] "), codexStderrTailBytes)
cmd.Stderr = stderrBuf
if err := cmd.Start(); err != nil {
cancel()
return nil, fmt.Errorf("start codex: %w", err)
}
b.cfg.Logger.Info("codex started app-server", "pid", cmd.Process.Pid, "cwd", opts.Cwd)
msgCh := make(chan Message, 256)
resCh := make(chan Result, 1)
semanticActivityCh := make(chan string, 256)
var outputMu sync.Mutex
var output strings.Builder
// turnDone is set before starting the reader goroutine so there is no
// race between the lifecycle goroutine writing and the reader reading.
turnDone := make(chan bool, 1) // true = aborted
c := &codexClient{
cfg: b.cfg,
stdin: stdin,
pending: make(map[int]*pendingRPC),
processDone: make(chan struct{}),
notificationProtocol: "unknown",
onMessage: func(msg Message) {
logCodexAgentMessage(b.cfg.Logger, msg)
if msg.Type == MessageText {
outputMu.Lock()
output.WriteString(msg.Content)
outputMu.Unlock()
}
trySend(msgCh, msg)
trySendString(semanticActivityCh, describeCodexSemanticActivity(msg))
},
onSemanticActivity: func(description string) {
b.cfg.Logger.Debug("codex semantic activity observed", "activity", description)
trySendString(semanticActivityCh, description)
},
onTurnDone: func(aborted bool) {
select {
case turnDone <- aborted:
default:
}
},
}
// Start reading stdout in background
readerDone := make(chan struct{})
go func() {
defer close(readerDone)
scanner := bufio.NewScanner(stdout)
scanner.Buffer(make([]byte, 0, 1024*1024), 10*1024*1024)
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
if line == "" {
continue
}
c.handleLine(line)
}
if err := scanner.Err(); err != nil {
c.markProcessExited(fmt.Errorf("%w: %v", errCodexProcessExited, err))
return
}
c.markProcessExited(errCodexProcessExited)
}()
// drainAndWait closes stdin so codex shuts down, then joins cmd.Wait().
// cmd.Wait() is the only Go-stdlib-documented synchronization point for
// os/exec's internal stderr/stdout copy goroutines — until it returns,
// stderrBuf may not have observed every byte codex wrote before it
// exited, and stderrBuf.Tail() can come back empty or truncated. Any
// code that reads stderrBuf.Tail() must call drainAndWait() first.
// sync.Once makes it safe to call from both error paths and the deferred
// cleanup.
//
// drainAndWait is also the cleanup safety net for the scanner-overflow
// path (#4520). When codex emits a single stdout line larger than the
// scanner's MaxScanTokenSize, the reader goroutine returns with
// scanner.Err() set, fails all in-flight RPCs via markProcessExited, and
// closes readerDone — but the codex child process is still alive and is
// now blocked trying to write the rest of the oversized line into a
// stdout pipe nobody is reading. A naive stdin.Close()+cmd.Wait() then
// hangs forever: codex never reaches its stdin-read syscall, so it never
// sees EOF, never exits, and cmd.Wait() never returns. The lifecycle
// goroutine therefore never sends a failed Result, the outer daemon
// blocks on its result channel, and the higher-level fresh-session
// fallback never fires.
//
// To stay correct under both clean shutdown and the stuck-child case,
// drainAndWait runs in two bounded phases:
//
// 1. Close stdin and wait for the reader goroutine to finish, capped by
// codexGracefulShutdownTimeout. The reader exits when codex closes
// stdout on its own (clean shutdown — gives OTEL batch exporters a
// chance to flush) OR when the scanner errors out (overflow case —
// readerDone is already closed and the select returns immediately).
// Per os/exec docs, calling cmd.Wait() while reads are still
// in-flight on a StdoutPipe-returned pipe is incorrect because Wait
// closes the pipe and turns pending reads into spurious errors, so
// we must wait for the reader first.
//
// 2. Wait for cmd.Wait() to return, capped by another
// codexGracefulShutdownTimeout. Normally this returns immediately
// because the process has already exited. In the stuck-child case
// the process is still alive — we cancel the runCtx, which fires
// cmd.Cancel (the group-SIGKILL helper installed above), and
// cmd.WaitDelay then guarantees cmd.Wait() returns even if pipes
// stay open.
var waitOnce sync.Once
drainAndWait := func() {
waitOnce.Do(func() {
stdin.Close()
grace := codexGracefulShutdown()
// Phase 1: let the reader finish before invoking cmd.Wait().
select {
case <-readerDone:
// reader drained cleanly (codex shutdown closed stdout)
// or aborted early (e.g. scanner overflow). Either way it
// is now safe to call cmd.Wait().
case <-time.After(grace):
// codex did not close stdout within the grace window. Force
// the shutdown via context cancellation — cmd.Cancel
// group-kills the tree, the reader unblocks when stdout
// EOFs, and we proceed to phase 2.
b.cfg.Logger.Warn("codex did not close stdout after stdin EOF; forcing shutdown",
"pid", cmd.Process.Pid,
"grace", grace.String(),
)
cancel()
<-readerDone
}
// Phase 2: bound cmd.Wait() in case the process is still alive
// (scanner-overflow case: reader exited early on its own while
// codex stayed blocked writing into a full stdout pipe).
waitCh := make(chan struct{})
go func() {
_ = cmd.Wait()
close(waitCh)
}()
select {
case <-waitCh:
// reaped cleanly.
case <-time.After(grace):
b.cfg.Logger.Warn("codex process still alive after reader exited; forcing shutdown",
"pid", cmd.Process.Pid,
"grace", grace.String(),
)
cancel()
// WaitDelay (10s) is the final backstop: even if the
// group-kill races with an open pipe held by a
// descendant, cmd.Wait() returns within WaitDelay of the
// cancel.
<-waitCh
}
})
}
// Drive the session lifecycle in a goroutine.
// Shutdown sequence: lifecycle goroutine closes stdin + cancels context →
// codex process exits → reader goroutine's scanner.Scan() returns false →
// readerDone closes → lifecycle goroutine collects final output and sends Result.
go func() {
defer cancel()
defer close(msgCh)
defer close(resCh)
defer drainAndWait()
startTime := time.Now()
finalStatus := "completed"
var finalError string
// 1. Initialize handshake
_, err := c.request(runCtx, "initialize", map[string]any{
"clientInfo": map[string]any{
"name": "multica-agent-sdk",
"title": "Multica Agent SDK",
"version": "0.2.0",
},
"capabilities": map[string]any{
"experimentalApi": true,
},
})
if err != nil {
drainAndWait() // flush os/exec stderr goroutine before sampling Tail
finalStatus = "failed"
finalError = withAgentStderr(fmt.Sprintf("codex initialize failed: %v", err), "codex", stderrBuf.Tail())
resCh <- Result{Status: finalStatus, Error: finalError, DurationMs: time.Since(startTime).Milliseconds()}
return
}
c.notify("initialized")
// 2. Start a new thread, or resume the prior one for this issue. When
// resume fails (thread GCed on the server, schema drift, etc.) we fall
// back to a fresh thread so the task still makes progress.
threadID, resumed, err := c.startOrResumeThread(runCtx, opts, b.cfg.Logger)
if err != nil {
drainAndWait() // flush os/exec stderr goroutine before sampling Tail
finalStatus = "failed"
finalError = withAgentStderr(err.Error(), "codex", stderrBuf.Tail())
resCh <- Result{Status: finalStatus, Error: finalError, DurationMs: time.Since(startTime).Milliseconds()}
return
}
c.threadID = threadID
if resumed {
b.cfg.Logger.Info("codex thread resumed", "thread_id", threadID)
} else {
b.cfg.Logger.Info("codex thread started", "thread_id", threadID)
}
// 3. Send turn and wait for completion
turnParams := map[string]any{
"threadId": threadID,
"input": []map[string]any{
{"type": "text", "text": prompt},
},
}
// Per-turn reasoning override. Mirrors the per-thread injection in
// startOrResumeThread; keeping both in sync is enforced by the
// shared `codexReasoningInjection` fixture in codex_test.go (see
// MUL-2339 — Trump's constraint that the three injection points
// must not drift independently).
applyCodexReasoningEffort(turnParams, opts.ThinkingLevel)
_, err = c.request(runCtx, "turn/start", turnParams)
if err != nil {
drainAndWait() // flush os/exec stderr goroutine before sampling Tail
finalStatus = "failed"
finalError = withAgentStderr(fmt.Sprintf("codex turn/start failed: %v", err), "codex", stderrBuf.Tail())
resCh <- Result{Status: finalStatus, Error: finalError, DurationMs: time.Since(startTime).Milliseconds()}
return
}
lastSemanticActivity := time.Now()
lastSemanticActivityDescription := "turn/start"
semanticTimer := time.NewTimer(semanticInactivityTimeout)
defer semanticTimer.Stop()
firstTurnNoProgressTimeout := codexFirstTurnNoProgressTimeout(semanticInactivityTimeout)
var firstTurnNoProgressTimer *time.Timer
var firstTurnNoProgressTimerC <-chan time.Time
firstTurnStarted := false
firstTurnProgressObserved := false
stopFirstTurnNoProgressTimer := func() {
if firstTurnNoProgressTimer == nil {
return
}
stopTimer(firstTurnNoProgressTimer)
firstTurnNoProgressTimerC = nil
}
defer stopFirstTurnNoProgressTimer()
waitingForTurn := true
var timeoutDiagnostic codexTimeoutDiagnostic
var processExitErr error
finishTurn := func(aborted bool) {
waitingForTurn = false
switch {
case aborted:
finalStatus = "aborted"
if errMsg := c.getTurnError(); errMsg != "" {
finalError = errMsg
} else {
finalError = "turn was aborted"
}
default:
if errMsg := c.getTurnError(); errMsg != "" {
finalStatus = "failed"
finalError = errMsg
}
}
}
finishRunContextDone := func() {
waitingForTurn = false
if runCtx.Err() == context.DeadlineExceeded {
finalStatus = "timeout"
finalError = fmt.Sprintf("codex timed out after %s", timeout)
} else {
finalStatus = "aborted"
finalError = "execution cancelled"
}
}
for waitingForTurn {
select {
case aborted := <-turnDone:
finishTurn(aborted)
case activity := <-semanticActivityCh:
lastSemanticActivity = time.Now()
lastSemanticActivityDescription = activity
resetTimer(semanticTimer, semanticInactivityTimeout)
if activity == "status:running" && !firstTurnStarted {
firstTurnStarted = true
firstTurnNoProgressTimer = time.NewTimer(firstTurnNoProgressTimeout)
firstTurnNoProgressTimerC = firstTurnNoProgressTimer.C
} else if firstTurnStarted && !firstTurnProgressObserved && isCodexFirstTurnProgressActivity(activity) {
firstTurnProgressObserved = true
stopFirstTurnNoProgressTimer()
}
case <-firstTurnNoProgressTimerC:
waitingForTurn = false
finalStatus = "timeout"
timeoutDiagnostic = codexTimeoutDiagnostic{
Kind: codexTimeoutFirstTurnNoProgress,
Timeout: firstTurnNoProgressTimeout,
LastActivity: lastSemanticActivityDescription,
ThreadID: threadID,
TurnID: c.turnID,
Model: opts.Model,
}
b.cfg.Logger.Warn(CodexFirstTurnNoProgressMarker,
"pid", cmd.Process.Pid,
"thread_id", threadID,
"turn_id", c.turnID,
"timeout", firstTurnNoProgressTimeout.String(),
"last_activity", lastSemanticActivityDescription,
)
case <-semanticTimer.C:
waitingForTurn = false
finalStatus = "timeout"
timeoutDiagnostic = codexTimeoutDiagnostic{
Kind: codexTimeoutSemanticInactivity,
Timeout: semanticInactivityTimeout,
LastActivity: lastSemanticActivityDescription,
ThreadID: threadID,
TurnID: c.turnID,
Model: opts.Model,
}
b.cfg.Logger.Warn(CodexSemanticInactivityMarker,
"pid", cmd.Process.Pid,
"thread_id", threadID,
"turn_id", c.turnID,
"timeout", semanticInactivityTimeout.String(),
"last_activity", lastSemanticActivityDescription,
"idle_for", time.Since(lastSemanticActivity).Round(time.Millisecond).String(),
)
case <-runCtx.Done():
finishRunContextDone()
case <-c.processDone:
select {
case aborted := <-turnDone:
finishTurn(aborted)
default:
if runCtx.Err() != nil {
finishRunContextDone()
} else {
waitingForTurn = false
finalStatus = "failed"
processExitErr = c.getProcessErr()
if processExitErr == nil {
processExitErr = errCodexProcessExited
}
finalError = processExitErr.Error()
}
}
}
}
duration := time.Since(startTime)
b.cfg.Logger.Info("codex finished", "pid", cmd.Process.Pid, "status", finalStatus, "duration", duration.Round(time.Millisecond).String())
// Run cleanup. drainAndWait handles the graceful-then-cancel pattern
// in two bounded phases (see its declaration): wait for the reader,
// then wait for cmd.Wait(), force-cancelling either if the grace
// window expires. A clean shutdown lets codex flush OTEL telemetry;
// a stuck process is killed via the process-group SIGKILL.
drainAndWait()
if processExitErr != nil {
finalError = withAgentStderr(processExitErr.Error(), "codex", stderrBuf.Tail())
}
if timeoutDiagnostic.Kind != codexTimeoutNone {
timeoutDiagnostic.CodexVersion = detectCodexVersionForDiagnostics(context.Background(), execPath, cmd.Env, b.cfg.Logger)
finalError = buildCodexTimeoutDiagnosticError(timeoutDiagnostic, stderrBuf.Tail())
}
outputMu.Lock()
finalOutput := output.String()
outputMu.Unlock()
// Build usage map from accumulated codex usage.
// First check JSON-RPC notifications (often empty for Codex).
var usageMap map[string]TokenUsage
c.usageMu.Lock()
u := c.usage
c.usageMu.Unlock()
// Fallback: if no usage from JSON-RPC, scan Codex session JSONL logs.
// Codex writes token_count events to ~/.codex/sessions/YYYY/MM/DD/*.jsonl.
if u.InputTokens == 0 && u.OutputTokens == 0 {
if scanned := scanCodexSessionUsage(startTime); scanned != nil {
u = scanned.usage
if scanned.model != "" && opts.Model == "" {
opts.Model = scanned.model
}
}
}
if u.InputTokens > 0 || u.OutputTokens > 0 || u.CacheReadTokens > 0 || u.CacheWriteTokens > 0 {
model := opts.Model
if model == "" {
model = "unknown"
}
usageMap = map[string]TokenUsage{model: u}
}
resCh <- Result{
Status: finalStatus,
Output: finalOutput,
Error: finalError,
SessionID: threadID,
DurationMs: duration.Milliseconds(),
Usage: usageMap,
}
}()
return &Session{Messages: msgCh, Result: resCh}, nil
}
// startOrResumeThread picks between Codex's thread/resume and thread/start
// based on opts.ResumeSessionID. When a prior thread ID is provided it first
// tries thread/resume; recoverable protocol errors (unknown thread, schema
// mismatch) fall back to thread/start so the task still executes, while
// transport/process failures fail fast because the app-server can no longer
// answer a fresh start request. The returned threadID is what subsequent
// turn/start calls must reference, and resumed indicates whether the prior
// thread was picked up (only useful for logging).
func (c *codexClient) startOrResumeThread(ctx context.Context, opts ExecOptions, logger *slog.Logger) (string, bool, error) {
if priorThreadID := opts.ResumeSessionID; priorThreadID != "" {
// thread/resume reuses the thread's persisted model and reasoning
// effort; only override fields the daemon actually cares about.
resumeParams := map[string]any{
"threadId": priorThreadID,
"cwd": opts.Cwd,
"model": nilIfEmpty(opts.Model),
"developerInstructions": nilIfEmpty(opts.SystemPrompt),
}
// Explicit override of the persisted reasoning effort: without
// this, a Codex resume silently reuses whatever level the prior
// session was created with, even when the user has flipped the
// agent's thinking_level since. See MUL-2339 — Elon flagged that
// resume must honour the live config, not the stored one.
applyCodexReasoningEffort(resumeParams, opts.ThinkingLevel)
resumeResult, err := c.request(ctx, "thread/resume", resumeParams)
if err == nil {
if threadID := extractThreadID(resumeResult); threadID != "" {
return threadID, true, nil
}
logger.Warn("codex thread/resume returned no thread ID; falling back to thread/start", "prior_thread_id", priorThreadID)
} else {
if isCodexTransportError(err) {
logger.Warn("codex thread/resume failed due to transport error; not falling back to thread/start", "prior_thread_id", priorThreadID, "error", err)
return "", false, fmt.Errorf("codex thread/resume failed: %w", err)
}
logger.Warn("codex thread/resume failed; falling back to thread/start", "prior_thread_id", priorThreadID, "error", err)
}
}
startParams := map[string]any{
"model": nilIfEmpty(opts.Model),
"modelProvider": nil,
"profile": nil,
"cwd": opts.Cwd,
"approvalPolicy": nil,
"sandbox": nil,
"config": nil,
"baseInstructions": nil,
"developerInstructions": nilIfEmpty(opts.SystemPrompt),
"compactPrompt": nil,
"includeApplyPatchTool": nil,
"experimentalRawEvents": false,
"persistExtendedHistory": true,
}
applyCodexReasoningEffort(startParams, opts.ThinkingLevel)
startResult, err := c.request(ctx, "thread/start", startParams)
if err != nil {
return "", false, fmt.Errorf("codex thread/start failed: %w", err)
}
threadID := extractThreadID(startResult)
if threadID == "" {
return "", false, fmt.Errorf("codex thread/start returned no thread ID")
}
c.trySetThreadName(ctx, threadID, opts.ThreadName, logger)
return threadID, false, nil
}
func (c *codexClient) trySetThreadName(ctx context.Context, threadID, name string, logger *slog.Logger) {
name = strings.TrimSpace(name)
if name == "" {
return
}
if err := c.setThreadName(ctx, threadID, name); err != nil {
logger.Warn("codex thread/name/set failed; continuing without provider-native thread title",
"thread_id", threadID, "error", err)
}
}
func (c *codexClient) setThreadName(ctx context.Context, threadID, name string) error {
_, err := c.request(ctx, "thread/name/set", map[string]any{
"threadId": threadID,
"name": name,
})
return err
}
// applyCodexReasoningEffort writes the per-agent thinking_level into a
// Codex app-server request. The three points — thread/start.config,
// thread/resume.config, turn/start.effort — all flow through this helper
// so any future protocol/key change touches one site rather than three
// (per Trump's MUL-2339 review constraint).
//
// The shape is detected from the params keys:
// - turn/start always carries `input`, and the schema exposes the
// reasoning override as the top-level `effort` field.
// - thread/start and thread/resume nest it under
// `config.model_reasoning_effort`.
//
// Empty `level` is a no-op: we deliberately do NOT emit a key when the
// caller didn't request an override, so the upstream defaults (config
// file, account-scoped model preference) stay in charge. This also
// guarantees `effort: ""` never reaches the CLI — Codex rejects empty
// strings on this field.
func applyCodexReasoningEffort(params map[string]any, level string) {
if params == nil || level == "" {
return
}
if _, isTurnStart := params["input"]; isTurnStart {
params["effort"] = level
return
}
cfg, _ := params["config"].(map[string]any)
if cfg == nil {
cfg = map[string]any{}
}
cfg["model_reasoning_effort"] = level
params["config"] = cfg
}
func resetTimer(timer *time.Timer, d time.Duration) {
if !timer.Stop() {
select {
case <-timer.C:
default:
}
}
timer.Reset(d)
}
func stopTimer(timer *time.Timer) {
if timer == nil {
return
}
if !timer.Stop() {
select {
case <-timer.C:
default:
}
}
}
func codexFirstTurnNoProgressTimeout(semanticInactivityTimeout time.Duration) time.Duration {
if semanticInactivityTimeout <= 0 || semanticInactivityTimeout > defaultCodexFirstTurnNoProgressTimeout {
return defaultCodexFirstTurnNoProgressTimeout
}
scaled := semanticInactivityTimeout * 4 / 5
if scaled <= 0 {
return semanticInactivityTimeout
}
return scaled
}
func isCodexFirstTurnProgressActivity(activity string) bool {
return activity != "" && activity != "status:running" && activity != "error:retry"
}
func buildCodexTimeoutDiagnosticError(diag codexTimeoutDiagnostic, stderrTail string) string {
var msg string
switch diag.Kind {
case codexTimeoutFirstTurnNoProgress:
msg = fmt.Sprintf("%s after %s: received turn start but no item, message, tool, turn/completed, or error event (%s)",
CodexFirstTurnNoProgressMarker,
diag.Timeout,
formatCodexDiagnosticFields(diag),
)
case codexTimeoutSemanticInactivity:
msg = fmt.Sprintf("%s after %s without agent progress (last activity: %s; %s)",
CodexSemanticInactivityMarker,
diag.Timeout,
nonEmptyCodexDiagnosticValue(diag.LastActivity),
formatCodexDiagnosticFields(diag),
)
default:
msg = "codex timed out"
}
msg = appendCodexKnownStderrHint(msg, stderrTail)
return withAgentStderr(msg, "codex", stderrTail)
}
func formatCodexDiagnosticFields(diag codexTimeoutDiagnostic) string {
return fmt.Sprintf("codex_version=%q thread_id=%q turn_id=%q model=%q",
nonEmptyCodexDiagnosticValue(diag.CodexVersion),
nonEmptyCodexDiagnosticValue(diag.ThreadID),
nonEmptyCodexDiagnosticValue(diag.TurnID),
formatCodexDiagnosticModel(diag.Model),
)
}
func nonEmptyCodexDiagnosticValue(value string) string {
if strings.TrimSpace(value) == "" {
return "unknown"
}
return value
}
func formatCodexDiagnosticModel(model string) string {
if strings.TrimSpace(model) == "" {
return "default(empty)"
}
return model
}
func appendCodexKnownStderrHint(msg, stderrTail string) string {
if strings.Contains(stderrTail, codexModelCatalogRefreshTimeoutSignal) {
return msg + "; diagnosis: Codex stderr shows the model catalog refresh timed out. Try setting an explicit model, switching Codex CLI versions, or using another runtime while Codex app-server recovers"
}
return msg
}
func detectCodexVersionForDiagnostics(ctx context.Context, execPath string, env []string, logger *slog.Logger) string {
versionCtx, cancel := context.WithTimeout(ctx, codexVersionDiagnosticTimeout)
defer cancel()
cmd := exec.CommandContext(versionCtx, execPath, "--version")
cmd.Env = env
data, err := cmd.Output()
if err != nil {
if logger != nil {
logger.Debug("codex version diagnostic failed", "error", err)
}
return "unknown"
}
version := extractVersionLine(string(data))
if strings.TrimSpace(version) == "" {
return "unknown"
}
return version
}
func trySendString(ch chan<- string, value string) {
select {
case ch <- value:
default:
}
}
func logCodexAgentMessage(logger *slog.Logger, msg Message) {
if logger == nil {
return
}
attrs := []any{
"type", string(msg.Type),
"tool", msg.Tool,
"call_id", msg.CallID,
"status", msg.Status,
"content_len", len(msg.Content),
"output_len", len(msg.Output),
}
logger.Info("codex agent message received", attrs...)
if msg.Type == MessageToolResult {
logger.Info("codex tool_result observed", "tool", msg.Tool, "call_id", msg.CallID, "output_len", len(msg.Output))
}
}
func describeCodexSemanticActivity(msg Message) string {
switch msg.Type {
case MessageToolUse, MessageToolResult:
if msg.Tool != "" {
return fmt.Sprintf("%s:%s", msg.Type, msg.Tool)
}
case MessageStatus:
if msg.Status != "" {
return fmt.Sprintf("%s:%s", msg.Type, msg.Status)
}
}
return string(msg.Type)
}
// ── codexClient: JSON-RPC 2.0 transport ──
type codexClient struct {
cfg Config
stdin interface{ Write([]byte) (int, error) }
mu sync.Mutex
nextID int
pending map[int]*pendingRPC
processDone chan struct{}
processErr error
threadID string
turnID string
onMessage func(Message)
onSemanticActivity func(description string)
onTurnDone func(aborted bool)
notificationProtocol string // "unknown", "legacy", "raw"
turnStarted bool
completedTurnIDs map[string]bool
usageMu sync.Mutex
usage TokenUsage // accumulated from turn events
turnErrorMu sync.Mutex
turnError string // captured from turn/completed status=failed or terminal error notifications
}
func (c *codexClient) setTurnError(msg string) {
if msg == "" {
return
}
c.turnErrorMu.Lock()
defer c.turnErrorMu.Unlock()
if c.turnError == "" {
c.turnError = msg
}
}
func (c *codexClient) getTurnError() string {
c.turnErrorMu.Lock()
defer c.turnErrorMu.Unlock()
return c.turnError
}
type pendingRPC struct {
ch chan rpcResult
method string
}
type rpcResult struct {
result json.RawMessage
err error
}
func (c *codexClient) request(ctx context.Context, method string, params any) (json.RawMessage, error) {
if err := ctx.Err(); err != nil {
return nil, err
}
c.mu.Lock()
if c.processErr != nil {
err := c.processErr
c.mu.Unlock()
return nil, err
}
if c.processDone == nil {
c.processDone = make(chan struct{})
}
processDone := c.processDone
c.nextID++
id := c.nextID
pr := &pendingRPC{ch: make(chan rpcResult, 1), method: method}
c.pending[id] = pr
c.mu.Unlock()
msg := map[string]any{
"jsonrpc": "2.0",
"id": id,
"method": method,
"params": params,
}
data, err := json.Marshal(msg)
if err != nil {
c.mu.Lock()
delete(c.pending, id)
c.mu.Unlock()
return nil, err
}
data = append(data, '\n')
if _, err := c.stdin.Write(data); err != nil {
c.mu.Lock()
delete(c.pending, id)
c.mu.Unlock()
return nil, fmt.Errorf("write %s: %w", method, err)
}
if method == "turn/start" {
threadID := ""
if paramMap, ok := params.(map[string]any); ok {
threadID, _ = paramMap["threadId"].(string)
}
c.cfg.Logger.Info("codex turn/start sent", "request_id", id, "thread_id", threadID)
}
select {
case res := <-pr.ch:
return res.result, res.err
case <-processDone:
c.mu.Lock()
delete(c.pending, id)
err := c.processErr
c.mu.Unlock()
if ctxErr := ctx.Err(); ctxErr != nil {
return nil, ctxErr
}
if err == nil {
err = errCodexProcessExited
}
return nil, err
case <-ctx.Done():
c.mu.Lock()
delete(c.pending, id)
c.mu.Unlock()
return nil, ctx.Err()
}
}
func (c *codexClient) notify(method string) {
msg := map[string]any{
"jsonrpc": "2.0",
"method": method,
}
data, _ := json.Marshal(msg)
data = append(data, '\n')
_, _ = c.stdin.Write(data)
}
func (c *codexClient) respond(id int, result any) {
msg := map[string]any{
"jsonrpc": "2.0",
"id": id,
"result": result,
}
data, _ := json.Marshal(msg)
data = append(data, '\n')
_, _ = c.stdin.Write(data)
}
func (c *codexClient) respondError(id int, code int, message string) {
msg := map[string]any{
"jsonrpc": "2.0",
"id": id,
"error": map[string]any{
"code": code,
"message": message,
},
}
data, _ := json.Marshal(msg)
data = append(data, '\n')
_, _ = c.stdin.Write(data)
}
func (c *codexClient) closeAllPending(err error) {
c.mu.Lock()
defer c.mu.Unlock()
for id, pr := range c.pending {
pr.ch <- rpcResult{err: err}
delete(c.pending, id)
}
}
func (c *codexClient) markProcessExited(err error) {
if err == nil {
err = errCodexProcessExited
}
c.mu.Lock()
defer c.mu.Unlock()
if c.processErr == nil {
c.processErr = err
if c.processDone != nil {
close(c.processDone)
}
}
for id, pr := range c.pending {
pr.ch <- rpcResult{err: err}
delete(c.pending, id)
}
}
func (c *codexClient) getProcessErr() error {
c.mu.Lock()
defer c.mu.Unlock()
return c.processErr
}
func isCodexTransportError(err error) bool {
if err == nil {
return false
}
if errors.Is(err, errCodexProcessExited) {
return true
}
return strings.HasPrefix(err.Error(), "write ")
}
func (c *codexClient) handleLine(line string) {
var raw map[string]json.RawMessage
if err := json.Unmarshal([]byte(line), &raw); err != nil {
return
}
// Check if it's a response to our request
if _, hasID := raw["id"]; hasID {
if _, hasResult := raw["result"]; hasResult {
c.handleResponse(raw)
return
}
if _, hasError := raw["error"]; hasError {
c.handleResponse(raw)
return
}
// Server request (has id + method)
if _, hasMethod := raw["method"]; hasMethod {
c.handleServerRequest(raw)
return
}
}
// Notification (no id, has method)
if _, hasMethod := raw["method"]; hasMethod {
c.handleNotification(raw)
}
}
func (c *codexClient) handleResponse(raw map[string]json.RawMessage) {
var id int
if err := json.Unmarshal(raw["id"], &id); err != nil {
return
}
c.mu.Lock()
pr, ok := c.pending[id]
if ok {
delete(c.pending, id)
}
c.mu.Unlock()
if !ok {
return
}
if errData, hasErr := raw["error"]; hasErr {
var rpcErr struct {
Code int `json:"code"`
Message string `json:"message"`
}
_ = json.Unmarshal(errData, &rpcErr)
pr.ch <- rpcResult{err: fmt.Errorf("%s: %s (code=%d)", pr.method, rpcErr.Message, rpcErr.Code)}
} else {
pr.ch <- rpcResult{result: raw["result"]}
}
}
func (c *codexClient) handleServerRequest(raw map[string]json.RawMessage) {
var id int
_ = json.Unmarshal(raw["id"], &id)
var method string
_ = json.Unmarshal(raw["method"], &method)
// Auto-approve all exec/patch requests in daemon mode
switch method {
case "item/commandExecution/requestApproval", "execCommandApproval":
c.respond(id, map[string]any{"decision": "accept"})
case "item/fileChange/requestApproval", "applyPatchApproval":
c.respond(id, map[string]any{"decision": "accept"})
case "item/permissions/requestApproval":
c.respond(id, codexPermissionsApprovalResponse(raw["params"], c.cfg.Logger))
case "mcpServer/elicitation/request":
c.respond(id, map[string]any{"action": "accept", "content": nil, "_meta": nil})
default:
msg := fmt.Sprintf("unsupported codex app-server request: %s", method)
c.cfg.Logger.Warn("codex: unhandled server request", "method", method, "id", id)
c.setTurnError(msg)
c.respondError(id, -32601, msg)
}
}
// codexPermissionsApprovalResponse builds the auto-grant reply for a Codex
// item/permissions/requestApproval server request. In daemon mode there is no
// human to approve, so we echo back the requested network / fileSystem profile
// and scope it to the current turn, mirroring the other auto-accept branches in
// handleServerRequest.
//
// The grant is intentionally limited to the network / fileSystem keys we
// understand. A parse failure and any dropped key are logged so that a future
// app-server protocol that adds a new permission shape is visible in daemon
// logs instead of being silently narrowed away.
func codexPermissionsApprovalResponse(params json.RawMessage, logger *slog.Logger) map[string]any {
var payload struct {
Permissions map[string]any `json:"permissions"`
}
if err := json.Unmarshal(params, &payload); err != nil && logger != nil {
logger.Warn("codex: failed to parse permission approval request; granting empty turn-scoped profile", "error", err)
}
granted := map[string]any{}
var dropped []string
for key, value := range payload.Permissions {
switch key {
case "network", "fileSystem":
if value != nil {
granted[key] = value
}
default:
dropped = append(dropped, key)
}
}
if len(dropped) > 0 && logger != nil {
sort.Strings(dropped)
logger.Warn("codex: dropping unrecognized permission keys from approval request; add explicit handling if the app-server protocol expanded", "keys", dropped)
}
return map[string]any{
"permissions": granted,
"scope": "turn",
}
}
func (c *codexClient) handleNotification(raw map[string]json.RawMessage) {
var method string
_ = json.Unmarshal(raw["method"], &method)
var params map[string]any
if p, ok := raw["params"]; ok {
_ = json.Unmarshal(p, &params)
}
// Legacy codex/event notifications
if method == "codex/event" || strings.HasPrefix(method, "codex/event/") {
c.notificationProtocol = "legacy"
msgData, ok := params["msg"]
if !ok {
return
}
msgMap, ok := msgData.(map[string]any)
if !ok {
return
}
c.handleEvent(msgMap)
return
}
// Raw v2 notifications
if c.notificationProtocol != "legacy" {
if c.notificationProtocol == "unknown" &&
(method == "turn/started" || method == "turn/completed" ||
method == "thread/started" || strings.HasPrefix(method, "item/")) {
c.notificationProtocol = "raw"
}
if c.notificationProtocol == "raw" {
c.handleRawNotification(method, params)
}
}
}
func (c *codexClient) handleEvent(msg map[string]any) {
msgType, _ := msg["type"].(string)
switch msgType {
case "task_started":
c.turnStarted = true
if c.onMessage != nil {
c.onMessage(Message{Type: MessageStatus, Status: "running", SessionID: c.threadID})
}
case "agent_message":
text, _ := msg["message"].(string)
if text != "" && c.onMessage != nil {
c.onMessage(Message{Type: MessageText, Content: text})
}
case "exec_command_begin":
callID, _ := msg["call_id"].(string)
command, _ := msg["command"].(string)
if c.onMessage != nil {
c.onMessage(Message{
Type: MessageToolUse,
Tool: "exec_command",
CallID: callID,
Input: map[string]any{"command": command},
})
}
case "exec_command_end":
callID, _ := msg["call_id"].(string)
output, _ := msg["output"].(string)
if c.onMessage != nil {
c.onMessage(Message{
Type: MessageToolResult,
Tool: "exec_command",
CallID: callID,
Output: output,
})
}
case "patch_apply_begin":
callID, _ := msg["call_id"].(string)
if c.onMessage != nil {
c.onMessage(Message{
Type: MessageToolUse,
Tool: "patch_apply",
CallID: callID,
})
}
case "patch_apply_end":
callID, _ := msg["call_id"].(string)
if c.onMessage != nil {
c.onMessage(Message{
Type: MessageToolResult,
Tool: "patch_apply",
CallID: callID,
})
}
case "task_complete":
// Extract usage from legacy task_complete if present.
c.extractUsageFromMap(msg)
if c.onTurnDone != nil {
c.onTurnDone(false)
}
case "turn_aborted":
if c.onTurnDone != nil {
c.onTurnDone(true)
}
}
}
func (c *codexClient) handleRawNotification(method string, params map[string]any) {
// Ignore notifications from threads other than the one we are tracking.
// Codex multiplexes subagent threads (e.g. memory consolidation) on the
// same stdio pipe; only our thread should drive turn lifecycle and output.
//
// The v2 app-server-protocol schema guarantees a top-level threadId on
// every notification, so this dispatch-level guard transparently covers
// every handler below. If a future codex revision introduces notifications
// without threadId, they fall through (ok=false) — re-audit this guard
// when bumping codex.
if threadID, ok := params["threadId"].(string); ok && c.threadID != "" && threadID != c.threadID {
return
}
switch method {
case "turn/started":
c.turnStarted = true
if turnID := extractNestedString(params, "turn", "id"); turnID != "" {
c.turnID = turnID
}
if c.onMessage != nil {
c.onMessage(Message{Type: MessageStatus, Status: "running", SessionID: c.threadID})
}
case "turn/completed":
turnID := extractNestedString(params, "turn", "id")
status := extractNestedString(params, "turn", "status")
threadID, _ := params["threadId"].(string)
c.cfg.Logger.Info("codex turn/completed received", "thread_id", threadID, "turn_id", turnID, "status", status)
aborted := status == "cancelled" || status == "canceled" ||
status == "aborted" || status == "interrupted"
// Capture the error message from failed turns so callers can surface
// a real reason instead of falling back to "empty output".
if status == "failed" {
errMsg := extractNestedString(params, "turn", "error", "message")
if errMsg == "" {
errMsg = "codex turn failed"
}
c.setTurnError(errMsg)
}
if c.completedTurnIDs == nil {
c.completedTurnIDs = map[string]bool{}
}
if turnID != "" {
if c.completedTurnIDs[turnID] {
return
}
c.completedTurnIDs[turnID] = true
}
// Extract usage from turn/completed if present (e.g. params.turn.usage).
if turn, ok := params["turn"].(map[string]any); ok {
c.extractUsageFromMap(turn)
}
if c.onTurnDone != nil {
c.onTurnDone(aborted)
}
case "error":
// Top-level protocol error. Retrying notifications (willRetry=true) are
// transient reconnect attempts; only capture terminal errors so we
// don't stomp on a real failure later with a retry placeholder.
willRetry, _ := params["willRetry"].(bool)
errMsg := extractNestedString(params, "error", "message")
if errMsg == "" {
errMsg = extractNestedString(params, "message")
}
if errMsg != "" {
c.cfg.Logger.Warn("codex error notification", "message", errMsg, "will_retry", willRetry)
if c.onSemanticActivity != nil {
if willRetry {
c.onSemanticActivity("error:retry")
} else {
c.onSemanticActivity("error:terminal")
}
}
if !willRetry {
c.setTurnError(errMsg)
if c.onTurnDone != nil {
c.onTurnDone(false)
}
}
}
case "thread/status/changed":
statusType := extractNestedString(params, "status", "type")
if statusType == "idle" && c.turnStarted {
if c.onTurnDone != nil {
c.onTurnDone(false)
}
}
default:
if strings.HasPrefix(method, "item/") {
c.handleItemNotification(method, params)
}
}
}
func (c *codexClient) handleItemNotification(method string, params map[string]any) {
item, _ := params["item"].(map[string]any)
itemType, _ := item["type"].(string)
itemID, _ := item["id"].(string)
if isCodexItemProgressActivity(method) && c.onSemanticActivity != nil {
c.onSemanticActivity(describeCodexItemProgressActivity(method, itemType, itemID))
}
if item == nil {
return
}
switch {
case method == "item/started" && itemType == "commandExecution":
command, _ := item["command"].(string)
if c.onMessage != nil {
c.onMessage(Message{
Type: MessageToolUse,
Tool: "exec_command",
CallID: itemID,
Input: map[string]any{"command": command},
})
}
case method == "item/completed" && itemType == "commandExecution":
output, _ := item["aggregatedOutput"].(string)
if c.onMessage != nil {
c.onMessage(Message{
Type: MessageToolResult,
Tool: "exec_command",
CallID: itemID,
Output: output,
})
}
case method == "item/started" && itemType == "fileChange":
if c.onMessage != nil {
c.onMessage(Message{
Type: MessageToolUse,
Tool: "patch_apply",
CallID: itemID,
})
}
case method == "item/completed" && itemType == "fileChange":
if c.onMessage != nil {
c.onMessage(Message{
Type: MessageToolResult,
Tool: "patch_apply",
CallID: itemID,
})
}
case method == "item/completed" && itemType == "agentMessage":
text, _ := item["text"].(string)
if text != "" && c.onMessage != nil {
c.onMessage(Message{Type: MessageText, Content: text})
}
phase, _ := item["phase"].(string)
if phase == "final_answer" && c.turnStarted {
if c.onTurnDone != nil {
c.onTurnDone(false)
}
}
}
}
func isCodexItemProgressActivity(method string) bool {
return strings.HasPrefix(method, "item/")
}
func describeCodexItemProgressActivity(method, itemType, itemID string) string {
if itemType == "" {
itemType = "unknown"
}
if itemID == "" {
return fmt.Sprintf("%s:%s", method, itemType)
}
return fmt.Sprintf("%s:%s:%s", method, itemType, itemID)
}
// extractUsageFromMap extracts token usage from a map that may contain
// "usage", "token_usage", or "tokens" fields. Handles various Codex formats.
func (c *codexClient) extractUsageFromMap(data map[string]any) {
// Try common field names for usage data.
var usageMap map[string]any
for _, key := range []string{"usage", "token_usage", "tokens"} {
if v, ok := data[key].(map[string]any); ok {
usageMap = v
break
}
}
if usageMap == nil {
return
}
c.usageMu.Lock()
defer c.usageMu.Unlock()
// Codex reports cached input as a prompt-token detail: cached_input_tokens
// are included in input_tokens. Persist mutually-exclusive buckets so
// dashboard cost math does not charge cached input twice.
inputTokens := codexInt64(usageMap, "input_tokens", "input", "prompt_tokens")
cacheReadTokens := codexInt64(usageMap, "cached_input_tokens", "cache_read_tokens", "cache_read_input_tokens")
c.usage.InputTokens += codexUncachedInputTokens(inputTokens, cacheReadTokens)
c.usage.OutputTokens += codexInt64(usageMap, "output_tokens", "output", "completion_tokens")
c.usage.CacheReadTokens += cacheReadTokens
c.usage.CacheWriteTokens += codexInt64(usageMap, "cache_write_tokens", "cache_creation_input_tokens")
}
func codexUncachedInputTokens(inputTokens, cachedInputTokens int64) int64 {
uncached := inputTokens - cachedInputTokens
if uncached < 0 {
return 0
}
return uncached
}
// codexInt64 returns the first non-zero int64 value from the map for the given keys.
func codexInt64(m map[string]any, keys ...string) int64 {
for _, key := range keys {
switch v := m[key].(type) {
case float64:
if v != 0 {
return int64(v)
}
case int64:
if v != 0 {
return v
}
}
}
return 0
}
// ── Codex session log scanner ──
// codexSessionUsage holds usage extracted from a Codex session JSONL file.
type codexSessionUsage struct {
usage TokenUsage
model string
}
// scanCodexSessionUsage scans Codex session JSONL files written after startTime
// to extract token usage. Codex writes token_count events to
// ~/.codex/sessions/YYYY/MM/DD/*.jsonl.
func scanCodexSessionUsage(startTime time.Time) *codexSessionUsage {
root := codexSessionRoot()
if root == "" {
return nil
}
// Look in today's session directory.
dateDir := filepath.Join(root,
fmt.Sprintf("%04d", startTime.Year()),
fmt.Sprintf("%02d", int(startTime.Month())),
fmt.Sprintf("%02d", startTime.Day()),
)
files, err := filepath.Glob(filepath.Join(dateDir, "*.jsonl"))
if err != nil || len(files) == 0 {
return nil
}
// Only scan files modified after startTime (this task's session).
var result codexSessionUsage
for _, f := range files {
info, err := os.Stat(f)
if err != nil || info.ModTime().Before(startTime) {
continue
}
if u := parseCodexSessionFile(f); u != nil {
// Take the last matching file's data (usually there's only one per task).
result = *u
}
}
if result.usage.InputTokens == 0 && result.usage.OutputTokens == 0 {
return nil
}
return &result
}
// codexSessionRoot returns the Codex sessions directory.
func codexSessionRoot() string {
if codexHome := os.Getenv("CODEX_HOME"); codexHome != "" {
dir := filepath.Join(codexHome, "sessions")
if info, err := os.Stat(dir); err == nil && info.IsDir() {
return dir
}
}
home, err := os.UserHomeDir()
if err != nil {
return ""
}
dir := filepath.Join(home, ".codex", "sessions")
if info, err := os.Stat(dir); err == nil && info.IsDir() {
return dir
}
return ""
}
// codexSessionTokenCount represents a token_count event in Codex JSONL.
type codexSessionTokenCount struct {
Type string `json:"type"`
Payload *struct {
Type string `json:"type"`
Info *struct {
TotalTokenUsage *struct {
InputTokens int64 `json:"input_tokens"`
OutputTokens int64 `json:"output_tokens"`
CachedInputTokens int64 `json:"cached_input_tokens"`
CacheReadInputTokens int64 `json:"cache_read_input_tokens"`
ReasoningOutputTokens int64 `json:"reasoning_output_tokens"`
} `json:"total_token_usage"`
LastTokenUsage *struct {
InputTokens int64 `json:"input_tokens"`
OutputTokens int64 `json:"output_tokens"`
CachedInputTokens int64 `json:"cached_input_tokens"`
CacheReadInputTokens int64 `json:"cache_read_input_tokens"`
ReasoningOutputTokens int64 `json:"reasoning_output_tokens"`
} `json:"last_token_usage"`
Model string `json:"model"`
} `json:"info"`
Model string `json:"model"`
} `json:"payload"`
}
// parseCodexSessionFile extracts the final token_count from a Codex session file.
func parseCodexSessionFile(path string) *codexSessionUsage {
f, err := os.Open(path)
if err != nil {
return nil
}
defer f.Close()
var result codexSessionUsage
found := false
scanner := bufio.NewScanner(f)
scanner.Buffer(make([]byte, 0, 256*1024), 1024*1024)
for scanner.Scan() {
line := scanner.Bytes()
// Fast pre-filter.
if !bytesContainsStr(line, "token_count") && !bytesContainsStr(line, "turn_context") {
continue
}
var evt codexSessionTokenCount
if err := json.Unmarshal(line, &evt); err != nil || evt.Payload == nil {
continue
}
// Track model from turn_context events.
if evt.Type == "turn_context" && evt.Payload.Model != "" {
result.model = evt.Payload.Model
continue
}
// Extract token usage from token_count events.
if evt.Payload.Type == "token_count" && evt.Payload.Info != nil {
usage := evt.Payload.Info.TotalTokenUsage
if usage == nil {
usage = evt.Payload.Info.LastTokenUsage
}
if usage != nil {
cachedTokens := usage.CachedInputTokens
if cachedTokens == 0 {
cachedTokens = usage.CacheReadInputTokens
}
result.usage = TokenUsage{
InputTokens: codexUncachedInputTokens(usage.InputTokens, cachedTokens),
OutputTokens: usage.OutputTokens + usage.ReasoningOutputTokens,
CacheReadTokens: cachedTokens,
}
if evt.Payload.Info.Model != "" {
result.model = evt.Payload.Info.Model
}
found = true
}
}
}
if !found {
return nil
}
return &result
}
// bytesContainsStr checks if b contains the string s (without allocating).
func bytesContainsStr(b []byte, s string) bool {
return strings.Contains(string(b), s)
}
// ── Helpers ──
func extractThreadID(result json.RawMessage) string {
var r struct {
Thread struct {
ID string `json:"id"`
} `json:"thread"`
}
if err := json.Unmarshal(result, &r); err != nil {
return ""
}
return r.Thread.ID
}
func extractNestedString(m map[string]any, keys ...string) string {
current := any(m)
for _, key := range keys {
obj, ok := current.(map[string]any)
if !ok {
return ""
}
current = obj[key]
}
s, _ := current.(string)
return s
}
func nilIfEmpty(s string) any {
if s == "" {
return nil
}
return s
}