mirror of
https://github.com/multica-ai/multica.git
synced 2026-07-25 20:15:37 +02:00
* feat(db): add Lark integration migration (MUL-2671) Introduces seven tables for the 飞书 Bot integration MVP — per-agent PersonalAgent installations, user/chat bindings, inbound dedup + non-content drop audit, outbound card mapping, and short-lived single-use member binding tokens. Schema notes: - chat_session schema unchanged; Lark routes through a separate binding table rather than adding a metadata JSONB column. - Outbound card mapping is task/message scoped so multiple runs on the same session can't stomp each other's cards. - lark_inbound_audit stores routing / identity / drop_reason ONLY, never message body — the audit channel for unbound users and group messages that don't address the Bot. - app_secret stores ciphertext (encryption helper lands in a follow-up commit on this branch); DB never sees plaintext. Co-authored-by: multica-agent <github@multica.ai> * feat(util): add secretbox AES-256-GCM helper for at-rest secrets First consumer is lark_installation.app_secret (MUL-2671 §4.4), but the helper is intentionally generic — future per-tenant secrets that must not appear in a DB dump can reuse it. Construction: AES-256-GCM with a per-message random nonce, providing authenticated encryption. Tampered ciphertext fails Open instead of silently decrypting to garbage. Master key loaded from a base64 env var via LoadKey; key rotation is not in scope yet. Co-authored-by: multica-agent <github@multica.ai> * refactor(issues): extract IssueService.Create as single create entry (MUL-2671) Establishes the service-layer boundary mandated by Elon's 二审 of MUL-2671 §4.8: issue creation no longer lives inside the HTTP handler. Both the HTTP POST /issues handler and the future Lark /issue command call into service.IssueService.Create, so duplicate guard, issue numbering, attachment linking, broadcast, analytics, and agent/squad enqueue stay aligned. Handler responsibilities shrink to parsing the HTTP request, doing actor resolution / validation (transport-specific), and converting service results into the IssueResponse + 201. The transaction-wrapped core, attachment link, event publish, analytics capture, and agent/squad enqueue all move into service.IssueService.Create. A BroadcastPayload callback on the service keeps the WS broadcast shape (the full IssueResponse) without forcing the service to depend on handler-layer response types. Co-authored-by: multica-agent <github@multica.ai> * feat(integrations): add Lark package skeleton (MUL-2671) Establishes the architectural boundaries Elon's 二审 mandated as first-PR blockers without dragging in OAuth, WebSocket, or card-patching code (those land in follow-up PRs): - ChatSessionService interface — channel-aware chat-session entry point for Lark, deliberately separate from the HTTP SendChatMessage handler. The HTTP handler's single-creator guard (creator_id == request user_id) is correct for the browser client but rejects group chat_sessions by construction; Lark needs its own service. - AuditLogger interface — the only path for recording dropped events. Its signature deliberately omits message body, enforcing the drop-audit policy (MUL-2671 §4.7) at the type level: unbound users and non-addressed group messages can't accidentally end up in chat_session. - Typed IDs (OpenID, ChatID) prevent UUIDs from being conflated with Lark-side identifiers at compile time. - DropReason constants align dashboard/audit queries across callers. Co-authored-by: multica-agent <github@multica.ai> * refactor(issues): move parent/project workspace check into IssueService (MUL-2671) Parent existence and project workspace membership now live inside IssueService.Create, inside the same transaction as the duplicate guard and counter increment. The HTTP handler stops re-implementing the lookup; every future create entry (Lark /issue, MCP, API keys) inherits the same boundary without copy-pasting the SQL. Adds two error sentinels (ErrParentIssueNotFound, ErrProjectNotFound) so transports can translate to their own error shapes. Handler-level cross-workspace tests guard the boundary against future regressions. Co-authored-by: multica-agent <github@multica.ai> * fix(db): harden Lark migration safety底座 — TTL cap + workspace FK (MUL-2671) Two storage-layer hardenings that move the must-fix line off "the app layer enforces it" and onto the schema itself, so future write paths or hand-inserted rows cannot regress the invariants. 1) lark_binding_token TTL cap. The DB CHECK was 1 hour as defense-in-depth while the app constant was 15 minutes; the CHECK now matches the product cap (15 minutes). Application constant docstring updated to reflect that storage enforces the same bound. 2) lark_user_binding workspace membership. The table previously only FK'd to workspace / user / installation independently, so a binding could exist for a user no longer in the workspace, or claim a workspace different from its installation's. Two composite FKs close the gap structurally: * (installation_id, workspace_id) → lark_installation(id, workspace_id) — guarantees a binding's workspace_id always matches its installation's workspace_id. A new UNIQUE (id, workspace_id) on lark_installation is added as the FK target. * (workspace_id, multica_user_id) → member(workspace_id, user_id) with ON DELETE CASCADE — when a user is removed from the workspace, the binding cascades away in the same transaction. There is no longer a path where lark_user_binding outlives workspace membership. These two FKs are the schema-level proof for §4.3's "unbound or non-workspace members cannot leak content into chat_session" invariant. Co-authored-by: multica-agent <github@multica.ai> * feat(integrations/lark): inbound services + /issue dispatcher (MUL-2671) Lands the inbound service layer for the Lark Bot MVP, sitting on top of the migration + service-boundary scaffold from the previous commits. What ships: - sqlc queries for all seven lark_* tables (idempotent dedup insert, CAS WS-lease, single-use binding-token consume, etc.) plus GetMostRecentUserChatMessage for the /issue fallback. - AuditLogger backed by lark_inbound_audit; signature deliberately body-free so callers cannot leak content into the drop log. - ChatSessionService: find-or-create chat_session via the binding table (winner-takes-all on the UNIQUE race), append-with-dedup, /issue parser, "previous user message" fallback for bare `/issue` invocation. - Dispatcher orchestrates the inbound pipeline in one place: installation routing → group-mention filter → identity check → ensure session → append+dedup → /issue → enqueue chat task. Group sessions use the installer as creator (stable workspace identity); p2p uses the sender. Agent-offline path falls through with OutcomeAgentOffline so the WS adapter can reply with the offline notice from §4.6. - BindingTokenService: random URL-safe token, SHA-256 stored hash, 15-min TTL pinned at the application AND the DB CHECK; Redeem returns the same opaque error for all rejection cases (no timing oracle on replay). - Unit tests for the parser (13 cases), dispatcher (8 cases via fake Queries/Chat/Audit/IssueCreator/Enqueuer), and binding-token hash/entropy. Real-DB integration tests for OAuth + token redeem land alongside the HTTP handlers in the next commit. Out of scope for this commit (next ones on the same feature branch): OAuth callback, HTTP routes, WebSocket hub, outbound card patcher, frontend. Co-authored-by: multica-agent <github@multica.ai> * feat(integrations/lark): installation HTTP surface + secretbox-gated wiring (MUL-2671) Lands the HTTP boundary on top of the inbound services from the previous commit. What ships: - InstallationService.Upsert: the only path that writes lark_installation. Encrypts app_secret with the secretbox passed in at construction time; refuses to fall back to plaintext storage (returns an error from the constructor if no Box is supplied), so a misconfigured dev environment cannot accidentally land a row with cleartext credentials. Revoke flips status without DELETE so audit trail survives. - HTTP handlers under /api/workspaces/{id}/lark/: * GET /installations — member-visible (Integrations tab renders for non-admins). Soft 200 with empty list + configured:false when MULTICA_LARK_SECRET_KEY is unset, so the tab does not error on self-host that has not opted in. * POST /installations — admin-only; 503 when not configured. Re-validates agent_id ∈ workspace before accepting credentials so a cross-workspace agent UUID is rejected. * DELETE /installations/{id} — admin-only; workspace-scoped lookup so one workspace cannot revoke another's installation by UUID guess. - POST /api/lark/binding/redeem (user-scoped, no workspace context): the only path that mints a lark_user_binding row from user action. Redeemer identity comes from the session, not the token, so a stolen link cannot bind an open_id to an attacker's Multica user. The composite FK on lark_user_binding cascades the binding away if the user is not (or no longer) a workspace member, so a non-member who steals the link gets 403 at the DB layer. - Two new event-bus types in protocol.events: EventLarkInstallationCreated, EventLarkInstallationRevoked. - Router wiring: MULTICA_LARK_SECRET_KEY drives a conditional initialization of h.LarkInstallations + h.LarkBindingTokens. When unset, the integration disables itself with an INFO log and the rest of the server boots normally. - Handler tests cover all four not-configured short-circuits. Happy-path integration tests (real DB, full create→list→revoke cycle and token mint→redeem) ship alongside the WS hub PR. Co-authored-by: multica-agent <github@multica.ai> * fix(integrations/lark): close binding-token rebind & typed task errors (MUL-2671) Two must-fixes from PR review on HEAD87ad15e1: 1. Binding-token redeem could be used to grab an already-bound Lark open_id. Two changes harden the path: - lark.sql `CreateLarkUserBinding` now gates ON CONFLICT DO UPDATE on `multica_user_id = EXCLUDED.multica_user_id`, so a cross-user rebind via a second valid token returns zero rows instead of silently switching ownership. - `BindingTokenService.RedeemAndBind` consumes the token and writes the binding row inside one transaction. A failed bind no longer burns the token; a successful bind never leaves a consumed-but- unused token. Distinct typed errors: ErrBindingTokenInvalid (410), ErrBindingAlreadyAssigned (409), ErrBindingNotWorkspaceMember (403). The handler maps each to its own status code. 2. Dispatcher collapsed every `EnqueueChatTask` error to `OutcomeAgentOffline`, hiding infra failure and misusing the "offline" label for cases (e.g. archived agent) where it doesn't fit. Now: - `service.EnqueueChatTask` returns `ErrChatTaskAgentNoRuntime` and `ErrChatTaskAgentArchived` as sentinel errors; DB / load / insert failures stay wrapped as ordinary errors. - Dispatcher uses `errors.Is` to map only the productizable cases (`OutcomeAgentOffline`, new `OutcomeAgentArchived`); any other error is returned to the WS adapter so it can retry or page instead of disguising the outage as an offline card. A daemon that's merely disconnected is still NOT an error — as long as `agent.runtime_id` is set the chat task enqueues and waits for the daemon to claim it on next online (returns `OutcomeIngested`). Co-authored-by: multica-agent <github@multica.ai> * ci: re-trigger workflow on lark MVP must-fix HEAD Co-authored-by: multica-agent <github@multica.ai> * ci: re-trigger workflow on lark MVP must-fix HEAD (retry) Co-authored-by: multica-agent <github@multica.ai> * test(integrations/lark): guard binding-token sentinel contract (MUL-2671) Two unit tests that document and protect the must-fix invariants without requiring a DB: 1. TestRedeemAndBindRequiresTxStarter — if a future refactor wires up BindingTokenService without a TxStarter, RedeemAndBind must fail fast with a clear error rather than nil-panic on Begin. The atomicity contract (consume + bind commit together) depends on that transaction existing. 2. TestBindingErrorSentinelsAreDistinct — the HTTP handler maps ErrBindingTokenInvalid → 410, ErrBindingAlreadyAssigned → 409, ErrBindingNotWorkspaceMember → 403. Accidentally aliasing them (e.g. var ErrBindingAlreadyAssigned = ErrBindingTokenInvalid) would silently regress the response codes without any other test catching it. Co-authored-by: multica-agent <github@multica.ai> * feat(integrations/lark): WS hub orchestrator + outbound card patcher (MUL-2671) The hub owns one supervisor goroutine per active installation. Each supervisor acquires the WS lease via the existing CAS query, runs an EventConnector (interface — real Lark wire protocol lands in a follow-up behind it), renews the lease on a tighter cadence than the TTL, and backs off (with jitter) on connector failure. Lease loss tears the connector down cleanly; revocation is reaped on the next sweep. Per- process node id satisfies §4.4 multi-replica safety: at most one Hub globally holds the lease for any installation. The patcher subscribes to task / chat-done events on the existing events.Bus and keeps the per-task Lark interactive card in sync (thinking → streaming → final | error). Card binding is per-task as required by §4.5; throttled patches via an in-memory last-patched map; final / error transitions bypass the throttle so the user always sees the terminal state. The Renderer is plug-replaceable so the product card template can evolve without touching transport. The APIClient interface centralizes the Lark Open Platform surface this package needs (send card, patch card, send binding prompt, exchange OAuth code). The default stubAPIClient returns ErrAPIClientNotConfigured for every transport call so a misconfigured deployment fails loudly instead of dropping cards silently. Real implementation lands in a follow-up; OAuth callback + frontend entries land in the next commits on this branch. Co-authored-by: multica-agent <github@multica.ai> * feat(integrations/lark): OAuth install start / callback (MUL-2671) OAuthService builds a signed-state Lark authorization URL the frontend can render as a QR (or open directly), then on callback verifies the HMAC-protected state, exchanges the OAuth code for installation credentials via APIClient.ExchangeOAuthCode, and persists the row via InstallationService.Upsert (which keeps app_secret encryption inside a single chokepoint). State token format: workspaceID.agentID.initiatorID.expiresUnix.nonce.sig — HMAC-SHA256 over the first five fields with a deployment-level secret. TTL defaults to 10 minutes (covered by tests). Three failure modes (invalid state / expired state / missing code) map to typed errors so the HTTP handler can emit a single lark_error= query param the frontend uses to pick copy. Both endpoints degrade cleanly: the at-rest key gate (already in place) returns 503 from /install/start when the InstallationService is nil, and the OAuth gate (MULTICA_LARK_OAUTH_APP_ID / _SECRET / _REDIRECT_URI / _STATE_SECRET) returns configured:false from /install/start so the frontend can render "configure manually instead" without an error banner. /install/callback always finishes with a redirect to /settings?tab=lark carrying either lark_installed=1 or lark_error=<code>. Tests cover signed-URL shape, missing-config rejection, tampered state, expired state, propagated exchange error, and the no-config redirect path on the HTTP handler. Co-authored-by: multica-agent <github@multica.ai> * feat(views/lark): settings tab + agent bind button + /lark/bind redemption page (MUL-2671) Adds the user-facing Lark surface across the shared packages: - packages/core/types/lark.ts — wire shapes that mirror server/internal/ handler/lark.go. Optional fields default to undefined so older desktop builds keep parsing if the server adds new keys (CLAUDE.md → API Response Compatibility). - packages/core/lark/{queries,index}.ts — Tanstack Query options keyed by workspace id; realtime sync invalidates `installations(wsId)` on `lark_installation:*` events. - packages/core/api/client.ts — listLarkInstallations, getLarkInstallURL, deleteLarkInstallation, redeemLarkBindingToken. - packages/views/settings/components/lark-tab.tsx — Settings → Lark panel. Listing is member-visible (matches backend); disconnect is admin-only. Empty state points users at the per-Agent bind entry, matching the (workspace_id, agent_id) UNIQUE: there is no "pick an agent" UI here because the bind URL is per-agent. - LarkAgentBindButton (same file) is the per-Agent CTA the Agent detail page imports. Opens the OAuth URL in a new tab; the callback bounces back to /settings?tab=lark with a query param the panel reads for inline confirmation copy. - packages/views/lark/bind-page.tsx — the Bot's "you need to bind" destination. Requires session before redeeming, distinguishes the 410/409/403 backend responses into distinct copy. - apps/web/app/lark/bind/page.tsx — Next.js route wrapping the shared bind page in a Suspense boundary (Next 15 useSearchParams rule). i18n: all user-facing strings land in en/zh-Hans, settings tab nav includes a Sparkles-iconed Lark entry, bind-page copy lives under common.lark_bind so it works pre-workspace-context too. typecheck + lint clean. Co-authored-by: multica-agent <github@multica.ai> * chore(integrations/lark): wire outbound Patcher into server bootstrap (MUL-2671) Constructs the Patcher next to the existing Installation/BindingToken wiring in router.go and Register()s it on the event bus. With the stub APIClient any actual transport call surfaces ErrAPIClientNotConfigured; once the real Lark client lands, swap NewStubAPIClient for the real implementation here without touching the Patcher's subscription logic. doc.go updated to reflect everything the package now contains (Hub, Patcher, OAuthService, APIClient interface). The Hub itself is NOT booted here yet — it needs an EventConnector implementation for the Lark long-connection wire protocol, which lands in a follow-up; the orchestrator code and its unit tests are in place so that follow-up can focus on the WS protocol rather than lifecycle plumbing. Co-authored-by: multica-agent <github@multica.ai> * fix(integrations/lark): address Elon 二审 5 must-fix items (MUL-2671) - Hub: renewer cancels run ctx on lease loss so the connector exits even if its wire I/O is blocked, keeping the §4.4 ownership invariant intact under lease theft. - Hub: EventEmitter returns (DispatchResult, error) so the real connector can post the matching Lark-side card (needs_binding, agent_offline, agent_archived) and react to infra failures instead of silently logging at the seam. - Dispatcher: top-level message_id dedup runs before group filter and identity check, so a reconnect storm cannot re-fire binding prompts or re-spam not_addressed_in_group audit rows; the in- AppendUserMessage dedup is removed since the table-level UNIQUE is the ultimate backstop. - OAuth: HandleCallback auto-binds the installer via the new InstallerBinder seam (BindingTokenService implements it), so the §2.1 "scan to bind, you're done" promise holds end-to-end. validateExchangeResult now requires installer open_id; new error reason codes wired through the callback redirect. - Frontend / handler: install_supported listing field + StartLark- Install short-circuit on stub APIClient hide install entry points (Settings tab + per-agent button) while no real Lark HTTP client is wired, so users do not land in an OAuth flow that fails at exchange. Includes tests for each fix (lease-loss cancel, emit error propagation, dedup ordering, OAuth installer-bind contract, stub- client install gate) and i18n strings for the new preview state. Co-authored-by: multica-agent <github@multica.ai> * fix(integrations/lark): two-phase dedup so infra failures do not swallow messages (MUL-2671) The pre-fix top-level dedup wrote the lark_inbound_message_dedup row before EnsureChatSession / AppendUserMessage. An infra error in either step left the row in place and a WS-adapter retry was mis-classified as a duplicate, so the user's Lark message was permanently lost without ever landing in chat_session. Make dedup two-phase: - ClaimLarkInboundDedup acquires an in-flight claim (processed_at NULL). Stale claims older than 60 s are re-takeable so a process crash does not strand the message_id. - MarkLarkInboundDedupProcessed flips processed_at on durable success (audit row OR chat_message + session touch). - ReleaseLarkInboundDedup deletes the in-flight row on infra failure before any durable side effect, so the retry can re-claim immediately. Dispatcher.Handle now finalizes the claim exactly once based on whether the inner pipeline reached a durable outcome — chat_message commit being the transition point (errors past it Mark, errors before it Release). Regression tests cover the two failure variants Elon flagged plus the inverse invariants (durable-error Marks, drops Mark, in-flight replays drop, stale claims re-claim). Co-authored-by: multica-agent <github@multica.ai> * fix(integrations/lark): owner-fence dedup claim to close the double-write windows (MUL-2671) The two-phase Claim/Mark/Release fix from the previous commit closed the "infra error swallows a replay" gap but left two windows that could still write a chat_message twice for the same Lark message_id: 1. Stale-reclaim race. Worker A claims at t=0, runs slowly past the 60 s staleness TTL but is still alive. Worker B sees the row as stale and re-takes the claim. A reaches AppendUserMessage and commits a second chat_message. 2. Mark window. Worker A commits chat_message but the post-pipeline MarkLarkInboundDedupProcessed fails (DB hiccup) or the process crashes before it runs. 60 s later a retry treats the in-flight row as stale, re-claims it, and writes a second chat_message. Close both with owner fencing + same-tx Mark: - lark_inbound_message_dedup now carries a `claim_token` UUID; ClaimLarkInboundDedup mints a fresh one on insert and on stale re-take, so a reclaim ROTATES the token. - MarkLarkInboundDedupProcessed and ReleaseLarkInboundDedup are fenced on (message_id, claim_token, processed_at IS NULL) and return rowsAffected. Zero means our token is no longer live, and the caller treats it as a no-op (not an error). - AppendUserMessage invokes MarkLarkInboundDedupProcessed INSIDE its chat_message+session tx (qtx). If the token has been rotated by a concurrent reclaim, the Mark matches zero rows and the method returns ErrClaimLost; the deferred Rollback unwinds the chat_message insert, so the other holder is the sole writer. The durable write and the Mark therefore commit (or roll back) atomically — there is no "committed but not yet Marked" window for a crash or retry to exploit. Dispatcher.processClaimed now returns a tri-state dedupFinalize directive (none / mark / release): finalizeNone for the in-tx Mark path (and ErrClaimLost), finalizeMark for audit-drop branches and the defensive post-Append-success fallback, finalizeRelease for pre-durable infra errors. ErrClaimLost is translated into OutcomeDropped + DropReason- Duplicate at the Handle boundary, matching what the WS adapter expects for a "another worker is the writer" outcome. Regression tests: - TestDispatcher_StaleReclaimRaceDoesNotDoubleWrite injects worker B's reclaim via a beforeAppend hook so the claim_token rotates between Claim and AppendUserMessage. Asserts worker A's AppendUserMessage returns ErrClaimLost (no chat_message committed), the dispatcher surfaces a duplicate drop, the token rotated to a value distinct from A's original, and a follow-up replay still duplicate-drops. - TestDispatcher_InTxMarkPreventsPostCommitReclaim verifies the "Mark window" case is unreachable: a successful in-tx Mark produces exactly one Mark call (no post-finalize duplicate), the row is terminal, and a retry with dedupReclaim=true still duplicate-drops without re-rotating the token. - TestDispatcher_InTxMarkSucceedsAndSkipsPostFinalize pins the positive contract: DedupMarked=true must make applyFinalize a no-op (no extra Mark, no Release). fakeQueries gains a fakeDedupRow model carrying (processed, token, rotations) so the test seam matches production's UPDATE-with-WHERE semantics; fakeChat gains a beforeAppend hook to inject race timing. go test ./... and go vet ./... pass. Co-authored-by: multica-agent <github@multica.ai> * feat(integrations/lark): real Lark HTTP APIClient for IM v1 send/patch (MUL-2671) Lands the production Lark Open Platform HTTP APIClient that replaces the stub for outbound transport. The patcher's "thinking → streaming → final | error" card lifecycle and the dispatcher's binding-prompt card both now reach Lark for real once MULTICA_LARK_HTTP_ENABLED=true. Scope of this stage: - tenant_access_token retrieval via /open-apis/auth/v3/ tenant_access_token/internal, cached in-process per app_id with a 60s safety margin against Lark's `expire` value. Sub-2-minute expires are clamped to 120s so we never cache an entry that's already past its safe window. - SendInteractiveCard: POST /open-apis/im/v1/messages?receive_id_type=chat_id returning the Lark message_id the Patcher persists in lark_outbound_card_message for later patches. - PatchInteractiveCard: PATCH /open-apis/im/v1/messages/:id with the full re-rendered card body (Lark's update endpoint replaces, not deep-merges). - SendBindingPromptCard: open_id-targeted interactive card with a primary "去绑定" CTA pointing at the redemption URL. Template is co-located with the transport so the dispatcher never has to know about Lark's card schema. - Token-error invalidation: Lark codes 99991663 (expired) / 99991664 (invalid) drop the cached token so the next call refreshes from /tenant_access_token/internal instead of looping on a stale entry. Out of scope (deferred to follow-up stages): - ExchangeOAuthCode stays unimplemented behind ErrAPIClientNotConfigured. The PersonalAgent install handshake's response shape (returning per-installation app credentials in a single call) is not yet verified against the production endpoint, and a silent mis-fill of OAuthExchangeResult would corrupt lark_installation rows past validateExchangeResult. Operators continue to use the manual-paste InstallationService path until the OAuth stage lands. - Inbound WS EventConnector — Hub's ConnectorFactory still needs a real wire-protocol implementation. Wiring: - MULTICA_LARK_HTTP_ENABLED=true switches router.go from the stub to the real client. MULTICA_LARK_HTTP_BASE_URL overrides the default open.feishu.cn host (set to open.larksuite.com for the Lark international tenant, or to an httptest URL for integration tests). - The OAuth handler now also receives the real client (its ExchangeOAuthCode still surfaces ErrAPIClientNotConfigured, so callback behavior is unchanged until that stage lands). Tests (19 new cases against an httptest.Server fake): - happy path send/patch/binding-prompt round trips, asserting URL query params, body shape, Authorization header - token cache: 3 sends share one /tenant_access_token/internal hit - token refresh after clock-driven expiry - sub-margin expire clamping (10s expire → cached for >= safety margin of wall-clock) - Lark error code surfacing (230001 send, 230002 patch, 10003 auth) - token-expired (99991663) invalidates the cache; caller's retry re-fetches and succeeds - non-2xx HTTP status surfaces "http 500: …" - input validation: missing chat_id short-circuits BEFORE auth round-trip, missing card json / open_id / bind url all fail pre-flight without hitting Lark - ExchangeOAuthCode still returns ErrAPIClientNotConfigured - binding-prompt template carries the BindURL and the localized "去绑定" CTA in valid JSON go build ./..., go vet ./..., and go test ./internal/integrations/lark/... pass. Pre-existing handler/router integration tests that require a real Postgres connection are unaffected by this change. Co-authored-by: multica-agent <github@multica.ai> * fix(integrations/lark): split outbound vs OAuth-install capability + card update_multi (MUL-2671) Address Elon's two must-fix items from the HEADa09993b1review: 1. HTTP outbound and OAuth-install are now distinct APIClient capabilities. The new SupportsOAuthInstall() reports whether the install flow can succeed end-to-end (i.e. ExchangeOAuthCode is implemented); the real httpAPIClient still returns IsConfigured() = true (send / patch / binding prompt work) but SupportsOAuthInstall() = false until the PersonalAgent install-time response shape is pinned. Handler-side `install_supported` and StartLarkInstall now gate on SupportsOAuthInstall, so a half-wired client never reveals the scan-to-bind UI. larkOAuthErrorReason also maps ErrAPIClientNotConfigured to a dedicated `oauth_exchange_unimplemented` reason so a raw callback hit no longer masquerades as `internal_error`. 2. defaultRenderer now emits config.update_multi=true on every Kind. Lark refuses to apply PatchInteractiveCard to a card whose initial config doesn't declare it shared/updatable, so the absent flag would make every patch after the first send silently no-op on the wire while the local outbound status row still flipped to streaming/final. Tests cover both halves of each fix: - TestHTTPClient_SupportsOAuthInstall_FalseUntilExchangeLands + TestHTTPClient_StubReportsBothCapabilitiesFalse pin the new capability surface. - TestStartLarkInstall_TransportOnlyClientReportsNotConfigured + TestListLarkInstallations_TransportOnlyClientReportsInstallNotSupported pin the handler gate at exactly the half-wired state. - TestLarkOAuthErrorReason_APIClientNotConfigured pins the mapping for both the bare sentinel and the fmt.Errorf-wrapped form HandleCallback produces. - TestDefaultRendererConfigCarriesUpdateMulti covers every CardKind. - TestHTTPClient_(Send|Patch)InteractiveCard_DefaultRendererBodyHasUpdateMulti verify the wire body Lark actually receives carries update_multi through both send and patch transport paths. Co-authored-by: multica-agent <github@multica.ai> * feat(integrations/lark): real OAuth code exchange + agent-detail bind entry (MUL-2671) Stages the install side of the MVP critical path on top of the real HTTP outbound work: - httpAPIClient.ExchangeOAuthCode runs the production Lark v2 OAuth flow: POST /authen/v2/oauth/token to swap the authorization code for the installer's open_id, then GET /bot/v3/info under the parent app's tenant_access_token to fetch bot_open_id. Result feeds InstallationParams unchanged so OAuthService.HandleCallback's auto-bind step lights up automatically. - HTTPClientConfig gains OAuthAppID/OAuthAppSecret, read from the same MULTICA_LARK_OAUTH_APP_ID/_APP_SECRET env vars the OAuthConfig consumes. SupportsOAuthInstall now mirrors that pair so the install capability gate is honest: outbound transport without OAuth creds reports configured-but-not-install-supported, exactly like before. - Agent detail inspector wires the LarkAgentBindButton in a new Integrations section, viewer-hidden by canEdit. The button still self-hides when SupportsOAuthInstall is false, so a deployment without OAuth creds renders the section empty rather than CTA-broken. - Capability wording cleaned across handler / router / lark-tab to say "OAuth-install capability" instead of "real APIClient wired", and the misleading TransportOnly... test was renamed/refocused on the early-return branch it actually exercises (Elon non-blocking note). Co-authored-by: multica-agent <github@multica.ai> * fix(integrations/lark): identity-only OAuth + atomic bind (MUL-2671) Addresses Elon's round-4 must-fix items on PR #3277: 1. OAuth v2 token → user_info chain now matches Lark's official user-OAuth shape. `httpAPIClient.ExchangeOAuthCode` POSTs /open-apis/authen/v2/oauth/token (RFC 6749: top-level access_token, NO open_id), then GETs /open-apis/authen/v1/user_info with the user_access_token as Bearer to obtain the installer's open_id / union_id. The test fixture now reflects the real wire shape (separate user_info handler; no synthetic open_id in the token response). 2. `OAuthExchangeResult` is identity-only — drops the synthesized shared-parent AppID / AppSecret / BotOpenID return that broke the UNIQUE(app_id) constraint and the dispatcher's per-app_id routing. `OAuthService.HandleCallback` no longer Upserts an installation row: it looks up the lark_installation already provisioned via the manual-paste POST /lark/installations route and binds the installer onto it. Two new typed errors — ErrInstallationNotProvisioned and ErrInstallationRevoked — map to `installation_not_provisioned` / `installation_revoked` reasons at the HTTP boundary so the UI can guide the admin. The PersonalAgent install API (which would deliver per-installation bot credentials at scan time) remains a follow-up; until it lands the OAuth flow is identity-binding only and the agent-detail bind button stays hidden on deployments without OAuth env (capability gate unchanged). 3. The installation lookup + installer bind run inside a single DB transaction so a concurrent revoke / re-provision between the read and the binding insert cannot leak a half-applied state. `InstallerBinder.BindInstaller` is renamed to `BindInstallerTx` and accepts the OAuth-service-owned transaction's qtx; the binding_token redemption path is unchanged. 4. `validateExchangeResult` is simplified to require only the installer's open_id; the obsolete ErrExchangeMissingAppID / AppSecret / BotOpenID sentinels are removed (no caller can trip them now). The oauth_test suite is rewritten to use a stub failTxStarter so tests covering state-token verification and exchange-error propagation remain DB-free, while a new TestOAuthCallbackOpensTxAfterValidExchange pins the post-must-fix order (state ok + exchange ok ⇒ Begin runs before any lookup or bind, and a Begin failure aborts cleanly with no bind). Verified locally: - go build ./... / go vet ./... clean - go test ./internal/integrations/lark/... ✓ - go test ./internal/handler -run 'Lark|Binding|OAuth' ✓ - go test ./internal/util/secretbox/... ./internal/service/... ✓ Co-authored-by: multica-agent <github@multica.ai> * feat(integrations/lark): device-flow scan-to-install (MUL-2671) Replaces the manual paste-credentials install path + identity-only OAuth callback (rejected in product review: too many steps before a user sees value) with a true single-step scan-to-install built on Lark's RFC 8628 device-flow registration endpoint (POST accounts.feishu.cn/oauth/v1/app/registration) — the same protocol the official larksuite/oapi-sdk-go/scene/registration package and zarazhangrui/feishu-claude-code-bridge use. User journey: admin clicks "Bind to Lark" on the Agent detail page → QR dialog opens → admin scans in the Lark app on their phone → authorizes the new PersonalAgent → dialog auto-closes with the new installation visible. No app_id / app_secret to copy, no Lark developer console visit, no Multica-side OAuth env to configure. Backend (server/internal/integrations/lark): - registration.go — inline ~280-line RFC 8628 client. Begin posts archetype=PersonalAgent / auth_method=client_secret / request_user_info=open_id; Poll follows the upstream SDK's state machine including the tenant-brand mid-stream domain swap to accounts.larksuite.com when a Lark-international account authorizes. SDK is NOT vendored — one endpoint isn't worth dragging the full oapi-sdk-go + transitive deps. - registration_service.go — owns the in-process session store + background polling goroutine. On success calls APIClient.GetBotInfo (the new IM-side endpoint added below) and writes lark_installation + the installer's lark_user_binding inside one DB transaction so a half-applied install can never land. Stable error_reason codes (expired / access_denied / lark_protocol_error / bot_info_failed / installation_conflict / installer_bind_failed / internal_error) drive the UI copy without parsing prose. - client.go / http_client.go — drops ExchangeOAuthCode and SupportsOAuthInstall (no longer applicable: device-flow returns identity alongside credentials in one response); adds GetBotInfo which mints a tenant_access_token from the freshly-minted client_id / client_secret and calls /open-apis/bot/v3/info for the bot_open_id. install_supported now gates on IsConfigured() (real HTTP client wired) instead of a separate OAuth capability. - binding_token.go — absorbs InstallerBindParams / InstallerBinder (previously in oauth.go), retargets the doc-comment from the OAuth caller to the device-flow caller. - Deletes oauth.go + oauth_test.go entirely. Handler & router (server/internal/handler, server/cmd/server): - POST /api/workspaces/{id}/lark/install/begin — opens a new registration session, returns {session_id, qr_code_url, expires_in_seconds, poll_interval_seconds}. Admin-only. - GET /api/workspaces/{id}/lark/install/{sessionId}/status — polling endpoint, returns {status, installation_id?, error_reason?, error_message?}. Workspace-scoped lookup so a stolen session_id cannot be polled from another workspace. Admin-only. - Removes POST /lark/installations (paste form), GET /lark/install/start (OAuth-redirect entry), and GET /api/lark/install/callback (OAuth redirect target). - Removes MULTICA_LARK_OAUTH_APP_ID / _APP_SECRET / _REDIRECT_URI / _STATE_SECRET / _AUTHORIZE_URL / _SUCCESS_URL env vars. Self-host operators no longer need a parent Lark app at all. Frontend (packages/core, packages/views): - New types BeginLarkInstallResponse / LarkInstallStatusResponse + matching API methods (beginLarkInstall / getLarkInstallStatus); drops getLarkInstallURL. - LarkAgentBindButton opens LarkInstallDialog instead of a window.open() to Lark's authorize page. The dialog uses react-qr-code (catalog) to render the verification_uri_complete inline as SVG (no external CDN image), polls status at the server-supplied cadence, auto-closes on success, offers "scan again" on terminal failure. Per CLAUDE.md "Enum drift downgrades, not crashes", error_reason switch has a default fallback so an older desktop build on a newer server still renders the generic failure copy. - Adds the device-flow strings to en + zh-Hans settings.json; removes the obsolete OAuth-not-configured copy. Verified locally: - go build ./... / go vet ./... clean - go test ./internal/integrations/lark/... — all green (existing tests + 15 new registration / GetBotInfo tests) - go test ./internal/handler -run 'Lark|Binding' — all green - pnpm typecheck — all 6 packages clean - pnpm lint — 0 errors (15 pre-existing warnings, none in changed files) - pnpm --filter @multica/views test — 859/859 pass Pre-existing failures in server/internal/middleware (column "profile_description" missing from local test DB) reproduce against the parent commit and are unrelated to this change. Co-authored-by: multica-agent <github@multica.ai> * fix(integrations/lark): gate bind CTA to workspace admins, terminate QR polling on 4xx (MUL-2671) Two frontend must-fixes from the PR #3277 二审: 1. LarkAgentBindButton now self-hides for non-admin viewers in addition to the existing install_supported check. The agent-detail page mounts the button under `canEdit`, which canEditAgent lets agent owners through even when they are not workspace admins — but the backend gates POST /lark/install/begin and the status poll on owner/admin (router.go:478-487), so the previous behavior shipped a CTA that was guaranteed to 403. The new gate reads workspace role from the same member list the settings tab already uses. 2. The status polling loop now terminates on 404 (session gone — server restarted, multi-instance routing, or in-process GC swept it) and 403/401 (permission revoked mid-session). Previously every error path scheduled another setTimeout, which trapped the user on a stale QR forever. ApiError gives us the HTTP status verbatim; terminal responses set status=error with stable error_reason codes (session_lost, forbidden) that flow through the existing dialog switch + retry/close affordances. 5xx + network blips still retry. i18n: new install_error_session_lost / install_error_forbidden in en and zh-Hans, with default fallback preserved per the enum-drift rule. Coverage: 6 new vitest cases — admin/owner allow, member deny, unsupported-install deny, and the two terminal-error polling paths using fake timers to assert the loop stops scheduling. Also clears a handful of stale OAuth/manual-install doc comments flagged in the review (non-blocker cleanup): doc.go's §10 now points at RegistrationService, installation.go's input-shape doc loses the OAuth-callback half, and client.go's stubAPIClient comments no longer reference OAuth callbacks. Co-authored-by: multica-agent <github@multica.ai> * docs(integrations/lark): describe gate as device-flow install in agent-detail integrations comment (MUL-2671) The comment block above the agent-detail Integrations section still described the capability gate as 'server-side OAuth-install'. The OAuth path is gone — install is now device-flow per RFC 8628 — so the comment now reads 'server-side device-flow install capability gate'. Pure comment change; behavior is unchanged. Cleans up the nit Elon called out in PR #3277 二审 (MUL-2671). Co-authored-by: multica-agent <github@multica.ai> * feat(integrations/lark): wire inbound pipeline + WS Hub at boot (MUL-2671) Stage 3.a of MUL-2671. Hub class, Dispatcher, ChatSessionService and AuditLogger have all been implemented and tested in prior PRs but none of them was constructed at boot, so the in-process plumbing was never exercised end-to-end. This change wires them together behind the same `MULTICA_LARK_SECRET_KEY` gate that already gates InstallationService / RegistrationService, and starts the Hub under the existing `sweepCtx` so it winds down alongside the other long-running workers after HTTP drain. The real long-conn EventConnector is still pending; the factory hands every supervisor a shared NoopConnector that holds the lease and emits nothing. That lets staging exercise the lease / supervisor / shutdown lifecycle against real DB rows without committing to the Lark wire protocol implementation. Swapping in the real connector is a single line change in the same router block; the Dispatcher / ChatSessionService / Hub seams stay frozen. ## Why a noop placeholder, not a stub-or-skip The Hub's value is mostly its lifecycle: §4.4 ownership lease, LeaseRenewInterval / LeaseTTL, supervisor reap on revoke, clean release on shutdown. None of that runs unless the Hub is actually started. Holding off until the real connector lands means the next PR has to debut both pieces simultaneously; wiring the supervisor loop first lets the real connector PR be a focused, reviewable swap. ## Changes - `internal/integrations/lark/noop_connector.go` — `NoopConnector` implementing `EventConnector`: blocks on ctx until the Hub cancels (lease loss / shutdown / revoke), emits no events, logs on enter/exit so operators see exactly which installation the supervisor is holding the lease for. - `internal/integrations/lark/noop_connector_test.go` — verifies the connector blocks until ctx cancel, returns nil on clean exit, never invokes the emit callback, and the factory shares a single connector instance across installations. - `internal/handler/handler.go` — new `LarkHub *lark.Hub` field on `Handler`. Nil when the Lark integration is disabled. - `cmd/server/router.go` — inside the existing Lark wiring block, construct `AuditLogger`, `ChatSessionService` (with `*pgxpool.Pool` for the in-tx dedup Mark), `Dispatcher` (wiring `h.IssueService` and `h.TaskService` so `/issue`-created issues share counter / duplicate guard / project boundary / broadcast / analytics with the rest of the product), and the `Hub` with the `NoopConnectorFactory`. `NewRouterWithOptions` now returns `(chi.Router, *handler.Handler)` so main.go can drive Hub lifecycle; `NewRouter` discards the handler. - `cmd/server/main.go` — start the Hub under `sweepCtx` after the other background workers, and `Wait` on it after HTTP drain + sweep cancel so the lease renewer can issue a final release before exit. Skipped entirely when `h.LarkHub == nil`. ## Test plan - [x] `go build ./...` clean - [x] `go vet ./...` clean - [x] `go test ./internal/integrations/lark/...` (new noop tests + existing hub / dispatcher / chat_service / registration / binding_token / outbound / issue_command suites) — all pass - [x] `go test ./internal/handler -run 'TestLark|TestRedeemLarkBinding'` pass — handler-side Lark surfaces unchanged - [x] `go test ./internal/service/... ./internal/util/secretbox/...` pass - [x] `pnpm --filter @multica/views exec vitest run settings/components/lark-tab` pass (6/6) — frontend lark surfaces unchanged - [ ] Local broad `go test ./internal/handler/...` still blocked by the pre-existing test DB schema drift Elon flagged in the previous round (`column "metadata" does not exist`, unrelated to this change); CI is the authoritative check. - [ ] Manual end-to-end deferred until the real long-conn EventConnector lands (next stage). MUL-2671 Co-authored-by: multica-agent <github@multica.ai> * fix(integrations/lark): bound Hub lease release + shutdown wait (MUL-2671) Lease release used context.Background(); a stalled DB pool could pin shutdown indefinitely. Add LeaseReleaseTimeout (5s default) and ShutdownTimeout (15s default) to HubConfig, route releaseLease through a bounded context, and expose WaitWithTimeout for main.go so a wedged supervisor degrades to LeaseTTL expiry on the next replica instead of blocking process exit. Also correct the LarkHub field comment in handler.go: the Hub is wired whenever the at-rest secret key is set, independent of whether the outbound HTTP APIClient is configured. Co-authored-by: multica-agent <github@multica.ai> * feat(integrations/lark): real WS long-conn connector + ctx-cancel-breaks-read (MUL-2671) Replaces NoopConnectorFactory with a production EventConnector that opens Lark's event-subscription WebSocket. Gated behind MULTICA_LARK_WS_ENABLED so staging boots stay on the noop path until operators opt in, and falls back to noop with a warning when the WS flag is set without MULTICA_LARK_HTTP_ENABLED (the real connector needs the cached tenant_access_token). Why this connector exists separately from the Hub: gorilla/websocket ReadMessage blocks on the underlying TCP socket and does not observe context. The watchdog goroutine inside WSLongConnConnector.Run closes the conn the moment ctx fires, so lease loss / shutdown breaks the blocking read in bounded time — exactly the invariant Hub renewLeaseUntil's runCancel depends on for the "at most one active WS per installation across replicas" guarantee. Tests cover this explicitly (TestWSConnectorRunReturnsOnCtxCancelEvenWhenReadIsBlocked). The Lark wire surface is split into three swappable seams so the transport layer stays tested in isolation: - EndpointFetcher (POST /event-subscription/v1/connection_token) resolves a one-shot wss URL per Run. No caching — replaying a one-shot token would look like a Lark outage. - FrameDecoder turns one raw JSON envelope into an InboundMessage or a "control / heartbeat / drop" verdict. Decoder errors log + drop the frame; they do NOT tear down the connection. - CredentialsProvider wraps InstallationService.DecryptAppSecret so plaintext app_secret lives in memory only during a Run. Also fixes the handler.go LarkHub comment: it still said "joins on Wait during graceful shutdown" but main.go has used WaitWithTimeout (bounded wait) for several commits. Comment now matches. Co-authored-by: multica-agent <github@multica.ai> * feat(integrations/lark): align WS to official binary Frame protocol + DispatchResult outbound replies (MUL-2671) Two must-fix items from Elon's review of PR #3277: 1. WS protocol layer rewritten to match the official Lark Go SDK (`larksuite/oapi-sdk-go/v3/ws`): - Bootstrap is `POST /callback/ws/endpoint` with AppID/AppSecret in the body (no tenant_access_token bearer). Response carries wss URL + ClientConfig (PingInterval / ReconnectInterval / ReconnectNonce / ReconnectCount). - `service_id` is parsed from the wss URL query and used as Frame.Service on every outbound frame. - Wire envelope is the binary protobuf `pbbp2.Frame` (hand-rolled via protowire to avoid pulling the whole SDK in, byte-identical field tags). JSON payloads are nested inside Frame.Payload. - Inbound data frames are ACKed with a `Response{code:200,...}` JSON payload that reuses the inbound headers; infra failures produce code=500 so Lark retries. - Ping is the app-layer binary `NewPingFrame(serviceID)` at the server-supplied cadence; WebSocket protocol PING is removed (Lark ignores it). Server-initiated pings get a pong reply. - ctx-cancel-breaks-read invariant preserved via the watchdog goroutine that closes the conn on ctx.Done; the read loop and ping goroutine serialize their writes through a single mutex. 2. `DispatchResult` outbound replies wired via a new `OutcomeReplier`: - `OutcomeNeedsBinding` mints a one-shot binding token and sends the binding prompt card to the sender's open_id. - `OutcomeAgentOffline` / `OutcomeAgentArchived` push a notice card into the chat with the agent name + Chinese copy matching §4.6. - `OutcomeIngested` stays owned by the Patcher; `OutcomeDropped` is silent. - The replier is best-effort: outbound failures are logged and swallowed so a Lark outage cannot stall the inbound pipeline. - Hub installs the noop replier by default; router wires the production `LarkOutcomeReplier` when APIClient.IsConfigured(). PersonalAgent long-conn risk surfaced (open per Feishu docs: `长连接模式仅支持企业自建应用`). The implementation works for any app archetype; the open question is whether `/callback/ws/endpoint` accepts PersonalAgent credentials in practice. Surfacing the Lark code+msg verbatim from the bootstrap response so an operator running the smoke test sees the exact failure rather than a generic timeout. Co-authored-by: multica-agent <github@multica.ai> * fix(integrations/lark): byte-compat Frame marshal, chunk reassembly, ACK off reply critical path (MUL-2671) Three protocol blockers from Elon's review of9540008a: 1. Frame.Marshal is now byte-identical to oapi-sdk-go/v3/ws/pbbp2.Frame: - SeqID/LogID/Service/Method (proto2 req) emit unconditionally even at zero - PayloadEncoding/PayloadType/LogIDNew emit unconditionally per gogo generated MarshalToSizedBuffer (no zero-guard) - Payload uses the SDK's `!= nil` guard (nil omits, []byte{} emits 0-length) - ACK payload JSON matches SDK's NewResponseByCode + json.Marshal output ({"code":N,"headers":null,"data":null}) Golden tests pin exact byte sequences for ping/pong/ACK/full/zero frames; verified against the real SDK pbbp2.pb.go MarshalToSizedBuffer producing identical bytes. 2. Multi-frame events (sum>1) are reassembled via the new chunkAssembler: - 5s sliding TTL (matches SDK combine() cache TTL) - Lazy GC on admit (no separate sweeper goroutine) - Out-of-order seq + duplicate seq idempotent - Partial chunks are NOT ACKed (SDK behaviour: only the final chunk's ACK confirms the whole event so Lark can retry on partial loss) - Connector wires assembler per-Run; state dies with the session 3. OutcomeReplier detached from ACK critical path: - HubConfig.ReplyTimeout default 2.5s, strictly under Lark's 3s ACK deadline - handleEvent dispatches synchronously (fast DB path), then spawns the replier under a fresh background ctx with WithTimeout(ReplyTimeout) - Hub.replyWg tracks in-flight replies; Hub.Wait / WaitWithTimeout drain them so shutdown is bounded - Noop replier short-circuits inline (no goroutine cost when outbound APIClient isn't configured) Proof tests: - TestHubScheduleReplyReturnsImmediately: scheduleReply with a 10s slow replier returns in <50ms - TestHubReplyTimeoutCancelsHungReplier: hung replier ctx fires at ReplyTimeout - TestHubWaitDrainsInFlightReplies: Wait blocks until replies finish - TestHubACKNotBlockedByOutboundReply: end-to-end through the connector — data-frame ACK lands within 500ms even when the replier hangs 5s PersonalAgent real-env smoke remains Bohan's decision; this PR closes the technical blockers Elon flagged. Co-authored-by: multica-agent <github@multica.ai> * docs(service/issue): narrow position concurrency claim to create-create (MUL-2671) Elon's review of the merge resolution flagged that the comment on the new NextTopPosition call promised more than the code guarantees: concurrent manual reorder via UpdateIssue(position) does NOT take the workspace row lock that IncrementIssueCounter holds, so a create racing a reorder can still land on the same position. Rewrite the comment to only claim create-create serialization, which is the behaviour the lock actually delivers. No code change. Co-authored-by: multica-agent <github@multica.ai> * fix(integrations/lark): keep device-flow polling on RFC 8628 HTTP 400 (MUL-2671) Lark's device-flow polling endpoint returns HTTP 400 with the JSON body `{"error":"authorization_pending"}` while the user hasn't scanned the QR yet — this is the RFC 8628 spec, and the upstream oapi-sdk-go implements the same handling. Our previous doForm treated ANY non-2xx as a terminal protocol error, so every install session was killed by the first poll (~5s after begin) and the install dialog appeared silently empty: the frontend received status=error + lark_protocol_error before the user could even read the description. Fix: doForm now decodes the JSON body first; if it parses, the caller (Begin / Poll) routes on the body's `error` field, where the existing switch correctly maps authorization_pending / slow_down to "keep polling" and access_denied / expired_token to terminal failure. Only unparseable bodies (5xx HTML proxy pages, gateway timeouts) still surface as a typed http_NNN RegistrationError. Three regression tests pin the new behaviour: - HTTP 400 + authorization_pending → res.Status="authorization_pending" - HTTP 400 + access_denied → res.Err.Code="access_denied" (terminal) - HTTP 502 + HTML body → http_502 RegistrationError Verified against the live local env: install/begin -> 200, status stays "pending" through the first poll cycle, no longer flips to "error" within seconds. Co-authored-by: multica-agent <github@multica.ai> * fix(views/lark): reset closedRef on every mount so StrictMode double-mount renders QR (MUL-2671) Empty QR dialog body in the dev env: Bohan opened the bind dialog and got an empty white area where the QR should have been — no QR, no "starting" placeholder, no error text. Backend was returning the QR URL correctly; the bug was on the frontend. Root cause: React 19 / Next.js dev StrictMode mounts every component twice (mount → cleanup → mount). The component instance is REUSED across the simulated remount, which means useRef objects are preserved. The dialog's `closedRef` lifecycle: 1. Mount #1: closedRef={current:false}, beginSession() kicked off (HTTP request still in flight) 2. Cleanup runs: closedRef.current=true 3. Mount #2: beginSession() kicked off again, BUT the ref still reads {current:true} from step 2 4. Both promises resolve. Both hit the post-await guard `if (closedRef.current) return;` and bail out before setSession(). 5. Result: session stays null forever. Every conditional in the dialog body (beginning/session-pending/success/error) is false → empty body. Fix: reset closedRef.current=false at the START of the effect, not just at component construction. The cleanup-then-mount pair now re-arms the guard so subsequent setSession calls actually land. Regression test wraps the dialog in <StrictMode> and asserts the QR appears within 2s with the correct value — fails closed if anyone removes the reset. Co-authored-by: multica-agent <github@multica.ai> * fix(integrations/lark): drop EventTaskCompleted subscription so the chat reply doesn't get overwritten by "Done." (MUL-2671) Bohan reproduced on the live dev env: agent replies show only a card saying "Done." in Lark, even though Multica's own chat panel has the real "Hello! I'm cc…" reply. Tasks succeed end-to-end, but the user loses the reply on the Lark side. Root cause: TaskService.CompleteTask publishes two events for every chat task IN ORDER: 1. broadcastChatDone(...) → ChatDonePayload{Content: "Hello!..."} 2. broadcastTaskEvent(Completed) → map[string]any{task_id, agent_id,...} (no `content` key) The Patcher subscribed to BOTH and routed each to finalize(). The first patch correctly rendered the reply text, the second patched the same card with an empty payload — chatDoneContent() returned "" and the renderer fell back to "Done." (default empty-body copy). The second patch wins because Lark stores whatever was last applied. Fix: stop subscribing to EventTaskCompleted in the Patcher and remove the corresponding switch arm. EventChatDone is the canonical "agent finished replying" signal for the Lark card path; EventTaskCompleted is still emitted to the bus for other listeners (web UI, analytics, task usage) where the lack of content doesn't matter. Regression test TestPatcherIgnoresEventTaskCompletedForChatTasks emits ChatDone followed by TaskCompleted on a streaming card and asserts: exactly one patch, body contains the agent reply, body does NOT contain "Done.". If anyone re-adds the EventTaskCompleted subscription, this fails immediately. Co-authored-by: multica-agent <github@multica.ai> * feat(integrations/lark): chat replies as plain text IM messages, not card chrome (MUL-2671) Bohan reported on the live dev env that even with the agent's reply shown correctly, every message is wrapped in an interactive card with the agent name as the header — it feels like a system notification, not a normal chat reply. He wants the reply to land as a regular Lark text bubble. Changes: - Add APIClient.SendTextMessage backed by Lark's /open-apis/im/v1/messages with msg_type=text. JSON-encodes the {"text": ...} envelope Lark requires so callers pass raw strings. - Patcher.Register no longer subscribes to EventTaskQueued / EventTaskRunning. There is no more thinking → running → final card lifecycle on the success path: it added card chrome without buying anything for free-form chat. - On EventChatDone, the new sendChatReply path posts the assistant message content as plain text. Empty content is silently dropped rather than rendered as "Done." (the prior fallback that confused Bohan). - Failure path keeps a one-shot error card on EventTaskFailed — the visual distinction from a normal reply is genuinely useful, and failures are rare enough that the chrome isn't noisy. - Throttle / lastPatched map / MinPatchInterval / shouldPatch / markPatched / loadCardOrSkip are all removed; nothing in the new flow patches. Tests: - TestPatcherSendsPlainTextOnChatDone pins the new contract: exactly one SendTextMessage call, no card sends or patches, content matches the ChatDonePayload. - TestPatcherDropsEmptyChatReply pins the "no more Done. fallback" decision — empty content drops, period. - TestPatcherFailEventSendsErrorCard pins the failure path still uses a card (one-shot, no patching). - TestPatcherIgnoresEventTaskCompletedForChatTasks rewritten for text path: ChatDone then TaskCompleted yields exactly one text send, no duplicate. - TestPatcherSkipsWhenNoChatSessionBinding and TestPatcherSwallowsInstallationLoadErrors rewritten to drive EventChatDone (the new entry point) instead of TaskQueued. - TestPatcherSendsThinkingCardOnTaskQueued deleted (no more thinking card). Co-authored-by: multica-agent <github@multica.ai> * feat(integrations/lark): pre-fill PersonalAgent bot name as "<agent> - Multica" (MUL-2823) (#3520) The device-flow install left the bot at Lark's auto-generated "{用户姓名}的智能助手". Lark's registration scene supports pre-filling the name via a `name` query param on the verification/QR URL (mirrors the upstream SDK's AppPreset.Name) — a user-editable default that rides on the QR URL, not the begin POST body (which has no name field). BeginInstall already loads the agent for its ownership check, so we keep it and thread `<agent.Name> - Multica` through Begin → decorateQRCodeURL. A blank name degrades to plain "Multica". There is no post-install rename API (bot/v3 is read-only; no bot/v3/update), so the install-time pre-fill is the only programmatic lever; the user can still edit the name on the creation form. Co-authored-by: J <j@multica.ai> Co-authored-by: multica-agent <github@multica.ai> * fix(integrations/lark): restore /issue confirmation + pin SendTextMessage wire (MUL-2671) Two recovered/added contracts off Trump's review of HEADfe381a07: 1) /issue confirmation in Lark was a casualty of the plain-text refactor. The pre-refactor `RenderInput.IssueNumber` field was declared but never actually rendered into the card body, so even in the original card-based flow the user never saw a "Created [MUL-42]" confirmation. Now the OutcomeReplier handles OutcomeIngested + IssueID.Valid by sending a plain text message: Created MUL-42 — fix login bug https://multica.example/issues/MUL-42 Composed from a new DispatchResult.IssueIdentifier + IssueTitle, populated by the Dispatcher from workspace.IssuePrefix + issue.Number / issue.Title. Workspace lookup is best-effort: a Postgres blip on workspace gets a "#42" fallback rather than silently dropping the confirmation. The agent's own chat reply (if any) continues to land separately via the Patcher on EventChatDone — these are two semantically distinct messages and the user benefits from seeing both. 2) SendTextMessage is the wire layer Trump flagged for missing coverage. Three new wire tests pin: - happy path: POST /open-apis/im/v1/messages?receive_id_type=chat_id, msg_type=text, Bearer <tenant_access_token>, double-JSON content envelope - special-character round trip: newlines, double quotes, backslashes, tabs, Chinese + emoji, JSON-lookalike strings. The inner {"text": ...} is encoded once at JSON.Marshal time and once again when the outer body serializes; losing either pass corrupts the message and the bug is invisible without a contract pin. - Lark error path: non-zero `code` surfaces as a wrapped error with the code embedded. Tests: - TestDispatcher_IssueCreationFromCommand asserts IssueIdentifier ("MUL-42") and IssueTitle propagate through DispatchResult. - TestDispatcher_IssueIdentifierFallsBackToNumberOnWorkspaceLookupErr pins the "#7" degrade-graceful fallback. - TestLarkOutcomeReplierIssueCreatedSendsConfirmation pins the text body (identifier + title + deep link) and asserts no card send on this path. - TestLarkOutcomeReplierOutcomeIngestedSilentWithoutIssue pins the silent-on-plain-chat default so we don't accidentally start emitting a confirmation for every message. - TestHTTPClient_SendTextMessage_* covers the wire contract. Frontend locale parity (en + zh-Hans, 53 tests) is currently green on this HEAD; no changes needed. Co-authored-by: multica-agent <github@multica.ai> * fix(views/locales): add missing ko keys for Lark MVP (MUL-2671) Trump flagged on PR #3277 review that the ko bundle was missing the Lark-MVP-only keys that en + zh-Hans both carry. The parity test caught it cleanly after main was merged in (Korean PR landed on main between the prior review and this one): common.lark_bind.* (13 keys) settings.page.tabs.lark (1 key) settings.lark.* (45 keys) agents.inspector.section_integrations (1 key) Korean translations are professional/concise — "Lark" stays as the brand name (matches how en keeps "Lark" + "(飞书)" parenthetically; ko/users searching for the product expect "Lark"), and product copy follows the zh-Hans tone where Multica nouns ("에이전트", "워크스페이스") are romanized loan words consistent with the rest of the ko bundle. Slot ordering preserved against EN: - page.tabs.lark sits between github and integrations - inspector.section_integrations sits right after section_skills Verified: pnpm exec vitest run locales/parity → 105/105 pass. Co-authored-by: multica-agent <github@multica.ai> * fix(integrations/lark): /issue origin_type CHECK + Hub restart on credentials rotation (MUL-2671) Two live-env bugs Bohan reproduced: 1) /issue command crashed the WS connector. Dispatcher writes origin_type='lark_chat' on issues born from `/issue`, but the issue_origin_type_check CHECK constraint was last extended in migration 060 for quick_create — it doesn't list lark_chat, so every Lark /issue tripped SQLSTATE 23514 and bubbled up as an infra error. The infra error tore down the WS connector, Lark retried the same message, the new connector tripped the same constraint and crashed again. Repro in the live env: three crashes from the same /issue event over ~40s, each leaving the user with no confirmation in Lark. Migration 111 extends the CHECK list: CHECK (origin_type IN ('autopilot', 'quick_create', 'lark_chat')) 2) Re-scanning an already-bound agent silenced the bot. The device flow re-registers with Lark, which mints a brand-new bot (fresh app_id + app_secret); RegistrationService.finishSuccess upserts into lark_installation by agent_id, so the row's credentials rotate in place. But the running supervisor held the OLD inst struct by value and kept a WS open against the OLD bot's app_id — so all events to the NEW bot went nowhere. Bohan's "claude code 现在不能在飞书里回复了" symptom maps exactly to this: log timeline: 16:29:57 cc connector connected with app_id=cli_aa9398dd... (OLD) 16:34:07 lark registration: install complete (rotation) → row.app_id is now cli_aa93f36f... (NEW) → old WS still subscribed to OLD app_id; new app_id receives nothing Fix: Hub.sweep now compares each installation row's credentials fingerprint (app_id + bot_open_id + sha256(app_secret_encrypted)) against the snapshot the running supervisor was started with. On diff, cancel the old supervisor and start a fresh one inline. A monotonic gen counter on the supervisor entry disambiguates the old goroutine's deferred cleanup from the new entry the rotation path already swapped in. Tests: - TestHubRestartsSupervisorOnCredentialsRotation pins the new path: starts hub on app_one, rotates the row to app_two, asserts the connector factory is called again with the fresh AppID. - TestHubDoesNotRestartSupervisorOnUnchangedRow pins the negative case so an unchanged row doesn't degenerate into a per-sweep busy-loop. - Existing hub tests (lease, supervise, shutdown, ACK timing, noop replier) all green. Verification: - go test ./internal/integrations/lark/... -race -count=1 ok - go build ./... clean - migration applied locally; \d+ issue confirms lark_chat in CHECK Co-authored-by: multica-agent <github@multica.ai> * fix(integrations/lark): per-supervisor lease token to fence rotation handoff (MUL-2671) Elon flagged a race in HEAD be8d4cef's rotation path: both the old and the new supervisors of the same Hub used the hub-wide nodeID as their WS lease token, so an old supervisor's post-cancel releaseLease(nodeID) would CAS-match the lease row the successor had just acquired with the SAME token and DELETE it. Symptom would be a silently empty lease row a few hundred ms after every device-flow re-scan — no replica owning the install, no events delivered, the "bot goes quiet" pattern Bohan hit the first time but now from the fencing side rather than the credentials side. Fix: leaseToken(nodeID, gen) composes "<nodeID>-g<gen>", where gen is the monotonic counter already attached to each supervisorEntry. The nodeID prefix keeps cross-replica observability (an operator inspecting lark_installation.ws_lease_token can still map back to a process) while the -g suffix makes the OLD supervisor's release target the OLD row state. Once the rotation path swaps in the new supervisor, the row's CurrentToken is the new -g(N+1) token, so the old -gN release's WHERE clause no-ops instead of clobbering. acquireLease / renewLeaseUntil / releaseLease now take an explicit token argument; supervise threads its leaseToken through. The plumbing isn't pretty, but having an explicit argument at every call site is the only way the rotation invariant survives subsequent refactors — without it, a future caller could quietly reintroduce "just use h.nodeID" and the race is back. Two regression tests: - TestHubRotationStaleReleaseDoesNotClearSuccessorLease drives the fake lease state machine directly: 1. old acquires(tokenA) 2. rotation lands; new acquires(tokenB) 3. old's stale release(tokenA) fires Asserts owner ends up still tokenB. Hub-wide-nodeID code would fail step 3 by clearing the entry. - TestHubRotationEndToEndKeepsSuccessorLeased runs the same scenario through the live supervise loop: starts hub, rotates the row, waits for sup2 to take over with a distinct token, sleeps past sup1's unwind, asserts the row is still held by a non-sup1 token. Catches the bug even when the goroutine timing is non-deterministic. Verification: go test ./internal/integrations/lark/... -race -count=1 ok go build ./... clean go vet ./... clean Co-authored-by: multica-agent <github@multica.ai> * fix(integrations/lark): route group @-mentions via union_id, not open_id (MUL-2671) In a Lark group with multiple Multica bots installed, the bot whose WS received the event sometimes failed to recognize that it was the @-target while the OTHER bot's supervisor falsely fired. Bohan's controlled three- message test (only @A, only @B, @both) hit this: @A and @B alone went unanswered, @both got picked up by A only. Root cause: the `mentions[].id.open_id` field Lark puts on the WS event is structurally INVERSE to `/bot/v3/info`'s `bot.open_id` across the two WSes. From A's WS perspective, the wire-form open_id for "A was @-ed" is NOT equal to A's API-side open_id, but IS equal to what B's WS sees on its side, and vice versa. The decoder's `mention.open_id == inst.BotOpenID` match therefore fires on the wrong bot in multi-bot groups. Only `union_id` (the Lark-tenant-scoped stable identifier) is consistent across both WSes. Changes: - migration 112 adds nullable `lark_installation.bot_union_id` - sqlc query exposes UpsertLarkInstallation/CreateLarkInstallation with bot_union_id, plus a focused SetLarkInstallationBotUnionID for the backfill path - httpAPIClient.GetBotInfo now follows /bot/v3/info with /contact/v3/ users/{open_id}?user_id_type=open_id and returns both identifiers on BotInfo. Soft-fails on contact-scope denial: install still succeeds with an empty UnionID, and the decoder falls back to the legacy open_id match for single-bot deployments. - RegistrationService.finishSuccess persists union_id alongside open_id during the device-flow finalize. - ws_frame_decoder.containsMention prefers union_id and only walks open_id when the installation row has not been backfilled yet. - BackfillBotUnionIDs runs once at server boot for installations created before migration 112; bounded per-row 10s timeout and a pure soft-fail policy so a slow Lark round-trip cannot block startup. - regression tests cover the three decoder paths: union_id match wins over open_id mismatch, union_id mismatch overrides open_id match, and open_id fallback when union_id is unknown. Co-authored-by: multica-agent <github@multica.ai> * chore: drop trailing blank lines at EOF on four files (MUL-2671) git diff --check origin/main..origin/pr-3277 flagged these as new blank lines at EOF; clearing so the diff stays clean for review. Co-authored-by: multica-agent <github@multica.ai> * fix(views/locales): add missing ja keys for Lark MVP + section_integrations (MUL-2671) CI frontend job tripped on the ja locale parity check: ja is missing the lark_bind block in common.json, the lark block + page.tabs.lark in settings.json, and inspector.section_integrations in agents.json. The ko fix earlier covered Korean; ja was added separately on main and the merge surfaced these gaps. Translations mirror the en source and follow the same voice as the existing ja bundle. Co-authored-by: multica-agent <github@multica.ai> * fix(integrations/lark): rewrite @_user_N placeholders into clean body (MUL-2671) When Lark dispatches a group `im.message.receive_v1`, the message text contains opaque `@_user_1`, `@_user_2`, … placeholders and the real identity is in `mentions[]`. We were forwarding the raw text to the agent, so a Bohan-typed "@Bot ping test" arrived as "@_user_1 ping test" — neither human-readable nor useful as LLM context, and the agent was paying tokens to figure out which `@_user_N` was even itself. The new resolveMentions pass: * strips the bot's own mention entirely (the dispatcher already routes the event on AddressedToBot; re-emitting @<self> in front of every message adds zero signal and pollutes context), * substitutes other participants with `@<displayName>` so a follow-up "@Alice" reads naturally, * collapses horizontal whitespace introduced by the strip while preserving original newlines. Bot identity check uses the same union_id-preferred + open_id fallback as containsMention, so the rewrite stays consistent with the routing path. Tests cover the four shapes: bot self-mention, mixed bot + other-user mention, multi-line body with stripped mention, and a no-mention body that should be left untouched. Co-authored-by: multica-agent <github@multica.ai> * fix(integrations/lark): union_id-first self mention strip + token-aware scan + local whitespace cleanup (MUL-2671) Three review blockers on the mention rewrite from PR review: 1. isBotMention now mirrors containsMention's union_id-first policy. When the installation row knows our union_id, we trust it exclusively (open_id is structurally inverted in multi-bot groups — matching on it would re-introduce the routing bug we fixed two commits ago). open_id fallback fires only when union_id is absent. New tests: @-ing both bots in one message correctly strips only self and renders the sibling as @<name>; open_id-matches-but-union_id-differs does NOT strip. 2. resolveMentions no longer collapses or trims whitespace globally. Indentation, tabs, code blocks, tables — all preserved verbatim. When the self mention is removed we eat exactly one adjacent horizontal space (the one after the placeholder, or, when the mention sits at end-of-input, a single space already emitted right before it). New test exercises a multi-line indented + tabbed body and asserts the whole shape survives. 3. Prefix-collision-safe replacement. A chat with 11+ participants exposes both `@_user_1` and `@_user_10`; naive ReplaceAll for `@_user_1` would mangle the substring of `@_user_10`. The resolver now does a single-pass token scan with the mention list sorted longest-key-first, so the longer placeholder always wins at any scan position. New test covers the @_user_1 / @_user_10 case explicitly. Also drops the temporary INFO-level diag logging the previous commit added — root cause was confirmed (union_id swap in the manual backfill; not a decoder bug). Co-authored-by: multica-agent <github@multica.ai> * fix(integrations/lark): scope inbound dedup per (installation_id, message_id) (MUL-2671) Root cause of the residual "@Cc gets dropped as not_addressed_in_group" even after the union_id swap landed: lark_inbound_message_dedup was keyed on `message_id` alone. In a Lark group chat where the workspace has multiple Multica bots installed, Lark delivers the SAME message_id to every bot's WS supervisor. Whichever WS claimed first then ran its own AddressedToBot check; the bot that was actually @-ed lost the dedup race, found the row already terminal (`processed_at IS NOT NULL`), and was dropped as `duplicate` BEFORE it could evaluate its own mention. Net: every @ silently disappeared if Lark happened to route the OTHER bot's WS first. The dedup gate's original purpose (idempotency against WS reconnect replay) is per-installation by definition, so the right key is composite (installation_id, message_id). Changes: - migration 113 drops + recreates lark_inbound_message_dedup with installation_id NOT NULL REFERENCES lark_installation(id) ON DELETE CASCADE and PRIMARY KEY (installation_id, message_id). The table is a 24h transient cache, so dropping existing rows is safe. - sqlc queries: ClaimLarkInboundDedup / MarkLarkInboundDedupProcessed / ReleaseLarkInboundDedup all now take installation_id. - AppendUserMessageParams carries InstallationID through to the in-tx Mark call so the chat_message+dedup atomicity stays intact. - Dispatcher passes inst.ID to claim + applyFinalize + AppendUserMessage. - Test fakes key dedup state on (installation_id, message_id) via a composite map key; all existing pre-seeded rows use a seedDedupKey helper bound to the default activeInstallation fixture so the prior staleness / token-rotation / in-tx mark tests still exercise the same regression they did before. - New regression TestDispatcher_DedupIsScopedPerInstallation pins the multi-bot invariant: a row pre-seeded for installation A does NOT block installation B's first delivery of the same message_id; B runs through its own group-filter / identity / ingest pipeline. Co-authored-by: multica-agent <github@multica.ai> * feat(integrations/lark): render markdown chat replies via schema-2.0 card (MUL-2671) The agent's chat replies were going out as msg_type=text, so every `**bold**`, fenced code block, list, table, and link in the body showed up as literal markdown characters in Lark — the user saw raw asterisks, hashes, pipes instead of formatted text. Bohan reported this and pointed at zarazhangrui/lark-coding-agent-bridge as the shape to emulate. The bridge repo uses Lark interactive cards with the schema-2.0 envelope and a `tag: "markdown"` body element; Lark's client renders that to formatted text (GFM-ish: bold/italic, headings, lists, links, fenced code blocks, tables, blockquotes). They expose multiple reply modes (card / markdown-as-post / text) gated by user config; we go a step simpler — auto-detect markdown syntax in the agent's body and route accordingly: - containsMarkdown(): cheap substring + regex pass for fenced code blocks, headings, list markers, bold/italic, tables, links, blockquotes, horizontal rules, inline code. Biases toward false- positive — wrapping prose in a card still renders fine, but missing a real markdown block leaves raw characters visible. - APIClient gains SendMarkdownCard / SendMarkdownCardParams. Implementation marshals the schema-2.0 envelope verbatim: {schema:"2.0", body:{elements:[{tag:"markdown", content: md}]}}. Stub returns ErrAPIClientNotConfigured. - Patcher.sendChatReply now branches on containsMarkdown: markdown → SendMarkdownCard, plain prose → SendTextMessage. A one-liner "sure, on it" stays as a normal IM bubble (no card chrome); anything with markdown gets the rendered card. Tests: TestContainsMarkdown pins the heuristic across plain prose and ten markdown shapes; TestPatcherRoutesMarkdownReplyToCard and TestPatcherRoutesPlainReplyToText cover the router; new HTTP wire test TestHTTPClient_SendMarkdownCard_HappyPath contract-pins the card envelope (msg_type=interactive, schema 2.0, markdown tag, verbatim body). Full lark suite passes. Co-authored-by: multica-agent <github@multica.ai> * fix(service/issue): route analytics.IssueCreated through obsmetrics.RecordEvent (MUL-2671) CI's TestNoNakedAnalyticsCaptureInHandlersOrServices guard caught the post-merge analytics call in IssueService.captureCreatedAnalytics that still used s.Analytics.Capture(...) directly. Main added that lint to prevent the Prometheus and PostHog sides from drifting — any new analytics.* event must go through obsmetrics.RecordEvent so the business-metrics collector and the PostHog client fire from the same call site. Fix mirrors how TaskService handles it: IssueService gains a Metrics *obsmetrics.BusinessMetrics field (router wires it via h.IssueService.Metrics = opts.BusinessMetrics next to the existing TaskService line), and the in-service Capture call becomes obsmetrics.RecordEvent(s.Analytics, s.Metrics, ...). nil-safe by construction — RecordEvent treats a nil Metrics as PostHog-only. Co-authored-by: multica-agent <github@multica.ai> * feat(views/lark): swap Bind CTA for Connected+Manage link when agent already has an installation (MUL-2671) Bohan reported the agent-detail Bind button keeps inviting the user to re-scan the QR even when the agent already has an active Lark PersonalAgent connected — and re-scanning silently upserts the installation row, leaving the previously-created Lark bot dangling as a zombie. Frustrating UX and an actual product footgun. Anti-zombie guard at the only entry point: LarkAgentBindButton now checks the cached installations listing for an active row pinned to this agent_id. When one exists, the install CTA is gone — replaced by a small Connected pill + an "Manage in Lark" link that opens the Bot's app page in Lark's developer console (open.feishu.cn/app/<app_id>) in a new tab. That's where scopes, display name, and additional permission requests actually live; re-scanning never was the right answer for managing an existing bot. Scoping is per-agent: an active installation on a DIFFERENT agent in the same workspace doesn't affect this agent's button, and a revoked installation falls back to the bind CTA so the user can re-create. Tests cover all four states (own-active / own-revoked / other-agent-active / no-installation) and pin the Manage link's href + target=_blank + noopener. i18n: three new keys in settings.json (en / zh-Hans / ja / ko): agent_bot_connected_label, agent_bot_manage_link, agent_bot_manage_tooltip. Locale parity test still 157/157. The dev console host is hardcoded to open.feishu.cn — operators on the Lark international tenant currently get the wrong host; future-proof fix wants the backend to surface a per-installation dev_console_url on the listings response, called out in a code comment. Co-authored-by: multica-agent <github@multica.ai> * feat(views/settings): collapse Lark into Integrations + render agent identity (MUL-2671) Lark was its own top-level workspace settings tab while Integrations sat empty next to it. As more integrations land, the sidebar would balloon with one tab per provider. Move the Lark surface into Integrations as the first hosted integration; the old ?tab=lark URL redirects through LEGACY_WORKSPACE_TAB_REDIRECTS so bookmarks still resolve. The Connected bots list was leaking the raw Lark app_id (cli_…) as the row title with bot_open_id (ou_…) underneath — meaningless to product users. Since the binding is 1:1 with a Multica Agent, join on agent_id and render the agent's avatar + name via the workspace-standard ActorAvatar + useActorName.getAgentName. Deleted agents fall back to "Unknown Agent" so the row is still actionable for cleanup. Tests: stub useActorName + ActorAvatar in lark-tab.test.tsx and add LarkTab connected-bot tests covering the agent identity render and the deleted-agent fallback. Drop the now-dead integrations.* + page.tabs.lark + lark.bot_open_id_label keys across all four locales — parity still 157/157, views suite 1141/1141. Co-authored-by: multica-agent <github@multica.ai> * feat(views/settings): wrap Lark in a named section inside Integrations (MUL-2671) Integrations is meant to host multiple providers (Slack, Linear etc. as they land), so the Lark content should sit under a Lark heading rather than fill the tab directly — otherwise the first additional integration would feel like it broke the IA. Add a "Lark" / "飞书" section heading above LarkTab using the same h2 chrome the other settings tabs use, and pin lark.section_title across all four locales (parity 169/169). Co-authored-by: multica-agent <github@multica.ai> --------- Co-authored-by: multica-agent <github@multica.ai> Co-authored-by: J <j@multica.ai>
2424 lines
90 KiB
Go
2424 lines
90 KiB
Go
package service
|
|
|
|
import (
|
|
"context"
|
|
"encoding/json"
|
|
"errors"
|
|
"fmt"
|
|
"log/slog"
|
|
"strconv"
|
|
"strings"
|
|
"sync"
|
|
"time"
|
|
|
|
"github.com/jackc/pgx/v5"
|
|
"github.com/jackc/pgx/v5/pgtype"
|
|
"github.com/multica-ai/multica/server/internal/analytics"
|
|
"github.com/multica-ai/multica/server/internal/events"
|
|
"github.com/multica-ai/multica/server/internal/mention"
|
|
obsmetrics "github.com/multica-ai/multica/server/internal/metrics"
|
|
"github.com/multica-ai/multica/server/internal/realtime"
|
|
"github.com/multica-ai/multica/server/internal/util"
|
|
db "github.com/multica-ai/multica/server/pkg/db/generated"
|
|
"github.com/multica-ai/multica/server/pkg/protocol"
|
|
"github.com/multica-ai/multica/server/pkg/redact"
|
|
"github.com/multica-ai/multica/server/pkg/taskfailure"
|
|
)
|
|
|
|
type TaskService struct {
|
|
Queries *db.Queries
|
|
TxStarter TxStarter
|
|
Hub *realtime.Hub
|
|
Bus *events.Bus
|
|
Analytics analytics.Client
|
|
Metrics *obsmetrics.BusinessMetrics
|
|
Wakeup TaskWakeupNotifier
|
|
// EmptyClaim caches "this runtime has no queued task" so the daemon
|
|
// poll path can skip a Postgres scan on the steady-state empty case.
|
|
// Optional — a nil cache disables the fast path and every claim
|
|
// goes through the DB. Wired in router.go from the shared Redis
|
|
// client.
|
|
EmptyClaim *EmptyClaimCache
|
|
|
|
analyticsContextMu sync.Mutex
|
|
analyticsContextCache map[string]analytics.TaskContext
|
|
analyticsContextOrder []string
|
|
}
|
|
|
|
type TaskWakeupNotifier interface {
|
|
NotifyTaskAvailable(runtimeID, taskID string)
|
|
}
|
|
|
|
// triggerSummaryMaxLen caps the snapshot length so the row stays cheap to
|
|
// transmit (it ends up in every task list response). 200 is enough for a
|
|
// recognisable preview of a one-paragraph comment.
|
|
const triggerSummaryMaxLen = 200
|
|
|
|
// truncateForSummary returns s shortened to maxRunes, with a trailing
|
|
// `…` when truncated. Operates on runes (not bytes) so multibyte characters
|
|
// — Chinese / emoji — count as one each. Strips surrounding whitespace
|
|
// first so a leading newline doesn't waste budget.
|
|
func truncateForSummary(s string, maxRunes int) string {
|
|
// strings.Builder + Grow avoids the O(N²) realloc cycle of `+=` in
|
|
// a loop. Grow uses byte length, which is an upper bound for the
|
|
// rune-equivalent output (replacing \n/\r/\t with space is byte-equal
|
|
// for ASCII whitespace), so we never reallocate.
|
|
var b strings.Builder
|
|
b.Grow(len(s))
|
|
for _, r := range s {
|
|
switch r {
|
|
case '\n', '\r', '\t':
|
|
b.WriteByte(' ')
|
|
default:
|
|
b.WriteRune(r)
|
|
}
|
|
}
|
|
rs := []rune(strings.TrimSpace(b.String()))
|
|
if len(rs) <= maxRunes {
|
|
return string(rs)
|
|
}
|
|
return string(rs[:maxRunes]) + "…"
|
|
}
|
|
|
|
const (
|
|
taskAnalyticsContextCacheMax = 4096
|
|
// claimResponseRecoveryWindow must exceed daemon client.Timeout for
|
|
// /tasks/claim (30s) plus /tasks/{id}/start (30s) plus scheduling slack, so
|
|
// an in-flight StartTask cannot be reclaimed and double-dispatched.
|
|
claimResponseRecoveryWindow = 90 * time.Second
|
|
)
|
|
|
|
// buildCommentTriggerSummary fetches the comment content and truncates
|
|
// it for storage on the task row. Returns an invalid pgtype.Text when
|
|
// the comment is missing (deleted / wrong workspace / etc) so the column
|
|
// stays NULL — front-end falls back to a structural label in that case.
|
|
func (s *TaskService) buildCommentTriggerSummary(ctx context.Context, commentID pgtype.UUID) pgtype.Text {
|
|
if !commentID.Valid {
|
|
return pgtype.Text{}
|
|
}
|
|
comment, err := s.Queries.GetComment(ctx, commentID)
|
|
if err != nil {
|
|
return pgtype.Text{}
|
|
}
|
|
summary := truncateForSummary(comment.Content, triggerSummaryMaxLen)
|
|
if summary == "" {
|
|
return pgtype.Text{}
|
|
}
|
|
return pgtype.Text{String: summary, Valid: true}
|
|
}
|
|
|
|
func NewTaskService(q *db.Queries, tx TxStarter, hub *realtime.Hub, bus *events.Bus, wakeups ...TaskWakeupNotifier) *TaskService {
|
|
var wakeup TaskWakeupNotifier
|
|
if len(wakeups) > 0 {
|
|
wakeup = wakeups[0]
|
|
}
|
|
return &TaskService{Queries: q, TxStarter: tx, Hub: hub, Bus: bus, Wakeup: wakeup}
|
|
}
|
|
|
|
var trivialDoneMarkers = []string{
|
|
"done",
|
|
"готово",
|
|
"готова",
|
|
"сделано",
|
|
"完成",
|
|
"完了",
|
|
}
|
|
|
|
func isTrivialDoneOutput(output string) bool {
|
|
normalized := strings.TrimSpace(strings.ToLower(output))
|
|
normalized = strings.Trim(normalized, ".!!。… ")
|
|
for _, marker := range trivialDoneMarkers {
|
|
if normalized == marker {
|
|
return true
|
|
}
|
|
}
|
|
return false
|
|
}
|
|
|
|
func (s *TaskService) captureTaskQueued(ctx context.Context, task db.AgentTaskQueue) {
|
|
if s.Metrics != nil {
|
|
source, runtimeMode, _ := s.taskMetricsContext(ctx, task)
|
|
s.Metrics.RecordTaskEnqueued(source, runtimeMode)
|
|
}
|
|
s.captureTaskEvent(ctx, analytics.AgentTaskQueued(s.taskAnalyticsContext(ctx, task)))
|
|
}
|
|
|
|
func (s *TaskService) captureTaskDispatched(ctx context.Context, task db.AgentTaskQueue) {
|
|
if s.Metrics != nil {
|
|
source, runtimeMode, _ := s.taskMetricsContext(ctx, task)
|
|
s.Metrics.RecordTaskDispatched(util.UUIDToString(task.ID), source, runtimeMode, taskQueueWaitSeconds(task))
|
|
}
|
|
s.captureTaskEvent(ctx, analytics.AgentTaskDispatched(s.taskAnalyticsContext(ctx, task)))
|
|
}
|
|
|
|
func (s *TaskService) AnalyticsContextForTask(ctx context.Context, task db.AgentTaskQueue) analytics.TaskContext {
|
|
return s.taskAnalyticsContext(ctx, task)
|
|
}
|
|
|
|
func (s *TaskService) captureTaskStarted(ctx context.Context, task db.AgentTaskQueue) {
|
|
if s.Metrics != nil {
|
|
source, runtimeMode, provider := s.taskMetricsContext(ctx, task)
|
|
s.Metrics.RecordTaskStarted(source, runtimeMode, provider)
|
|
}
|
|
s.captureTaskEvent(ctx, analytics.AgentTaskStarted(s.taskAnalyticsContext(ctx, task)))
|
|
}
|
|
|
|
func (s *TaskService) captureTaskCompleted(ctx context.Context, task db.AgentTaskQueue) {
|
|
if s.Metrics != nil {
|
|
source, runtimeMode, _ := s.taskMetricsContext(ctx, task)
|
|
s.Metrics.RecordTaskTerminal(util.UUIDToString(task.ID), source, runtimeMode, task.Status, taskRunSeconds(task), taskTotalSeconds(task), task.Attempt)
|
|
}
|
|
s.captureTaskEvent(ctx, analytics.AgentTaskCompleted(
|
|
s.taskAnalyticsContext(ctx, task),
|
|
taskDurationMS(task),
|
|
))
|
|
}
|
|
|
|
func (s *TaskService) captureTaskFailed(ctx context.Context, task db.AgentTaskQueue) {
|
|
failureReason := taskFailureReason(task)
|
|
if s.Metrics != nil {
|
|
source, runtimeMode, _ := s.taskMetricsContext(ctx, task)
|
|
s.Metrics.RecordTaskTerminal(util.UUIDToString(task.ID), source, runtimeMode, task.Status, taskRunSeconds(task), taskTotalSeconds(task), task.Attempt)
|
|
s.Metrics.RecordTaskFailed(source, runtimeMode, failureReason)
|
|
}
|
|
s.captureTaskEvent(ctx, analytics.AgentTaskFailed(
|
|
s.taskAnalyticsContext(ctx, task),
|
|
taskDurationMS(task),
|
|
failureReason,
|
|
taskErrorType(failureReason),
|
|
s.willRetryTask(task),
|
|
))
|
|
}
|
|
|
|
func (s *TaskService) captureTaskCancelled(ctx context.Context, task db.AgentTaskQueue) {
|
|
if s.Metrics != nil {
|
|
source, runtimeMode, _ := s.taskMetricsContext(ctx, task)
|
|
s.Metrics.RecordTaskTerminal(util.UUIDToString(task.ID), source, runtimeMode, task.Status, taskRunSeconds(task), taskTotalSeconds(task), task.Attempt)
|
|
}
|
|
s.captureTaskEvent(ctx, analytics.AgentTaskCancelled(
|
|
s.taskAnalyticsContext(ctx, task),
|
|
taskDurationMS(task),
|
|
))
|
|
// Revoke any mat_ task tokens minted for this task. Cancellation is
|
|
// a terminal transition, so the running agent process no longer
|
|
// needs to call back; eagerly deleting the token closes the
|
|
// window where a compromised process could keep authenticating
|
|
// against the API until the 24h expiry. Failure is non-fatal — the
|
|
// expiry / FK cascade are the durable guards. MUL-2600.
|
|
if err := s.Queries.DeleteTaskTokensByTask(ctx, task.ID); err != nil {
|
|
slog.Warn("cancel task: failed to revoke task tokens",
|
|
"task_id", util.UUIDToString(task.ID), "error", err)
|
|
}
|
|
}
|
|
|
|
func (s *TaskService) CaptureTaskUsage(ctx context.Context, task db.AgentTaskQueue, provider, model string, inputTokens, outputTokens, cacheReadTokens, cacheWriteTokens int64) {
|
|
if s.Metrics == nil {
|
|
return
|
|
}
|
|
source, runtimeMode, _ := s.taskMetricsContext(ctx, task)
|
|
s.Metrics.RecordLLMUsage(source, runtimeMode, provider, model, inputTokens, outputTokens, cacheReadTokens, cacheWriteTokens)
|
|
}
|
|
|
|
func (s *TaskService) CaptureQueuedExpiredTasks(ctx context.Context, tasks []db.AgentTaskQueue) {
|
|
if s.Metrics == nil {
|
|
return
|
|
}
|
|
for _, task := range tasks {
|
|
source, runtimeMode, _ := s.taskMetricsContext(ctx, task)
|
|
s.Metrics.RecordTaskQueuedExpired(source, runtimeMode)
|
|
}
|
|
}
|
|
|
|
func (s *TaskService) CaptureLeaseExpiredTasks(ctx context.Context, tasks []db.AgentTaskQueue) {
|
|
if s.Metrics == nil {
|
|
return
|
|
}
|
|
for _, task := range tasks {
|
|
source, _, _ := s.taskMetricsContext(ctx, task)
|
|
s.Metrics.RecordTaskLeaseExpired(source)
|
|
}
|
|
}
|
|
|
|
func (s *TaskService) captureTaskEvent(ctx context.Context, event analytics.Event) {
|
|
if s.Analytics == nil {
|
|
return
|
|
}
|
|
if event.WorkspaceID == "" {
|
|
return
|
|
}
|
|
s.Analytics.Capture(event)
|
|
}
|
|
|
|
func (s *TaskService) cachedTaskAnalyticsContext(task db.AgentTaskQueue) (analytics.TaskContext, bool) {
|
|
key := taskAnalyticsContextKey(task)
|
|
if key == "" {
|
|
return analytics.TaskContext{}, false
|
|
}
|
|
s.analyticsContextMu.Lock()
|
|
defer s.analyticsContextMu.Unlock()
|
|
if s.analyticsContextCache == nil {
|
|
return analytics.TaskContext{}, false
|
|
}
|
|
tc, ok := s.analyticsContextCache[key]
|
|
return tc, ok
|
|
}
|
|
|
|
func (s *TaskService) storeTaskAnalyticsContext(task db.AgentTaskQueue, tc analytics.TaskContext) {
|
|
if tc.WorkspaceID == "" {
|
|
return
|
|
}
|
|
key := taskAnalyticsContextKey(task)
|
|
if key == "" {
|
|
return
|
|
}
|
|
s.analyticsContextMu.Lock()
|
|
defer s.analyticsContextMu.Unlock()
|
|
if s.analyticsContextCache == nil {
|
|
s.analyticsContextCache = make(map[string]analytics.TaskContext)
|
|
}
|
|
if _, ok := s.analyticsContextCache[key]; !ok {
|
|
s.analyticsContextOrder = append(s.analyticsContextOrder, key)
|
|
if len(s.analyticsContextOrder) > taskAnalyticsContextCacheMax {
|
|
oldest := s.analyticsContextOrder[0]
|
|
s.analyticsContextOrder = s.analyticsContextOrder[1:]
|
|
delete(s.analyticsContextCache, oldest)
|
|
}
|
|
}
|
|
s.analyticsContextCache[key] = tc
|
|
}
|
|
|
|
func taskAnalyticsContextKey(task db.AgentTaskQueue) string {
|
|
taskID := util.UUIDToString(task.ID)
|
|
if taskID == "" {
|
|
return ""
|
|
}
|
|
return strings.Join([]string{
|
|
taskID,
|
|
util.UUIDToString(task.RuntimeID),
|
|
util.UUIDToString(task.IssueID),
|
|
util.UUIDToString(task.ChatSessionID),
|
|
util.UUIDToString(task.AutopilotRunID),
|
|
}, "|")
|
|
}
|
|
|
|
func (s *TaskService) taskMetricsContext(ctx context.Context, task db.AgentTaskQueue) (source, runtimeMode, provider string) {
|
|
tc := s.taskAnalyticsContext(ctx, task)
|
|
source = "other"
|
|
switch {
|
|
case task.ChatSessionID.Valid:
|
|
source = "chat"
|
|
case task.IssueID.Valid:
|
|
if tc.Source == analytics.SourceAutopilot {
|
|
source = "autopilot_issue"
|
|
} else {
|
|
source = "issue"
|
|
}
|
|
case task.AutopilotRunID.Valid:
|
|
source = "autopilot"
|
|
default:
|
|
if _, ok := s.parseQuickCreateContext(task); ok {
|
|
source = "quick_create"
|
|
} else if tc.Source != "" {
|
|
source = tc.Source
|
|
}
|
|
}
|
|
return source, tc.RuntimeMode, tc.Provider
|
|
}
|
|
|
|
func (s *TaskService) taskAnalyticsContext(ctx context.Context, task db.AgentTaskQueue) analytics.TaskContext {
|
|
if tc, ok := s.cachedTaskAnalyticsContext(task); ok {
|
|
return tc
|
|
}
|
|
tc := analytics.TaskContext{
|
|
AgentID: util.UUIDToString(task.AgentID),
|
|
TaskID: util.UUIDToString(task.ID),
|
|
Source: analytics.SourceManual,
|
|
}
|
|
if task.IssueID.Valid {
|
|
tc.IssueID = util.UUIDToString(task.IssueID)
|
|
}
|
|
if task.ChatSessionID.Valid {
|
|
tc.ChatSessionID = util.UUIDToString(task.ChatSessionID)
|
|
tc.Source = analytics.SourceChat
|
|
}
|
|
if task.AutopilotRunID.Valid {
|
|
tc.AutopilotRunID = util.UUIDToString(task.AutopilotRunID)
|
|
tc.Source = analytics.SourceAutopilot
|
|
}
|
|
|
|
if task.RuntimeID.Valid {
|
|
if rt, err := s.Queries.GetAgentRuntime(ctx, task.RuntimeID); err == nil {
|
|
tc.WorkspaceID = util.UUIDToString(rt.WorkspaceID)
|
|
tc.RuntimeMode = rt.RuntimeMode
|
|
tc.Provider = rt.Provider
|
|
}
|
|
}
|
|
if tc.WorkspaceID == "" || tc.RuntimeMode == "" {
|
|
if agent, err := s.Queries.GetAgent(ctx, task.AgentID); err == nil {
|
|
if tc.WorkspaceID == "" {
|
|
tc.WorkspaceID = util.UUIDToString(agent.WorkspaceID)
|
|
}
|
|
if tc.RuntimeMode == "" {
|
|
tc.RuntimeMode = agent.RuntimeMode
|
|
}
|
|
}
|
|
}
|
|
|
|
if task.IssueID.Valid {
|
|
if issue, err := s.Queries.GetIssue(ctx, task.IssueID); err == nil {
|
|
tc.WorkspaceID = util.UUIDToString(issue.WorkspaceID)
|
|
if issue.CreatorType == "member" {
|
|
tc.UserID = util.UUIDToString(issue.CreatorID)
|
|
}
|
|
if issue.OriginType.Valid {
|
|
switch issue.OriginType.String {
|
|
case "autopilot":
|
|
tc.Source = analytics.SourceAutopilot
|
|
if ap, err := s.Queries.GetAutopilot(ctx, issue.OriginID); err == nil {
|
|
if ap.CreatedByType == "member" {
|
|
tc.UserID = util.UUIDToString(ap.CreatedByID)
|
|
}
|
|
}
|
|
case "quick_create":
|
|
tc.Source = analytics.SourceManual
|
|
}
|
|
}
|
|
}
|
|
}
|
|
if task.ChatSessionID.Valid {
|
|
if cs, err := s.Queries.GetChatSession(ctx, task.ChatSessionID); err == nil {
|
|
tc.WorkspaceID = util.UUIDToString(cs.WorkspaceID)
|
|
tc.UserID = util.UUIDToString(cs.CreatorID)
|
|
}
|
|
}
|
|
if task.AutopilotRunID.Valid {
|
|
if run, err := s.Queries.GetAutopilotRun(ctx, task.AutopilotRunID); err == nil {
|
|
if ap, err := s.Queries.GetAutopilot(ctx, run.AutopilotID); err == nil {
|
|
tc.WorkspaceID = util.UUIDToString(ap.WorkspaceID)
|
|
if ap.CreatedByType == "member" {
|
|
tc.UserID = util.UUIDToString(ap.CreatedByID)
|
|
}
|
|
}
|
|
}
|
|
}
|
|
if qc, ok := s.parseQuickCreateContext(task); ok {
|
|
tc.WorkspaceID = qc.WorkspaceID
|
|
tc.UserID = qc.RequesterID
|
|
tc.Source = analytics.SourceManual
|
|
}
|
|
s.storeTaskAnalyticsContext(task, tc)
|
|
return tc
|
|
}
|
|
|
|
func taskDurationMS(task db.AgentTaskQueue) int64 {
|
|
if !task.CompletedAt.Valid {
|
|
return 0
|
|
}
|
|
start := task.CreatedAt
|
|
if task.StartedAt.Valid {
|
|
start = task.StartedAt
|
|
} else if task.DispatchedAt.Valid {
|
|
start = task.DispatchedAt
|
|
}
|
|
if !start.Valid {
|
|
return 0
|
|
}
|
|
ms := task.CompletedAt.Time.Sub(start.Time).Milliseconds()
|
|
if ms < 0 {
|
|
return 0
|
|
}
|
|
return ms
|
|
}
|
|
|
|
func taskQueueWaitSeconds(task db.AgentTaskQueue) float64 {
|
|
return durationSeconds(task.CreatedAt, task.DispatchedAt)
|
|
}
|
|
|
|
func taskRunSeconds(task db.AgentTaskQueue) float64 {
|
|
return durationSeconds(task.StartedAt, task.CompletedAt)
|
|
}
|
|
|
|
func taskTotalSeconds(task db.AgentTaskQueue) float64 {
|
|
return durationSeconds(task.CreatedAt, task.CompletedAt)
|
|
}
|
|
|
|
func durationSeconds(start, end pgtype.Timestamptz) float64 {
|
|
if !start.Valid || !end.Valid {
|
|
return -1
|
|
}
|
|
seconds := end.Time.Sub(start.Time).Seconds()
|
|
if seconds < 0 {
|
|
return 0
|
|
}
|
|
return seconds
|
|
}
|
|
|
|
func taskFailureReason(task db.AgentTaskQueue) string {
|
|
if task.FailureReason.Valid && task.FailureReason.String != "" {
|
|
return task.FailureReason.String
|
|
}
|
|
return "agent_error"
|
|
}
|
|
|
|
func taskErrorType(reason string) string {
|
|
switch reason {
|
|
case "runtime_offline", "runtime_recovery":
|
|
return "runtime"
|
|
case "timeout", "codex_semantic_inactivity":
|
|
return "timeout"
|
|
case "iteration_limit", "agent_fallback_message":
|
|
return "agent_output"
|
|
case "cancelled", "user_cancelled":
|
|
return "cancelled"
|
|
default:
|
|
return "agent_error"
|
|
}
|
|
}
|
|
|
|
func (s *TaskService) willRetryTask(task db.AgentTaskQueue) bool {
|
|
reason := taskFailureReason(task)
|
|
if !retryableReasons[reason] {
|
|
return false
|
|
}
|
|
if task.Attempt >= task.MaxAttempts {
|
|
return false
|
|
}
|
|
if task.AutopilotRunID.Valid {
|
|
return false
|
|
}
|
|
return task.IssueID.Valid || task.ChatSessionID.Valid
|
|
}
|
|
|
|
// EnqueueTaskForIssue creates a queued task for an agent-assigned issue.
|
|
// No context snapshot is stored — the agent fetches all data it needs at
|
|
// runtime via the multica CLI.
|
|
func (s *TaskService) EnqueueTaskForIssue(ctx context.Context, issue db.Issue, triggerCommentID ...pgtype.UUID) (db.AgentTaskQueue, error) {
|
|
var commentID pgtype.UUID
|
|
if len(triggerCommentID) > 0 {
|
|
commentID = triggerCommentID[0]
|
|
}
|
|
return s.enqueueIssueTask(ctx, issue, commentID, false)
|
|
}
|
|
|
|
// enqueueIssueTask is the shared implementation behind EnqueueTaskForIssue
|
|
// and the manual rerun path. forceFreshSession=true marks the task so the
|
|
// daemon claim handler skips the (agent_id, issue_id) resume lookup — the
|
|
// user already judged the prior output bad, a fresh agent session is the
|
|
// expected behavior.
|
|
func (s *TaskService) enqueueIssueTask(ctx context.Context, issue db.Issue, triggerCommentID pgtype.UUID, forceFreshSession bool) (db.AgentTaskQueue, error) {
|
|
if !issue.AssigneeID.Valid {
|
|
slog.Error("task enqueue failed", "issue_id", util.UUIDToString(issue.ID), "error", "issue has no assignee")
|
|
return db.AgentTaskQueue{}, fmt.Errorf("issue has no assignee")
|
|
}
|
|
|
|
agent, err := s.Queries.GetAgent(ctx, issue.AssigneeID)
|
|
if err != nil {
|
|
slog.Error("task enqueue failed", "issue_id", util.UUIDToString(issue.ID), "error", err)
|
|
return db.AgentTaskQueue{}, fmt.Errorf("load agent: %w", err)
|
|
}
|
|
if agent.ArchivedAt.Valid {
|
|
slog.Debug("task enqueue skipped: agent is archived", "issue_id", util.UUIDToString(issue.ID), "agent_id", util.UUIDToString(agent.ID))
|
|
return db.AgentTaskQueue{}, fmt.Errorf("agent is archived")
|
|
}
|
|
if !agent.RuntimeID.Valid {
|
|
slog.Error("task enqueue failed", "issue_id", util.UUIDToString(issue.ID), "error", "agent has no runtime")
|
|
return db.AgentTaskQueue{}, fmt.Errorf("agent has no runtime")
|
|
}
|
|
|
|
task, err := s.Queries.CreateAgentTask(ctx, db.CreateAgentTaskParams{
|
|
AgentID: issue.AssigneeID,
|
|
RuntimeID: agent.RuntimeID,
|
|
IssueID: issue.ID,
|
|
Priority: priorityToInt(issue.Priority),
|
|
TriggerCommentID: triggerCommentID,
|
|
TriggerSummary: s.buildCommentTriggerSummary(ctx, triggerCommentID),
|
|
ForceFreshSession: pgtype.Bool{Bool: forceFreshSession, Valid: forceFreshSession},
|
|
})
|
|
if err != nil {
|
|
slog.Error("task enqueue failed", "issue_id", util.UUIDToString(issue.ID), "error", err)
|
|
return db.AgentTaskQueue{}, fmt.Errorf("create task: %w", err)
|
|
}
|
|
|
|
slog.Info("task enqueued",
|
|
"task_id", util.UUIDToString(task.ID),
|
|
"issue_id", util.UUIDToString(issue.ID),
|
|
"agent_id", util.UUIDToString(issue.AssigneeID),
|
|
"force_fresh_session", forceFreshSession,
|
|
)
|
|
// Order matters: broadcast first, notify daemon second. notifyTaskAvailable
|
|
// kicks an in-process channel that the daemon picks up over HTTP and
|
|
// claims; the claim path then emits its own task:dispatch. Doing the
|
|
// queued broadcast afterwards risks the dispatch event reaching clients
|
|
// before the queued one (rare but unsafe-by-construction). Publishing
|
|
// in the desired observe-order makes correctness independent of timing.
|
|
s.broadcastTaskEvent(ctx, protocol.EventTaskQueued, task)
|
|
s.NotifyTaskEnqueued(ctx, task)
|
|
return task, nil
|
|
}
|
|
|
|
// EnqueueTaskForMention creates a queued task for a mentioned agent on an issue.
|
|
// Unlike EnqueueTaskForIssue, this takes an explicit agent ID rather than
|
|
// deriving it from the issue assignee.
|
|
func (s *TaskService) EnqueueTaskForMention(ctx context.Context, issue db.Issue, agentID pgtype.UUID, triggerCommentID pgtype.UUID) (db.AgentTaskQueue, error) {
|
|
return s.enqueueMentionTask(ctx, issue, agentID, triggerCommentID, false, false)
|
|
}
|
|
|
|
// EnqueueTaskForSquadLeader is the leader-role variant of EnqueueTaskForMention.
|
|
// The resulting task carries is_leader_task=true so that downstream
|
|
// self-trigger guards can distinguish a comment posted while the agent was
|
|
// acting as the squad's leader (skip) from one posted while it was acting
|
|
// as a worker (do not skip). This matters for agents that are simultaneously
|
|
// the leader and a worker of the same squad — see migration 090.
|
|
func (s *TaskService) EnqueueTaskForSquadLeader(ctx context.Context, issue db.Issue, leaderID pgtype.UUID, triggerCommentID pgtype.UUID) (db.AgentTaskQueue, error) {
|
|
return s.enqueueMentionTask(ctx, issue, leaderID, triggerCommentID, true, false)
|
|
}
|
|
|
|
func (s *TaskService) enqueueMentionTask(ctx context.Context, issue db.Issue, agentID pgtype.UUID, triggerCommentID pgtype.UUID, isLeader bool, forceFreshSession bool) (db.AgentTaskQueue, error) {
|
|
agent, err := s.Queries.GetAgent(ctx, agentID)
|
|
if err != nil {
|
|
slog.Error("mention task enqueue failed: agent not found", "issue_id", util.UUIDToString(issue.ID), "agent_id", util.UUIDToString(agentID), "error", err)
|
|
return db.AgentTaskQueue{}, fmt.Errorf("load agent: %w", err)
|
|
}
|
|
if agent.ArchivedAt.Valid {
|
|
slog.Debug("mention task enqueue skipped: agent is archived", "issue_id", util.UUIDToString(issue.ID), "agent_id", util.UUIDToString(agentID))
|
|
return db.AgentTaskQueue{}, fmt.Errorf("agent is archived")
|
|
}
|
|
if !agent.RuntimeID.Valid {
|
|
slog.Error("mention task enqueue failed: agent has no runtime", "issue_id", util.UUIDToString(issue.ID), "agent_id", util.UUIDToString(agentID))
|
|
return db.AgentTaskQueue{}, fmt.Errorf("agent has no runtime")
|
|
}
|
|
|
|
task, err := s.Queries.CreateAgentTask(ctx, db.CreateAgentTaskParams{
|
|
AgentID: agentID,
|
|
RuntimeID: agent.RuntimeID,
|
|
IssueID: issue.ID,
|
|
Priority: priorityToInt(issue.Priority),
|
|
TriggerCommentID: triggerCommentID,
|
|
TriggerSummary: s.buildCommentTriggerSummary(ctx, triggerCommentID),
|
|
IsLeaderTask: pgtype.Bool{Bool: isLeader, Valid: isLeader},
|
|
ForceFreshSession: pgtype.Bool{Bool: forceFreshSession, Valid: forceFreshSession},
|
|
})
|
|
if err != nil {
|
|
slog.Error("mention task enqueue failed", "issue_id", util.UUIDToString(issue.ID), "agent_id", util.UUIDToString(agentID), "error", err)
|
|
return db.AgentTaskQueue{}, fmt.Errorf("create task: %w", err)
|
|
}
|
|
|
|
slog.Info("mention task enqueued", "task_id", util.UUIDToString(task.ID), "issue_id", util.UUIDToString(issue.ID), "agent_id", util.UUIDToString(agentID), "is_leader_task", isLeader)
|
|
// See EnqueueTaskForIssue for ordering rationale.
|
|
s.broadcastTaskEvent(ctx, protocol.EventTaskQueued, task)
|
|
s.NotifyTaskEnqueued(ctx, task)
|
|
return task, nil
|
|
}
|
|
|
|
// QuickCreateContext is the JSON payload stored on a quick-create task's
|
|
// context column. The daemon detects this variant via Type == "quick_create"
|
|
// and switches to the quick-create prompt template; the completion path
|
|
// uses RequesterID + WorkspaceID to write the inbox notification.
|
|
//
|
|
// ProjectID is the optional project the user picked in the modal. When
|
|
// non-empty the daemon claim handler resolves the project's title +
|
|
// resources, and the prompt template instructs the agent to pass
|
|
// `--project <uuid>` so the new issue lands in that project.
|
|
//
|
|
// SquadID is non-empty when the user picked a squad (rather than an agent)
|
|
// in the modal. The task is still enqueued against the squad's leader
|
|
// agent (Queries.CreateQuickCreateTask is agent-scoped); SquadID is the
|
|
// hint the daemon claim handler uses to layer the squad-leader briefing
|
|
// onto the agent's Instructions, matching the behavior of issue-bound
|
|
// tasks assigned to the squad.
|
|
type QuickCreateContext struct {
|
|
Type string `json:"type"`
|
|
Prompt string `json:"prompt"`
|
|
RequesterID string `json:"requester_id"`
|
|
WorkspaceID string `json:"workspace_id"`
|
|
ProjectID string `json:"project_id,omitempty"`
|
|
SquadID string `json:"squad_id,omitempty"`
|
|
// ParentIssueID is the optional UUID of the parent issue the new issue
|
|
// should be filed under. Set when the user opens the modal from "Add
|
|
// sub issue" on an existing issue; the daemon claim handler resolves the
|
|
// parent's identifier and the prompt template instructs the agent to
|
|
// pass `--parent <uuid>` so the sub-issue relationship is preserved
|
|
// across the manual→agent mode flip.
|
|
ParentIssueID string `json:"parent_issue_id,omitempty"`
|
|
}
|
|
|
|
// QuickCreateContextType marks a task as a quick-create job.
|
|
const QuickCreateContextType = "quick_create"
|
|
|
|
// EnqueueQuickCreateTask creates a queued task that has no issue / chat /
|
|
// autopilot link — the user's natural-language prompt is stored in the
|
|
// task's context JSONB and the agent is expected to translate it into a
|
|
// `multica issue create` call. Pre-validates that the agent is reachable
|
|
// (not archived, has a runtime) so the API can reject up-front rather than
|
|
// queue a task no one will ever claim.
|
|
//
|
|
// projectID is optional (zero-valued pgtype.UUID when the user didn't pick
|
|
// one). The handler is responsible for validating it belongs to the same
|
|
// workspace before passing it in.
|
|
//
|
|
// squadID is non-empty (Valid) when the user picked a squad as the actor.
|
|
// The handler has already resolved it to the squad's leader agent for
|
|
// agentID; the squadID hint is stamped into the task context so the daemon
|
|
// claim handler can inject the squad-leader briefing on dispatch.
|
|
//
|
|
// parentIssueID is optional (zero-valued pgtype.UUID when the user didn't
|
|
// open the modal from "Add sub issue"). The handler is responsible for
|
|
// validating it belongs to the same workspace before passing it in.
|
|
func (s *TaskService) EnqueueQuickCreateTask(ctx context.Context, workspaceID, requesterID pgtype.UUID, agentID, squadID pgtype.UUID, prompt string, projectID, parentIssueID pgtype.UUID) (db.AgentTaskQueue, error) {
|
|
agent, err := s.Queries.GetAgent(ctx, agentID)
|
|
if err != nil {
|
|
return db.AgentTaskQueue{}, fmt.Errorf("load agent: %w", err)
|
|
}
|
|
if agent.ArchivedAt.Valid {
|
|
return db.AgentTaskQueue{}, fmt.Errorf("agent is archived")
|
|
}
|
|
if !agent.RuntimeID.Valid {
|
|
return db.AgentTaskQueue{}, fmt.Errorf("agent has no runtime")
|
|
}
|
|
|
|
payload := QuickCreateContext{
|
|
Type: QuickCreateContextType,
|
|
Prompt: prompt,
|
|
RequesterID: util.UUIDToString(requesterID),
|
|
WorkspaceID: util.UUIDToString(workspaceID),
|
|
}
|
|
if projectID.Valid {
|
|
payload.ProjectID = util.UUIDToString(projectID)
|
|
}
|
|
if squadID.Valid {
|
|
payload.SquadID = util.UUIDToString(squadID)
|
|
}
|
|
if parentIssueID.Valid {
|
|
payload.ParentIssueID = util.UUIDToString(parentIssueID)
|
|
}
|
|
contextJSON, err := json.Marshal(payload)
|
|
if err != nil {
|
|
return db.AgentTaskQueue{}, fmt.Errorf("marshal quick-create context: %w", err)
|
|
}
|
|
|
|
task, err := s.Queries.CreateQuickCreateTask(ctx, db.CreateQuickCreateTaskParams{
|
|
AgentID: agentID,
|
|
RuntimeID: agent.RuntimeID,
|
|
Priority: priorityToInt("high"),
|
|
Context: contextJSON,
|
|
})
|
|
if err != nil {
|
|
return db.AgentTaskQueue{}, fmt.Errorf("create quick-create task: %w", err)
|
|
}
|
|
|
|
slog.Info("quick-create task enqueued",
|
|
"task_id", util.UUIDToString(task.ID),
|
|
"agent_id", util.UUIDToString(agentID),
|
|
"squad_id", payload.SquadID,
|
|
"requester_id", util.UUIDToString(requesterID),
|
|
"workspace_id", util.UUIDToString(workspaceID),
|
|
"project_id", payload.ProjectID,
|
|
"parent_issue_id", payload.ParentIssueID,
|
|
)
|
|
// Match every other Enqueue* path: kick the daemon WS so the task
|
|
// gets claimed promptly instead of waiting for the next 30 s poll
|
|
// cycle. Without this the user perceives "quick create never
|
|
// triggered" because the modal closes immediately and the task
|
|
// sits in 'queued' until the next sleepWithContextOrWakeup tick.
|
|
s.NotifyTaskEnqueued(ctx, task)
|
|
return task, nil
|
|
}
|
|
|
|
// ErrChatTaskAgentArchived signals that EnqueueChatTask refused to
|
|
// queue work because the destination agent has been archived. This
|
|
// is a productizable state — surface it to the user as "this agent
|
|
// has been archived" rather than retrying.
|
|
var ErrChatTaskAgentArchived = errors.New("chat task: agent archived")
|
|
|
|
// ErrChatTaskAgentNoRuntime signals that EnqueueChatTask refused to
|
|
// queue work because the agent has never been associated with a
|
|
// runtime (agent.runtime_id IS NULL). This is the "agent has no
|
|
// daemon configured" case — productizable as "agent offline".
|
|
//
|
|
// IMPORTANT: this is NOT the same as "the daemon is currently
|
|
// disconnected". When agent.runtime_id IS set, EnqueueChatTask
|
|
// enqueues the task and the daemon claims it on next online; that
|
|
// path returns a task row, not this error.
|
|
var ErrChatTaskAgentNoRuntime = errors.New("chat task: agent has no runtime")
|
|
|
|
// EnqueueChatTask creates a queued task for a chat session.
|
|
// Unlike issue tasks, chat tasks have no issue_id.
|
|
//
|
|
// Errors split into two layers:
|
|
//
|
|
// - Productizable rejections (agent archived, no runtime) return
|
|
// the sentinel errors above. Callers (e.g. the Lark dispatcher)
|
|
// can errors.Is them to decide a user-visible outcome.
|
|
//
|
|
// - Infrastructure failures (DB load / insert errors) are wrapped
|
|
// as ordinary errors. The caller should treat them as retryable
|
|
// or page-worthy, NOT as user-facing state.
|
|
func (s *TaskService) EnqueueChatTask(ctx context.Context, chatSession db.ChatSession) (db.AgentTaskQueue, error) {
|
|
agent, err := s.Queries.GetAgent(ctx, chatSession.AgentID)
|
|
if err != nil {
|
|
slog.Error("chat task enqueue failed", "chat_session_id", util.UUIDToString(chatSession.ID), "error", err)
|
|
return db.AgentTaskQueue{}, fmt.Errorf("load agent: %w", err)
|
|
}
|
|
if agent.ArchivedAt.Valid {
|
|
return db.AgentTaskQueue{}, ErrChatTaskAgentArchived
|
|
}
|
|
if !agent.RuntimeID.Valid {
|
|
return db.AgentTaskQueue{}, ErrChatTaskAgentNoRuntime
|
|
}
|
|
|
|
task, err := s.Queries.CreateChatTask(ctx, db.CreateChatTaskParams{
|
|
AgentID: chatSession.AgentID,
|
|
RuntimeID: agent.RuntimeID,
|
|
Priority: 2, // medium priority for chat
|
|
ChatSessionID: chatSession.ID,
|
|
})
|
|
if err != nil {
|
|
slog.Error("chat task enqueue failed", "chat_session_id", util.UUIDToString(chatSession.ID), "error", err)
|
|
return db.AgentTaskQueue{}, fmt.Errorf("create chat task: %w", err)
|
|
}
|
|
|
|
slog.Info("chat task enqueued", "task_id", util.UUIDToString(task.ID), "chat_session_id", util.UUIDToString(chatSession.ID), "agent_id", util.UUIDToString(chatSession.AgentID))
|
|
// See EnqueueTaskForIssue for ordering rationale.
|
|
s.broadcastTaskEvent(ctx, protocol.EventTaskQueued, task)
|
|
s.NotifyTaskEnqueued(ctx, task)
|
|
return task, nil
|
|
}
|
|
|
|
// CancelTasksForIssue cancels every active task on the issue, reconciles each
|
|
// affected agent's status, and broadcasts task:cancelled events so frontends
|
|
// clear their live cards.
|
|
//
|
|
// Before #1587 this path was "cancel rows and return" — issue-status flips
|
|
// (e.g. user marks the issue `done` or `cancelled` while a task is still
|
|
// running) left the agent stuck at status="working" indefinitely, requiring a
|
|
// manual `multica agent update <id> --status idle` to unwedge. Matches the
|
|
// pattern already used by CancelTask and RerunIssue.
|
|
func (s *TaskService) CancelTasksForIssue(ctx context.Context, issueID pgtype.UUID) error {
|
|
cancelled, err := s.Queries.CancelAgentTasksByIssue(ctx, issueID)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
for _, t := range cancelled {
|
|
s.captureTaskCancelled(ctx, t)
|
|
s.ReconcileAgentStatus(ctx, t.AgentID)
|
|
s.broadcastTaskEvent(ctx, protocol.EventTaskCancelled, t)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// CancelTasksForAgent cancels every active task belonging to an agent
|
|
// (queued + dispatched + running), reconciles the agent's status, and
|
|
// broadcasts task:cancelled events. Used by the agent-level "Cancel all
|
|
// tasks" action — same shape as CancelTasksForIssue but scoped on agent_id.
|
|
//
|
|
// Returns the cancelled rows so callers can report counts / log them.
|
|
func (s *TaskService) CancelTasksForAgent(ctx context.Context, agentID pgtype.UUID) ([]db.AgentTaskQueue, error) {
|
|
cancelled, err := s.Queries.CancelAgentTasksByAgent(ctx, agentID)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
for _, t := range cancelled {
|
|
s.captureTaskCancelled(ctx, t)
|
|
s.broadcastTaskEvent(ctx, protocol.EventTaskCancelled, t)
|
|
}
|
|
// Reconcile once after the loop — agent transitions from
|
|
// working→available based on remaining task counts, no need to call
|
|
// per row (the rows we just cancelled all belong to the same agent).
|
|
s.ReconcileAgentStatus(ctx, agentID)
|
|
return cancelled, nil
|
|
}
|
|
|
|
// CancelTasksByTriggerComment cancels active tasks whose trigger is the given
|
|
// comment. Called from DeleteComment so an agent does not run with the
|
|
// now-deleted content already embedded in its prompt. Must be invoked BEFORE
|
|
// the comment row is deleted because the FK ON DELETE SET NULL would
|
|
// otherwise nullify trigger_comment_id and we'd lose the ability to find
|
|
// the affected tasks.
|
|
func (s *TaskService) CancelTasksByTriggerComment(ctx context.Context, commentID pgtype.UUID) error {
|
|
cancelled, err := s.Queries.CancelAgentTasksByTriggerComment(ctx, commentID)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
for _, t := range cancelled {
|
|
s.captureTaskCancelled(ctx, t)
|
|
s.ReconcileAgentStatus(ctx, t.AgentID)
|
|
s.broadcastTaskEvent(ctx, protocol.EventTaskCancelled, t)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// BroadcastCancelledTasks reconciles each affected agent's status and emits
|
|
// task:cancelled for every row. Callers must invoke this AFTER committing the
|
|
// cancellation so subscribers don't observe a "cancelled" event for a row
|
|
// that the tx might still roll back.
|
|
func (s *TaskService) BroadcastCancelledTasks(ctx context.Context, cancelled []db.AgentTaskQueue) {
|
|
for _, t := range cancelled {
|
|
s.captureTaskCancelled(ctx, t)
|
|
s.ReconcileAgentStatus(ctx, t.AgentID)
|
|
s.broadcastTaskEvent(ctx, protocol.EventTaskCancelled, t)
|
|
}
|
|
}
|
|
|
|
func (s *TaskService) CaptureCancelledTasks(ctx context.Context, cancelled []db.AgentTaskQueue) {
|
|
for _, t := range cancelled {
|
|
s.captureTaskCancelled(ctx, t)
|
|
}
|
|
}
|
|
|
|
// CancelTask cancels a single task by ID. It broadcasts a task:cancelled event
|
|
// so frontends can update immediately.
|
|
func (s *TaskService) CancelTask(ctx context.Context, taskID pgtype.UUID) (*db.AgentTaskQueue, error) {
|
|
task, err := s.Queries.CancelAgentTask(ctx, taskID)
|
|
if errors.Is(err, pgx.ErrNoRows) {
|
|
existing, err := s.Queries.GetAgentTask(ctx, taskID)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("cancel task: %w", err)
|
|
}
|
|
return &existing, nil
|
|
}
|
|
if err != nil {
|
|
return nil, fmt.Errorf("cancel task: %w", err)
|
|
}
|
|
|
|
slog.Info("task cancelled", "task_id", util.UUIDToString(task.ID), "issue_id", util.UUIDToString(task.IssueID))
|
|
s.captureTaskCancelled(ctx, task)
|
|
|
|
// Reconcile agent status
|
|
s.ReconcileAgentStatus(ctx, task.AgentID)
|
|
|
|
// Broadcast cancellation as a task:failed event so frontends clear the live card
|
|
s.broadcastTaskEvent(ctx, protocol.EventTaskCancelled, task)
|
|
|
|
return &task, nil
|
|
}
|
|
|
|
// ClaimTask atomically claims the next queued task for an agent,
|
|
// respecting max_concurrent_tasks.
|
|
func (s *TaskService) ClaimTask(ctx context.Context, agentID pgtype.UUID) (*db.AgentTaskQueue, error) {
|
|
start := time.Now()
|
|
var (
|
|
outcome = "unknown"
|
|
getAgentMs, countRunningMs, claimAgentMs, updateStatusMs, dispatchMs int64
|
|
)
|
|
defer func() {
|
|
s.maybeLogClaimSlow(agentID, outcome, start, getAgentMs, countRunningMs, claimAgentMs, updateStatusMs, dispatchMs)
|
|
}()
|
|
|
|
t0 := start
|
|
agent, err := s.Queries.GetAgent(ctx, agentID)
|
|
getAgentMs = time.Since(t0).Milliseconds()
|
|
if err != nil {
|
|
outcome = "error_get_agent"
|
|
return nil, fmt.Errorf("agent not found: %w", err)
|
|
}
|
|
|
|
t0 = time.Now()
|
|
running, err := s.Queries.CountRunningTasks(ctx, agentID)
|
|
countRunningMs = time.Since(t0).Milliseconds()
|
|
if err != nil {
|
|
outcome = "error_count_running"
|
|
return nil, fmt.Errorf("count running tasks: %w", err)
|
|
}
|
|
if running >= int64(agent.MaxConcurrentTasks) {
|
|
slog.Debug("task claim: no capacity", "agent_id", util.UUIDToString(agentID), "running", running, "max", agent.MaxConcurrentTasks)
|
|
outcome = "no_capacity"
|
|
return nil, nil // No capacity
|
|
}
|
|
|
|
t0 = time.Now()
|
|
task, err := s.Queries.ClaimAgentTask(ctx, agentID)
|
|
claimAgentMs = time.Since(t0).Milliseconds()
|
|
if err != nil {
|
|
if errors.Is(err, pgx.ErrNoRows) {
|
|
slog.Debug("task claim: no tasks available", "agent_id", util.UUIDToString(agentID))
|
|
outcome = "no_tasks"
|
|
return nil, nil // No tasks available
|
|
}
|
|
outcome = "error_claim"
|
|
return nil, fmt.Errorf("claim task: %w", err)
|
|
}
|
|
|
|
slog.Info("task claimed", "task_id", util.UUIDToString(task.ID), "agent_id", util.UUIDToString(agentID))
|
|
s.captureTaskDispatched(ctx, task)
|
|
|
|
// Refresh agent status from active tasks. This avoids a stale unconditional
|
|
// working write racing after a just-cancelled claim.
|
|
t0 = time.Now()
|
|
s.ReconcileAgentStatus(ctx, agentID)
|
|
updateStatusMs = time.Since(t0).Milliseconds()
|
|
|
|
// Broadcast task:dispatch. ResolveTaskWorkspaceID inside this path can
|
|
// re-query issue/chat_session/autopilot_run, so it can also be a real
|
|
// contributor to claim latency.
|
|
t0 = time.Now()
|
|
s.broadcastTaskDispatch(ctx, task)
|
|
dispatchMs = time.Since(t0).Milliseconds()
|
|
|
|
outcome = "claimed"
|
|
return &task, nil
|
|
}
|
|
|
|
// ClaimTaskForRuntime claims the next runnable task for a runtime while
|
|
// still respecting each agent's max_concurrent_tasks limit.
|
|
//
|
|
// Empty-claim fast path: when EmptyClaim is configured and a recent
|
|
// check verified the runtime had no queued tasks, returns immediately
|
|
// without touching Postgres. The cache is invalidated synchronously on
|
|
// every enqueue (notifyTaskAvailable), so a queued task becomes
|
|
// claimable on the next call rather than waiting for the TTL.
|
|
func (s *TaskService) ClaimTaskForRuntime(ctx context.Context, runtimeID pgtype.UUID) (*db.AgentTaskQueue, error) {
|
|
start := time.Now()
|
|
var (
|
|
outcome = "no_task"
|
|
listMs, loopMs int64
|
|
listCount, tried int
|
|
claimedFlag bool
|
|
)
|
|
defer func() {
|
|
totalMs := time.Since(start).Milliseconds()
|
|
if totalMs < 300 {
|
|
return
|
|
}
|
|
slog.Info("claim_for_runtime slow",
|
|
"runtime_id", util.UUIDToString(runtimeID),
|
|
"outcome", outcome,
|
|
"total_ms", totalMs,
|
|
"list_pending_ms", listMs,
|
|
"list_pending_count", listCount,
|
|
"agents_tried", tried,
|
|
"claim_loop_ms", loopMs,
|
|
"claimed", claimedFlag,
|
|
)
|
|
}()
|
|
|
|
runtimeKey := util.UUIDToString(runtimeID)
|
|
// Check this before EmptyClaim: a lost claim response moves the task out of
|
|
// `queued`, so the empty-queued cache cannot represent recoverability.
|
|
stale, err := s.Queries.ReclaimStaleDispatchedTaskForRuntime(ctx, db.ReclaimStaleDispatchedTaskForRuntimeParams{
|
|
RuntimeID: runtimeID,
|
|
ClaimRecoverySecs: claimResponseRecoveryWindow.Seconds(),
|
|
})
|
|
if err == nil {
|
|
outcome = "reclaimed_dispatched"
|
|
claimedFlag = true
|
|
slog.Info("stale dispatched task reclaimed",
|
|
"task_id", util.UUIDToString(stale.ID),
|
|
"runtime_id", runtimeKey,
|
|
"agent_id", util.UUIDToString(stale.AgentID),
|
|
)
|
|
return &stale, nil
|
|
}
|
|
if !errors.Is(err, pgx.ErrNoRows) {
|
|
outcome = "error_reclaim_dispatched"
|
|
return nil, fmt.Errorf("reclaim stale dispatched task: %w", err)
|
|
}
|
|
|
|
if s.EmptyClaim.IsEmpty(ctx, runtimeKey) {
|
|
outcome = "empty_cache_hit"
|
|
return nil, nil
|
|
}
|
|
|
|
// Sample the invalidation version BEFORE the SELECT. If a
|
|
// concurrent enqueue Bumps between this read and the post-SELECT
|
|
// MarkEmpty, the next IsEmpty will see the empty key tagged with
|
|
// a stale version and reject it — closing the race that would
|
|
// otherwise stall the just-queued task until the empty key's TTL
|
|
// expired.
|
|
preSelectVersion := s.EmptyClaim.CurrentVersion(ctx, runtimeKey)
|
|
|
|
t0 := time.Now()
|
|
tasks, err := s.Queries.ListQueuedClaimCandidatesByRuntime(ctx, runtimeID)
|
|
listMs = time.Since(t0).Milliseconds()
|
|
listCount = len(tasks)
|
|
if err != nil {
|
|
outcome = "error_list"
|
|
return nil, fmt.Errorf("list queued claim candidates: %w", err)
|
|
}
|
|
|
|
if len(tasks) == 0 {
|
|
s.EmptyClaim.MarkEmpty(ctx, runtimeKey, preSelectVersion)
|
|
outcome = "empty_db"
|
|
return nil, nil
|
|
}
|
|
|
|
loopStart := time.Now()
|
|
triedAgents := map[string]struct{}{}
|
|
var claimed *db.AgentTaskQueue
|
|
for _, candidate := range tasks {
|
|
agentKey := util.UUIDToString(candidate.AgentID)
|
|
if _, seen := triedAgents[agentKey]; seen {
|
|
continue
|
|
}
|
|
triedAgents[agentKey] = struct{}{}
|
|
tried++
|
|
|
|
task, err := s.ClaimTask(ctx, candidate.AgentID)
|
|
if err != nil {
|
|
loopMs = time.Since(loopStart).Milliseconds()
|
|
outcome = "error_claim"
|
|
return nil, err
|
|
}
|
|
if task != nil && task.RuntimeID == runtimeID {
|
|
claimed = task
|
|
break
|
|
}
|
|
}
|
|
loopMs = time.Since(loopStart).Milliseconds()
|
|
if claimed != nil {
|
|
claimedFlag = true
|
|
outcome = "claimed"
|
|
}
|
|
|
|
return claimed, nil
|
|
}
|
|
|
|
// maybeLogClaimSlow emits one structured log per ClaimTask call when its total
|
|
// latency exceeds 300ms, so the prod tail can be diagnosed without flooding
|
|
// logs at normal poll rates. Called via defer so it captures the full path
|
|
// including post-claim updateAgentStatus / broadcastTaskDispatch (both of
|
|
// which can hit the DB) and any error exit.
|
|
func (s *TaskService) maybeLogClaimSlow(agentID pgtype.UUID, outcome string, start time.Time, getAgentMs, countRunningMs, claimAgentMs, updateStatusMs, dispatchMs int64) {
|
|
totalMs := time.Since(start).Milliseconds()
|
|
if totalMs < 300 {
|
|
return
|
|
}
|
|
slog.Info("claim_task slow",
|
|
"agent_id", util.UUIDToString(agentID),
|
|
"outcome", outcome,
|
|
"total_ms", totalMs,
|
|
"get_agent_ms", getAgentMs,
|
|
"count_running_ms", countRunningMs,
|
|
"claim_agent_ms", claimAgentMs,
|
|
"update_status_ms", updateStatusMs,
|
|
"dispatch_ms", dispatchMs,
|
|
)
|
|
}
|
|
|
|
// StartTask transitions a dispatched task to running.
|
|
// Issue status is NOT changed here — the agent manages it via the CLI.
|
|
func (s *TaskService) StartTask(ctx context.Context, taskID pgtype.UUID) (*db.AgentTaskQueue, error) {
|
|
task, err := s.Queries.StartAgentTask(ctx, taskID)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("start task: %w", err)
|
|
}
|
|
|
|
slog.Info("task started", "task_id", util.UUIDToString(task.ID), "issue_id", util.UUIDToString(task.IssueID))
|
|
s.captureTaskStarted(ctx, task)
|
|
// Tell every connected workspace WS client that this task transitioned
|
|
// (dispatched | waiting_local_directory) → running. Without this, the
|
|
// workspace-wide `agentTaskSnapshot` query only refreshes on the 30s
|
|
// staleTime, so any UI that distinguishes "queued" from "running" (e.g.
|
|
// the issue-card agent activity indicator) lags by up to half a minute
|
|
// on the transition users care about most.
|
|
s.broadcastTaskEvent(ctx, protocol.EventTaskRunning, task)
|
|
return &task, nil
|
|
}
|
|
|
|
// MarkTaskWaitingLocalDirectory parks a dispatched task in the
|
|
// waiting_local_directory state while the daemon waits for another in-flight
|
|
// task to release the project_resource path lock. reason carries a short
|
|
// human-readable hint (typically the contested path) that the UI surfaces
|
|
// next to the status. Returns the updated row so the daemon can confirm the
|
|
// transition and so the broadcast carries the up-to-date snapshot.
|
|
func (s *TaskService) MarkTaskWaitingLocalDirectory(ctx context.Context, taskID pgtype.UUID, reason string) (*db.AgentTaskQueue, error) {
|
|
reason = strings.TrimSpace(reason)
|
|
task, err := s.Queries.MarkAgentTaskWaitingLocalDirectory(ctx, db.MarkAgentTaskWaitingLocalDirectoryParams{
|
|
ID: taskID,
|
|
WaitReason: pgtype.Text{String: reason, Valid: reason != ""},
|
|
})
|
|
if err != nil {
|
|
return nil, fmt.Errorf("mark task waiting_local_directory: %w", err)
|
|
}
|
|
|
|
slog.Info("task waiting_local_directory",
|
|
"task_id", util.UUIDToString(task.ID),
|
|
"issue_id", util.UUIDToString(task.IssueID),
|
|
"reason", reason,
|
|
)
|
|
s.broadcastTaskEvent(ctx, protocol.EventTaskWaitingLocalDirectory, task)
|
|
return &task, nil
|
|
}
|
|
|
|
// CompleteTask marks a task as completed.
|
|
// Issue status is NOT changed here — the agent manages it via the CLI.
|
|
//
|
|
// For chat tasks, CompleteAgentTask and the chat_session resume-pointer
|
|
// update run in a single transaction. This closes a race where the next
|
|
// queued chat message could be claimed in the window between the task
|
|
// flipping to 'completed' and chat_session.session_id being refreshed,
|
|
// causing the new task to resume against a stale (or NULL) session.
|
|
func (s *TaskService) CompleteTask(ctx context.Context, taskID pgtype.UUID, result []byte, sessionID, workDir string) (*db.AgentTaskQueue, error) {
|
|
var task db.AgentTaskQueue
|
|
if err := s.runInTx(ctx, func(qtx *db.Queries) error {
|
|
t, err := qtx.CompleteAgentTask(ctx, db.CompleteAgentTaskParams{
|
|
ID: taskID,
|
|
Result: result,
|
|
SessionID: pgtype.Text{String: sessionID, Valid: sessionID != ""},
|
|
WorkDir: pgtype.Text{String: workDir, Valid: workDir != ""},
|
|
})
|
|
if err != nil {
|
|
return err
|
|
}
|
|
task = t
|
|
|
|
if t.ChatSessionID.Valid {
|
|
// Pin the chat_session's runtime_id alongside the session_id so the
|
|
// next claim can apply the runtime-guard. Both fields move together:
|
|
// when there's no session_id to record, leave runtime_id untouched
|
|
// (NULL → COALESCE keeps the existing value).
|
|
var sessionRuntimeID pgtype.UUID
|
|
if sessionID != "" {
|
|
sessionRuntimeID = t.RuntimeID
|
|
}
|
|
// COALESCE in SQL guarantees empty inputs don't wipe the
|
|
// existing resume pointer; we still surface DB errors.
|
|
if err := qtx.UpdateChatSessionSession(ctx, db.UpdateChatSessionSessionParams{
|
|
ID: t.ChatSessionID,
|
|
SessionID: pgtype.Text{String: sessionID, Valid: sessionID != ""},
|
|
WorkDir: pgtype.Text{String: workDir, Valid: workDir != ""},
|
|
RuntimeID: sessionRuntimeID,
|
|
}); err != nil {
|
|
return fmt.Errorf("update chat session resume pointer: %w", err)
|
|
}
|
|
}
|
|
return nil
|
|
}); err != nil {
|
|
// When parallel agents race, a task may already be completed,
|
|
// cancelled, or failed by the time this call runs. The UPDATE
|
|
// … WHERE status = 'running' returns no rows in that case.
|
|
// Treat it as an idempotent success — same pattern as CancelTask.
|
|
if existing, lookupErr := s.Queries.GetAgentTask(ctx, taskID); lookupErr == nil {
|
|
if errors.Is(err, pgx.ErrNoRows) {
|
|
slog.Info("complete task: already finalized",
|
|
"task_id", util.UUIDToString(taskID),
|
|
"current_status", existing.Status,
|
|
"agent_id", util.UUIDToString(existing.AgentID),
|
|
)
|
|
return &existing, nil
|
|
}
|
|
slog.Warn("complete task failed",
|
|
"task_id", util.UUIDToString(taskID),
|
|
"current_status", existing.Status,
|
|
"issue_id", util.UUIDToString(existing.IssueID),
|
|
"chat_session_id", util.UUIDToString(existing.ChatSessionID),
|
|
"agent_id", util.UUIDToString(existing.AgentID),
|
|
"error", err,
|
|
)
|
|
} else {
|
|
slog.Warn("complete task failed: task not found",
|
|
"task_id", util.UUIDToString(taskID),
|
|
"lookup_error", lookupErr,
|
|
)
|
|
}
|
|
return nil, fmt.Errorf("complete task: %w", err)
|
|
}
|
|
|
|
slog.Info("task completed", "task_id", util.UUIDToString(task.ID), "issue_id", util.UUIDToString(task.IssueID))
|
|
s.captureTaskCompleted(ctx, task)
|
|
|
|
// Invariant: every completed issue task must have at least one agent
|
|
// comment on the issue, so the user always sees something when a run
|
|
// ends. If the agent posted a comment during execution (result, progress
|
|
// ping, or CLI reply), HasAgentCommentedSince returns true and we skip.
|
|
// Otherwise, synthesize one from the final output. For comment-triggered
|
|
// tasks, TriggerCommentID threads the fallback under the original comment;
|
|
// for assignment-triggered tasks it is NULL and the fallback is top-level.
|
|
// Chat tasks have no IssueID and are handled separately below.
|
|
if task.IssueID.Valid {
|
|
suppressNoActionComment, err := HasSquadLeaderNoActionEvaluationForTask(ctx, s.Queries, task)
|
|
if err != nil {
|
|
slog.Warn("checking squad leader no_action evaluation failed",
|
|
"task_id", util.UUIDToString(task.ID),
|
|
"issue_id", util.UUIDToString(task.IssueID),
|
|
"agent_id", util.UUIDToString(task.AgentID),
|
|
"error", err,
|
|
)
|
|
}
|
|
agentCommented, _ := s.Queries.HasAgentCommentedSince(ctx, db.HasAgentCommentedSinceParams{
|
|
IssueID: task.IssueID,
|
|
AuthorID: task.AgentID,
|
|
Since: task.StartedAt,
|
|
})
|
|
if !suppressNoActionComment && !agentCommented {
|
|
var payload protocol.TaskCompletedPayload
|
|
if err := json.Unmarshal(result, &payload); err == nil {
|
|
if payload.Output != "" {
|
|
// Match the CLI's --content / --description behavior: agents that
|
|
// emit literal `\n` 4-char sequences (Python/JSON-style) get them
|
|
// decoded into real newlines before the comment hits the DB. See
|
|
// util.UnescapeBackslashEscapes for the exact contract.
|
|
body := util.UnescapeBackslashEscapes(payload.Output)
|
|
if task.TriggerCommentID.Valid && isTrivialDoneOutput(body) {
|
|
slog.Warn("suppressing trivial comment-trigger fallback output",
|
|
"task_id", util.UUIDToString(task.ID),
|
|
"issue_id", util.UUIDToString(task.IssueID),
|
|
"agent_id", util.UUIDToString(task.AgentID),
|
|
)
|
|
} else {
|
|
s.createAgentComment(ctx, task.IssueID, task.AgentID, redact.Text(body), "comment", task.TriggerCommentID)
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
// Quick-create tasks: locate the issue the agent just created and push
|
|
// an inbox confirmation to the requester. The agent has no issue / chat
|
|
// link, so the regular completion paths above don't apply. We find the
|
|
// new issue by querying for the most recent issue this agent created in
|
|
// the requester's workspace since the task started — more robust than
|
|
// parsing the agent's stdout for an identifier.
|
|
if qc, ok := s.parseQuickCreateContext(task); ok {
|
|
s.notifyQuickCreateCompleted(ctx, task, qc)
|
|
}
|
|
|
|
// For chat tasks, save assistant reply and broadcast chat:done. The
|
|
// resume pointer was already persisted inside the transaction above.
|
|
if task.ChatSessionID.Valid {
|
|
var assistantMsg *db.ChatMessage
|
|
var payload protocol.TaskCompletedPayload
|
|
if err := json.Unmarshal(result, &payload); err == nil && payload.Output != "" {
|
|
// Same unescape as the issue-comment path above: literal `\n` from
|
|
// agent stdout becomes a real newline so the chat panel renders
|
|
// paragraph breaks instead of one wall of prose.
|
|
body := util.UnescapeBackslashEscapes(payload.Output)
|
|
row, err := s.Queries.CreateChatMessage(ctx, db.CreateChatMessageParams{
|
|
ChatSessionID: task.ChatSessionID,
|
|
Role: "assistant",
|
|
Content: redact.Text(body),
|
|
TaskID: task.ID,
|
|
ElapsedMs: computeChatElapsedMs(task),
|
|
})
|
|
if err != nil {
|
|
slog.Error("failed to save assistant chat message", "task_id", util.UUIDToString(task.ID), "error", err)
|
|
} else {
|
|
assistantMsg = &row
|
|
// Event-driven unread: stamp unread_since on the first unread
|
|
// assistant message. No-op if the session already has unread.
|
|
// If the user is actively viewing the session, the frontend's
|
|
// auto-mark-read effect will clear this within a tick.
|
|
if err := s.Queries.SetUnreadSinceIfNull(ctx, task.ChatSessionID); err != nil {
|
|
slog.Warn("failed to set unread_since", "chat_session_id", util.UUIDToString(task.ChatSessionID), "error", err)
|
|
}
|
|
}
|
|
}
|
|
s.broadcastChatDone(ctx, task, assistantMsg)
|
|
}
|
|
|
|
// Reconcile agent status
|
|
s.ReconcileAgentStatus(ctx, task.AgentID)
|
|
|
|
// Broadcast
|
|
s.broadcastTaskEvent(ctx, protocol.EventTaskCompleted, task)
|
|
|
|
return &task, nil
|
|
}
|
|
|
|
// FailTask marks a task as failed.
|
|
// Issue status is NOT changed here — the agent manages it via the CLI.
|
|
//
|
|
// sessionID/workDir are optional: when the agent established a real session
|
|
// before failing (e.g. crashed mid-conversation, was cancelled, or hit a
|
|
// tool error), the daemon should pass them so we can preserve the resume
|
|
// pointer on both the task row and the chat_session — otherwise the next
|
|
// chat turn would silently start a brand-new session and lose memory.
|
|
//
|
|
// failureReason is a coarse classifier consumed by the auto-retry path.
|
|
// Pass "" when unknown — the server runs the raw error text through
|
|
// taskfailure.Classify so the persisted failure_reason still lands in
|
|
// the canonical refined taxonomy rather than the legacy "agent_error"
|
|
// coarse bucket. Daemon callers that already produced a refined reason
|
|
// (via classifyPoisonedError, the timeout / runtime classifier, etc.)
|
|
// will have their value preserved untouched.
|
|
func (s *TaskService) FailTask(ctx context.Context, taskID pgtype.UUID, errMsg, sessionID, workDir, failureReason string) (*db.AgentTaskQueue, error) {
|
|
// MUL-2946: synthesise a refined reason from the error text whenever the
|
|
// caller didn't supply one. This is the last write-path guard against
|
|
// "agent_error" coarse rows ending up in agent_task_queue.failure_reason
|
|
// — every other path either provides a classified reason directly
|
|
// (sweepers writing 'queued_expired' / 'runtime_offline' / 'timeout'
|
|
// / 'runtime_recovery' via SQL) or runs the daemon's classifyPoisonedError
|
|
// + taskfailure.Classify chain.
|
|
if failureReason == "" {
|
|
failureReason = taskfailure.Classify(errMsg).String()
|
|
}
|
|
var task db.AgentTaskQueue
|
|
if err := s.runInTx(ctx, func(qtx *db.Queries) error {
|
|
t, err := qtx.FailAgentTask(ctx, db.FailAgentTaskParams{
|
|
ID: taskID,
|
|
Error: pgtype.Text{String: errMsg, Valid: true},
|
|
FailureReason: pgtype.Text{String: failureReason, Valid: failureReason != ""},
|
|
SessionID: pgtype.Text{String: sessionID, Valid: sessionID != ""},
|
|
WorkDir: pgtype.Text{String: workDir, Valid: workDir != ""},
|
|
})
|
|
if err != nil {
|
|
return err
|
|
}
|
|
task = t
|
|
|
|
// Keep resume-unsafe sessions on the task row for observability, but
|
|
// do not promote them to the chat-level resume pointer.
|
|
if t.ChatSessionID.Valid && !resumeUnsafeFailureReason(failureReason) {
|
|
// Pin the chat_session's runtime_id alongside the session_id so the
|
|
// next claim can apply the runtime-guard. Both fields move together:
|
|
// when there's no session_id to record, leave runtime_id untouched
|
|
// (NULL → COALESCE keeps the existing value).
|
|
var sessionRuntimeID pgtype.UUID
|
|
if sessionID != "" {
|
|
sessionRuntimeID = t.RuntimeID
|
|
}
|
|
if err := qtx.UpdateChatSessionSession(ctx, db.UpdateChatSessionSessionParams{
|
|
ID: t.ChatSessionID,
|
|
SessionID: pgtype.Text{String: sessionID, Valid: sessionID != ""},
|
|
WorkDir: pgtype.Text{String: workDir, Valid: workDir != ""},
|
|
RuntimeID: sessionRuntimeID,
|
|
}); err != nil {
|
|
return fmt.Errorf("update chat session resume pointer: %w", err)
|
|
}
|
|
}
|
|
return nil
|
|
}); err != nil {
|
|
if existing, lookupErr := s.Queries.GetAgentTask(ctx, taskID); lookupErr == nil {
|
|
if errors.Is(err, pgx.ErrNoRows) {
|
|
slog.Info("fail task: already finalized",
|
|
"task_id", util.UUIDToString(taskID),
|
|
"current_status", existing.Status,
|
|
"agent_id", util.UUIDToString(existing.AgentID),
|
|
)
|
|
return &existing, nil
|
|
}
|
|
slog.Warn("fail task failed",
|
|
"task_id", util.UUIDToString(taskID),
|
|
"current_status", existing.Status,
|
|
"issue_id", util.UUIDToString(existing.IssueID),
|
|
"chat_session_id", util.UUIDToString(existing.ChatSessionID),
|
|
"agent_id", util.UUIDToString(existing.AgentID),
|
|
"error", err,
|
|
)
|
|
} else {
|
|
slog.Warn("fail task failed: task not found",
|
|
"task_id", util.UUIDToString(taskID),
|
|
"lookup_error", lookupErr,
|
|
)
|
|
}
|
|
return nil, fmt.Errorf("fail task: %w", err)
|
|
}
|
|
|
|
slog.Warn("task failed", "task_id", util.UUIDToString(task.ID), "issue_id", util.UUIDToString(task.IssueID), "error", errMsg, "failure_reason", failureReason)
|
|
s.captureTaskFailed(ctx, task)
|
|
|
|
// Auto-retry eligible failures (orphan, timeout, runtime_offline,
|
|
// runtime_recovery). The helper itself enforces attempt < max_attempts
|
|
// and only triggers for issue/chat tasks.
|
|
retried, _ := s.MaybeRetryFailedTask(ctx, task)
|
|
|
|
// Skip the per-failure system comment when we'll immediately retry —
|
|
// the new task will surface its own status to the user, and we don't
|
|
// want to spam the issue with "task timed out" messages on every
|
|
// daemon hiccup.
|
|
if errMsg != "" && task.IssueID.Valid && retried == nil {
|
|
s.createAgentComment(ctx, task.IssueID, task.AgentID, redact.Text(errMsg), "system", task.TriggerCommentID)
|
|
}
|
|
|
|
// Mirror the issue fallback for chat tasks: write an assistant
|
|
// chat_message tagged with the daemon-reported failure_reason so the
|
|
// conversation history shows what happened. Skip when auto-retry is
|
|
// pending (the new attempt will write its own outcome) — same guard as
|
|
// the issue path above.
|
|
if task.ChatSessionID.Valid && retried == nil {
|
|
if _, err := s.Queries.CreateChatMessage(ctx, db.CreateChatMessageParams{
|
|
ChatSessionID: task.ChatSessionID,
|
|
Role: "assistant",
|
|
Content: redact.Text(errMsg),
|
|
TaskID: pgtype.UUID{Bytes: task.ID.Bytes, Valid: true},
|
|
FailureReason: pgtype.Text{String: failureReason, Valid: failureReason != ""},
|
|
ElapsedMs: computeChatElapsedMs(task),
|
|
}); err != nil {
|
|
slog.Error("failed to save failure chat message",
|
|
"task_id", util.UUIDToString(task.ID),
|
|
"chat_session_id", util.UUIDToString(task.ChatSessionID),
|
|
"error", err)
|
|
} else if err := s.Queries.SetUnreadSinceIfNull(ctx, task.ChatSessionID); err != nil {
|
|
slog.Warn("failed to set unread_since on failure",
|
|
"chat_session_id", util.UUIDToString(task.ChatSessionID),
|
|
"error", err)
|
|
}
|
|
}
|
|
|
|
// Quick-create tasks: push a failure inbox notification to the
|
|
// requester so they can either retry or fall back to the advanced form
|
|
// without losing their original prompt. Skipped when an auto-retry is
|
|
// pending — the new attempt will write its own outcome.
|
|
if retried == nil {
|
|
if qc, ok := s.parseQuickCreateContext(task); ok {
|
|
s.notifyQuickCreateFailed(ctx, task, qc, errMsg)
|
|
}
|
|
}
|
|
// Reconcile agent status
|
|
s.ReconcileAgentStatus(ctx, task.AgentID)
|
|
|
|
// Broadcast
|
|
s.broadcastTaskEvent(ctx, protocol.EventTaskFailed, task)
|
|
|
|
return &task, nil
|
|
}
|
|
|
|
// retryableReasons enumerates failure reasons that the auto-retry path is
|
|
// allowed to act on. Agent-side errors (compile failures, model rejections,
|
|
// etc.) are intentionally excluded — those are real problems that the user
|
|
// should see, not infrastructure flakiness.
|
|
var retryableReasons = map[string]bool{
|
|
"runtime_offline": true,
|
|
"runtime_recovery": true,
|
|
"timeout": true,
|
|
"codex_semantic_inactivity": true,
|
|
}
|
|
|
|
func resumeUnsafeFailureReason(reason string) bool {
|
|
switch reason {
|
|
// Keep in sync with GetLastTaskSession / GetLastChatTaskSession and
|
|
// CreateRetryTask's fresh-session CASE WHEN.
|
|
case "iteration_limit", "agent_fallback_message", "api_invalid_request", "codex_semantic_inactivity":
|
|
return true
|
|
default:
|
|
return false
|
|
}
|
|
}
|
|
|
|
// MaybeRetryFailedTask spawns a fresh queued attempt for a recently-failed
|
|
// task when the failure was infrastructure-shaped (daemon crash, runtime
|
|
// went offline, dispatch/run timeout) and the task hasn't exhausted its
|
|
// max_attempts budget. The child task inherits agent/runtime/issue/chat
|
|
// links and, for resume-safe failures, the parent's session_id/work_dir so
|
|
// the agent can resume the conversation when the backend supports it. Returns
|
|
// the new task, or nil when no retry was created.
|
|
//
|
|
// Autopilot tasks are NOT auto-retried here; the autopilot scheduler owns
|
|
// its own re-run cadence and we don't want to double-fire it.
|
|
func (s *TaskService) MaybeRetryFailedTask(ctx context.Context, parent db.AgentTaskQueue) (*db.AgentTaskQueue, error) {
|
|
if parent.Status != "failed" {
|
|
return nil, nil
|
|
}
|
|
reason := ""
|
|
if parent.FailureReason.Valid {
|
|
reason = parent.FailureReason.String
|
|
}
|
|
if !retryableReasons[reason] {
|
|
return nil, nil
|
|
}
|
|
if parent.Attempt >= parent.MaxAttempts {
|
|
slog.Info("task auto-retry skipped: budget exhausted",
|
|
"task_id", util.UUIDToString(parent.ID),
|
|
"attempt", parent.Attempt,
|
|
"max_attempts", parent.MaxAttempts,
|
|
)
|
|
return nil, nil
|
|
}
|
|
if parent.AutopilotRunID.Valid {
|
|
// Autopilot has its own retry semantics; do not double-trigger.
|
|
return nil, nil
|
|
}
|
|
if !parent.IssueID.Valid && !parent.ChatSessionID.Valid {
|
|
return nil, nil
|
|
}
|
|
|
|
child, err := s.Queries.CreateRetryTask(ctx, parent.ID)
|
|
if err != nil {
|
|
slog.Warn("task auto-retry failed",
|
|
"parent_task_id", util.UUIDToString(parent.ID),
|
|
"reason", reason,
|
|
"error", err,
|
|
)
|
|
return nil, err
|
|
}
|
|
slog.Info("task auto-retry enqueued",
|
|
"parent_task_id", util.UUIDToString(parent.ID),
|
|
"child_task_id", util.UUIDToString(child.ID),
|
|
"reason", reason,
|
|
"attempt", child.Attempt,
|
|
"max_attempts", child.MaxAttempts,
|
|
)
|
|
// Retry creates a fresh queued row, same status transition (∅ → queued)
|
|
// as EnqueueTaskFor*. Broadcast queued first, then notify the daemon —
|
|
// see EnqueueTaskForIssue for ordering rationale.
|
|
s.broadcastTaskEvent(ctx, protocol.EventTaskQueued, child)
|
|
s.NotifyTaskEnqueued(ctx, child)
|
|
return &child, nil
|
|
}
|
|
|
|
// RerunIssue creates a fresh queued task for an agent on the issue. Used by
|
|
// the manual rerun endpoint.
|
|
//
|
|
// Target agent resolution:
|
|
// - sourceTaskID Valid: rerun the agent that ran that task (and reuse its
|
|
// leader/worker role). This is what the execution log retry button uses
|
|
// so a per-row retry survives a subsequent assignee change and correctly
|
|
// re-fires the squad worker or mention agent whose row was clicked. The
|
|
// source task's trigger_comment_id is also inherited (when the caller
|
|
// didn't pass one) so a per-row rerun of a comment- or mention-triggered
|
|
// task stays comment-triggered — the daemon's buildCommentPrompt path
|
|
// keys on TriggerCommentID, and losing it would degrade the rerun into
|
|
// a generic issue run that no longer carries the original comment.
|
|
// - sourceTaskID empty: fall back to the issue's current assignee (agent
|
|
// or squad leader). This preserves the CLI / API contract for callers
|
|
// that have an issue ID but no specific task to target.
|
|
//
|
|
// The new task is flagged force_fresh_session=true so the daemon starts a
|
|
// clean agent session instead of resuming the prior (agent_id, issue_id)
|
|
// session. A user clicking rerun has just judged the prior output bad —
|
|
// resuming the same conversation would replay the same poisoned state.
|
|
// Auto-retry of an orphaned mid-flight failure (HandleFailedTasks →
|
|
// MaybeRetryFailedTask → CreateRetryTask) does NOT take this path, so
|
|
// MUL-1128's mid-flight resume contract is preserved.
|
|
//
|
|
// Only tasks belonging to the target agent on this issue are cancelled.
|
|
// Tasks owned by other agents on the same issue (e.g. a parallel
|
|
// @-mention agent) are left alone — rerun must not collateral-cancel
|
|
// them.
|
|
func (s *TaskService) RerunIssue(ctx context.Context, issueID pgtype.UUID, sourceTaskID pgtype.UUID, triggerCommentID pgtype.UUID) (*db.AgentTaskQueue, error) {
|
|
issue, err := s.Queries.GetIssue(ctx, issueID)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("load issue: %w", err)
|
|
}
|
|
|
|
// Determine the target agent for the rerun.
|
|
var (
|
|
agentID pgtype.UUID
|
|
isLeader bool
|
|
)
|
|
if sourceTaskID.Valid {
|
|
sourceTask, err := s.Queries.GetAgentTask(ctx, sourceTaskID)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("load source task: %w", err)
|
|
}
|
|
if !sourceTask.IssueID.Valid || util.UUIDToString(sourceTask.IssueID) != util.UUIDToString(issueID) {
|
|
return nil, fmt.Errorf("source task does not belong to this issue")
|
|
}
|
|
agentID = sourceTask.AgentID
|
|
isLeader = sourceTask.IsLeaderTask
|
|
// Inherit trigger provenance so a per-row rerun of a comment- or
|
|
// mention-triggered task stays a comment-triggered task. Without
|
|
// this the daemon's buildCommentPrompt path is skipped (it keys on
|
|
// TriggerCommentID) and the rerun degrades into a generic issue
|
|
// run that has lost the original comment context. Only override
|
|
// when the caller didn't pass one explicitly.
|
|
if !triggerCommentID.Valid && sourceTask.TriggerCommentID.Valid {
|
|
triggerCommentID = sourceTask.TriggerCommentID
|
|
}
|
|
} else {
|
|
switch {
|
|
case issue.AssigneeType.String == "agent" && issue.AssigneeID.Valid:
|
|
agentID = issue.AssigneeID
|
|
case issue.AssigneeType.String == "squad" && issue.AssigneeID.Valid:
|
|
squad, err := s.Queries.GetSquad(ctx, issue.AssigneeID)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("issue is assigned to a squad but squad not found")
|
|
}
|
|
agentID = squad.LeaderID
|
|
isLeader = true
|
|
default:
|
|
return nil, fmt.Errorf("issue is not assigned to an agent or squad")
|
|
}
|
|
}
|
|
|
|
// Cancel only the target agent's active/queued tasks on this issue.
|
|
cancelled, err := s.Queries.CancelAgentTasksByIssueAndAgent(ctx, db.CancelAgentTasksByIssueAndAgentParams{
|
|
IssueID: issueID,
|
|
AgentID: agentID,
|
|
})
|
|
if err != nil {
|
|
slog.Warn("rerun: cancel prior tasks failed",
|
|
"issue_id", util.UUIDToString(issueID),
|
|
"agent_id", util.UUIDToString(agentID),
|
|
"error", err,
|
|
)
|
|
}
|
|
for _, t := range cancelled {
|
|
s.captureTaskCancelled(ctx, t)
|
|
s.ReconcileAgentStatus(ctx, t.AgentID)
|
|
s.broadcastTaskEvent(ctx, protocol.EventTaskCancelled, t)
|
|
}
|
|
|
|
task, err := s.enqueueRerunTask(ctx, issue, agentID, triggerCommentID, isLeader)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
slog.Info("issue rerun enqueued",
|
|
"task_id", util.UUIDToString(task.ID),
|
|
"issue_id", util.UUIDToString(issueID),
|
|
"agent_id", util.UUIDToString(agentID),
|
|
"source_task_id", util.UUIDToString(sourceTaskID),
|
|
"is_leader", isLeader,
|
|
"cancelled_prior", len(cancelled),
|
|
)
|
|
return &task, nil
|
|
}
|
|
|
|
// enqueueRerunTask enqueues a fresh task for the given agent on the issue.
|
|
// When the target agent is the issue's single-agent assignee we use the
|
|
// assignee-driven path (enqueueIssueTask) so the issue-assignee bookkeeping
|
|
// stays in sync; otherwise (squad member, prior assignee that has since been
|
|
// reassigned, mention agent) we use the mention path with the same
|
|
// force_fresh_session=true contract.
|
|
func (s *TaskService) enqueueRerunTask(ctx context.Context, issue db.Issue, agentID pgtype.UUID, triggerCommentID pgtype.UUID, isLeader bool) (db.AgentTaskQueue, error) {
|
|
if issue.AssigneeType.String == "agent" && issue.AssigneeID.Valid &&
|
|
util.UUIDToString(issue.AssigneeID) == util.UUIDToString(agentID) {
|
|
return s.enqueueIssueTask(ctx, issue, triggerCommentID, true)
|
|
}
|
|
return s.enqueueMentionTask(ctx, issue, agentID, triggerCommentID, isLeader, true)
|
|
}
|
|
|
|
// HandleFailedTasks runs the post-failure side effects for a batch of
|
|
// freshly-failed tasks: optional auto-retry, task:failed event broadcast,
|
|
// agent status reconciliation, and (when an issue has no remaining active
|
|
// task and isn't being retried) resetting the issue back to todo so the
|
|
// daemon can pick it up again.
|
|
//
|
|
// All callers that surface a task as failed — sweepers, FailTask,
|
|
// recover-orphans — funnel through here so the same UI-consistency
|
|
// guarantees apply on every code path.
|
|
func (s *TaskService) HandleFailedTasks(ctx context.Context, tasks []db.AgentTaskQueue) int {
|
|
if len(tasks) == 0 {
|
|
return 0
|
|
}
|
|
|
|
affectedAgents := make(map[string]pgtype.UUID)
|
|
processedIssues := make(map[string]bool)
|
|
retriedIssues := make(map[string]bool)
|
|
retried := 0
|
|
|
|
for _, t := range tasks {
|
|
// Auto-retry first so the issue stays in_progress rather than
|
|
// flapping todo → in_progress within a tick.
|
|
if child, _ := s.MaybeRetryFailedTask(ctx, t); child != nil {
|
|
retried++
|
|
if t.IssueID.Valid {
|
|
retriedIssues[util.UUIDToString(t.IssueID)] = true
|
|
}
|
|
}
|
|
|
|
failureReason := "agent_error"
|
|
if t.FailureReason.Valid && t.FailureReason.String != "" {
|
|
failureReason = t.FailureReason.String
|
|
}
|
|
s.captureTaskFailed(ctx, t)
|
|
|
|
workspaceID := ""
|
|
if t.IssueID.Valid {
|
|
if issue, err := s.Queries.GetIssue(ctx, t.IssueID); err == nil {
|
|
workspaceID = util.UUIDToString(issue.WorkspaceID)
|
|
// Reset stuck in_progress issues only when no other active
|
|
// task exists for the issue and no retry was just enqueued.
|
|
issueKey := util.UUIDToString(t.IssueID)
|
|
if issue.Status == "in_progress" && !processedIssues[issueKey] && !retriedIssues[issueKey] {
|
|
processedIssues[issueKey] = true
|
|
hasActive, checkErr := s.Queries.HasActiveTaskForIssue(ctx, t.IssueID)
|
|
if checkErr != nil {
|
|
slog.Warn("handle failed tasks: active check failed",
|
|
"issue_id", issueKey,
|
|
"error", checkErr,
|
|
)
|
|
} else if !hasActive {
|
|
if _, updateErr := s.Queries.UpdateIssueStatus(ctx, db.UpdateIssueStatusParams{
|
|
ID: t.IssueID,
|
|
Status: "todo",
|
|
WorkspaceID: issue.WorkspaceID,
|
|
}); updateErr != nil {
|
|
slog.Warn("handle failed tasks: reset stuck issue failed",
|
|
"issue_id", issueKey,
|
|
"error", updateErr,
|
|
)
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
if workspaceID == "" {
|
|
workspaceID = s.ResolveTaskWorkspaceID(ctx, t)
|
|
}
|
|
|
|
if workspaceID != "" {
|
|
s.Bus.Publish(events.Event{
|
|
Type: protocol.EventTaskFailed,
|
|
WorkspaceID: workspaceID,
|
|
ActorType: "system",
|
|
Payload: map[string]any{
|
|
"task_id": util.UUIDToString(t.ID),
|
|
"agent_id": util.UUIDToString(t.AgentID),
|
|
"issue_id": util.UUIDToString(t.IssueID),
|
|
"status": "failed",
|
|
"failure_reason": failureReason,
|
|
},
|
|
})
|
|
}
|
|
|
|
affectedAgents[util.UUIDToString(t.AgentID)] = t.AgentID
|
|
}
|
|
|
|
for _, agentID := range affectedAgents {
|
|
s.ReconcileAgentStatus(ctx, agentID)
|
|
}
|
|
return retried
|
|
}
|
|
|
|
// runInTx executes fn inside a single DB transaction. If TxStarter is nil
|
|
// (e.g. some tests construct TaskService directly), fn runs against the
|
|
// regular Queries handle without transactional guarantees.
|
|
func (s *TaskService) runInTx(ctx context.Context, fn func(*db.Queries) error) error {
|
|
if s.TxStarter == nil {
|
|
return fn(s.Queries)
|
|
}
|
|
tx, err := s.TxStarter.Begin(ctx)
|
|
if err != nil {
|
|
return fmt.Errorf("begin tx: %w", err)
|
|
}
|
|
defer tx.Rollback(ctx)
|
|
if err := fn(s.Queries.WithTx(tx)); err != nil {
|
|
return err
|
|
}
|
|
return tx.Commit(ctx)
|
|
}
|
|
|
|
// ReportProgress broadcasts a progress update via the event bus.
|
|
func (s *TaskService) ReportProgress(ctx context.Context, taskID string, workspaceID string, summary string, step, total int) {
|
|
s.Bus.Publish(events.Event{
|
|
Type: protocol.EventTaskProgress,
|
|
WorkspaceID: workspaceID,
|
|
ActorType: "system",
|
|
ActorID: "",
|
|
TaskID: taskID,
|
|
Payload: protocol.TaskProgressPayload{
|
|
TaskID: taskID,
|
|
Summary: summary,
|
|
Step: step,
|
|
Total: total,
|
|
},
|
|
})
|
|
}
|
|
|
|
// ReconcileAgentStatus refreshes agent status from the current active task set.
|
|
func (s *TaskService) ReconcileAgentStatus(ctx context.Context, agentID pgtype.UUID) {
|
|
agent, err := s.Queries.RefreshAgentStatusFromTasks(ctx, agentID)
|
|
if err != nil {
|
|
return
|
|
}
|
|
slog.Debug("agent status reconciled", "agent_id", util.UUIDToString(agentID), "status", agent.Status)
|
|
s.publishAgentStatus(agent)
|
|
}
|
|
|
|
func (s *TaskService) updateAgentStatus(ctx context.Context, agentID pgtype.UUID, status string) {
|
|
agent, err := s.Queries.UpdateAgentStatus(ctx, db.UpdateAgentStatusParams{
|
|
ID: agentID,
|
|
Status: status,
|
|
})
|
|
if err != nil {
|
|
return
|
|
}
|
|
s.publishAgentStatus(agent)
|
|
}
|
|
|
|
func (s *TaskService) publishAgentStatus(agent db.Agent) {
|
|
s.Bus.Publish(events.Event{
|
|
Type: protocol.EventAgentStatus,
|
|
WorkspaceID: util.UUIDToString(agent.WorkspaceID),
|
|
ActorType: "system",
|
|
ActorID: "",
|
|
Payload: map[string]any{"agent": agentToMap(agent)},
|
|
})
|
|
}
|
|
|
|
// LoadAgentSkills loads an agent's skills with their files for task execution.
|
|
func (s *TaskService) LoadAgentSkills(ctx context.Context, agentID pgtype.UUID) []AgentSkillData {
|
|
skills, err := s.Queries.ListAgentSkills(ctx, agentID)
|
|
if err != nil || len(skills) == 0 {
|
|
return nil
|
|
}
|
|
|
|
result := make([]AgentSkillData, 0, len(skills))
|
|
for _, sk := range skills {
|
|
data := AgentSkillData{
|
|
ID: util.UUIDToString(sk.ID),
|
|
Name: sk.Name,
|
|
Description: sk.Description,
|
|
Content: sk.Content,
|
|
}
|
|
files, _ := s.Queries.ListSkillFiles(ctx, sk.ID)
|
|
for _, f := range files {
|
|
data.Files = append(data.Files, AgentSkillFileData{Path: f.Path, Content: f.Content})
|
|
}
|
|
result = append(result, data)
|
|
}
|
|
return result
|
|
}
|
|
|
|
// AgentSkillData represents a skill for task execution responses.
|
|
type AgentSkillData struct {
|
|
ID string `json:"id"`
|
|
Name string `json:"name"`
|
|
Description string `json:"description,omitempty"`
|
|
Content string `json:"content"`
|
|
Files []AgentSkillFileData `json:"files,omitempty"`
|
|
}
|
|
|
|
// AgentSkillFileData represents a supporting file within a skill.
|
|
type AgentSkillFileData struct {
|
|
Path string `json:"path"`
|
|
Content string `json:"content"`
|
|
}
|
|
|
|
// computeChatElapsedMs returns the wall-clock duration from task creation
|
|
// (user hit send) to terminal state (completed/failed). Stored on the
|
|
// assistant chat_message so the UI can render "Replied in 38s" /
|
|
// "Failed after 12s". Uses created_at — not started_at — because users
|
|
// experience total wait time, including queue + dispatch, not just the
|
|
// daemon's actual run time.
|
|
func computeChatElapsedMs(task db.AgentTaskQueue) pgtype.Int8 {
|
|
if !task.CompletedAt.Valid || !task.CreatedAt.Valid {
|
|
return pgtype.Int8{}
|
|
}
|
|
ms := task.CompletedAt.Time.Sub(task.CreatedAt.Time).Milliseconds()
|
|
if ms < 0 {
|
|
ms = 0
|
|
}
|
|
return pgtype.Int8{Int64: ms, Valid: true}
|
|
}
|
|
|
|
func priorityToInt(p string) int32 {
|
|
switch p {
|
|
case "urgent":
|
|
return 4
|
|
case "high":
|
|
return 3
|
|
case "medium":
|
|
return 2
|
|
case "low":
|
|
return 1
|
|
default:
|
|
return 0
|
|
}
|
|
}
|
|
|
|
// NotifyTaskEnqueued is the cross-package shim for callers outside
|
|
// TaskService (e.g. AutopilotService.dispatchRunOnly) that insert a
|
|
// row into agent_task_queue directly. Invalidates the empty-claim
|
|
// cache and kicks the daemon WS so the new task is claimed without
|
|
// waiting for the next poll.
|
|
func (s *TaskService) NotifyTaskEnqueued(ctx context.Context, task db.AgentTaskQueue) {
|
|
s.captureTaskQueued(ctx, task)
|
|
s.notifyTaskAvailable(task)
|
|
}
|
|
|
|
// notifyTaskAvailable runs after a task has been inserted: bumps the
|
|
// runtime's invalidation version so any in-flight claim that is about
|
|
// to write an "empty" verdict will have it rejected on read, then
|
|
// kicks the daemon WS so the daemon claims without waiting for its
|
|
// next poll. Order matters — Bump must happen before the wakeup,
|
|
// otherwise the wakeup-driven claim could read the still-current
|
|
// empty verdict and return null.
|
|
func (s *TaskService) notifyTaskAvailable(task db.AgentTaskQueue) {
|
|
if !task.RuntimeID.Valid {
|
|
return
|
|
}
|
|
runtimeKey := util.UUIDToString(task.RuntimeID)
|
|
// Use a background context: the cache bump / wakeup must outlive
|
|
// the request that created the task, otherwise an early client
|
|
// disconnect could leave the empty verdict in place and stall the
|
|
// just-queued task until the TTL expires. The cache itself bounds
|
|
// every Redis call with a short timeout so a wedged Redis cannot
|
|
// block enqueue.
|
|
s.EmptyClaim.Bump(context.Background(), runtimeKey)
|
|
if s.Wakeup == nil {
|
|
return
|
|
}
|
|
s.Wakeup.NotifyTaskAvailable(runtimeKey, util.UUIDToString(task.ID))
|
|
}
|
|
|
|
func (s *TaskService) broadcastTaskDispatch(ctx context.Context, task db.AgentTaskQueue) {
|
|
var payload map[string]any
|
|
if task.Context != nil {
|
|
json.Unmarshal(task.Context, &payload)
|
|
}
|
|
if payload == nil {
|
|
payload = map[string]any{}
|
|
}
|
|
payload["task_id"] = util.UUIDToString(task.ID)
|
|
payload["runtime_id"] = util.UUIDToString(task.RuntimeID)
|
|
payload["issue_id"] = util.UUIDToString(task.IssueID)
|
|
payload["agent_id"] = util.UUIDToString(task.AgentID)
|
|
// chat_session_id is the routing key the chat window uses to writethrough
|
|
// `chatKeys.pendingTask` to status="running" the moment the daemon claims
|
|
// the task. Without it the pill stays stuck at "Queued" until completion.
|
|
if task.ChatSessionID.Valid {
|
|
payload["chat_session_id"] = util.UUIDToString(task.ChatSessionID)
|
|
}
|
|
|
|
workspaceID := s.ResolveTaskWorkspaceID(ctx, task)
|
|
if workspaceID == "" {
|
|
return
|
|
}
|
|
s.Bus.Publish(events.Event{
|
|
Type: protocol.EventTaskDispatch,
|
|
WorkspaceID: workspaceID,
|
|
ActorType: "system",
|
|
ActorID: "",
|
|
Payload: payload,
|
|
})
|
|
}
|
|
|
|
func (s *TaskService) broadcastTaskEvent(ctx context.Context, eventType string, task db.AgentTaskQueue) {
|
|
workspaceID := s.ResolveTaskWorkspaceID(ctx, task)
|
|
if workspaceID == "" {
|
|
return
|
|
}
|
|
payload := map[string]any{
|
|
"task_id": util.UUIDToString(task.ID),
|
|
"agent_id": util.UUIDToString(task.AgentID),
|
|
"issue_id": util.UUIDToString(task.IssueID),
|
|
"status": task.Status,
|
|
}
|
|
if task.ChatSessionID.Valid {
|
|
payload["chat_session_id"] = util.UUIDToString(task.ChatSessionID)
|
|
}
|
|
s.Bus.Publish(events.Event{
|
|
Type: eventType,
|
|
WorkspaceID: workspaceID,
|
|
ActorType: "system",
|
|
ActorID: "",
|
|
Payload: payload,
|
|
})
|
|
}
|
|
|
|
// ResolveTaskWorkspaceID determines the workspace ID for a task.
|
|
// For issue tasks, it comes from the issue. For chat tasks, from the chat session.
|
|
// For autopilot tasks, from the autopilot via its run.
|
|
// Returns "" when none of the links resolve — callers treat that as "not found".
|
|
func (s *TaskService) ResolveTaskWorkspaceID(ctx context.Context, task db.AgentTaskQueue) string {
|
|
if task.IssueID.Valid {
|
|
if issue, err := s.Queries.GetIssue(ctx, task.IssueID); err == nil {
|
|
return util.UUIDToString(issue.WorkspaceID)
|
|
}
|
|
}
|
|
if task.ChatSessionID.Valid {
|
|
if cs, err := s.Queries.GetChatSession(ctx, task.ChatSessionID); err == nil {
|
|
return util.UUIDToString(cs.WorkspaceID)
|
|
}
|
|
}
|
|
if task.AutopilotRunID.Valid {
|
|
if run, err := s.Queries.GetAutopilotRun(ctx, task.AutopilotRunID); err == nil {
|
|
if ap, err := s.Queries.GetAutopilot(ctx, run.AutopilotID); err == nil {
|
|
return util.UUIDToString(ap.WorkspaceID)
|
|
}
|
|
}
|
|
}
|
|
// Quick-create tasks have no issue / chat / autopilot link — workspace
|
|
// lives in the context JSONB. Returning "" here is what blocked
|
|
// requireDaemonTaskAccess (404 on /start, /progress, /complete, /fail
|
|
// for the daemon) and silently dropped task:dispatch / task:completed
|
|
// broadcasts, which is why quick-create tasks appeared stuck queued.
|
|
if qc, ok := s.parseQuickCreateContext(task); ok {
|
|
return qc.WorkspaceID
|
|
}
|
|
return ""
|
|
}
|
|
|
|
func (s *TaskService) broadcastChatDone(ctx context.Context, task db.AgentTaskQueue, msg *db.ChatMessage) {
|
|
workspaceID := s.ResolveTaskWorkspaceID(ctx, task)
|
|
if workspaceID == "" {
|
|
return
|
|
}
|
|
payload := protocol.ChatDonePayload{
|
|
ChatSessionID: util.UUIDToString(task.ChatSessionID),
|
|
TaskID: util.UUIDToString(task.ID),
|
|
}
|
|
if msg != nil {
|
|
payload.MessageID = util.UUIDToString(msg.ID)
|
|
payload.Content = msg.Content
|
|
if msg.CreatedAt.Valid {
|
|
payload.CreatedAt = msg.CreatedAt.Time.UTC().Format(time.RFC3339Nano)
|
|
}
|
|
if msg.ElapsedMs.Valid {
|
|
payload.ElapsedMs = msg.ElapsedMs.Int64
|
|
}
|
|
}
|
|
s.Bus.Publish(events.Event{
|
|
Type: protocol.EventChatDone,
|
|
WorkspaceID: workspaceID,
|
|
ActorType: "system",
|
|
ActorID: "",
|
|
ChatSessionID: util.UUIDToString(task.ChatSessionID),
|
|
Payload: payload,
|
|
})
|
|
}
|
|
|
|
func (s *TaskService) broadcastIssueUpdated(issue db.Issue) {
|
|
prefix := s.getIssuePrefix(issue.WorkspaceID)
|
|
s.Bus.Publish(events.Event{
|
|
Type: protocol.EventIssueUpdated,
|
|
WorkspaceID: util.UUIDToString(issue.WorkspaceID),
|
|
ActorType: "system",
|
|
ActorID: "",
|
|
Payload: map[string]any{"issue": issueToMap(issue, prefix)},
|
|
})
|
|
}
|
|
|
|
func (s *TaskService) getIssuePrefix(workspaceID pgtype.UUID) string {
|
|
ws, err := s.Queries.GetWorkspace(context.Background(), workspaceID)
|
|
if err != nil {
|
|
return ""
|
|
}
|
|
return ws.IssuePrefix
|
|
}
|
|
|
|
func (s *TaskService) createAgentComment(ctx context.Context, issueID, agentID pgtype.UUID, content, commentType string, parentID pgtype.UUID) {
|
|
if content == "" {
|
|
return
|
|
}
|
|
// Look up issue to get workspace ID for mention expansion and broadcasting.
|
|
issue, err := s.Queries.GetIssue(ctx, issueID)
|
|
if err != nil {
|
|
return
|
|
}
|
|
// Resolve the thread root for thread-level side effects without overwriting
|
|
// parentID. The stored parent_id must remain the exact comment being replied
|
|
// to; recursive thread reads recover the root when needed.
|
|
var rootComment *db.Comment
|
|
if parentID.Valid {
|
|
if root, err := s.Queries.GetThreadRoot(ctx, db.GetThreadRootParams{
|
|
CommentID: parentID,
|
|
WorkspaceID: issue.WorkspaceID,
|
|
}); err == nil {
|
|
rootComment = &root
|
|
}
|
|
}
|
|
// Expand bare issue identifiers (e.g. MUL-117) into mention links.
|
|
content = mention.ExpandIssueIdentifiers(ctx, s.Queries, issue.WorkspaceID, content)
|
|
comment, err := s.Queries.CreateComment(ctx, db.CreateCommentParams{
|
|
IssueID: issueID,
|
|
WorkspaceID: issue.WorkspaceID,
|
|
AuthorType: "agent",
|
|
AuthorID: agentID,
|
|
Content: content,
|
|
Type: commentType,
|
|
ParentID: parentID,
|
|
})
|
|
if err != nil {
|
|
return
|
|
}
|
|
s.Bus.Publish(events.Event{
|
|
Type: protocol.EventCommentCreated,
|
|
WorkspaceID: util.UUIDToString(issue.WorkspaceID),
|
|
ActorType: "agent",
|
|
ActorID: util.UUIDToString(agentID),
|
|
Payload: map[string]any{
|
|
"comment": map[string]any{
|
|
"id": util.UUIDToString(comment.ID),
|
|
"issue_id": util.UUIDToString(comment.IssueID),
|
|
"author_type": comment.AuthorType,
|
|
"author_id": util.UUIDToString(comment.AuthorID),
|
|
"content": comment.Content,
|
|
"type": comment.Type,
|
|
"parent_id": util.UUIDToPtr(comment.ParentID),
|
|
"created_at": comment.CreatedAt.Time.Format("2006-01-02T15:04:05Z"),
|
|
},
|
|
"issue_title": issue.Title,
|
|
"issue_status": issue.Status,
|
|
},
|
|
})
|
|
s.AutoUnresolveThreadOnReply(ctx, rootComment, util.UUIDToString(issue.WorkspaceID), "agent", util.UUIDToString(agentID))
|
|
}
|
|
|
|
// AutoUnresolveThreadOnReply clears resolved_at on the thread root when a
|
|
// reply lands in a resolved thread, and broadcasts comment:unresolved. Shared
|
|
// between the user-facing Handler.CreateComment path and the agent-facing
|
|
// TaskService.createAgentComment path so the resolved-then-replied state can
|
|
// never desync (one of the bugs Emacs flagged on PR #2300). Errors are logged
|
|
// — the reply itself already committed, the desync is recoverable on next read.
|
|
func (s *TaskService) AutoUnresolveThreadOnReply(ctx context.Context, parent *db.Comment, workspaceID, actorType, actorID string) {
|
|
if parent == nil || !parent.ResolvedAt.Valid {
|
|
return
|
|
}
|
|
updated, err := s.Queries.UnresolveComment(ctx, parent.ID)
|
|
if err != nil {
|
|
slog.Warn("auto-unresolve on reply failed", "error", err, "comment_id", util.UUIDToString(parent.ID))
|
|
return
|
|
}
|
|
s.Bus.Publish(events.Event{
|
|
Type: protocol.EventCommentUnresolved,
|
|
WorkspaceID: workspaceID,
|
|
ActorType: actorType,
|
|
ActorID: actorID,
|
|
Payload: map[string]any{
|
|
"comment": map[string]any{
|
|
"id": util.UUIDToString(updated.ID),
|
|
"issue_id": util.UUIDToString(updated.IssueID),
|
|
"author_type": updated.AuthorType,
|
|
"author_id": util.UUIDToString(updated.AuthorID),
|
|
"content": updated.Content,
|
|
"type": updated.Type,
|
|
"parent_id": util.UUIDToPtr(updated.ParentID),
|
|
"created_at": util.TimestampToString(updated.CreatedAt),
|
|
"updated_at": util.TimestampToString(updated.UpdatedAt),
|
|
"resolved_at": util.TimestampToPtr(updated.ResolvedAt),
|
|
"resolved_by_type": util.TextToPtr(updated.ResolvedByType),
|
|
"resolved_by_id": util.UUIDToPtr(updated.ResolvedByID),
|
|
},
|
|
},
|
|
})
|
|
}
|
|
|
|
func issueToMap(issue db.Issue, issuePrefix string) map[string]any {
|
|
return map[string]any{
|
|
"id": util.UUIDToString(issue.ID),
|
|
"workspace_id": util.UUIDToString(issue.WorkspaceID),
|
|
"number": issue.Number,
|
|
"identifier": issuePrefix + "-" + strconv.Itoa(int(issue.Number)),
|
|
"title": issue.Title,
|
|
"description": util.TextToPtr(issue.Description),
|
|
"status": issue.Status,
|
|
"priority": issue.Priority,
|
|
"assignee_type": util.TextToPtr(issue.AssigneeType),
|
|
"assignee_id": util.UUIDToPtr(issue.AssigneeID),
|
|
"creator_type": issue.CreatorType,
|
|
"creator_id": util.UUIDToString(issue.CreatorID),
|
|
"parent_issue_id": util.UUIDToPtr(issue.ParentIssueID),
|
|
"position": issue.Position,
|
|
"start_date": util.DateToPtr(issue.StartDate),
|
|
"due_date": util.DateToPtr(issue.DueDate),
|
|
"created_at": util.TimestampToString(issue.CreatedAt),
|
|
"updated_at": util.TimestampToString(issue.UpdatedAt),
|
|
}
|
|
}
|
|
|
|
// parseQuickCreateContext returns the quick-create payload if the task's
|
|
// context JSONB contains type == "quick_create"; otherwise the bool is
|
|
// false so callers can short-circuit. Tasks linked to an issue / chat /
|
|
// autopilot are never quick-create even if they happen to carry a
|
|
// context blob, so those are filtered up front.
|
|
func (s *TaskService) parseQuickCreateContext(task db.AgentTaskQueue) (QuickCreateContext, bool) {
|
|
if task.IssueID.Valid || task.ChatSessionID.Valid || task.AutopilotRunID.Valid {
|
|
return QuickCreateContext{}, false
|
|
}
|
|
if len(task.Context) == 0 {
|
|
return QuickCreateContext{}, false
|
|
}
|
|
var qc QuickCreateContext
|
|
if err := json.Unmarshal(task.Context, &qc); err != nil {
|
|
return QuickCreateContext{}, false
|
|
}
|
|
if qc.Type != QuickCreateContextType {
|
|
return QuickCreateContext{}, false
|
|
}
|
|
return qc, true
|
|
}
|
|
|
|
// notifyQuickCreateCompleted writes a success inbox notification to the
|
|
// requester pointing at the issue the agent just created. The issue is
|
|
// stamped with origin_type=quick_create + origin_id=<task_id> by the
|
|
// daemon-injected MULTICA_QUICK_CREATE_TASK_ID env var, so this lookup is
|
|
// deterministic — robust against the same agent creating other issues in
|
|
// parallel (e.g. assignment task running while max_concurrent_tasks > 1
|
|
// permits another quick-create alongside it).
|
|
func (s *TaskService) notifyQuickCreateCompleted(ctx context.Context, task db.AgentTaskQueue, qc QuickCreateContext) {
|
|
requesterID, err := util.ParseUUID(qc.RequesterID)
|
|
if err != nil {
|
|
slog.Warn("quick-create completion: invalid requester id", "task_id", util.UUIDToString(task.ID), "error", err)
|
|
return
|
|
}
|
|
workspaceID, err := util.ParseUUID(qc.WorkspaceID)
|
|
if err != nil {
|
|
slog.Warn("quick-create completion: invalid workspace id", "task_id", util.UUIDToString(task.ID), "error", err)
|
|
return
|
|
}
|
|
issue, err := s.Queries.GetIssueByOrigin(ctx, db.GetIssueByOriginParams{
|
|
WorkspaceID: workspaceID,
|
|
OriginType: pgtype.Text{String: "quick_create", Valid: true},
|
|
OriginID: task.ID,
|
|
})
|
|
if err != nil {
|
|
// No issue created — agent ran to completion but the CLI call must
|
|
// have failed. Surface as a failure inbox so the user sees something.
|
|
slog.Warn("quick-create completion: no issue found, writing failure inbox",
|
|
"task_id", util.UUIDToString(task.ID),
|
|
"agent_id", util.UUIDToString(task.AgentID),
|
|
"workspace_id", qc.WorkspaceID,
|
|
)
|
|
s.notifyQuickCreateFailed(ctx, task, qc, "agent finished without creating an issue")
|
|
return
|
|
}
|
|
|
|
// Link the new issue back to this task so subsequent reads of the task
|
|
// (Activity tab, Recent work, etc.) render it as a normal issue task
|
|
// (kind = "direct") instead of staying on the "Creating issue" active-
|
|
// wording label. Best-effort: a write failure here doesn't block the
|
|
// inbox notification, which is the more important signal to the user.
|
|
if err := s.Queries.LinkTaskToIssue(ctx, db.LinkTaskToIssueParams{
|
|
ID: task.ID,
|
|
IssueID: issue.ID,
|
|
}); err != nil {
|
|
slog.Warn("quick-create completion: link task→issue failed",
|
|
"task_id", util.UUIDToString(task.ID),
|
|
"issue_id", util.UUIDToString(issue.ID),
|
|
"error", err,
|
|
)
|
|
}
|
|
|
|
// Subscribe the requester so they receive notifications for follow-up
|
|
// comments and updates. The DB row's creator_type/creator_id is the
|
|
// agent (it ran the CLI), but the human who triggered the quick-create
|
|
// is the semantic creator from a UX perspective — without this they
|
|
// only see the one-shot completion inbox and miss everything after.
|
|
// Best-effort: log on failure but don't block the inbox notification.
|
|
if err := s.Queries.AddIssueSubscriber(ctx, db.AddIssueSubscriberParams{
|
|
IssueID: issue.ID,
|
|
UserType: "member",
|
|
UserID: requesterID,
|
|
Reason: "creator",
|
|
}); err != nil {
|
|
slog.Warn("quick-create completion: subscribe requester failed",
|
|
"task_id", util.UUIDToString(task.ID),
|
|
"issue_id", util.UUIDToString(issue.ID),
|
|
"requester_id", qc.RequesterID,
|
|
"error", err,
|
|
)
|
|
} else {
|
|
s.Bus.Publish(events.Event{
|
|
Type: protocol.EventSubscriberAdded,
|
|
WorkspaceID: qc.WorkspaceID,
|
|
ActorType: "agent",
|
|
ActorID: util.UUIDToString(task.AgentID),
|
|
Payload: map[string]any{
|
|
"issue_id": util.UUIDToString(issue.ID),
|
|
"user_type": "member",
|
|
"user_id": qc.RequesterID,
|
|
"reason": "creator",
|
|
},
|
|
})
|
|
}
|
|
prefix := s.getIssuePrefix(workspaceID)
|
|
identifier := fmt.Sprintf("%s-%d", prefix, issue.Number)
|
|
details, _ := json.Marshal(map[string]any{
|
|
"task_id": util.UUIDToString(task.ID),
|
|
"agent_id": util.UUIDToString(task.AgentID),
|
|
"issue_id": util.UUIDToString(issue.ID),
|
|
"identifier": identifier,
|
|
"original_prompt": qc.Prompt,
|
|
})
|
|
item, err := s.Queries.CreateInboxItem(ctx, db.CreateInboxItemParams{
|
|
WorkspaceID: workspaceID,
|
|
RecipientType: "member",
|
|
RecipientID: requesterID,
|
|
Type: "quick_create_done",
|
|
Severity: "info",
|
|
IssueID: issue.ID,
|
|
Title: issue.Title,
|
|
Body: pgtype.Text{},
|
|
ActorType: pgtype.Text{String: "agent", Valid: true},
|
|
ActorID: task.AgentID,
|
|
Details: details,
|
|
})
|
|
if err != nil {
|
|
slog.Error("quick-create completion: inbox write failed", "task_id", util.UUIDToString(task.ID), "error", err)
|
|
return
|
|
}
|
|
s.publishQuickCreateInbox(item, qc.WorkspaceID, util.UUIDToString(task.AgentID), issue.Status)
|
|
}
|
|
|
|
// notifyQuickCreateFailed writes a failure inbox notification carrying the
|
|
// original prompt + agent ID so the frontend can render an "Edit as
|
|
// advanced form" entry that pre-fills the legacy create-issue modal
|
|
// without asking the user to retype.
|
|
func (s *TaskService) notifyQuickCreateFailed(ctx context.Context, task db.AgentTaskQueue, qc QuickCreateContext, errMsg string) {
|
|
requesterID, err := util.ParseUUID(qc.RequesterID)
|
|
if err != nil {
|
|
return
|
|
}
|
|
workspaceID, err := util.ParseUUID(qc.WorkspaceID)
|
|
if err != nil {
|
|
return
|
|
}
|
|
if errMsg == "" {
|
|
errMsg = "Quick create did not finish successfully"
|
|
}
|
|
details, _ := json.Marshal(map[string]any{
|
|
"task_id": util.UUIDToString(task.ID),
|
|
"agent_id": util.UUIDToString(task.AgentID),
|
|
"original_prompt": qc.Prompt,
|
|
"error": redact.Text(errMsg),
|
|
})
|
|
item, err := s.Queries.CreateInboxItem(ctx, db.CreateInboxItemParams{
|
|
WorkspaceID: workspaceID,
|
|
RecipientType: "member",
|
|
RecipientID: requesterID,
|
|
Type: "quick_create_failed",
|
|
Severity: "action_required",
|
|
IssueID: pgtype.UUID{},
|
|
Title: "Quick create failed",
|
|
Body: pgtype.Text{String: redact.Text(errMsg), Valid: true},
|
|
ActorType: pgtype.Text{String: "agent", Valid: true},
|
|
ActorID: task.AgentID,
|
|
Details: details,
|
|
})
|
|
if err != nil {
|
|
slog.Error("quick-create failure: inbox write failed", "task_id", util.UUIDToString(task.ID), "error", err)
|
|
return
|
|
}
|
|
s.publishQuickCreateInbox(item, qc.WorkspaceID, util.UUIDToString(task.AgentID), "")
|
|
}
|
|
|
|
// publishQuickCreateInbox emits the WS event so the requester's inbox list
|
|
// updates immediately. Mirrors the payload shape used by the other inbox
|
|
// listeners (notification_listeners.go).
|
|
func (s *TaskService) publishQuickCreateInbox(item db.InboxItem, workspaceID, agentID, issueStatus string) {
|
|
resp := map[string]any{
|
|
"id": util.UUIDToString(item.ID),
|
|
"workspace_id": util.UUIDToString(item.WorkspaceID),
|
|
"recipient_type": item.RecipientType,
|
|
"recipient_id": util.UUIDToString(item.RecipientID),
|
|
"type": item.Type,
|
|
"severity": item.Severity,
|
|
"issue_id": util.UUIDToPtr(item.IssueID),
|
|
"title": item.Title,
|
|
"body": util.TextToPtr(item.Body),
|
|
"read": item.Read,
|
|
"archived": item.Archived,
|
|
"created_at": util.TimestampToString(item.CreatedAt),
|
|
"actor_type": util.TextToPtr(item.ActorType),
|
|
"actor_id": util.UUIDToPtr(item.ActorID),
|
|
"details": json.RawMessage(item.Details),
|
|
"issue_status": issueStatus,
|
|
}
|
|
s.Bus.Publish(events.Event{
|
|
Type: protocol.EventInboxNew,
|
|
WorkspaceID: workspaceID,
|
|
ActorType: "agent",
|
|
ActorID: agentID,
|
|
Payload: map[string]any{"item": resp},
|
|
})
|
|
}
|
|
|
|
// agentToMap builds a simple map for broadcasting agent status updates.
|
|
func agentToMap(a db.Agent) map[string]any {
|
|
var rc any
|
|
if a.RuntimeConfig != nil {
|
|
json.Unmarshal(a.RuntimeConfig, &rc)
|
|
}
|
|
return map[string]any{
|
|
"id": util.UUIDToString(a.ID),
|
|
"workspace_id": util.UUIDToString(a.WorkspaceID),
|
|
"runtime_id": util.UUIDToString(a.RuntimeID),
|
|
"name": a.Name,
|
|
"description": a.Description,
|
|
"avatar_url": util.TextToPtr(a.AvatarUrl),
|
|
"runtime_mode": a.RuntimeMode,
|
|
"runtime_config": rc,
|
|
"visibility": a.Visibility,
|
|
"status": a.Status,
|
|
"max_concurrent_tasks": a.MaxConcurrentTasks,
|
|
"owner_id": util.UUIDToPtr(a.OwnerID),
|
|
"skills": []any{},
|
|
"created_at": util.TimestampToString(a.CreatedAt),
|
|
"updated_at": util.TimestampToString(a.UpdatedAt),
|
|
"archived_at": util.TimestampToPtr(a.ArchivedAt),
|
|
"archived_by": util.UUIDToPtr(a.ArchivedBy),
|
|
}
|
|
}
|