highperfocused/multica
1
0
Fork 0
mirror of https://github.com/multica-ai/multica.git synced 2026-07-17 07:09:53 +02:00
Code Issues Packages Projects Releases Wiki Activity
4,123 Commits 1,702 Branches 124 Tags
agent/lambda/768b92e0
Commit Graph

8 Commits

Author SHA1 Message Date
Multica Eve
ac62f72c2a MUL-4480: make daemon workspace sync event-driven (#5354) 
* feat(daemon): make workspace sync event-driven

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon): preserve trailing workspace changes

Co-authored-by: multica-agent <github@multica.ai>

* fix(workspace): reconcile failed creates

Co-authored-by: multica-agent <github@multica.ai>

---------

Co-authored-by: Eve <eve@multica-ai.local>
Co-authored-by: multica-agent <github@multica.ai>
 2026-07-14 11:56:30 +08:00
Multica Eve
c3dd9ec845 Machine-level batch task claim endpoint (MUL-4257) (#5193) 
* feat(daemon-claim): machine-level batch task claim endpoint (MUL-4257)

Collapse the per-runtime /tasks/claim poll fan-out into a single machine-level
batch claim to cut /api/daemon claim request volume.

Server:
- agent.sql: = ANY(runtime_ids) batch variants of the claim queries
  (ListQueuedClaimCandidatesByRuntimes, PromoteDueDeferredTasksForRuntimes,
  ReclaimStaleDispatchedTasksForRuntimes); runtime.sql: GetAgentRuntimes(= ANY)
  so a whole machine's runtimes are resolved/promoted/reclaimed/listed in a
  constant number of queries instead of N.
- service.ClaimTasksForRuntimes: claim up to max_tasks across a runtime set,
  preserving per-(issue,agent) serialization, the concurrency cap, the
  empty-claim cache short-circuit, and every dispatch side effect. Batch
  promote replays the per-row side effects (task:queued + empty-cache Bump).
- handler.ClaimTasksByRuntime (canonical POST /api/daemon/tasks/claim, with a
  transitional /claim alias): validates daemon_id (required; must match the
  mdt_ token) and rejects runtimes bound to a different daemon (group-ownership
  check mirroring the WS path); resolves+authorizes each runtime_id; claims;
  and finalizes each task through the SAME FinalizeTaskClaim as the per-runtime
  endpoint (atomic token + delivered_comment_ids receipt), requeueing the exact
  claim and omitting it on failure. buildClaimedTaskResponse is extracted from
  the per-runtime handler and returns the delivered-comment ids plus a
  structured *claimBuildFailure so both paths share identical payload building
  and failure semantics (workspace-isolation, chat-input load/empty).
- max_tasks: negative -> 400, zero -> empty (never coerce to 1), positive
  capped at 32. runtime_ids parsed with non-panicking util.ParseUUID.

Daemon:
- Client.ClaimTasks posts daemon_id + runtime set + free-slot count to the
  canonical path under a short request-scoped timeout, bounding the
  head-of-line coupling the per-runtime pollers avoid (MUL-1744).

Tests: service batch drain / max_tasks cap / deferred-promote receipt /
finalize-failure rollback+requeue; handler routing + token, cross-workspace
skip, cross-daemon skip, daemon_id required, owner-missing cancel,
max_tasks=0/negative, invalid-uuid skip, comment delivery receipt, stale-reclaim
replacement receipt; client posts/parses (daemon_id + canonical path).

Follow-up: cut the daemon pollLoop over to a single batched poller (flips the
MUL-1744 isolation contract; needs its concurrency tests redesigned).

Co-authored-by: multica-agent <github@multica.ai>

* feat(daemon-ws): generic WS request/response transport for daemon RPC (MUL-4257)

Add a generic daemon->server request/response layer over the existing WS
control connection, the transport for WS-first claim (HTTP fallback):
- protocol: daemon:rpc_request / daemon:rpc_response envelopes with a
  correlation request_id + method + body, and an rpc-v1 capability gate.
- daemonws.Hub: SetRPCHandler + goroutine-dispatched handleRPCFrame (bounded
  by a per-connection in-flight cap) that echoes the request_id; missing
  handler / saturation return non-2xx so the daemon falls back to HTTP.
  Read limit raised to 64KB for rpc requests carrying a runtime set.
- hub tests: round-trip, handler-error->non-2xx, no-handler->503.

Co-authored-by: multica-agent <github@multica.ai>

* feat(daemon-ws): WS-first task claim over the generic RPC transport (MUL-4257)

Bind claim to the WS request/response layer, with HTTP fallback:
- server: handler.DaemonRPCHandler adapts a daemon:rpc_request (method
  tasks.claim) to the existing HTTP ClaimTasksByRuntime via a synthetic
  in-process request carrying the WS connection's identity (daemon_id +
  workspace + capabilities), so all auth / payload-building / finalization is
  reused unchanged. Wired via daemonHub.SetRPCHandler. ClientIdentity now
  captures X-Client-Capabilities so capability gating matches the HTTP path.
- daemon: wsRPCClient correlates responses by request_id over the shared WS
  connection; attached to the live connection's write channel (guarded so a
  Call racing teardown never sends on a closed channel) and detached on
  disconnect. rpc_response frames are routed in the read loop.
  Daemon.ClaimTasksWSFirst issues tasks.claim over WS and falls back to the
  HTTP claim endpoint on any transport failure (no conn / buffer full /
  timeout) — wired into the poller at the poller cutover.
- tests: handler tasks.claim RPC end-to-end (claims + dispatches) + unknown
  method 404; daemon wsRPCClient round-trip / timeout / unavailable /
  server-error / detach-fails-pending (all under -race).

Co-authored-by: multica-agent <github@multica.ai>

* feat(daemon): cut claim poller over to machine-level ClaimTasksWSFirst (MUL-4257)

Replace the per-runtime HTTP poll loop with a single batch poller: each cycle
acquires all free execution slots (slot-before-claim) and issues ONE
ClaimTasksWSFirst across every runtime the daemon hosts (WS-first, HTTP
fallback), dispatching each returned task to its runtime. Wakeups (targeted /
catch-up / runtime-set change) collapse to one nudge. Removes runRuntimePoller
+ runtimePollOffset. The WS handshake now advertises the same capabilities as
HTTP (+ rpc-v1) so WS-built claim payloads keep skill-ref / coalesced-comment
gating.

Trades per-runtime isolation (MUL-1744) for one request, bounded by the short
per-request WS timeout / client timeout. Tests: batch poller claims across
runtimes + skips-at-capacity + pollLoop shutdown drain (replacing the
per-runtime poller tests); heartbeat isolation + runtime-set watcher kept.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon-ws): WS RPC disconnect-race panic + batch stale-comment-plan repair (MUL-4257)

Two PR #5193 review blockers:

1) WS RPC send-on-closed-channel race, both ends:
   - server: give each connection a cancelable ctx (cancelled on readPump
     teardown) and run the RPC handler under it, so a slow claim stops on
     disconnect; guard c.send with sendMu/sendClosed (trySend) so a late RPC
     response goroutine never writes to the closed channel. Heartbeat ack routed
     through the same guard.
   - daemon: wsRPCClient.deliver now sends under the mutex, serialized with
     attach(nil)'s close+delete, so a delivered response can't hit a channel
     the detach path just closed.
   - regressions (-race): daemon deliver-vs-detach; server
     disconnect-during-handler-response.

2) batch claim now runs the stale-comment-plan repair: extracted the
   per-runtime handler's repair (trigger deleted, only coalesced survive ->
   cancel + replay survivors) into shared repairStaleCommentPlanIfNeeded, called
   by both claim paths. Prevents the batch path (now the default poller) from
   finalizing+dispatching a task with no comment input and silently dropping the
   surviving user comment. Regression: batch omits the stale task, cancels it,
   and rebuilds the survivor into a new trigger plan.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon-ws): server-side RPC deadline + legacy claim fallback (MUL-4257)

Two review blockers:

1) WS RPC timeout/fallback (GPT-Boy): the daemon's WS wait didn't cancel
   server-side claim, so a slow WS claim could commit after the daemon fell
   back to HTTP, leaking dispatched tasks and breaking the free-slot bound.
   Fix: RPC envelope carries TimeoutMs; the server bounds the handler ctx by it
   (so ClaimTasksByRuntime's tx is cancelled/rolled back at the deadline), and
   the daemon waits budget + grace so a claim that committed before the deadline
   still reports back. A committed-then-unreported claim degrades to the same
   stale-reclaim safety net as HTTP, never a double effective claim. Regression:
   server-side TimeoutMs cancels the handler.

2) Backward compat (Terra-Boy): a new daemon against a server without the batch
   route (/api/daemon/tasks/claim 404) couldn't claim. Fix: ClaimTasksWSFirst
   falls back to the legacy per-runtime ClaimTask loop on a batch 404 and caches
   'batch unsupported' (reset on WS reconnect to re-probe after a server
   upgrade). Regression: server exposing only the legacy route.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon-ws): no double-claim on WS teardown/detach (MUL-4257)

Sol-Boy review blocker: on reconnect, teardown failed the pending RPC (→ HTTP
fallback) but then flushed the queued tasks.claim frame to the still-alive
socket, so the server committed the WS claim on top of the HTTP one — double
claim, WS batch orphaned to stale reclaim, breaking the free-slot bound.

- Teardown now closes the connection FIRST, so runWSWriter discards the queued
  RPC frame (write error path) instead of delivering it.
- A detach while a claim's frame is already in flight now returns a distinct
  errWSRPCUncertain; ClaimTasksWSFirst does NOT HTTP-fall-back on uncertain (the
  WS claim may have committed) — it skips the cycle and lets reclaim / the next
  poll recover. Genuine 'not sent' / timeout still fall back (safe: the
  server-side deadline guarantees no uncommitted claim by budget+grace).
- Regression: detach during an in-flight WS claim asserts zero HTTP claims
  (at most one path claims); plus the existing detach/deliver-race and
  server-timeout tests.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon-ws): cancelable RPC frames close the backpressure double-claim (MUL-4257)

Sol-Boy review blocker: the client's response budget starts at enqueue, but
the socket write is async (10s write deadline). A backpressured writer could
hold a tasks.claim in the local queue past the client timeout — the daemon
HTTP-fell-back, then the writer woke and delivered the stale WS frame, so the
server committed it too: same free slots claimed twice. No detach occurs, so
the prior errWSRPCUncertain fix did not cover it.

- WS frames are now cancelable (wsOutbound{sent,canceled} under a mutex). The
  writer calls beginWrite() before WriteMessage and skips cancelled frames.
- On give-up (timeout / detach / ctx), Call cancels the queued frame: if it was
  still pending the cancel wins and the frame is guaranteed never delivered
  (errWSRPCUnavailable → safe HTTP fallback); if the writer already began
  sending it the cancel loses and the outcome is errWSRPCUncertain (no
  fallback). The decision is atomic, so at most one transport claims.

Tests: wsOutbound cancel-before-write vs write-before-cancel; Call timeout
cancels an unsent frame (writer then drops it) vs uncertain when already sent;
plus the updated detach and existing timeout/race tests.

Co-authored-by: multica-agent <github@multica.ai>

* fix(batch-claim): return partial success instead of dropping committed claims (MUL-4257)

Sol-Boy review blocker: ClaimTasksForRuntimes reclaims (step 2) and claims per
agent (step 6) in independent transactions, but a step-4 candidate-SELECT error
or a mid-loop ClaimTask error did 'return nil, err' — discarding tasks already
committed as dispatched. The handler 500s; the daemon sees a definite (non-
uncertain) 500 and HTTP-falls-back, claiming a SECOND batch into the same free
slots while the first batch waits for stale reclaim — the double-claim this PR
removes.

- Both error paths now prefer partial success: if any task has already
  committed (claimed non-empty), return it (nil error) so the handler finalizes
  and returns 200; the errored candidates stay queued for the next poll. The
  remaining error is logged. Only a genuinely empty result still returns the
  error (safe: no committed claim to lose, HTTP fallback just re-fails).

Regression (internal/service, DB-backed, fault-injected):
- PartialSuccessOnSecondAgentClaimFailure: fail the 2nd ClaimTask's Begin →
  the first agent's committed task is returned, not dropped.
- PartialSuccessOnCandidateQueryFailureAfterReclaim: a stale dispatched task is
  reclaimed, then the candidate SELECT fails → the reclaimed task is returned.

Co-authored-by: multica-agent <github@multica.ai>

---------

Co-authored-by: Eve <eve@multica-ai.local>
Co-authored-by: multica-agent <github@multica.ai>
 2026-07-14 11:53:42 +08:00
Multica Eve
69ad806881 MUL-4472: fetch runtime profiles on demand 
Merge approved after review; includes WS reconnect reconciliation and regression coverage.
 2026-07-13 18:07:10 +08:00
LinYushen
77ac17ef49 Make custom runtimes appear immediately (#4234) 
* Make custom runtimes appear immediately

* Scope daemon profile refresh by authorized runtimes

* Relay runtime profile refresh hints

* Localize runtime profile close label
 2026-06-17 16:00:22 +08:00
LinYushen
de900b2ba6 feat(server): funnel/community/commercial business metrics + PostHog pairing (MUL-2949) (#3698) 
* feat(server): funnel/community/commercial business metrics + PostHog pairing (MUL-2949)

PR3 of the Grafana board metrics split (parent MUL-2328).

Adds 23 new Prometheus counter/histogram families to the PR2 BusinessMetrics
collector covering the activation/community/commercial funnels, and binds
every PostHog event emission to a matching metric increment so the two sides
cannot drift.

Funnel: signup, workspace_created, team_invite_sent/accepted, onboarding_*,
cloud_waitlist_joined.
Content: issue_created, chat_message_sent, agent_created, squad_created,
autopilot_created, issue_executed.
Runtime: runtime_registered/ready/failed/offline + ready_seconds histogram,
daemon_ws_message_received_total.
Autopilot: autopilot_run_started/terminal/skipped.
Webhook/GitHub: webhook_delivery_total, github_event_received_total,
github_pr_review_total, github_pr_merge_seconds histogram.
CloudRuntime: cloudruntime_request_total + duration histogram, wired through
a small RequestRecorder interface so the cloudruntime package stays decoupled
from metrics.
Commercial: feedback_submitted, contact_sales_submitted.

The pairing helper metrics.RecordEvent(client, m, ev) emits the PostHog
event AND increments the matching counter via IncForEvent dispatch, reading
labels from the analytics event Properties. Every existing
h.Analytics.Capture(analytics.X(...)) call site has been migrated to the
helper across handler/, service/, and cmd/server/runtime_sweeper.go.

Lint enforcement (server/internal/metrics/business_pairing_test.go):
- TestEveryAnalyticsEventHasPrometheusCounter: every Event* constant in
  analytics/events.go either dispatches via IncForEvent or is in the
  taskMetricEvents allow-list (PR2 typed RecordTask* methods).
- TestNoNakedAnalyticsCaptureInHandlersOrServices: AST-walks handler/
  service/cmd-server for direct Analytics.Capture(...) calls — only
  service/task.go's captureTaskEvent helper is allow-listed.
- TestEveryAnalyticsRecordEventTakesAnalyticsHelper: validates the third
  arg of every metrics.RecordEvent call is built from analytics.*.

Cardinality protection: all new label values pass through fixed allow-lists
in labels_pr3.go; unknown values collapse to 'other'/'unknown'/'error'.

Refs:
- Spec MUL-2328 / MUL-2949.
- Builds on PR2 (MUL-2948) — collectors registered through the same
  BusinessMetrics struct, no separate Registry.
- Uses PR1's taskfailure.Reason (MUL-2946) for runtime_failed's failure_reason
  label via NormalizeFailureReason.

Out of scope: Sampler-class metrics (PR4 / MUL-2947), pr_review_total
emission point (no review event handler exists yet — counter is defined,
TODO to wire up when /api/webhooks/github grows pull_request_review handling).

Co-authored-by: multica-agent <github@multica.ai>

* fix(server): tighten PR3 review items — signup_source bucket, fill platform/kind/form_source enums, onboarding_started server emission, lint scope (MUL-2949)

Addresses 张大彪's review on #3698:

1. signup_source: NormalizeSignupSource added to labels_pr3.go with a
   fixed allow-list bucket (direct/google/twitter/linkedin/.../other).
   Parses JSON cookie payload for utm_source/source/referrer fields,
   strips URL schemes, maps well-known hostnames to channel buckets.
   PostHog event still ships the raw cookie value for analytics; only
   the Prometheus label is bucketed.

2. Filled the unknown/other label gaps:
   - analytics.IssueCreated and analytics.ChatMessageSent now take a
     platform parameter sourced from middleware.ClientMetadataFromContext
     (X-Client-Platform header) at the handler. Autopilot-originated
     issues stamp PlatformServer.
   - analytics.FeedbackSubmitted now takes a kind parameter; CreateFeedback
     reads req.Kind (default "general") so the picker selection lights up
     the metric's kind label instead of long-term "other".
   - analytics.ContactSalesSubmitted now takes a formSource (page /
     onboarding / agents_page); CreateContactSales reads req.Source.
     The metric reads ev.Properties["form_source"] so the analytics
     CoreProperties.Source ("marketing_contact_sales") stays
     backward-compat for PostHog dashboards.

3. analytics.OnboardingStarted helper added; server-side emission lives
   in PatchOnboarding, fired exactly once per user on the first PATCH
   that carries a non-empty questionnaire payload (firstTouch logic
   compares prior bytes against {} / null). Frontend onboarding_started
   keeps firing on page open; the server emission is what guarantees the
   Prometheus counter exists so Grafana can be cross-checked against the
   PostHog funnel without depending on the SDK roundtrip.

4. business_pairing_test.go tightened:
   - TestNoNakedAnalyticsCaptureInHandlersOrServices now allow-lists at
     function granularity (just captureTaskEvent in service/task.go), not
     whole-file. Any future naked Capture in the same file fails CI.
   - TestEveryAnalyticsRecordEventTakesAnalyticsHelper now does def-use
     tracking inside the enclosing FuncDecl: when RecordEvent's third
     arg is an *ast.Ident, the test walks the function body for the
     assignment that defined it and confirms the RHS is an
     analytics.<Helper>(...) call. Bare local idents that didn't
     originate from analytics are now caught.

5. gofmt -w applied across the touched files; gofmt -l clean.

Tests: go test ./internal/metrics/... ./internal/analytics/... pass.
Pre-existing TestClaimTask_/TestWebhook_MergedPR/TestDeleteIssueByIdentifier
failures on origin/main are DB-environment-dependent and not regressions
from this change.

Co-authored-by: multica-agent <github@multica.ai>

* fix(server): normalise onboarding_started platform label + regression test (MUL-2949)

Addresses 张大彪's last review nit:

- IncForEvent's EventOnboardingStarted case now wraps the platform
  property with NormalizePlatform, matching every other platform-bearing
  metric. A misbehaving frontend can no longer leak a raw X-Client-Platform
  header value into the multica_onboarding_started_total{platform=...}
  series.

- New labels_pr3_test.go covers every PR3 normalizer with both a happy-path
  value and an unknown value, asserting the unknown collapses to the
  documented fallback bucket. Includes a focused regression for
  onboarding_started: emits one event with an attacker-shaped platform
  string and asserts the metric only exposes web + unknown label values
  (no raw header bleed).

- testutil.go gains a small GatherForTest helper so the regression test
  can pull the typed MetricFamily map without re-implementing the
  registry-walk dance.

Co-authored-by: multica-agent <github@multica.ai>

* fix(server): NormalizeTaskSource on workspace_created + document lint limitations (MUL-2949)

Final review touch-ups before merge:

- IncForEvent's EventWorkspaceCreated case wraps source through
  NormalizeTaskSource, matching the other source-bearing dispatches
  (issue_created, agent_created, issue_executed). Closes the last raw
  property leak in the dispatcher table.

- business_pairing_test.go inline docstrings now spell out the two
  known limitations of the lint gate that 张大彪 / Eve flagged:
  analyticsBackedIdents matches by ident NAME (not SSA def-use, so a
  nested-scope shadow could pass) and isMetricsRecordEvent hard-codes
  the import alias set. PR description carries a Follow-ups section
  with the same two items so the work is visible after merge.

Co-authored-by: multica-agent <github@multica.ai>

---------

Co-authored-by: 魏和尚 <agent+wei@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
 2026-06-03 16:39:06 +08:00
AdamQQQ
fab0671332 feat(skills): support multi-select bulk import in Copy from runtime (#2686) 
- Multi-select UI for batch importing skills from a local runtime
- Server batch-dispatches up to 10 import requests per heartbeat cycle
- WS heartbeat now reads supports_batch_import from daemon payload
  instead of hardcoding true, so old daemons correctly fall back to
  one-at-a-time dispatch
- Raised server pending timeout to 3min and client poll timeout to 4min
  to accommodate daemons that pop only one import per 15s heartbeat

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-18 16:56:27 +08:00
Bohan Jiang
286ecf04b1 feat(daemon): add WebSocket heartbeat with HTTP fallback 
Adds daemon WebSocket heartbeat acknowledgements while preserving HTTP heartbeat fallback and HTTP task claim/result paths. Keeps old daemon compatibility and task wakeup behavior intact.
 2026-04-29 17:17:55 +08:00
devv-eve
9db91e89f5 feat: add daemon websocket task wakeups (#1772) 
* feat: add daemon websocket task wakeups

* feat: fan out daemon wakeups across nodes

* fix: dedupe daemon wakeup loopback events

* fix: lengthen daemon polling fallback interval

---------

Co-authored-by: Eve <eve@multica.ai>
 2026-04-28 16:07:24 +08:00