mirror of
https://github.com/multica-ai/multica.git
synced 2026-07-26 20:45:37 +02:00
agent/lambda/5825e31d
7 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Multica Eve
|
945d684afd |
MUL-2965 fix(metrics): harden business sampler quality (#3738)
* fix(metrics): harden business sampler quality (MUL-2965) Co-authored-by: multica-agent <github@multica.ai> * fix(metrics): alert on sampler acquire failures Co-authored-by: multica-agent <github@multica.ai> --------- Co-authored-by: Eve <eve@multica-ai.local> Co-authored-by: multica-agent <github@multica.ai> |
||
|
Bohan Jiang
|
b92c3fbc93 |
chore(analytics): stop shipping operational events to PostHog (MUL-2967) (#3720)
* chore(analytics): stop shipping operational events to PostHog (MUL-2967) Operational / execution-lifecycle telemetry dominated PostHog event volume and drove the bill: runtime_offline alone was ~54% of ~22.6M events/mo, and ~99% of events were billed at the higher identified-event rate. These signals already have Prometheus counters (Grafana), so the PostHog copies were redundant cost. - Add analytics.IsMetricsOnly; metrics.RecordEvent now skips the PostHog Capture for runtime_* and autopilot_run_* while still incrementing their Prometheus counter (their analytics.Event constructors are retained to feed the metric label set via IncForEvent). - Remove the agent_task_* PostHog path entirely: drop captureTaskEvent and the AgentTask* constructors/constants. Their Prometheus side is unchanged via the typed BusinessMetrics.RecordTask* methods. Also remove the now-dead taskDurationMS / willRetryTask helpers. - Update the pairing lint test (no agent_task allow-list, no naked-Capture exception), add a RecordEvent skip test + IsMetricsOnly test, and update docs/analytics.md (taxonomy, per-event banners, reconciliation). Product/funnel events (signup, onboarding, issue_created, issue_executed, chat_message_sent, agent_created, autopilot_created, etc.) are unchanged and still ship to PostHog. Co-authored-by: multica-agent <github@multica.ai> * docs(analytics): correct agent_task Prometheus metric contract (MUL-2967) Address PR review: the agent_task_* "Prometheus-only" banner claimed the old PostHog event properties (task_id, agent_id, duration_ms, error_type, will_retry, ...) were the metric label set. They are not — the real labels are only source/runtime_mode/provider/terminal_status/failure_reason. - Replace the agent_task_* sections with the actual metric names and labels (multica_agent_task_*; see business.go / labels.go), and explain that completed/failed/cancelled are terminal_status values on multica_agent_task_terminal_total, with wall-clock in the *_seconds histograms. - Tighten the runtime_*/autopilot_run_* banners so id properties aren't mistaken for labels. - Drop the stale AgentTask allow-list reference from the pairing lint test header comment. Co-authored-by: multica-agent <github@multica.ai> --------- Co-authored-by: J <j@multica.ai> Co-authored-by: multica-agent <github@multica.ai> |
||
|
LinYushen
|
9c9afd4a66 |
feat(metrics): BusinessSamplerCollector for active users / queued / runtime gauges (MUL-2947) (#3706)
* feat(metrics): scrape-time BusinessSamplerCollector for active users / queued / runtime gauges (MUL-2947)
Adds an opt-in prometheus.Collector that runs a fixed set of read-only
SQL queries on every /metrics scrape and exposes the results as gauges:
- multica_active_users{window=5m|1h|24h}
- multica_active_workspaces{window=...}
- multica_agent_task_queued{source}
- multica_agent_task_running{source,runtime_mode}
- multica_agent_task_stuck_total{source}
- multica_runtime_online{runtime_mode,provider}
- multica_runtime_heartbeat_age_seconds{runtime_mode} (histogram)
- multica_workspace_total
Plus a self-introspection histogram
multica_business_sampler_query_seconds{name=...} and a counter
multica_business_sampler_query_errors_total{name=...} so the sampler's
own behaviour is observable on /metrics.
Production-safety contract per the PR4 brief:
- every query runs in its own BEGIN READ ONLY tx with
SET LOCAL statement_timeout = '500ms' (configurable)
- the sampler takes a dedicated *pgxpool.Pool option so operators
can isolate it from business traffic
- successful results are cached for 5–10s (default 8s) to absorb
concurrent scrapes from multiple Prometheus replicas
- every SQL has a hard LIMIT 100 fallback
- all label values flow through the existing BusinessMetrics
NormalizeTaskSource / NormalizeRuntimeMode / NormalizeRuntimeProvider
whitelists, so a misbehaving runtime cannot inflate cardinality
- sampler is OPT-IN via RegistryOptions.BusinessSampler — existing
callers that only pass Pool keep their current behaviour and never
start hitting the DB on /metrics
Tests cover: emit shape, TTL cache (one DB call per N scrapes),
bounded cardinality under malicious labels, opt-out (no leakage), and
DB-hang isolation (unreachable host -> /metrics returns within 5s,
query_errors_total advances).
Refs MUL-2947 (depends on PR2 / MUL-2948, merged in #3695).
Co-authored-by: multica-agent <github@multica.ai>
* fix(metrics): address PR4 review — wire sampler in main.go, fix LIMIT bug, add live-DB statement_timeout test
Three fixes from 大彪's review on #3706:
1. main.go was building NewRegistry without the BusinessSampler option,
so the collector was effectively dead code in prod. Now constructs a
dedicated 2-conn pgxpool (newSamplerDBPool) from the same DATABASE_URL
when METRICS_ADDR is set, plumbs it into RegistryOptions.BusinessSampler,
and defers Close() at shutdown. A pool-build failure logs and disables
the sampler instead of taking down the server.
2. queryActiveUsers / queryActiveWorkspaces previously wrapped the
distinct-user/workspace subquery in a 'LIMIT 100', then COUNT(*)'d
the result — capping the active-user gauge at 100 regardless of
reality. Removed the inner LIMIT; the COUNT scalar is one row anyway,
and metric cardinality is bounded by the fixed samplerWindows
allow-list, not by the SQL shape.
3. The previous DB-hang test only exercised the acquire-fails path. Added
business_sampler_pgsleep_test.go which connects to a live Postgres
(skips cleanly when DATABASE_URL is not set), runs SELECT pg_sleep(2)
inside a sampler-style tx with SET LOCAL statement_timeout = '500ms',
and asserts:
- the call returns in well under 1.5 s (proving the server-side
cancellation, not just our caller-side context)
- query_errors_total{name=pg_sleep_canary} advances
- the duration histogram records the cancellation
Verified locally: 550 ms, SQLSTATE 57014 'canceling statement due to
statement timeout' — exactly the safety net the PR claims.
Refs MUL-2947 / PR #3706.
Co-authored-by: multica-agent <github@multica.ai>
* test(metrics): assert SQLSTATE 57014 on pg_sleep cancellation
The previous assertion only checked that the query was cut off in well
under the sleep duration, which a caller-side context cancellation
would also satisfy. Capturing the inner pgconn.PgError and asserting
Code == "57014" ("query_canceled") nails down that Postgres itself
cancelled the statement because of the SET LOCAL statement_timeout —
so a regression that drops the SET LOCAL line fails this test loudly
instead of silently passing on context cancellation.
Refs MUL-2947 / PR #3706 review nit.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
|
||
|
LinYushen
|
de900b2ba6 |
feat(server): funnel/community/commercial business metrics + PostHog pairing (MUL-2949) (#3698)
* feat(server): funnel/community/commercial business metrics + PostHog pairing (MUL-2949) PR3 of the Grafana board metrics split (parent MUL-2328). Adds 23 new Prometheus counter/histogram families to the PR2 BusinessMetrics collector covering the activation/community/commercial funnels, and binds every PostHog event emission to a matching metric increment so the two sides cannot drift. Funnel: signup, workspace_created, team_invite_sent/accepted, onboarding_*, cloud_waitlist_joined. Content: issue_created, chat_message_sent, agent_created, squad_created, autopilot_created, issue_executed. Runtime: runtime_registered/ready/failed/offline + ready_seconds histogram, daemon_ws_message_received_total. Autopilot: autopilot_run_started/terminal/skipped. Webhook/GitHub: webhook_delivery_total, github_event_received_total, github_pr_review_total, github_pr_merge_seconds histogram. CloudRuntime: cloudruntime_request_total + duration histogram, wired through a small RequestRecorder interface so the cloudruntime package stays decoupled from metrics. Commercial: feedback_submitted, contact_sales_submitted. The pairing helper metrics.RecordEvent(client, m, ev) emits the PostHog event AND increments the matching counter via IncForEvent dispatch, reading labels from the analytics event Properties. Every existing h.Analytics.Capture(analytics.X(...)) call site has been migrated to the helper across handler/, service/, and cmd/server/runtime_sweeper.go. Lint enforcement (server/internal/metrics/business_pairing_test.go): - TestEveryAnalyticsEventHasPrometheusCounter: every Event* constant in analytics/events.go either dispatches via IncForEvent or is in the taskMetricEvents allow-list (PR2 typed RecordTask* methods). - TestNoNakedAnalyticsCaptureInHandlersOrServices: AST-walks handler/ service/cmd-server for direct Analytics.Capture(...) calls — only service/task.go's captureTaskEvent helper is allow-listed. - TestEveryAnalyticsRecordEventTakesAnalyticsHelper: validates the third arg of every metrics.RecordEvent call is built from analytics.*. Cardinality protection: all new label values pass through fixed allow-lists in labels_pr3.go; unknown values collapse to 'other'/'unknown'/'error'. Refs: - Spec MUL-2328 / MUL-2949. - Builds on PR2 (MUL-2948) — collectors registered through the same BusinessMetrics struct, no separate Registry. - Uses PR1's taskfailure.Reason (MUL-2946) for runtime_failed's failure_reason label via NormalizeFailureReason. Out of scope: Sampler-class metrics (PR4 / MUL-2947), pr_review_total emission point (no review event handler exists yet — counter is defined, TODO to wire up when /api/webhooks/github grows pull_request_review handling). Co-authored-by: multica-agent <github@multica.ai> * fix(server): tighten PR3 review items — signup_source bucket, fill platform/kind/form_source enums, onboarding_started server emission, lint scope (MUL-2949) Addresses 张大彪's review on #3698: 1. signup_source: NormalizeSignupSource added to labels_pr3.go with a fixed allow-list bucket (direct/google/twitter/linkedin/.../other). Parses JSON cookie payload for utm_source/source/referrer fields, strips URL schemes, maps well-known hostnames to channel buckets. PostHog event still ships the raw cookie value for analytics; only the Prometheus label is bucketed. 2. Filled the unknown/other label gaps: - analytics.IssueCreated and analytics.ChatMessageSent now take a platform parameter sourced from middleware.ClientMetadataFromContext (X-Client-Platform header) at the handler. Autopilot-originated issues stamp PlatformServer. - analytics.FeedbackSubmitted now takes a kind parameter; CreateFeedback reads req.Kind (default "general") so the picker selection lights up the metric's kind label instead of long-term "other". - analytics.ContactSalesSubmitted now takes a formSource (page / onboarding / agents_page); CreateContactSales reads req.Source. The metric reads ev.Properties["form_source"] so the analytics CoreProperties.Source ("marketing_contact_sales") stays backward-compat for PostHog dashboards. 3. analytics.OnboardingStarted helper added; server-side emission lives in PatchOnboarding, fired exactly once per user on the first PATCH that carries a non-empty questionnaire payload (firstTouch logic compares prior bytes against {} / null). Frontend onboarding_started keeps firing on page open; the server emission is what guarantees the Prometheus counter exists so Grafana can be cross-checked against the PostHog funnel without depending on the SDK roundtrip. 4. business_pairing_test.go tightened: - TestNoNakedAnalyticsCaptureInHandlersOrServices now allow-lists at function granularity (just captureTaskEvent in service/task.go), not whole-file. Any future naked Capture in the same file fails CI. - TestEveryAnalyticsRecordEventTakesAnalyticsHelper now does def-use tracking inside the enclosing FuncDecl: when RecordEvent's third arg is an *ast.Ident, the test walks the function body for the assignment that defined it and confirms the RHS is an analytics.<Helper>(...) call. Bare local idents that didn't originate from analytics are now caught. 5. gofmt -w applied across the touched files; gofmt -l clean. Tests: go test ./internal/metrics/... ./internal/analytics/... pass. Pre-existing TestClaimTask_/TestWebhook_MergedPR/TestDeleteIssueByIdentifier failures on origin/main are DB-environment-dependent and not regressions from this change. Co-authored-by: multica-agent <github@multica.ai> * fix(server): normalise onboarding_started platform label + regression test (MUL-2949) Addresses 张大彪's last review nit: - IncForEvent's EventOnboardingStarted case now wraps the platform property with NormalizePlatform, matching every other platform-bearing metric. A misbehaving frontend can no longer leak a raw X-Client-Platform header value into the multica_onboarding_started_total{platform=...} series. - New labels_pr3_test.go covers every PR3 normalizer with both a happy-path value and an unknown value, asserting the unknown collapses to the documented fallback bucket. Includes a focused regression for onboarding_started: emits one event with an attacker-shaped platform string and asserts the metric only exposes web + unknown label values (no raw header bleed). - testutil.go gains a small GatherForTest helper so the regression test can pull the typed MetricFamily map without re-implementing the registry-walk dance. Co-authored-by: multica-agent <github@multica.ai> * fix(server): NormalizeTaskSource on workspace_created + document lint limitations (MUL-2949) Final review touch-ups before merge: - IncForEvent's EventWorkspaceCreated case wraps source through NormalizeTaskSource, matching the other source-bearing dispatches (issue_created, agent_created, issue_executed). Closes the last raw property leak in the dispatcher table. - business_pairing_test.go inline docstrings now spell out the two known limitations of the lint gate that 张大彪 / Eve flagged: analyticsBackedIdents matches by ident NAME (not SSA def-use, so a nested-scope shadow could pass) and isMetricsRecordEvent hard-codes the import alias set. PR description carries a Follow-ups section with the same two items so the work is visible after merge. Co-authored-by: multica-agent <github@multica.ai> --------- Co-authored-by: 魏和尚 <agent+wei@multica.ai> Co-authored-by: multica-agent <github@multica.ai> |
||
|
Multica Eve
|
a72fb020de |
Add business metrics collectors (#3695)
Co-authored-by: Eve <eve@multica-ai.local> Co-authored-by: multica-agent <github@multica.ai> |
||
|
devv-eve
|
9db91e89f5 |
feat: add daemon websocket task wakeups (#1772)
* feat: add daemon websocket task wakeups * feat: fan out daemon wakeups across nodes * fix: dedupe daemon wakeup loopback events * fix: lengthen daemon polling fallback interval --------- Co-authored-by: Eve <eve@multica.ai> |
||
|
devv-eve
|
f864a07bd5 |
feat: add server Prometheus metrics endpoint
Add Prometheus metrics endpoint with local-bind listener support and baseline metrics collectors. |