* chore(analytics): stop shipping operational events to PostHog (MUL-2967)
Operational / execution-lifecycle telemetry dominated PostHog event volume
and drove the bill: runtime_offline alone was ~54% of ~22.6M events/mo, and
~99% of events were billed at the higher identified-event rate. These signals
already have Prometheus counters (Grafana), so the PostHog copies were
redundant cost.
- Add analytics.IsMetricsOnly; metrics.RecordEvent now skips the PostHog
Capture for runtime_* and autopilot_run_* while still incrementing their
Prometheus counter (their analytics.Event constructors are retained to feed
the metric label set via IncForEvent).
- Remove the agent_task_* PostHog path entirely: drop captureTaskEvent and the
AgentTask* constructors/constants. Their Prometheus side is unchanged via the
typed BusinessMetrics.RecordTask* methods. Also remove the now-dead
taskDurationMS / willRetryTask helpers.
- Update the pairing lint test (no agent_task allow-list, no naked-Capture
exception), add a RecordEvent skip test + IsMetricsOnly test, and update
docs/analytics.md (taxonomy, per-event banners, reconciliation).
Product/funnel events (signup, onboarding, issue_created, issue_executed,
chat_message_sent, agent_created, autopilot_created, etc.) are unchanged and
still ship to PostHog.
Co-authored-by: multica-agent <github@multica.ai>
* docs(analytics): correct agent_task Prometheus metric contract (MUL-2967)
Address PR review: the agent_task_* "Prometheus-only" banner claimed the old
PostHog event properties (task_id, agent_id, duration_ms, error_type,
will_retry, ...) were the metric label set. They are not — the real labels are
only source/runtime_mode/provider/terminal_status/failure_reason.
- Replace the agent_task_* sections with the actual metric names and labels
(multica_agent_task_*; see business.go / labels.go), and explain that
completed/failed/cancelled are terminal_status values on
multica_agent_task_terminal_total, with wall-clock in the *_seconds
histograms.
- Tighten the runtime_*/autopilot_run_* banners so id properties aren't
mistaken for labels.
- Drop the stale AgentTask allow-list reference from the pairing lint test
header comment.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: J <j@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
* feat(metrics): scrape-time BusinessSamplerCollector for active users / queued / runtime gauges (MUL-2947)
Adds an opt-in prometheus.Collector that runs a fixed set of read-only
SQL queries on every /metrics scrape and exposes the results as gauges:
- multica_active_users{window=5m|1h|24h}
- multica_active_workspaces{window=...}
- multica_agent_task_queued{source}
- multica_agent_task_running{source,runtime_mode}
- multica_agent_task_stuck_total{source}
- multica_runtime_online{runtime_mode,provider}
- multica_runtime_heartbeat_age_seconds{runtime_mode} (histogram)
- multica_workspace_total
Plus a self-introspection histogram
multica_business_sampler_query_seconds{name=...} and a counter
multica_business_sampler_query_errors_total{name=...} so the sampler's
own behaviour is observable on /metrics.
Production-safety contract per the PR4 brief:
- every query runs in its own BEGIN READ ONLY tx with
SET LOCAL statement_timeout = '500ms' (configurable)
- the sampler takes a dedicated *pgxpool.Pool option so operators
can isolate it from business traffic
- successful results are cached for 5–10s (default 8s) to absorb
concurrent scrapes from multiple Prometheus replicas
- every SQL has a hard LIMIT 100 fallback
- all label values flow through the existing BusinessMetrics
NormalizeTaskSource / NormalizeRuntimeMode / NormalizeRuntimeProvider
whitelists, so a misbehaving runtime cannot inflate cardinality
- sampler is OPT-IN via RegistryOptions.BusinessSampler — existing
callers that only pass Pool keep their current behaviour and never
start hitting the DB on /metrics
Tests cover: emit shape, TTL cache (one DB call per N scrapes),
bounded cardinality under malicious labels, opt-out (no leakage), and
DB-hang isolation (unreachable host -> /metrics returns within 5s,
query_errors_total advances).
Refs MUL-2947 (depends on PR2 / MUL-2948, merged in #3695).
Co-authored-by: multica-agent <github@multica.ai>
* fix(metrics): address PR4 review — wire sampler in main.go, fix LIMIT bug, add live-DB statement_timeout test
Three fixes from 大彪's review on #3706:
1. main.go was building NewRegistry without the BusinessSampler option,
so the collector was effectively dead code in prod. Now constructs a
dedicated 2-conn pgxpool (newSamplerDBPool) from the same DATABASE_URL
when METRICS_ADDR is set, plumbs it into RegistryOptions.BusinessSampler,
and defers Close() at shutdown. A pool-build failure logs and disables
the sampler instead of taking down the server.
2. queryActiveUsers / queryActiveWorkspaces previously wrapped the
distinct-user/workspace subquery in a 'LIMIT 100', then COUNT(*)'d
the result — capping the active-user gauge at 100 regardless of
reality. Removed the inner LIMIT; the COUNT scalar is one row anyway,
and metric cardinality is bounded by the fixed samplerWindows
allow-list, not by the SQL shape.
3. The previous DB-hang test only exercised the acquire-fails path. Added
business_sampler_pgsleep_test.go which connects to a live Postgres
(skips cleanly when DATABASE_URL is not set), runs SELECT pg_sleep(2)
inside a sampler-style tx with SET LOCAL statement_timeout = '500ms',
and asserts:
- the call returns in well under 1.5 s (proving the server-side
cancellation, not just our caller-side context)
- query_errors_total{name=pg_sleep_canary} advances
- the duration histogram records the cancellation
Verified locally: 550 ms, SQLSTATE 57014 'canceling statement due to
statement timeout' — exactly the safety net the PR claims.
Refs MUL-2947 / PR #3706.
Co-authored-by: multica-agent <github@multica.ai>
* test(metrics): assert SQLSTATE 57014 on pg_sleep cancellation
The previous assertion only checked that the query was cut off in well
under the sleep duration, which a caller-side context cancellation
would also satisfy. Capturing the inner pgconn.PgError and asserting
Code == "57014" ("query_canceled") nails down that Postgres itself
cancelled the statement because of the SET LOCAL statement_timeout —
so a regression that drops the SET LOCAL line fails this test loudly
instead of silently passing on context cancellation.
Refs MUL-2947 / PR #3706 review nit.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
* feat(server): funnel/community/commercial business metrics + PostHog pairing (MUL-2949)
PR3 of the Grafana board metrics split (parent MUL-2328).
Adds 23 new Prometheus counter/histogram families to the PR2 BusinessMetrics
collector covering the activation/community/commercial funnels, and binds
every PostHog event emission to a matching metric increment so the two sides
cannot drift.
Funnel: signup, workspace_created, team_invite_sent/accepted, onboarding_*,
cloud_waitlist_joined.
Content: issue_created, chat_message_sent, agent_created, squad_created,
autopilot_created, issue_executed.
Runtime: runtime_registered/ready/failed/offline + ready_seconds histogram,
daemon_ws_message_received_total.
Autopilot: autopilot_run_started/terminal/skipped.
Webhook/GitHub: webhook_delivery_total, github_event_received_total,
github_pr_review_total, github_pr_merge_seconds histogram.
CloudRuntime: cloudruntime_request_total + duration histogram, wired through
a small RequestRecorder interface so the cloudruntime package stays decoupled
from metrics.
Commercial: feedback_submitted, contact_sales_submitted.
The pairing helper metrics.RecordEvent(client, m, ev) emits the PostHog
event AND increments the matching counter via IncForEvent dispatch, reading
labels from the analytics event Properties. Every existing
h.Analytics.Capture(analytics.X(...)) call site has been migrated to the
helper across handler/, service/, and cmd/server/runtime_sweeper.go.
Lint enforcement (server/internal/metrics/business_pairing_test.go):
- TestEveryAnalyticsEventHasPrometheusCounter: every Event* constant in
analytics/events.go either dispatches via IncForEvent or is in the
taskMetricEvents allow-list (PR2 typed RecordTask* methods).
- TestNoNakedAnalyticsCaptureInHandlersOrServices: AST-walks handler/
service/cmd-server for direct Analytics.Capture(...) calls — only
service/task.go's captureTaskEvent helper is allow-listed.
- TestEveryAnalyticsRecordEventTakesAnalyticsHelper: validates the third
arg of every metrics.RecordEvent call is built from analytics.*.
Cardinality protection: all new label values pass through fixed allow-lists
in labels_pr3.go; unknown values collapse to 'other'/'unknown'/'error'.
Refs:
- Spec MUL-2328 / MUL-2949.
- Builds on PR2 (MUL-2948) — collectors registered through the same
BusinessMetrics struct, no separate Registry.
- Uses PR1's taskfailure.Reason (MUL-2946) for runtime_failed's failure_reason
label via NormalizeFailureReason.
Out of scope: Sampler-class metrics (PR4 / MUL-2947), pr_review_total
emission point (no review event handler exists yet — counter is defined,
TODO to wire up when /api/webhooks/github grows pull_request_review handling).
Co-authored-by: multica-agent <github@multica.ai>
* fix(server): tighten PR3 review items — signup_source bucket, fill platform/kind/form_source enums, onboarding_started server emission, lint scope (MUL-2949)
Addresses 张大彪's review on #3698:
1. signup_source: NormalizeSignupSource added to labels_pr3.go with a
fixed allow-list bucket (direct/google/twitter/linkedin/.../other).
Parses JSON cookie payload for utm_source/source/referrer fields,
strips URL schemes, maps well-known hostnames to channel buckets.
PostHog event still ships the raw cookie value for analytics; only
the Prometheus label is bucketed.
2. Filled the unknown/other label gaps:
- analytics.IssueCreated and analytics.ChatMessageSent now take a
platform parameter sourced from middleware.ClientMetadataFromContext
(X-Client-Platform header) at the handler. Autopilot-originated
issues stamp PlatformServer.
- analytics.FeedbackSubmitted now takes a kind parameter; CreateFeedback
reads req.Kind (default "general") so the picker selection lights up
the metric's kind label instead of long-term "other".
- analytics.ContactSalesSubmitted now takes a formSource (page /
onboarding / agents_page); CreateContactSales reads req.Source.
The metric reads ev.Properties["form_source"] so the analytics
CoreProperties.Source ("marketing_contact_sales") stays
backward-compat for PostHog dashboards.
3. analytics.OnboardingStarted helper added; server-side emission lives
in PatchOnboarding, fired exactly once per user on the first PATCH
that carries a non-empty questionnaire payload (firstTouch logic
compares prior bytes against {} / null). Frontend onboarding_started
keeps firing on page open; the server emission is what guarantees the
Prometheus counter exists so Grafana can be cross-checked against the
PostHog funnel without depending on the SDK roundtrip.
4. business_pairing_test.go tightened:
- TestNoNakedAnalyticsCaptureInHandlersOrServices now allow-lists at
function granularity (just captureTaskEvent in service/task.go), not
whole-file. Any future naked Capture in the same file fails CI.
- TestEveryAnalyticsRecordEventTakesAnalyticsHelper now does def-use
tracking inside the enclosing FuncDecl: when RecordEvent's third
arg is an *ast.Ident, the test walks the function body for the
assignment that defined it and confirms the RHS is an
analytics.<Helper>(...) call. Bare local idents that didn't
originate from analytics are now caught.
5. gofmt -w applied across the touched files; gofmt -l clean.
Tests: go test ./internal/metrics/... ./internal/analytics/... pass.
Pre-existing TestClaimTask_/TestWebhook_MergedPR/TestDeleteIssueByIdentifier
failures on origin/main are DB-environment-dependent and not regressions
from this change.
Co-authored-by: multica-agent <github@multica.ai>
* fix(server): normalise onboarding_started platform label + regression test (MUL-2949)
Addresses 张大彪's last review nit:
- IncForEvent's EventOnboardingStarted case now wraps the platform
property with NormalizePlatform, matching every other platform-bearing
metric. A misbehaving frontend can no longer leak a raw X-Client-Platform
header value into the multica_onboarding_started_total{platform=...}
series.
- New labels_pr3_test.go covers every PR3 normalizer with both a happy-path
value and an unknown value, asserting the unknown collapses to the
documented fallback bucket. Includes a focused regression for
onboarding_started: emits one event with an attacker-shaped platform
string and asserts the metric only exposes web + unknown label values
(no raw header bleed).
- testutil.go gains a small GatherForTest helper so the regression test
can pull the typed MetricFamily map without re-implementing the
registry-walk dance.
Co-authored-by: multica-agent <github@multica.ai>
* fix(server): NormalizeTaskSource on workspace_created + document lint limitations (MUL-2949)
Final review touch-ups before merge:
- IncForEvent's EventWorkspaceCreated case wraps source through
NormalizeTaskSource, matching the other source-bearing dispatches
(issue_created, agent_created, issue_executed). Closes the last raw
property leak in the dispatcher table.
- business_pairing_test.go inline docstrings now spell out the two
known limitations of the lint gate that 张大彪 / Eve flagged:
analyticsBackedIdents matches by ident NAME (not SSA def-use, so a
nested-scope shadow could pass) and isMetricsRecordEvent hard-codes
the import alias set. PR description carries a Follow-ups section
with the same two items so the work is visible after merge.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: 魏和尚 <agent+wei@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
