Follow-up to #5188 addressing the second-round review.
- ValidateThinkingLevel now fails an empty codex model closed instead of
borrowing the flagged Default (gpt-5.6-sol). An empty model follows
config.toml, which can resolve to any installed model; Sol alone advertises
`ultra`, so the old borrow green-lit levels Luna / gpt-5.5 don't support and
Codex doesn't reject. Checked before ListModels so a discovery error can't
fail it open. Frontend pickModelEntry mirrors this (no per-model effort
preview for an empty codex model); the persisted-orphan clear path stays.
- parseCodexDebugModels drops efforts without a known label so the picker
never advertises a level the Create/Update enum gate would 400 on save; the
contract test now drives the real parser with an unknown effort instead of
comparing two hand-written maps.
- gpt-5.6 price aliases anchor to a literal dot (not the [.-] class), so
dashed variants like gpt-5-6-luna surface as unmapped on both backend and
frontend rather than silently borrowing a tier.
Co-authored-by: J <j@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
* feat(models): add Codex gpt-5.6 series (sol/terra/luna) to model list & pricing (MUL-4347)
Co-authored-by: multica-agent <github@multica.ai>
* fix(models): official gpt-5.6 pricing, exact aliases, max/ultra effort levels (MUL-4347)
- Replace provisional gpt-5.6 rates with OpenAI's official announcement
values (sol 5/30, terra 2.5/15, luna 1/6); cache read 0.1x input, cache
write 1.25x input (frontend + backend, kept in sync).
- Anchor gpt-5.6 price aliases to exact match so unknown suffixed variants
surface as unmapped instead of borrowing a tier.
- Add Codex 0.144.1 max/ultra effort levels to the label map and server
enum so the daemon-advertised catalog matches what the API can persist;
add a catalog->API contract test.
- Clarify that the codex Default flag is the effort-validation anchor, not a
user-facing badge.
- Note the cache-write measurement limitation (codex usage stream doesn't
report cache-write tokens yet).
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: J <j@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
* chore(analytics): retire redundant PostHog tracking (MUL-4127)
PostHog had become a chaotic, largely-unused second copy of data we already
query from the DB and Grafana. Remove the redundant instrumentation.
Server: every product event (signup, workspace_created, issue_created,
issue_executed, chat_message_sent, team_invite_*, onboarding_*, agent_created,
cloud_waitlist_joined, feedback_submitted, contact_sales_submitted,
squad_created, autopilot_created) is now in metricsOnlyEvents, so
metrics.RecordEvent still increments the Prometheus/Grafana counter but no longer
ships to PostHog. DB rows remain the source of truth. Runtime/autopilot/
agent_task lifecycle were already Prometheus-only.
Frontend: delete the PostHog-only funnel instrumentation — $pageview (+ web and
desktop trackers), download_intent_expressed/page_viewed/initiated, the
onboarding_started mirror, onboarding_runtime_path_selected/detected,
feedback_opened, and source_backfill_*. The source-backfill modal itself stays
(it PATCHes the questionnaire to the DB).
Kept on PostHog (frontend only): $exception autocapture and the
client_crash / client_unresponsive stability telemetry (no DB equivalent), plus
$identify/$set. captureSignupSource (attribution cookie) stays — it still feeds
the signup_source Prometheus label.
Verified: pnpm typecheck, pnpm lint (0 errors), vitest (core/views/web/desktop),
go test ./internal/analytics/... ./internal/metrics/...
Co-authored-by: multica-agent <github@multica.ai>
* docs(analytics): fix stale PostHog references after MUL-4127 (review follow-up)
Addresses review of #4996 — three spots still described server events as active
PostHog signals after they became metrics-only:
- docs/analytics.md: issue_executed is no longer a PostHog success signal; it is
Prometheus-only (multica_issue_executed_total) + issue.first_executed_at, in
both the event contract and the Reconciliation section.
- docs/analytics.md: the signup $set_once person properties (email, signup_source)
are no longer emitted — signup is Prometheus-only; only the bucketed
signup_source survives as the multica_signup_total label.
- server/internal/metrics/business_events.go: RecordEvent doc comment no longer
claims it ships product events to PostHog / "PostHog is reserved for
user/product-behaviour events" — every server event is now metrics-only.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: J <j@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
* feat(analytics): anonymous self-host onboarding source beacon (MUL-3708)
Production self-host servers now report the anonymous onboarding "how did
you hear about us" channel to Multica's public write-only ingest, so the
self-host source distribution becomes visible alongside official cloud.
Official cloud keeps its existing PostHog capture unchanged; this is a
submit-time beacon, not a background telemetry pipeline.
- server/internal/sourcebeacon: ShouldSend gate (production + non-local +
non-*.multica.ai app host, fail-closed — judged by the app/frontend host,
not the backend URL, which official often leaves unset), per-instance
salted hashing, deterministic event uuid, fire-and-forget sender.
- POST /api/telemetry/self-host-source: public, write-only, per-IP
rate-limited, 4 KiB body cap, channel allowlist, strict unknown-field
rejection. Lands in PostHog as self_host_source_channel with a
deterministic uuid (best-effort dedup), $process_person_profile=false,
and deployment=self_host — a distinct event name so it never pollutes the
official onboarding funnel.
- Hook in PatchOnboarding fires once when the source is first set; never
blocks onboarding. Only channel enum(s) + two per-instance hashes leave
the box — never user_id/email/name/workspace/org/domain/role/use_case/the
source_other free-text/IP.
- migration 128: system_settings singleton holding instance_salt.
- frontend: self-host-only anonymous-collection notice on the source step,
gated by a new /api/config self_host_source_notice flag (en/zh-Hans/ko/ja).
- analytics.Event gains an optional top-level uuid; docs/analytics.md,
SELF_HOSTING.md and .env.example document exactly what is/isn't sent and
how to disable it (ANALYTICS_DISABLED). Also fixes the long-standing
team_size→source drift in docs/analytics.md.
Verified locally: go build/vet, go test (sourcebeacon, analytics, handler),
pnpm typecheck (all packages), locale parity (157), step-source (6) + core
config/schema (69) vitest, lint (0 errors).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: multica-agent <github@multica.ai>
* fix(analytics): wire self-host source beacon through metrics, guard nil pool (MUL-3708)
Addresses Howard CI blockers on #4691 (no product-direction change):
- loadInstanceSalt returns "" on nil pool; salt is only loaded when
ShouldSendFromEnv() is true, via a bounded (5s) context — restores the
"router constructible without a DB" invariant (nil-pool routing tests).
- Add multica_self_host_source_channel_total counter (by source) + an
IncForEvent case, so every analytics event is paired with a Prometheus
counter. NormalizeSourceChannel reuses sourcebeacon allowlist (no 3rd copy).
- Beacon handler now builds the event via the analytics.SelfHostSourceChannel
helper and ships it through obsmetrics.RecordEvent (no naked Capture); not
IsMetricsOnly, so it still reaches PostHog.
- Prime the new family in the registry-families test.
Verified: go build/vet, go test ./internal/metrics ./internal/sourcebeacon
./internal/handler ./cmd/server (incl. the 3 named blockers + registry +
record-event-helper lints) all green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: multica-agent <github@multica.ai>
* chore(analytics): stop shipping operational events to PostHog (MUL-2967)
Operational / execution-lifecycle telemetry dominated PostHog event volume
and drove the bill: runtime_offline alone was ~54% of ~22.6M events/mo, and
~99% of events were billed at the higher identified-event rate. These signals
already have Prometheus counters (Grafana), so the PostHog copies were
redundant cost.
- Add analytics.IsMetricsOnly; metrics.RecordEvent now skips the PostHog
Capture for runtime_* and autopilot_run_* while still incrementing their
Prometheus counter (their analytics.Event constructors are retained to feed
the metric label set via IncForEvent).
- Remove the agent_task_* PostHog path entirely: drop captureTaskEvent and the
AgentTask* constructors/constants. Their Prometheus side is unchanged via the
typed BusinessMetrics.RecordTask* methods. Also remove the now-dead
taskDurationMS / willRetryTask helpers.
- Update the pairing lint test (no agent_task allow-list, no naked-Capture
exception), add a RecordEvent skip test + IsMetricsOnly test, and update
docs/analytics.md (taxonomy, per-event banners, reconciliation).
Product/funnel events (signup, onboarding, issue_created, issue_executed,
chat_message_sent, agent_created, autopilot_created, etc.) are unchanged and
still ship to PostHog.
Co-authored-by: multica-agent <github@multica.ai>
* docs(analytics): correct agent_task Prometheus metric contract (MUL-2967)
Address PR review: the agent_task_* "Prometheus-only" banner claimed the old
PostHog event properties (task_id, agent_id, duration_ms, error_type,
will_retry, ...) were the metric label set. They are not — the real labels are
only source/runtime_mode/provider/terminal_status/failure_reason.
- Replace the agent_task_* sections with the actual metric names and labels
(multica_agent_task_*; see business.go / labels.go), and explain that
completed/failed/cancelled are terminal_status values on
multica_agent_task_terminal_total, with wall-clock in the *_seconds
histograms.
- Tighten the runtime_*/autopilot_run_* banners so id properties aren't
mistaken for labels.
- Drop the stale AgentTask allow-list reference from the pairing lint test
header comment.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: J <j@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
* feat(metrics): scrape-time BusinessSamplerCollector for active users / queued / runtime gauges (MUL-2947)
Adds an opt-in prometheus.Collector that runs a fixed set of read-only
SQL queries on every /metrics scrape and exposes the results as gauges:
- multica_active_users{window=5m|1h|24h}
- multica_active_workspaces{window=...}
- multica_agent_task_queued{source}
- multica_agent_task_running{source,runtime_mode}
- multica_agent_task_stuck_total{source}
- multica_runtime_online{runtime_mode,provider}
- multica_runtime_heartbeat_age_seconds{runtime_mode} (histogram)
- multica_workspace_total
Plus a self-introspection histogram
multica_business_sampler_query_seconds{name=...} and a counter
multica_business_sampler_query_errors_total{name=...} so the sampler's
own behaviour is observable on /metrics.
Production-safety contract per the PR4 brief:
- every query runs in its own BEGIN READ ONLY tx with
SET LOCAL statement_timeout = '500ms' (configurable)
- the sampler takes a dedicated *pgxpool.Pool option so operators
can isolate it from business traffic
- successful results are cached for 5–10s (default 8s) to absorb
concurrent scrapes from multiple Prometheus replicas
- every SQL has a hard LIMIT 100 fallback
- all label values flow through the existing BusinessMetrics
NormalizeTaskSource / NormalizeRuntimeMode / NormalizeRuntimeProvider
whitelists, so a misbehaving runtime cannot inflate cardinality
- sampler is OPT-IN via RegistryOptions.BusinessSampler — existing
callers that only pass Pool keep their current behaviour and never
start hitting the DB on /metrics
Tests cover: emit shape, TTL cache (one DB call per N scrapes),
bounded cardinality under malicious labels, opt-out (no leakage), and
DB-hang isolation (unreachable host -> /metrics returns within 5s,
query_errors_total advances).
Refs MUL-2947 (depends on PR2 / MUL-2948, merged in #3695).
Co-authored-by: multica-agent <github@multica.ai>
* fix(metrics): address PR4 review — wire sampler in main.go, fix LIMIT bug, add live-DB statement_timeout test
Three fixes from 大彪's review on #3706:
1. main.go was building NewRegistry without the BusinessSampler option,
so the collector was effectively dead code in prod. Now constructs a
dedicated 2-conn pgxpool (newSamplerDBPool) from the same DATABASE_URL
when METRICS_ADDR is set, plumbs it into RegistryOptions.BusinessSampler,
and defers Close() at shutdown. A pool-build failure logs and disables
the sampler instead of taking down the server.
2. queryActiveUsers / queryActiveWorkspaces previously wrapped the
distinct-user/workspace subquery in a 'LIMIT 100', then COUNT(*)'d
the result — capping the active-user gauge at 100 regardless of
reality. Removed the inner LIMIT; the COUNT scalar is one row anyway,
and metric cardinality is bounded by the fixed samplerWindows
allow-list, not by the SQL shape.
3. The previous DB-hang test only exercised the acquire-fails path. Added
business_sampler_pgsleep_test.go which connects to a live Postgres
(skips cleanly when DATABASE_URL is not set), runs SELECT pg_sleep(2)
inside a sampler-style tx with SET LOCAL statement_timeout = '500ms',
and asserts:
- the call returns in well under 1.5 s (proving the server-side
cancellation, not just our caller-side context)
- query_errors_total{name=pg_sleep_canary} advances
- the duration histogram records the cancellation
Verified locally: 550 ms, SQLSTATE 57014 'canceling statement due to
statement timeout' — exactly the safety net the PR claims.
Refs MUL-2947 / PR #3706.
Co-authored-by: multica-agent <github@multica.ai>
* test(metrics): assert SQLSTATE 57014 on pg_sleep cancellation
The previous assertion only checked that the query was cut off in well
under the sleep duration, which a caller-side context cancellation
would also satisfy. Capturing the inner pgconn.PgError and asserting
Code == "57014" ("query_canceled") nails down that Postgres itself
cancelled the statement because of the SET LOCAL statement_timeout —
so a regression that drops the SET LOCAL line fails this test loudly
instead of silently passing on context cancellation.
Refs MUL-2947 / PR #3706 review nit.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
* feat(server): funnel/community/commercial business metrics + PostHog pairing (MUL-2949)
PR3 of the Grafana board metrics split (parent MUL-2328).
Adds 23 new Prometheus counter/histogram families to the PR2 BusinessMetrics
collector covering the activation/community/commercial funnels, and binds
every PostHog event emission to a matching metric increment so the two sides
cannot drift.
Funnel: signup, workspace_created, team_invite_sent/accepted, onboarding_*,
cloud_waitlist_joined.
Content: issue_created, chat_message_sent, agent_created, squad_created,
autopilot_created, issue_executed.
Runtime: runtime_registered/ready/failed/offline + ready_seconds histogram,
daemon_ws_message_received_total.
Autopilot: autopilot_run_started/terminal/skipped.
Webhook/GitHub: webhook_delivery_total, github_event_received_total,
github_pr_review_total, github_pr_merge_seconds histogram.
CloudRuntime: cloudruntime_request_total + duration histogram, wired through
a small RequestRecorder interface so the cloudruntime package stays decoupled
from metrics.
Commercial: feedback_submitted, contact_sales_submitted.
The pairing helper metrics.RecordEvent(client, m, ev) emits the PostHog
event AND increments the matching counter via IncForEvent dispatch, reading
labels from the analytics event Properties. Every existing
h.Analytics.Capture(analytics.X(...)) call site has been migrated to the
helper across handler/, service/, and cmd/server/runtime_sweeper.go.
Lint enforcement (server/internal/metrics/business_pairing_test.go):
- TestEveryAnalyticsEventHasPrometheusCounter: every Event* constant in
analytics/events.go either dispatches via IncForEvent or is in the
taskMetricEvents allow-list (PR2 typed RecordTask* methods).
- TestNoNakedAnalyticsCaptureInHandlersOrServices: AST-walks handler/
service/cmd-server for direct Analytics.Capture(...) calls — only
service/task.go's captureTaskEvent helper is allow-listed.
- TestEveryAnalyticsRecordEventTakesAnalyticsHelper: validates the third
arg of every metrics.RecordEvent call is built from analytics.*.
Cardinality protection: all new label values pass through fixed allow-lists
in labels_pr3.go; unknown values collapse to 'other'/'unknown'/'error'.
Refs:
- Spec MUL-2328 / MUL-2949.
- Builds on PR2 (MUL-2948) — collectors registered through the same
BusinessMetrics struct, no separate Registry.
- Uses PR1's taskfailure.Reason (MUL-2946) for runtime_failed's failure_reason
label via NormalizeFailureReason.
Out of scope: Sampler-class metrics (PR4 / MUL-2947), pr_review_total
emission point (no review event handler exists yet — counter is defined,
TODO to wire up when /api/webhooks/github grows pull_request_review handling).
Co-authored-by: multica-agent <github@multica.ai>
* fix(server): tighten PR3 review items — signup_source bucket, fill platform/kind/form_source enums, onboarding_started server emission, lint scope (MUL-2949)
Addresses 张大彪's review on #3698:
1. signup_source: NormalizeSignupSource added to labels_pr3.go with a
fixed allow-list bucket (direct/google/twitter/linkedin/.../other).
Parses JSON cookie payload for utm_source/source/referrer fields,
strips URL schemes, maps well-known hostnames to channel buckets.
PostHog event still ships the raw cookie value for analytics; only
the Prometheus label is bucketed.
2. Filled the unknown/other label gaps:
- analytics.IssueCreated and analytics.ChatMessageSent now take a
platform parameter sourced from middleware.ClientMetadataFromContext
(X-Client-Platform header) at the handler. Autopilot-originated
issues stamp PlatformServer.
- analytics.FeedbackSubmitted now takes a kind parameter; CreateFeedback
reads req.Kind (default "general") so the picker selection lights up
the metric's kind label instead of long-term "other".
- analytics.ContactSalesSubmitted now takes a formSource (page /
onboarding / agents_page); CreateContactSales reads req.Source.
The metric reads ev.Properties["form_source"] so the analytics
CoreProperties.Source ("marketing_contact_sales") stays
backward-compat for PostHog dashboards.
3. analytics.OnboardingStarted helper added; server-side emission lives
in PatchOnboarding, fired exactly once per user on the first PATCH
that carries a non-empty questionnaire payload (firstTouch logic
compares prior bytes against {} / null). Frontend onboarding_started
keeps firing on page open; the server emission is what guarantees the
Prometheus counter exists so Grafana can be cross-checked against the
PostHog funnel without depending on the SDK roundtrip.
4. business_pairing_test.go tightened:
- TestNoNakedAnalyticsCaptureInHandlersOrServices now allow-lists at
function granularity (just captureTaskEvent in service/task.go), not
whole-file. Any future naked Capture in the same file fails CI.
- TestEveryAnalyticsRecordEventTakesAnalyticsHelper now does def-use
tracking inside the enclosing FuncDecl: when RecordEvent's third
arg is an *ast.Ident, the test walks the function body for the
assignment that defined it and confirms the RHS is an
analytics.<Helper>(...) call. Bare local idents that didn't
originate from analytics are now caught.
5. gofmt -w applied across the touched files; gofmt -l clean.
Tests: go test ./internal/metrics/... ./internal/analytics/... pass.
Pre-existing TestClaimTask_/TestWebhook_MergedPR/TestDeleteIssueByIdentifier
failures on origin/main are DB-environment-dependent and not regressions
from this change.
Co-authored-by: multica-agent <github@multica.ai>
* fix(server): normalise onboarding_started platform label + regression test (MUL-2949)
Addresses 张大彪's last review nit:
- IncForEvent's EventOnboardingStarted case now wraps the platform
property with NormalizePlatform, matching every other platform-bearing
metric. A misbehaving frontend can no longer leak a raw X-Client-Platform
header value into the multica_onboarding_started_total{platform=...}
series.
- New labels_pr3_test.go covers every PR3 normalizer with both a happy-path
value and an unknown value, asserting the unknown collapses to the
documented fallback bucket. Includes a focused regression for
onboarding_started: emits one event with an attacker-shaped platform
string and asserts the metric only exposes web + unknown label values
(no raw header bleed).
- testutil.go gains a small GatherForTest helper so the regression test
can pull the typed MetricFamily map without re-implementing the
registry-walk dance.
Co-authored-by: multica-agent <github@multica.ai>
* fix(server): NormalizeTaskSource on workspace_created + document lint limitations (MUL-2949)
Final review touch-ups before merge:
- IncForEvent's EventWorkspaceCreated case wraps source through
NormalizeTaskSource, matching the other source-bearing dispatches
(issue_created, agent_created, issue_executed). Closes the last raw
property leak in the dispatcher table.
- business_pairing_test.go inline docstrings now spell out the two
known limitations of the lint gate that 张大彪 / Eve flagged:
analyticsBackedIdents matches by ident NAME (not SSA def-use, so a
nested-scope shadow could pass) and isMetricsRecordEvent hard-codes
the import alias set. PR description carries a Follow-ups section
with the same two items so the work is visible after merge.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: 魏和尚 <agent+wei@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
