Files
multica/server/migrations/103_drop_legacy_daily_rollups.up.sql
Bohan Jiang 9a5d8a52f3 fix(timezone): harden hourly-rollup rollout against straight-through migrate MUL-2488 (#2998)
* fix(timezone): harden hourly-rollup rollout against straight-through migrate

MUL-2488

PR #2968 introduced the new task_usage_hourly rollup but assumed operators
would stop migrate between 102 and 103 to run the one-shot
cmd/backfill_task_usage_hourly. Two pieces made that unsafe in practice:

1. The Dockerfile only shipped server / multica / migrate, so a deployed
   container has no backfill binary to run between phases.
2. cmd/migrate has no per-version stop, and entrypoint.sh runs `migrate up`
   to the latest version, so 103 silently drops the legacy daily rollups
   even when nobody ran the backfill — leaving usage dashboards at zero
   despite source data being intact in task_usage.

Changes:

- Build cmd/backfill_task_usage_hourly into the runtime image alongside
  the other binaries so operators can `docker exec` the backfill instead
  of needing a source checkout.
- Add a fail-closed plpgsql guard at the top of migration 103 that
  aborts the migration when task_usage has rows but task_usage_hourly is
  empty. Fresh databases (no task_usage rows) are exempt because the new
  triggers from 102 will populate the hourly table on the first event.

Already-applied databases are unaffected — schema_migrations tracks by
version only, so 103 is not re-run.

Co-authored-by: multica-agent <github@multica.ai>

* fix(timezone): use watermark coverage for hourly-rollup guard

The previous check only required `task_usage_hourly` to be non-empty,
which an interrupted backfill or a manual `rollup_task_usage_hourly_window`
call both satisfy. The completion signal we actually trust is
`task_usage_hourly_rollup_state.watermark_at` — backfill only stamps it
to `now() - 5 min` after every monthly slice succeeded, and the cron
worker only advances it on a real tick. Default after migration 101 is
`1970-01-01`, so an unrun or partial backfill is trivially detected.

Also corrects the comment about fresh-install behavior: the triggers in
102 only enqueue dirty keys for agent_task_queue / issue / task_usage
DELETE — they do not write hourly rows. INSERT/UPDATE flows through the
`updated_at` watermark window of `rollup_task_usage_hourly()`, which
only runs once the operator registers it as a pg_cron job.

Co-authored-by: multica-agent <github@multica.ai>

---------

Co-authored-by: multica-agent <github@multica.ai>
2026-05-21 16:26:42 +08:00

151 lines
7.5 KiB
SQL

-- Drop the two legacy daily rollup pipelines now that `task_usage_hourly`
-- is the only read path (see docs/timezone-architecture-rfc.md §6,
-- Phase 3). Forward-only: there is no down migration that would put the
-- data back, since by the time this ships:
--
-- * The hourly rollup has been live and writing every bucket since
-- the hourly-pipeline migration.
-- * No handler reads from `task_usage_daily` or
-- `task_usage_dashboard_daily`. The runtime PATCH path that used
-- to delete/insert into `task_usage_daily` on tz change has been
-- replaced by a single UPDATE.
-- * The `cmd/backfill_task_usage_daily` and
-- `cmd/backfill_task_usage_dashboard_daily` commands have been
-- removed. The remaining `cmd/backfill_task_usage_hourly` is the
-- only seed path going forward.
--
-- The two pg_cron entries are unscheduled below before their functions
-- are dropped, so a still-registered job cannot tick into a
-- `function does not exist` error. The cron.unschedule calls are wrapped
-- so the migration still succeeds on instances without pg_cron at all
-- (same guard pattern as migration 076).
-- ---------------------------------------------------------------------------
-- Fail-closed guard: refuse to drop the legacy daily pipelines unless the
-- hourly rollup has actually been seeded across the existing `task_usage`
-- history. `migrate up` runs the migration set straight through (no
-- per-version stop), so a self-host operator who skips the documented
-- backfill step would otherwise silently land in a state where dashboards
-- show zeros (see SELF-HOST UPGRADE ORDER in
-- cmd/backfill_task_usage_hourly/main.go and
-- docs/timezone-architecture-rfc.md §6 / §7.1). Failing loud here is the
-- only thing that turns an undetected outage into a clear migration error.
--
-- The completion signal we trust is task_usage_hourly_rollup_state.watermark_at:
--
-- * cmd/backfill_task_usage_hourly stamps it to `now() - 5 minutes` only
-- after every monthly slice succeeded (server/cmd/backfill_task_usage_hourly/main.go).
-- * rollup_task_usage_hourly() (the pg_cron entry) advances it on every
-- tick (102_task_usage_hourly_pipeline.up.sql).
-- * The default after migration 101 is `1970-01-01`, so an unrun or
-- interrupted backfill is trivially detected.
--
-- A non-empty `task_usage_hourly` is NOT a safe proxy for "backfill done":
-- the triggers in 102 only enqueue dirty keys on agent_task_queue /
-- issue / task_usage DELETE — they do not write hourly rows themselves,
-- and they fire only on writes since 102 landed. A backfill that
-- crashed mid-run, or a manual `rollup_task_usage_hourly_window` call,
-- both leave the hourly table non-empty but with a stale watermark and
-- partial history; the legacy rollups would still be the only complete
-- read path.
--
-- A fresh database (no rows in task_usage) is exempt — there is no
-- history to backfill, and rollup_task_usage_hourly() (registered as a
-- pg_cron job by the operator) will populate task_usage_hourly from the
-- first event forward via its updated_at watermark window.
-- ---------------------------------------------------------------------------
DO $$
DECLARE
v_watermark TIMESTAMPTZ;
v_max_event TIMESTAMPTZ;
v_lag INTERVAL := INTERVAL '1 hour';
BEGIN
IF NOT EXISTS (SELECT 1 FROM task_usage LIMIT 1) THEN
RETURN;
END IF;
SELECT watermark_at INTO v_watermark
FROM task_usage_hourly_rollup_state
WHERE id = 1;
IF v_watermark IS NULL THEN
RAISE EXCEPTION
'refusing to drop legacy daily rollups: task_usage_hourly_rollup_state row is missing — apply migrations 100-102 first, then run cmd/backfill_task_usage_hourly';
END IF;
SELECT MAX(COALESCE(updated_at, created_at)) INTO v_max_event FROM task_usage;
-- A successful cmd/backfill_task_usage_hourly stamps watermark_at to
-- `now() - 5 min`; once pg_cron is registered the worker keeps it
-- within a similar window. If the watermark trails the latest event
-- by more than v_lag, one of these went wrong:
-- * the backfill was never run (watermark stuck at 1970-01-01),
-- * the backfill was interrupted before stampWatermark ran,
-- * someone hand-seeded task_usage_hourly without stamping,
-- * pg_cron has been off long enough to drift past the cap.
-- In every case, dropping the legacy rollups now would remove the
-- only read path for buckets the hourly pipeline has not proven it
-- owns yet.
IF v_watermark < v_max_event - v_lag THEN
RAISE EXCEPTION
'refusing to drop legacy daily rollups: task_usage_hourly_rollup_state.watermark_at (%) trails task_usage latest event (%) by more than % — backfill is incomplete or pg_cron is not running. Run cmd/backfill_task_usage_hourly (and let pg_cron catch up) before re-running migrate (see SELF-HOST UPGRADE ORDER in cmd/backfill_task_usage_hourly/main.go).',
v_watermark, v_max_event, v_lag;
END IF;
END
$$;
-- ---------------------------------------------------------------------------
-- Unschedule the legacy pg_cron jobs first (no-op when pg_cron is absent
-- or the job was never registered).
-- ---------------------------------------------------------------------------
DO $$
BEGIN
IF EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'pg_cron') THEN
PERFORM cron.unschedule('rollup_task_usage_daily')
FROM cron.job WHERE jobname = 'rollup_task_usage_daily';
PERFORM cron.unschedule('rollup_task_usage_dashboard_daily')
FROM cron.job WHERE jobname = 'rollup_task_usage_dashboard_daily';
END IF;
END
$$;
-- ---------------------------------------------------------------------------
-- task_usage_dashboard_daily pipeline (migration 084).
-- ---------------------------------------------------------------------------
DROP TRIGGER IF EXISTS trg_issue_project_dirty_dashboard ON issue;
DROP TRIGGER IF EXISTS trg_tu_dirty_dashboard ON task_usage;
DROP TRIGGER IF EXISTS trg_issue_delete_dirty_dashboard ON issue;
DROP TRIGGER IF EXISTS trg_atq_dirty_dashboard ON agent_task_queue;
DROP FUNCTION IF EXISTS task_usage_dashboard_rollup_lag_seconds();
DROP FUNCTION IF EXISTS rollup_task_usage_dashboard_daily();
DROP FUNCTION IF EXISTS rollup_task_usage_dashboard_daily_window(TIMESTAMPTZ, TIMESTAMPTZ);
DROP FUNCTION IF EXISTS enqueue_task_usage_dashboard_dirty_for_issue_project();
DROP FUNCTION IF EXISTS enqueue_task_usage_dashboard_dirty_for_issue_delete();
DROP FUNCTION IF EXISTS enqueue_task_usage_dashboard_dirty_for_tu();
DROP FUNCTION IF EXISTS enqueue_task_usage_dashboard_dirty_for_atq();
DROP TABLE IF EXISTS task_usage_dashboard_dirty;
DROP TABLE IF EXISTS task_usage_dashboard_rollup_state;
DROP TABLE IF EXISTS task_usage_dashboard_daily;
-- ---------------------------------------------------------------------------
-- task_usage_daily pipeline (migrations 073 / 077 / 082).
-- ---------------------------------------------------------------------------
DROP TRIGGER IF EXISTS trg_tu_dirty_rollup ON task_usage;
DROP TRIGGER IF EXISTS trg_atq_dirty_rollup ON agent_task_queue;
DROP FUNCTION IF EXISTS task_usage_rollup_lag_seconds();
DROP FUNCTION IF EXISTS rollup_task_usage_daily();
DROP FUNCTION IF EXISTS rollup_task_usage_daily_window(TIMESTAMPTZ, TIMESTAMPTZ);
DROP FUNCTION IF EXISTS enqueue_task_usage_daily_dirty_for_tu();
DROP FUNCTION IF EXISTS enqueue_task_usage_daily_dirty_for_atq();
DROP TABLE IF EXISTS task_usage_daily_dirty;
DROP TABLE IF EXISTS task_usage_rollup_state;
DROP TABLE IF EXISTS task_usage_daily;