mirror of
https://github.com/multica-ai/multica.git
synced 2026-07-27 04:56:20 +02:00
fix(timezone): use watermark coverage for hourly-rollup guard
The previous check only required `task_usage_hourly` to be non-empty, which an interrupted backfill or a manual `rollup_task_usage_hourly_window` call both satisfy. The completion signal we actually trust is `task_usage_hourly_rollup_state.watermark_at` — backfill only stamps it to `now() - 5 min` after every monthly slice succeeded, and the cron worker only advances it on a real tick. Default after migration 101 is `1970-01-01`, so an unrun or partial backfill is trivially detected. Also corrects the comment about fresh-install behavior: the triggers in 102 only enqueue dirty keys for agent_task_queue / issue / task_usage DELETE — they do not write hourly rows. INSERT/UPDATE flows through the `updated_at` watermark window of `rollup_task_usage_hourly()`, which only runs once the operator registers it as a pg_cron job. Co-authored-by: multica-agent <github@multica.ai>
This commit is contained in:
@@ -21,8 +21,8 @@
|
||||
-- (same guard pattern as migration 076).
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- Fail-closed guard: refuse to drop the legacy daily pipelines while the
|
||||
-- new hourly rollup is empty on a database that already has token-event
|
||||
-- Fail-closed guard: refuse to drop the legacy daily pipelines unless the
|
||||
-- hourly rollup has actually been seeded across the existing `task_usage`
|
||||
-- history. `migrate up` runs the migration set straight through (no
|
||||
-- per-version stop), so a self-host operator who skips the documented
|
||||
-- backfill step would otherwise silently land in a state where dashboards
|
||||
@@ -31,17 +31,66 @@
|
||||
-- docs/timezone-architecture-rfc.md §6 / §7.1). Failing loud here is the
|
||||
-- only thing that turns an undetected outage into a clear migration error.
|
||||
--
|
||||
-- A fresh database (no rows in task_usage) is exempt — there is nothing to
|
||||
-- backfill, and the new triggers installed in 102 will populate
|
||||
-- task_usage_hourly on the first event.
|
||||
-- The completion signal we trust is task_usage_hourly_rollup_state.watermark_at:
|
||||
--
|
||||
-- * cmd/backfill_task_usage_hourly stamps it to `now() - 5 minutes` only
|
||||
-- after every monthly slice succeeded (server/cmd/backfill_task_usage_hourly/main.go).
|
||||
-- * rollup_task_usage_hourly() (the pg_cron entry) advances it on every
|
||||
-- tick (102_task_usage_hourly_pipeline.up.sql).
|
||||
-- * The default after migration 101 is `1970-01-01`, so an unrun or
|
||||
-- interrupted backfill is trivially detected.
|
||||
--
|
||||
-- A non-empty `task_usage_hourly` is NOT a safe proxy for "backfill done":
|
||||
-- the triggers in 102 only enqueue dirty keys on agent_task_queue /
|
||||
-- issue / task_usage DELETE — they do not write hourly rows themselves,
|
||||
-- and they fire only on writes since 102 landed. A backfill that
|
||||
-- crashed mid-run, or a manual `rollup_task_usage_hourly_window` call,
|
||||
-- both leave the hourly table non-empty but with a stale watermark and
|
||||
-- partial history; the legacy rollups would still be the only complete
|
||||
-- read path.
|
||||
--
|
||||
-- A fresh database (no rows in task_usage) is exempt — there is no
|
||||
-- history to backfill, and rollup_task_usage_hourly() (registered as a
|
||||
-- pg_cron job by the operator) will populate task_usage_hourly from the
|
||||
-- first event forward via its updated_at watermark window.
|
||||
-- ---------------------------------------------------------------------------
|
||||
|
||||
DO $$
|
||||
DECLARE
|
||||
v_watermark TIMESTAMPTZ;
|
||||
v_max_event TIMESTAMPTZ;
|
||||
v_lag INTERVAL := INTERVAL '1 hour';
|
||||
BEGIN
|
||||
IF EXISTS (SELECT 1 FROM task_usage LIMIT 1)
|
||||
AND NOT EXISTS (SELECT 1 FROM task_usage_hourly LIMIT 1) THEN
|
||||
IF NOT EXISTS (SELECT 1 FROM task_usage LIMIT 1) THEN
|
||||
RETURN;
|
||||
END IF;
|
||||
|
||||
SELECT watermark_at INTO v_watermark
|
||||
FROM task_usage_hourly_rollup_state
|
||||
WHERE id = 1;
|
||||
|
||||
IF v_watermark IS NULL THEN
|
||||
RAISE EXCEPTION
|
||||
'refusing to drop legacy daily rollups: task_usage_hourly is empty but task_usage has data — apply migrations 100-102 first, then run cmd/backfill_task_usage_hourly, then re-run migrate (see SELF-HOST UPGRADE ORDER in cmd/backfill_task_usage_hourly/main.go)';
|
||||
'refusing to drop legacy daily rollups: task_usage_hourly_rollup_state row is missing — apply migrations 100-102 first, then run cmd/backfill_task_usage_hourly';
|
||||
END IF;
|
||||
|
||||
SELECT MAX(COALESCE(updated_at, created_at)) INTO v_max_event FROM task_usage;
|
||||
|
||||
-- A successful cmd/backfill_task_usage_hourly stamps watermark_at to
|
||||
-- `now() - 5 min`; once pg_cron is registered the worker keeps it
|
||||
-- within a similar window. If the watermark trails the latest event
|
||||
-- by more than v_lag, one of these went wrong:
|
||||
-- * the backfill was never run (watermark stuck at 1970-01-01),
|
||||
-- * the backfill was interrupted before stampWatermark ran,
|
||||
-- * someone hand-seeded task_usage_hourly without stamping,
|
||||
-- * pg_cron has been off long enough to drift past the cap.
|
||||
-- In every case, dropping the legacy rollups now would remove the
|
||||
-- only read path for buckets the hourly pipeline has not proven it
|
||||
-- owns yet.
|
||||
IF v_watermark < v_max_event - v_lag THEN
|
||||
RAISE EXCEPTION
|
||||
'refusing to drop legacy daily rollups: task_usage_hourly_rollup_state.watermark_at (%) trails task_usage latest event (%) by more than % — backfill is incomplete or pg_cron is not running. Run cmd/backfill_task_usage_hourly (and let pg_cron catch up) before re-running migrate (see SELF-HOST UPGRADE ORDER in cmd/backfill_task_usage_hourly/main.go).',
|
||||
v_watermark, v_max_event, v_lag;
|
||||
END IF;
|
||||
END
|
||||
$$;
|
||||
|
||||
Reference in New Issue
Block a user