Files
multica/server/internal/handler/autopilot_webhook.go
Bohan Jiang 2323b72710 feat(autopilots): webhook delivery layer + idempotency/signature/replay (MUL-2334) [PR1] (#2774)
* feat(autopilots): webhook delivery layer + idempotency / signature / replay (MUL-2334)

Splits "inbound webhook receipt" from "autopilot run creation" so we can
record duplicate attempts, signature outcomes, and ignored/skipped
deliveries — and replay a delivery on demand. v1 ingress wrote straight
into autopilot_run.trigger_payload, which collapsed the two concerns and
left run_only autopilots vulnerable to provider retry storms.

Backend only (PR1). UI Deliveries tab follows in PR2.

Schema (migration 093):
  - autopilot_trigger.provider: 'generic' | 'github' (default 'generic').
  - autopilot_trigger.signing_secret: nullable plaintext (HMAC needs it
    cleartext; mirrors how webhook_token is stored).
  - webhook_delivery: one row per inbound POST. Carries raw_body,
    selected_headers, dedupe_key/source, signature_status,
    autopilot_run_id, replayed_from_delivery_id, response_status / body.
  - Partial unique index on (trigger_id, dedupe_key) excludes NULL and
    'rejected' rows, so a wrong-secret 401 does NOT permanently block a
    future retry with the same X-GitHub-Delivery once the operator fixes
    the secret.

Ingress flow (autopilot_webhook.go), persist-first + sync dispatch:
  1. IP rate limit -> 2. token lookup -> 3. token rate limit ->
  4. read raw body -> 5. autopilot/workspace cross-check ->
  6. normalize JSON (400 without persistence on parse failure) ->
  7. compute dedupe key + signature status ->
  8. INSERT delivery (status=queued). On (trigger_id, dedupe_key)
     unique-violation: bump attempt_count on existing row and return
     the original delivery_id + autopilot_run_id with 200 ->
  9. invalid/missing signature: UPDATE -> rejected, return 401 with
     delivery_id (no dispatch, not replayable) ->
 10. trigger disabled / autopilot paused/archived: UPDATE -> ignored,
     return 200 ->
 11. DispatchAutopilot synchronously, UPDATE -> dispatched/skipped/failed
     with autopilot_run_id and the response body we returned ->
 12. TouchAutopilotTriggerFiredAt and return 200.

No new long-running worker. A stale 'queued' row only happens if the
process dies between INSERT and UPDATE; that's a follow-up sweeper, not
this PR.

Authenticated API:
  - GET    /api/autopilots/{id}/deliveries (slim list)
  - GET    /api/autopilots/{id}/deliveries/{deliveryId} (with raw_body)
  - POST   /api/autopilots/{id}/deliveries/{deliveryId}/replay -> creates
    a new delivery row (replayed_from_delivery_id set), dispatches a
    new run, never collapses onto the original via dedupe.
  - PUT    /api/autopilots/{id}/triggers/{triggerId}/signing-secret
    Write-only; trigger response surfaces has_signing_secret +
    signing_secret_hint (last 4 chars), never the secret itself.

Signature verification reuses the GitHub-compatible
X-Hub-Signature-256: sha256=<hex(hmac(body, secret))> scheme; the
HMAC helper is constant-time. Invalid/missing signatures still count
against per-IP and per-token rate limits.

autopilot_run.trigger_payload is intentionally preserved — delivery
records the HTTP receipt; run records the normalized envelope handed
to the agent. They are two different views.

Tests (Postgres-backed):
  - delivery persistence on accept
  - dedupe via Idempotency-Key and X-GitHub-Delivery; run_only retry
    storm pin (3 retries -> 1 run)
  - invalid signature: 401 + rejected row + no run linkage
  - missing signature when secret configured: 401 + 'missing' state
  - valid signature dispatches
  - signing secret never echoed in trigger responses; hint shows last 4
  - min-length and clear-by-empty for signing secret PUT
  - replay creates a NEW delivery + new run; rejected deliveries cannot
    be replayed
  - list omits raw_body; detail includes it; cross-autopilot ID returns
    404 (workspace isolation defense in depth)
  - provider validation: unknown -> 400, github -> 201 round-trips
  - bad-signature stream still counts against per-token rate limit

Co-authored-by: multica-agent <github@multica.ai>

* fix(autopilots): address PR review on webhook delivery layer (MUL-2334)

- Exclude `failed` from the (trigger_id, dedupe_key) partial unique index
  alongside `rejected`, so a transient ingress failure does not strand the
  provider's stable X-GitHub-Delivery / Idempotency-Key retry. Update the
  dedupe lookup to prefer non-terminal rows under the same predicate.
- Tighten delivery status enum: drop `skipped` from the CHECK constraint
  and from the handler. A run that was admission-skipped (e.g. runtime
  offline) is now recorded as delivery=`dispatched` linked to the
  skipped run, with the response payload carrying status=`skipped`.
  Source of truth for skipped-ness is autopilot_run.status, not the
  delivery row — keeps the Deliveries UI enum unambiguous.
- On dispatch error, link the (possibly non-nil) autopilot_run returned
  by DispatchAutopilot to the failed delivery so Deliveries UI can
  navigate to the run row for debugging.
- Slim list projection: ListWebhookDeliveriesByAutopilot no longer pulls
  raw_body / selected_headers / response_body — a 100-row page × 256 KiB
  would otherwise round-trip ~25 MiB from Postgres per Deliveries reload.
  Detail endpoint continues to return the full row.
- Fix backend CI: TestGetDelivery_ReturnsFullPayload now decodes the
  response and asserts on the parsed raw_body instead of substring-
  matching against an escaped JSON string; raise the test-suite default
  webhook rate limits in TestMain so the shared 192.0.2.1 IP bucket
  doesn't fill across the suite and leak 429s into unrelated tests.
- Add regression coverage for the dedupe-after-failure path.

cd server && go test ./... is green locally.

Co-authored-by: multica-agent <github@multica.ai>

---------

Co-authored-by: multica-agent <github@multica.ai>
2026-05-18 14:59:40 +08:00

787 lines
28 KiB
Go

package handler
import (
"crypto/hmac"
"crypto/rand"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"net/netip"
"strings"
"time"
"github.com/go-chi/chi/v5"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgtype"
"github.com/multica-ai/multica/server/internal/middleware"
db "github.com/multica-ai/multica/server/pkg/db/generated"
)
// maxWebhookBodyBytes is the request body size cap for webhook ingress.
// 256 KiB is plenty for normal provider webhooks (a max-size GitHub PR
// payload comes in well under this) and small enough that an attacker
// cannot wedge agent context windows by sending megabytes of arbitrary JSON.
const maxWebhookBodyBytes = 256 * 1024
// webhookTokenPrefix makes a leaked token recognisable in logs / audit trails
// without revealing the entropy bytes themselves. 32 random bytes encoded as
// URL-safe base64 (no padding) is 43 chars, so a full token is "awt_" + 43 = 47
// chars. URL-safe base64 keeps the token URL-friendly without escaping.
const webhookTokenPrefix = "awt_"
// generateWebhookToken returns a cryptographically random bearer token used as
// the public webhook URL secret. Format: "awt_" + URL-safe base64(32 bytes,
// no padding). UUIDs are intentionally not used here — they are lower entropy
// (122 bits vs 256) and visually overlap with internal IDs, which made
// accidental token-vs-ID confusion easy in early prototypes.
func generateWebhookToken() (string, error) {
b := make([]byte, 32)
if _, err := rand.Read(b); err != nil {
return "", fmt.Errorf("rand: %w", err)
}
return webhookTokenPrefix + base64.RawURLEncoding.EncodeToString(b), nil
}
// signature_status values mirror the CHECK constraint on webhook_delivery.
const (
sigStatusNotRequired = "not_required"
sigStatusValid = "valid"
sigStatusInvalid = "invalid"
sigStatusMissing = "missing"
)
// delivery status values mirror the CHECK constraint on webhook_delivery.
//
// "Duplicate" is a *response* status, not a delivery status — duplicates
// don't get their own row; they bump attempt_count on the existing dedupe
// target. Likewise "skipped" is a *response* status reported when the
// autopilot service skipped the run (e.g. runtime offline); the delivery
// row itself records `dispatched` and links the skipped run via
// autopilot_run_id, because from the ingress's perspective we DID hand
// the payload to the autopilot machinery.
const (
deliveryStatusQueued = "queued"
deliveryStatusDispatched = "dispatched"
deliveryStatusRejected = "rejected"
deliveryStatusIgnored = "ignored"
deliveryStatusFailed = "failed"
)
// ── Payload normalization ───────────────────────────────────────────────────
// WebhookEnvelope is the canonical shape stored in autopilot_run.trigger_payload
// and surfaced to the agent. The handler normalises arbitrary JSON bodies into
// this shape so downstream consumers (run_only daemon prompt, create_issue
// description appendix) can rely on a stable schema regardless of which
// provider sent the webhook.
type WebhookEnvelope struct {
Event string `json:"event"`
EventPayload json.RawMessage `json:"eventPayload"`
Request WebhookRequest `json:"request"`
}
type WebhookRequest struct {
ReceivedAt string `json:"receivedAt"`
ContentType string `json:"contentType,omitempty"`
}
// normalizeWebhookPayload parses an incoming webhook body and returns a
// WebhookEnvelope. Rules:
//
// 1. Body must be a valid JSON object or array. Scalars / invalid JSON
// return an error so the handler can respond 400.
// 2. If the body is an object containing a string `event` and any
// `eventPayload`, those are preserved as-is.
// 3. Otherwise `event` is inferred from headers/body fields, and the entire
// original body becomes `eventPayload`.
// 4. The default event is `webhook.received`.
//
// Inference order:
//
// X-GitHub-Event (combined with body.action when present),
// X-Gitlab-Event, X-Event-Type, body.event, body.type, body.action.
func normalizeWebhookPayload(body []byte, headers http.Header) (WebhookEnvelope, error) {
body = stripBOM(body)
if len(body) == 0 {
return WebhookEnvelope{}, errors.New("empty body")
}
// First, validate JSON shape (object or array). Reject scalars early —
// `"hello"` is technically valid JSON but has no useful interpretation
// as a webhook payload and would land in the agent prompt as a bare
// string.
var asAny any
if err := json.Unmarshal(body, &asAny); err != nil {
return WebhookEnvelope{}, fmt.Errorf("invalid json: %w", err)
}
switch asAny.(type) {
case map[string]any, []any:
// ok
default:
return WebhookEnvelope{}, errors.New("body must be a JSON object or array")
}
now := time.Now().UTC().Format(time.RFC3339)
contentType := headers.Get("Content-Type")
if i := strings.Index(contentType, ";"); i >= 0 {
contentType = strings.TrimSpace(contentType[:i])
}
env := WebhookEnvelope{
Request: WebhookRequest{
ReceivedAt: now,
ContentType: contentType,
},
}
// 1. Caller-provided envelope.
if obj, ok := asAny.(map[string]any); ok {
if eventStr, ok := obj["event"].(string); ok && eventStr != "" {
if rawPayload, ok := obj["eventPayload"]; ok {
inner, err := json.Marshal(rawPayload)
if err == nil {
env.Event = eventStr
env.EventPayload = inner
return env, nil
}
}
// `event` present but no eventPayload: still preserve event
// string, fall through to use whole body as payload.
env.Event = eventStr
env.EventPayload = json.RawMessage(body)
return env, nil
}
}
// 2. Inferred event.
event := inferEvent(headers, asAny)
env.Event = event
env.EventPayload = json.RawMessage(body)
return env, nil
}
// inferEvent returns a best-effort event identifier from headers and body.
func inferEvent(headers http.Header, body any) string {
if gh := headers.Get("X-GitHub-Event"); gh != "" {
if obj, ok := body.(map[string]any); ok {
if action, ok := obj["action"].(string); ok && action != "" {
return "github." + gh + "." + action
}
}
return "github." + gh
}
if gl := headers.Get("X-Gitlab-Event"); gl != "" {
return "gitlab." + gl
}
if xe := headers.Get("X-Event-Type"); xe != "" {
return xe
}
if obj, ok := body.(map[string]any); ok {
if e, ok := obj["event"].(string); ok && e != "" {
return e
}
if t, ok := obj["type"].(string); ok && t != "" {
return t
}
if a, ok := obj["action"].(string); ok && a != "" {
return a
}
}
return "webhook.received"
}
// stripBOM removes a leading UTF-8 byte-order-mark, which some clients
// (notably PowerShell-based scripts) prepend to JSON bodies.
func stripBOM(b []byte) []byte {
if len(b) >= 3 && b[0] == 0xEF && b[1] == 0xBB && b[2] == 0xBF {
return b[3:]
}
return b
}
// ── Dedupe + signature helpers ──────────────────────────────────────────────
// extractDedupeKey returns the provider-specific idempotency identifier from
// request headers, plus a short tag naming the header it came from. Returns
// ("", "") when no recognised header is present.
//
// github -> X-GitHub-Delivery
// generic -> Idempotency-Key
//
// Other providers fall back to the generic header to keep manual replays from
// Postman / curl behaving the same way regardless of trigger config.
func extractDedupeKey(provider string, headers http.Header) (string, string) {
if v := strings.TrimSpace(headers.Get("X-GitHub-Delivery")); v != "" && provider == "github" {
return v, "x-github-delivery"
}
if v := strings.TrimSpace(headers.Get("Idempotency-Key")); v != "" {
return v, "idempotency-key"
}
if v := strings.TrimSpace(headers.Get("X-GitHub-Delivery")); v != "" {
return v, "x-github-delivery"
}
return "", ""
}
// verifyWebhookSignatureForProvider returns one of sigStatus* describing the
// outcome of HMAC verification for the configured trigger.
//
// When no signing secret is configured the result is `not_required` — the
// trigger has opted into bearer-token-only authentication. When a secret IS
// configured the request must carry the expected header; otherwise the
// outcome is `missing` (caller still records a rejected delivery).
//
// github -> X-Hub-Signature-256: sha256=<hex>
// generic -> X-Hub-Signature-256 (same shape; lets curl/Postman opt in)
func verifyWebhookSignatureForProvider(provider, secret string, headers http.Header, rawBody []byte) string {
if secret == "" {
return sigStatusNotRequired
}
sig := headers.Get("X-Hub-Signature-256")
if sig == "" {
return sigStatusMissing
}
if !verifyHubSignature(secret, sig, rawBody) {
return sigStatusInvalid
}
_ = provider
return sigStatusValid
}
// verifyHubSignature implements the GitHub-compatible HMAC-SHA256 scheme:
// `X-Hub-Signature-256: sha256=<hex(hmac(body, secret))>`. The hmac.Equal
// comparison is constant-time so partial-prefix attacks cannot leak timing.
func verifyHubSignature(secret, header string, body []byte) bool {
const prefix = "sha256="
if !strings.HasPrefix(header, prefix) {
return false
}
want, err := hex.DecodeString(strings.TrimPrefix(header, prefix))
if err != nil {
return false
}
mac := hmac.New(sha256.New, []byte(secret))
mac.Write(body)
return hmac.Equal(mac.Sum(nil), want)
}
// selectedHeadersJSON returns the small, debugging-friendly subset of request
// headers we persist on a delivery row. Signature header is recorded as
// present/absent only — never the value, so a delivery dump cannot leak the
// HMAC of a sensitive body.
func selectedHeadersJSON(headers http.Header) []byte {
out := map[string]any{}
add := func(name string) {
if v := headers.Get(name); v != "" {
out[strings.ToLower(name)] = v
}
}
add("User-Agent")
add("X-GitHub-Event")
add("X-GitHub-Delivery")
add("X-Gitlab-Event")
add("X-Event-Type")
add("Idempotency-Key")
if v := headers.Get("X-Hub-Signature-256"); v != "" {
out["x-hub-signature-256-present"] = true
}
b, err := json.Marshal(out)
if err != nil {
return []byte("{}")
}
return b
}
// ── Public ingress ──────────────────────────────────────────────────────────
// HandleAutopilotWebhook is the public entry point for webhook-triggered
// autopilots. It runs OUTSIDE the authenticated route group: the bearer
// token in the URL path IS the credential.
//
// Flow (persist-first, sync-dispatch):
//
// 1. Per-IP rate limit (gate before any DB I/O).
// 2. Token lookup. ErrNoRows → 404; other DB errors → 500.
// 3. Per-token rate limit.
// 4. Read raw body (capped). Oversized → 413.
// 5. Normalize JSON envelope. Invalid → 400 (no persistence — there is no
// dedupe identifier we can trust from an unparsable body).
// 6. Extract dedupe key from headers per provider.
// 7. Verify signature (or `not_required` when no secret is configured).
// 8. INSERT webhook_delivery row (status=queued). On dedupe collision (23505
// against `(trigger_id, dedupe_key)`) treat as duplicate: bump
// attempt_count on the existing row and return its delivery_id +
// autopilot_run_id with 200.
// 9. If signature invalid/missing: UPDATE delivery → rejected, return 401.
// 10. If trigger disabled / autopilot paused / archived: UPDATE delivery →
// ignored, return 200.
// 11. Dispatch the autopilot synchronously. UPDATE delivery → dispatched
// (with autopilot_run_id) or failed. Return 200 (skipped runs surface
// their `reason`).
// 12. Bump last_fired_at after dispatch — even on the skipped path — so the
// trigger's "last seen" is accurate.
//
// Response shapes:
// - 200 {"status":"accepted", "delivery_id", "run_id", "autopilot_id", "trigger_id"}
// - 200 {"status":"skipped", "delivery_id", "run_id", "reason"}
// - 200 {"status":"ignored", "delivery_id", "reason"}
// - 200 {"status":"duplicate", "delivery_id", "run_id?"}
// - 400 {"error":"..."} — invalid JSON / scalar / empty
// - 401 {"status":"rejected", "delivery_id", "reason":"..."} — signature failure
// - 404 {"error":"webhook not found"} — unknown token
// - 413 {"error":"payload too large"} — body exceeded cap
// - 429 {"error":"rate limit exceeded"} — over per-IP/token budget
// - 500 {"error":"..."} — internal failure
func (h *Handler) HandleAutopilotWebhook(w http.ResponseWriter, r *http.Request) {
token := chi.URLParam(r, "token")
if token == "" {
writeError(w, http.StatusNotFound, "webhook not found")
return
}
// 1. Per-IP rate limit BEFORE we hit Postgres. Bounds the DB-probe blast
// radius for an attacker spraying random tokens. A spray of bad
// signatures still counts here — fast-path 429 stops budget burn.
if h.WebhookIPRateLimiter != nil {
if ip := h.clientIPForRateLimit(r); ip != "" {
if !h.WebhookIPRateLimiter.Allow(r.Context(), ip) {
writeError(w, http.StatusTooManyRequests, "rate limit exceeded")
return
}
}
}
// 2. Token lookup. Distinguish "no row" from "DB error": collapsing both
// to 404 means a transient DB blip silently drops real deliveries
// (providers like GitHub don't retry on 404). For no-row we still
// return a generic message so we don't leak which tokens existed.
trigRow, err := h.Queries.GetWebhookTriggerByToken(r.Context(), pgtype.Text{String: token, Valid: true})
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
writeError(w, http.StatusNotFound, "webhook not found")
return
}
slog.Error("webhook: token lookup failed", "error", err)
writeError(w, http.StatusInternalServerError, "internal error")
return
}
middleware.SetWebhookTriggerID(r, uuidToString(trigRow.ID))
// 3. Per-token rate limit.
if h.WebhookRateLimiter != nil {
if !h.WebhookRateLimiter.Allow(r.Context(), token) {
writeError(w, http.StatusTooManyRequests, "rate limit exceeded")
return
}
}
// 4. Body size cap + JSON validation. http.MaxBytesReader stops the read
// mid-stream once the cap is exceeded so an oversized payload is
// rejected before being fully buffered.
r.Body = http.MaxBytesReader(w, r.Body, maxWebhookBodyBytes)
body, err := io.ReadAll(r.Body)
if err != nil {
var mbe *http.MaxBytesError
if errors.As(err, &mbe) {
writeError(w, http.StatusRequestEntityTooLarge, "payload too large")
return
}
writeError(w, http.StatusBadRequest, "failed to read request body")
return
}
// 5. Cross-check autopilot/workspace consistency BEFORE we persist the
// delivery — webhook_delivery.workspace_id is NOT NULL and a stale FK
// row would otherwise fail INSERT after we've already paid the body
// read. Same ErrNoRows-vs-DB-error split as token lookup.
autopilot, err := h.Queries.GetAutopilot(r.Context(), trigRow.AutopilotID)
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
writeError(w, http.StatusNotFound, "webhook not found")
return
}
slog.Error("webhook: autopilot lookup failed",
"error", err,
"trigger_id", uuidToString(trigRow.ID),
)
writeError(w, http.StatusInternalServerError, "internal error")
return
}
if uuidToString(autopilot.WorkspaceID) != uuidToString(trigRow.AutopilotWorkspaceID) {
slog.Warn("webhook: trigger workspace mismatch",
"trigger_id", uuidToString(trigRow.ID),
"autopilot_id", uuidToString(autopilot.ID),
)
writeError(w, http.StatusNotFound, "webhook not found")
return
}
// 6. Normalize body. Invalid JSON → 400 without persistence: we have no
// dedupe identifier from the body, and replaying an unparsable payload
// is not useful.
envelope, err := normalizeWebhookPayload(body, r.Header)
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
envelopeBytes, err := json.Marshal(envelope)
if err != nil {
writeError(w, http.StatusInternalServerError, "failed to encode envelope")
return
}
// 7. Provider + dedupe + signature.
provider := trigRow.Provider
if provider == "" {
provider = "generic"
}
dedupeKey, dedupeSource := extractDedupeKey(provider, r.Header)
sigStatus := verifyWebhookSignatureForProvider(provider, trigRow.SigningSecret.String, r.Header, body)
// 8. Persist (INSERT delivery). Dedupe collision → bump existing row.
delivery, dup, err := h.persistInboundDelivery(r, persistDeliveryInput{
WorkspaceID: autopilot.WorkspaceID,
AutopilotID: autopilot.ID,
TriggerID: trigRow.ID,
Provider: provider,
Event: envelope.Event,
DedupeKey: dedupeKey,
DedupeSource: dedupeSource,
SignatureStatus: sigStatus,
ContentType: envelope.Request.ContentType,
RawBody: body,
SelectedHeaders: selectedHeadersJSON(r.Header),
})
if err != nil {
slog.Error("webhook: persist delivery failed",
"error", err,
"trigger_id", uuidToString(trigRow.ID),
)
writeError(w, http.StatusInternalServerError, "internal error")
return
}
if dup {
// A previous delivery already covered this dedupe key. Return the
// original delivery_id + (possibly empty) run_id with 200 so the
// caller can correlate.
resp := map[string]any{
"status": "duplicate",
"delivery_id": uuidToString(delivery.ID),
}
if delivery.AutopilotRunID.Valid {
resp["run_id"] = uuidToString(delivery.AutopilotRunID)
}
writeJSON(w, http.StatusOK, resp)
return
}
// 9. Signature failure → rejected delivery + 401. No dispatch, no replay.
// Providers will look for 4xx feedback when their secret is wrong.
if sigStatus == sigStatusInvalid || sigStatus == sigStatusMissing {
reason := "invalid_signature"
if sigStatus == sigStatusMissing {
reason = "missing_signature"
}
respBody := map[string]any{
"status": "rejected",
"delivery_id": uuidToString(delivery.ID),
"reason": reason,
}
h.finaliseDeliveryTerminal(r, delivery.ID, deliveryStatusRejected, http.StatusUnauthorized, respBody, reason)
writeJSON(w, http.StatusUnauthorized, respBody)
return
}
// 10. Trigger disabled / autopilot paused / archived → ignored. We return
// 200 so the sender's webhook-retry machinery doesn't keep hammering
// us; the "ignored" status + delivery row makes the no-op visible if
// the operator inspects the delivery log.
if !trigRow.Enabled {
respBody := map[string]any{"status": "ignored", "delivery_id": uuidToString(delivery.ID), "reason": "trigger_disabled"}
h.finaliseDeliveryTerminal(r, delivery.ID, deliveryStatusIgnored, http.StatusOK, respBody, "trigger_disabled")
writeJSON(w, http.StatusOK, respBody)
return
}
if autopilot.Status == "archived" {
respBody := map[string]any{"status": "ignored", "delivery_id": uuidToString(delivery.ID), "reason": "autopilot_archived"}
h.finaliseDeliveryTerminal(r, delivery.ID, deliveryStatusIgnored, http.StatusOK, respBody, "autopilot_archived")
writeJSON(w, http.StatusOK, respBody)
return
}
if autopilot.Status != "active" {
respBody := map[string]any{"status": "ignored", "delivery_id": uuidToString(delivery.ID), "reason": "autopilot_paused"}
h.finaliseDeliveryTerminal(r, delivery.ID, deliveryStatusIgnored, http.StatusOK, respBody, "autopilot_paused")
writeJSON(w, http.StatusOK, respBody)
return
}
// 11. Dispatch synchronously. DispatchAutopilot publishes WS events,
// persists trigger_payload on autopilot_run, runs the admission
// check (offline runtime → skipped), and bumps last_run_at.
run, err := h.AutopilotService.DispatchAutopilot(
r.Context(),
autopilot,
trigRow.ID,
"webhook",
envelopeBytes,
)
if err != nil {
slog.Warn("webhook dispatch failed",
"trigger_id", uuidToString(trigRow.ID),
"autopilot_id", uuidToString(autopilot.ID),
"error", err,
)
respBody := map[string]any{"error": "failed to dispatch autopilot"}
// DispatchAutopilot may return a non-nil run alongside an error
// (e.g. when the run row was created but the downstream dispatch
// failed). Link the run on the delivery anyway so the Deliveries
// UI can show which run row corresponds to the failure.
if run != nil {
h.finaliseDeliveryWithRun(r, delivery.ID, deliveryStatusFailed, run.ID, http.StatusInternalServerError, respBody)
} else {
h.finaliseDeliveryTerminal(r, delivery.ID, deliveryStatusFailed, http.StatusInternalServerError, respBody, err.Error())
}
writeJSON(w, http.StatusInternalServerError, respBody)
return
}
// 12. Bump last_fired_at after dispatch returns — including the skipped
// path — so paused early-returns above don't corrupt "last fired".
if err := h.Queries.TouchAutopilotTriggerFiredAt(r.Context(), trigRow.ID); err != nil {
slog.Warn("webhook: failed to touch last_fired_at",
"trigger_id", uuidToString(trigRow.ID),
"error", err,
)
}
// 13. Persist the linkage delivery → run.
//
// The delivery row is always `dispatched` once we reach here: from the
// ingress's perspective we handed the payload off to the autopilot
// machinery and got a run id back. The autopilot may have skipped the
// run (e.g. runtime offline) — that's reflected in the response status
// + reason and in the linked run row, not in the delivery status. This
// keeps the delivery enum tight and the Deliveries UI unambiguous
// (`run.status` is the source of truth for what the run did).
respBody := map[string]any{
"status": "accepted",
"delivery_id": uuidToString(delivery.ID),
"run_id": uuidToString(run.ID),
"autopilot_id": uuidToString(autopilot.ID),
"trigger_id": uuidToString(trigRow.ID),
}
if run.Status == "skipped" {
respBody = map[string]any{
"status": "skipped",
"delivery_id": uuidToString(delivery.ID),
"run_id": uuidToString(run.ID),
}
if run.FailureReason.Valid {
respBody["reason"] = run.FailureReason.String
}
}
h.finaliseDeliveryWithRun(r, delivery.ID, deliveryStatusDispatched, run.ID, http.StatusOK, respBody)
writeJSON(w, http.StatusOK, respBody)
}
// ── Persistence helpers ─────────────────────────────────────────────────────
type persistDeliveryInput struct {
WorkspaceID pgtype.UUID
AutopilotID pgtype.UUID
TriggerID pgtype.UUID
Provider string
Event string
DedupeKey string
DedupeSource string
SignatureStatus string
ContentType string
RawBody []byte
SelectedHeaders []byte
}
// persistInboundDelivery INSERTs a fresh `queued` delivery, returning (row,
// false, nil) on the happy path. On dedupe-key unique-violation it returns
// (existing-row, true, nil) after bumping attempt_count on the prior row.
// Any other error bubbles up so the handler can 500 cleanly.
func (h *Handler) persistInboundDelivery(r *http.Request, in persistDeliveryInput) (db.WebhookDelivery, bool, error) {
params := db.CreateWebhookDeliveryParams{
WorkspaceID: in.WorkspaceID,
AutopilotID: in.AutopilotID,
TriggerID: in.TriggerID,
Provider: in.Provider,
Event: in.Event,
SignatureStatus: in.SignatureStatus,
Status: deliveryStatusQueued,
SelectedHeaders: in.SelectedHeaders,
RawBody: in.RawBody,
}
if in.DedupeKey != "" {
params.DedupeKey = pgtype.Text{String: in.DedupeKey, Valid: true}
params.DedupeSource = pgtype.Text{String: in.DedupeSource, Valid: true}
}
if in.ContentType != "" {
params.ContentType = pgtype.Text{String: in.ContentType, Valid: true}
}
delivery, err := h.Queries.CreateWebhookDelivery(r.Context(), params)
if err == nil {
return delivery, false, nil
}
if !isUniqueViolation(err) || in.DedupeKey == "" {
return db.WebhookDelivery{}, false, err
}
// Dedupe collision: fetch the original row, bump attempt count.
existing, lookupErr := h.Queries.GetWebhookDeliveryByTriggerAndDedupe(r.Context(), db.GetWebhookDeliveryByTriggerAndDedupeParams{
TriggerID: in.TriggerID,
DedupeKey: pgtype.Text{String: in.DedupeKey, Valid: true},
})
if lookupErr != nil {
return db.WebhookDelivery{}, false, fmt.Errorf("lookup duplicate delivery: %w", lookupErr)
}
bumped, bumpErr := h.Queries.BumpWebhookDeliveryAttempt(r.Context(), existing.ID)
if bumpErr != nil {
// Still treat as duplicate; just log the bump failure so the
// operator can investigate, returning the row we DID read.
slog.Warn("webhook: failed to bump attempt_count",
"delivery_id", uuidToString(existing.ID),
"error", bumpErr,
)
return existing, true, nil
}
return bumped, true, nil
}
// finaliseDeliveryTerminal records a non-dispatched outcome (rejected,
// ignored, failed). HTTP status and full response body are captured so a
// future Deliveries UI can show exactly what we returned.
func (h *Handler) finaliseDeliveryTerminal(
r *http.Request,
id pgtype.UUID,
status string,
httpStatus int,
responseBody any,
errMsg string,
) {
bodyJSON, _ := json.Marshal(responseBody)
params := db.UpdateWebhookDeliveryTerminalParams{
ID: id,
Status: status,
ResponseStatus: pgtype.Int4{Int32: int32(httpStatus), Valid: true},
ResponseBody: pgtype.Text{String: string(bodyJSON), Valid: true},
}
if errMsg != "" {
params.Error = pgtype.Text{String: errMsg, Valid: true}
}
if _, err := h.Queries.UpdateWebhookDeliveryTerminal(r.Context(), params); err != nil {
slog.Warn("webhook: finalise terminal failed",
"delivery_id", uuidToString(id),
"status", status,
"error", err,
)
}
}
// finaliseDeliveryWithRun records a delivery that produced (or was admission-
// skipped to) an autopilot_run. Same response-capture as the terminal path.
func (h *Handler) finaliseDeliveryWithRun(
r *http.Request,
id pgtype.UUID,
status string,
runID pgtype.UUID,
httpStatus int,
responseBody any,
) {
bodyJSON, _ := json.Marshal(responseBody)
params := db.UpdateWebhookDeliveryDispatchedParams{
ID: id,
Status: status,
AutopilotRunID: runID,
ResponseStatus: pgtype.Int4{Int32: int32(httpStatus), Valid: true},
ResponseBody: pgtype.Text{String: string(bodyJSON), Valid: true},
}
if _, err := h.Queries.UpdateWebhookDeliveryDispatched(r.Context(), params); err != nil {
slog.Warn("webhook: finalise with run failed",
"delivery_id", uuidToString(id),
"run_id", uuidToString(runID),
"error", err,
)
}
}
// ── Rate-limit / IP plumbing ────────────────────────────────────────────────
// clientIPForRateLimit returns the IP used as a rate-limit bucket key.
//
// Default behaviour: use the host portion of r.RemoteAddr. Forwarded
// headers (X-Forwarded-For, X-Real-IP) are IGNORED unless the operator
// has explicitly opted in via MULTICA_TRUSTED_PROXIES — and even then
// only when r.RemoteAddr is itself inside one of the listed CIDRs.
func (h *Handler) clientIPForRateLimit(r *http.Request) string {
remoteIP := remoteAddrHost(r.RemoteAddr)
if len(h.cfg.TrustedProxies) == 0 {
return remoteIP
}
remoteAddr, ok := parseNetIPAddr(remoteIP)
if !ok || !addrInPrefixes(remoteAddr, h.cfg.TrustedProxies) {
// Source isn't a trusted proxy — headers can't be believed.
return remoteIP
}
if xff := r.Header.Get("X-Forwarded-For"); xff != "" {
if i := strings.IndexByte(xff, ','); i >= 0 {
return strings.TrimSpace(xff[:i])
}
return strings.TrimSpace(xff)
}
if xri := r.Header.Get("X-Real-IP"); xri != "" {
return strings.TrimSpace(xri)
}
return remoteIP
}
func remoteAddrHost(remote string) string {
if remote == "" {
return ""
}
if strings.HasPrefix(remote, "[") {
if end := strings.IndexByte(remote, ']'); end > 0 {
return remote[1:end]
}
}
if i := strings.LastIndexByte(remote, ':'); i >= 0 && !strings.Contains(remote, "]") {
if strings.Count(remote, ":") == 1 {
return remote[:i]
}
}
return remote
}
func parseNetIPAddr(s string) (netip.Addr, bool) {
if s == "" {
return netip.Addr{}, false
}
addr, err := netip.ParseAddr(s)
if err != nil {
return netip.Addr{}, false
}
return addr.Unmap(), true
}
func addrInPrefixes(addr netip.Addr, prefixes []netip.Prefix) bool {
for _, p := range prefixes {
if p.Contains(addr) {
return true
}
}
return false
}