Files
multica/server/internal/daemon/reconcile.go
xiawiie 93028d303b fix(daemon): reconcile in-flight task and workspace state on WS reconnect (#4718)
After a WebSocket disconnect, the daemon's view of running tasks and
workspace state can lag the server for up to 5s (per-task cancellation
poll) or 30s (workspace sync) because both loops park on coarse tickers
that do not observe the WS wakeup channel.

This change adds a small fan-out broadcaster (`reconcileBroadcaster`)
that the WS connect path fires once per (re)connect. `watchTaskCancellation`
and `workspaceSyncLoop` subscribe and re-check immediately on broadcast,
without disturbing the ticker cadence. The broadcaster is edge-triggered
with a one-slot replay so a broadcast that lands before a subscriber is
ready is not lost (closes the daemon-startup race), and back-to-back
broadcasts inside 1s are debounced so a flapping connection cannot fan
out into a request stampede.

Existing behaviour is preserved: shouldInterruptAgent still decides
whether to interrupt, the 5s/30s ticker still bounds the worst case,
and the WS heartbeat / HTTP heartbeat coordination is untouched.

Closes #4665
2026-06-30 13:50:18 +08:00

99 lines
3.8 KiB
Go

package daemon
import (
"sync"
"time"
)
// reconcileBroadcaster fans out a "reconcile now" signal to any number of
// listeners using the close-and-replace channel pattern. Subscribers call
// notify() to obtain a snapshot channel that closes on the next broadcast;
// after waking, a subscriber re-acquires a fresh snapshot via notify() to
// receive the next signal.
//
// The broadcaster exists because some daemon loops run on coarse tickers
// (task-cancellation polling at 5s, workspace sync at 30s). When the daemon's
// WS connection drops and reconnects, anything the server changed during the
// gap is invisible to those loops until their next tick fires. broadcast()
// lets the WS connect path nudge every waiter to re-check immediately,
// without disturbing the ticker cadence.
//
// The broadcaster is edge-triggered with one-slot replay: if broadcast()
// fires while nobody is subscribed, the next notify() call returns an
// already-closed channel so the late subscriber observes the missed event
// exactly once. This closes the daemon-startup race where workspaceSyncLoop
// (or another late subscriber) parks on its ticker just after a broadcast
// landed during the WS connect, and would otherwise wait a full ticker
// period to learn about it.
//
// broadcast() debounces back-to-back calls inside minBroadcastInterval so a
// flapping WS connection cannot translate a network blip into a stampede of
// GetTaskStatus / ListWorkspaces requests. The interval is intentionally
// short: a real reconnect still converts quickly, but ten reconnects in a
// second collapse into one immediate fan-out plus, at most, one follow-up
// after the interval elapses.
type reconcileBroadcaster struct {
mu sync.Mutex
ch chan struct{}
pending bool
lastBroadcast time.Time
minBroadcastInterval time.Duration
// now is injected in tests to make the debounce window deterministic;
// production code uses time.Now.
now func() time.Time
}
func newReconcileBroadcaster() *reconcileBroadcaster {
return &reconcileBroadcaster{
ch: make(chan struct{}),
minBroadcastInterval: time.Second,
now: time.Now,
}
}
// notify returns a channel that closes on the next broadcast. Subscribers
// should call notify() again after waking to receive the next signal — the
// returned channel is single-shot.
//
// If a broadcast arrived while there were no subscribers, the channel
// returned by the next notify() call is already closed. The replay flag is
// cleared by that call: a second concurrent late subscriber does NOT see
// the same replayed event. Callers that need broadcast delivery to every
// goroutine must subscribe before the broadcast happens.
func (b *reconcileBroadcaster) notify() <-chan struct{} {
b.mu.Lock()
defer b.mu.Unlock()
if b.pending {
// Replay the missed broadcast exactly once: hand back a fresh,
// already-closed channel without disturbing b.ch for real
// subscribers, and clear pending so the next notify() resumes
// edge-triggered behaviour.
b.pending = false
replay := make(chan struct{})
close(replay)
return replay
}
return b.ch
}
// broadcast wakes every current subscriber, then installs a fresh channel
// for future subscribers. If there are no current subscribers, a one-slot
// replay flag is set so the next notify() observes the missed event once.
//
// Calls within minBroadcastInterval of the previous broadcast are dropped;
// the function reports whether the signal fired so callers can log
// debug-level traces of suppressed broadcasts.
func (b *reconcileBroadcaster) broadcast() bool {
b.mu.Lock()
defer b.mu.Unlock()
now := b.now()
if !b.lastBroadcast.IsZero() && now.Sub(b.lastBroadcast) < b.minBroadcastInterval {
return false
}
b.lastBroadcast = now
b.pending = true
close(b.ch)
b.ch = make(chan struct{})
return true
}