Files
multica/SELF_HOSTING_ADVANCED.md
Multica Eve cac453fe01 MUL-5505: fix(selfhost): one source of truth for the backend host port (#6168)
* fix(selfhost): read the published host port back from Compose (MUL-5505)

`make selfhost` polled the wrong port when the backend host port was passed
through the environment. Make and Compose disagree on variable precedence: an
assignment in an included makefile (the Makefile `include`s .env) overrides a
real environment variable, while for Compose the environment overrides .env.
Because .env.example ships BACKEND_PORT uncommented, .env always pinned the
recipe's view of the port, so `BACKEND_PORT=9000 make selfhost` published the
stack on 9000 while the health check hammered 8080 for 60s and then reported
"Services are still starting" on a perfectly healthy install. The success
message printed the same wrong backend URL. FRONTEND_PORT had the same
inversion, affecting the printed frontend URL.

Stop re-deriving the host port in the recipe and ask Compose, which is the only
authority on what it published. The wait-and-report block (three port
references per target, duplicated across selfhost and selfhost-build) moves into
scripts/selfhost-wait.sh, which resolves both host ports via `docker compose
port` and falls back to the compose default chain when the stack is not up.

Also fix the input contract: `PORT` was the only port variable documented in
SELF_HOSTING_ADVANCED.md, yet compose read only BACKEND_PORT, so setting the
documented variable was silently ignored. The backend host port mapping now
falls back to `${PORT:-8080}`, matching Makefile and scripts/local-env.sh, and
the docs describe both variables as host ports.

Container-internal ports (8080/3000), the global PORT derivation used by local
dev, and the 127.0.0.1 bindings are unchanged; the default self-host experience
is byte-identical.

Fixes #6145

Co-authored-by: multica-agent <github@multica.ai>

* fix(selfhost): make PORT the real backend port and test the whole chain

Addresses review feedback on #6168.

The first round's root-cause claim was wrong. It said `BACKEND_PORT=9100 make
selfhost` published on 9100 while the health check probed 8080. It does not:
when make drives Compose, Compose inherits make's exported values, so both
sides agree on 8080 and the override is silently dropped instead. Re-measured
through the real recipe, the genuine disagreements were:

  - `.env` setting PORT with no BACKEND_PORT: probe followed PORT, Compose
    published ${BACKEND_PORT:-8080} — 9100 vs 8080.
  - `make selfhost PORT=8080` over a .env with BACKEND_PORT=9100: probe 8080,
    Compose published 9100.

Both are the "wrong port variable" defect the GitHub issue names, and reading
the port back from Compose fixes both. The comments and PR text that blamed
make/Compose precedence are corrected.

The PORT fallback added to docker-compose.selfhost.yml was also unreachable:
.env.example shipped BACKEND_PORT=8080 uncommented, so the fallback never
engaged and `SELF_HOSTING_AI.md`'s "edit PORT and FRONTEND_PORT" instruction did
nothing. Pick one contract and apply it everywhere: PORT is the backend port to
edit, BACKEND_PORT/API_PORT/SERVER_PORT are optional aliases that override it.
That is already what Makefile, scripts/local-env.sh, scripts/install.sh and
install.ps1 do; compose and the web dev fallback were the outliers.

  - .env.example ships BACKEND_PORT commented out, like its sibling aliases, so
    editing PORT works out of the box.
  - apps/web/config/runtime-urls.ts walks the same alias chain instead of
    reading BACKEND_PORT alone, so `pnpm dev` follows an edited PORT.
  - SELF_HOSTING_AI.md and SELF_HOSTING_ADVANCED.md state the contract.

Configuration that cannot take effect is now reported instead of ignored.
scripts/selfhost-preflight.sh warns when an alias in the env file overrides an
edited PORT, and when a port from the shell environment is overridden by the env
file. The Makefile captures the pristine origins before its include, because
afterwards there is no way to tell that `PORT=9000 make selfhost` was asked for.

Tests now cross environment -> make -> compose -> report for real. The docker
stub is no longer told what to publish: on `up` it asks the real
`docker compose config` to interpolate the compose file with the environment the
recipe actually handed it, records that, and answers `port` from the recording.
Verified failing on the old code — with the old recipe and compose file the
suite reports published 8080 against probe 9100, the exact defect.

Co-authored-by: multica-agent <github@multica.ai>

* ci: gate the self-host test on scripts/selfhost-preflight.sh too

The new preflight script was missing from the frontend path filter, so a
change to it alone would skip the test that covers it.

Co-authored-by: multica-agent <github@multica.ai>

* fix(selfhost): honour the full backend port alias chain in Compose

Addresses the second review round on #6168.

The contract this PR documents is

  BACKEND_PORT -> API_PORT -> SERVER_PORT -> PORT -> 8080

and Makefile, scripts/local-env.sh, scripts/install.sh, scripts/install.ps1 and
apps/web/config/runtime-urls.ts all implement it. docker-compose.selfhost.yml
implemented only BACKEND_PORT -> PORT -> 8080, so `API_PORT=9100` or
`SERVER_PORT=9200` in .env still published 8080.

That is not only a direct-Compose concern. scripts/install.sh:407 derives its
health-check port from the full chain and then runs this compose file, so an
alias it honours but Compose ignores puts the probe on 9100 while the stack
listens on 8080 — the original #6145 defect, reached through the installer.

  - docker-compose.selfhost.yml publishes the full nested chain, verified to
    keep the same precedence and the same 8080 default.
  - scripts/selfhost-wait.sh's fallback matches, so the degraded path agrees
    with what Compose would have published.
  - The Makefile captures the pristine origins of API_PORT and SERVER_PORT too,
    and the preflight reports them, so no alias can be silently overridden by
    the env file while the others are reported.

Tests pin the chain on the boundaries a recipe-level test cannot see, because
Makefile normalises all four aliases into PORT before any recipe runs: a matrix
over the direct Compose path asserts each alias moves the published port with
the right precedence, and every case additionally asserts that
scripts/install.sh's resolver returns the same value Compose published. That
invariant is the one that actually protects the installer.

Verified failing without the fix: with the two-level chain restored the suite
reports `expected 9200 / compose published 8080 / installer resolved 9200`, and
with the alias warnings narrowed it reports the missing API_PORT notice.

Co-authored-by: multica-agent <github@multica.ai>

* fix(selfhost): resolve one winner before reporting ignored port config

Addresses the third review round on #6168.

The preflight walked each alias independently, so it made a separate "wins"
claim per alias. With

  PORT=9000
  BACKEND_PORT=8000
  API_PORT=7000
  SERVER_PORT=6000

in .env it announced that BACKEND_PORT, API_PORT and SERVER_PORT each won, when
only BACKEND_PORT=8000 can. Two of the three notices were simply false.

It also compared only same-named shell and file variables, so shadowing across
aliases stayed silent: .env with PORT=8000 and BACKEND_PORT=8000 plus a shell
API_PORT=7000 produced no output at all, even though API_PORT was discarded.

Resolve the winner along the documented chain first — within a variable the env
file beats the environment, then BACKEND_PORT beats API_PORT beats SERVER_PORT
beats PORT — and report only the inputs that winner actually shadows. Output now
names one winning input and lists the rest, whichever variable or source they
came from. An input carrying the winning value is not reported: it is redundant
but the user still gets the port they asked for, so there is nothing to fix.

Tests cover both cases the old logic got wrong: every alias set at once must
yield exactly one winner claim and list the others as unused, and an env-file
alias beating a lower-priority shell alias must be reported. Also asserted that
a redundant same-value alias and a default configuration both stay silent.

Measured against the previous implementation, phrasing aside: with all four
values set it made 3 winner claims where there is now 1, and on the cross-alias
case it printed nothing where API_PORT is now reported.

Co-authored-by: multica-agent <github@multica.ai>

* fix(selfhost): track env-file existence so empty port assignments resolve right

Addresses the fourth review round on #6168.

The Makefile passed only values to the preflight, so the script inferred "the
env file does not set this" from an empty string. An explicit empty assignment is
a different input from an absent one: make lets `BACKEND_PORT=` in the env file
override `BACKEND_PORT=9000` from the environment, and the variable then drops out
of the alias chain instead of setting a port.

With .env holding PORT=8080 and BACKEND_PORT=, invoked as
`BACKEND_PORT=9000 make selfhost`, the stack published 8080 and the health check
probed 8080 — correct — while the preflight announced

  the backend host port resolves to 9000 from BACKEND_PORT (environment)
  Set but unused: PORT=8080 (.env)

naming the ignored value as the winner and the winning value as unused. A report
that contradicts the startup is worse than no report. FRONTEND_PORT= had the
matching hole: it shadowed the environment, Compose fell back to 3000, and the
preflight said nothing.

The Makefile now captures ENV_FILE_<VAR>_IS_SET from $(origin) alongside the
value, for all five port variables via one $(foreach) instead of hand-written
lines. The preflight treats a variable the file defines as taking the file's
value even when empty, so it shadows the environment, and an empty value never
wins — resolution continues down the chain and falls back to 8080. Inputs the
winner shadows are still listed, including ones shadowed by an empty assignment.
The frontend check mirrors this and now names the port that actually resolved.

Recipe-level tests cover an empty alias over a shell value, every link of the
chain emptied at once, and the frontend equivalent — each asserting the reported
winner, the Compose published port and the probed port all agree. Against the
previous implementation the first case fails with `reported: 9000 / actual: 8080`.

Co-authored-by: multica-agent <github@multica.ai>

* fix(selfhost): installers read the published port; drop the second parser

Addresses all four blockers from the consolidated review on #6168.

1. Both installers probed and printed the wrong port. scripts/install.sh:402 and
   scripts/install.ps1 start Compose inheriting the current environment, and
   Compose lets that environment outrank .env — but the installers then derived
   the probe port and the summary URL from .env alone. Measured on this branch
   before the fix, every port variable diverged:

     ambient PORT=9100        -> compose 9100, installer 8080
     ambient BACKEND_PORT=9200 -> compose 9200, installer 8080
     ambient API_PORT=9300     -> compose 9300, installer 8080
     ambient SERVER_PORT=9400  -> compose 9400, installer 8080
     ambient FRONTEND_PORT=3100 -> compose 3100, installer 3000

   Both now read the ports back once with `docker compose port` after `up -d`
   and reuse that single result for the health check and the summary. If the
   query fails they fail loudly instead of claiming success. The .env-derived
   helpers are deleted rather than left beside the new path.

2. The Make preflight is removed entirely, as the review recommended. It
   modelled `origin=environment` and `origin=file` but not `command line`, so
   `make selfhost BACKEND_PORT=9000` over a .env with API_PORT=7000 announced
   7000 while Compose published 9000. That was its fourth wrong report in four
   rounds, because it was a second parser of precedence rules — the very thing
   this PR removes elsewhere. `docker compose port` is the runtime truth, so the
   script, the Makefile captures and the macro are gone.

3. Tests now cover the installer paths that were unguarded. scripts/install.test.sh
   gains a `--with-server` matrix (defaults, .env PORT, all four backend aliases,
   FRONTEND_PORT, ambient overriding .env for all five, and explicit-empty
   fallback) plus a case proving an unresolvable port fails loudly. A new
   scripts/install.ps1.test.ps1 drives the same matrix through Start-LocalInstall.
   Every case asserts Compose's published port == the probed URL == the printed
   URL. Ambient variables are explicitly cleared per case so a runner's own PORT
   cannot leak. The Makefile matrix gains the command-line origin cases. CI runs
   the PowerShell suite on windows-latest in the always-on installer job, and the
   frontend filter now includes both installers.

4. Docs state the real contract: the alias order is shared, but which *source*
   wins is per entry point — Compose prefers the environment, make prefers the
   included env file, and a make command-line assignment outranks both. The
   removed preflight's promise is gone from SELF_HOSTING_AI.md, and the compose
   header no longer claims the installer derives its own health-check port.

Verified failing without the fixes: the Bash matrix reports
`[ambient PORT beats .env] compose published 9500 / probed 9100 / printed 9100`
against the old installer, and the PowerShell matrix reports
`printed backend=8080` against an injected summary divergence.

Co-authored-by: multica-agent <github@multica.ai>

* fix(selfhost): keep web dev proxy off frontend port

Co-authored-by: multica-agent <github@multica.ai>

---------

Co-authored-by: Eve <eve@multica-ai.local>
Co-authored-by: multica-agent <github@multica.ai>
2026-07-31 14:46:13 +08:00

33 KiB
Raw Permalink Blame History

Self-Hosting — Advanced Configuration

This document covers advanced configuration for self-hosted Multica deployments. For the quick start guide, see SELF_HOSTING.md.

Configuration

All configuration is done via environment variables. Copy .env.example as a starting point.

Required Variables

Variable Description Example
DATABASE_URL PostgreSQL connection string postgres://multica:multica@localhost:5432/multica?sslmode=disable
JWT_SECRET Must change from default. Secret key for signing JWT tokens. Use a long random string. openssl rand -hex 32
FRONTEND_ORIGIN URL where the frontend is served (used for CORS) https://app.example.com

Database Pool Tuning (Optional)

These have sensible defaults and only need to be set when tuning a large or constrained deployment. Precedence (highest first): env var → pool_* query params on DATABASE_URL → built-in default.

Variable Description Default
DATABASE_MAX_CONNS pgxpool max connections per pod. pod_count × DATABASE_MAX_CONNS should stay well below the Postgres max_connections ceiling. With a connection pooler (PgBouncer / RDS Proxy / Supavisor) in front, this can be raised significantly. 25
DATABASE_MIN_CONNS pgxpool warm baseline connections per pod. Auto-clamped to DATABASE_MAX_CONNS. 5

Email (Required for Authentication)

Multica supports two email backends. SMTP_HOST takes priority when set; otherwise RESEND_API_KEY is used. With neither configured, verification codes are printed to the server log — copy them from there to log in.

Variable Description
RESEND_API_KEY Your Resend API key
RESEND_FROM_EMAIL Sender email address (default: noreply@multica.ai)

Option B: SMTP relay (for self-hosted / on-premise deployments)

Use this option when your deployment cannot reach the public internet or you already have an internal mail relay (e.g. Exchange, Postfix, SendGrid on-prem).

Variable Description Default
SMTP_HOST SMTP relay hostname (setting this activates SMTP mode) -
SMTP_PORT SMTP port 25
SMTP_USERNAME SMTP username (leave empty for unauthenticated relay) -
SMTP_PASSWORD SMTP password -
SMTP_TLS TLS mode. implicit (aliases smtps, ssl) forces SMTPS on connect; port 465 auto-enables it. Unset / starttls upgrades via STARTTLS starttls
SMTP_TLS_INSECURE Set true to skip TLS certificate verification (self-signed / private CA certs) false
SMTP_EHLO_NAME EHLO/HELO name announced to the relay. Set a real FQDN when a strict relay (e.g. Google Workspace) rejects the default greeting from a public IP machine hostname

STARTTLS is used automatically when advertised by the server. Port 465 (SMTPS / implicit TLS) is supported and auto-enables implicit TLS; set SMTP_TLS=implicit (aliases smtps, ssl) to force it on a non-standard port.

Note: If neither Resend nor SMTP is configured, generated verification codes are printed to backend logs — copy them from there to log in. A fixed local testing code (e.g. 888888) is opt-in only: set MULTICA_DEV_VERIFICATION_CODE=888888 in .env and keep APP_ENV non-production. The Docker self-host stack pins APP_ENV=production, so the shortcut is ignored there. Never enable a fixed code on a publicly reachable instance.

Google OAuth (Optional)

Variable Description
GOOGLE_CLIENT_ID Google OAuth client ID
GOOGLE_CLIENT_SECRET Google OAuth client secret
GOOGLE_REDIRECT_URI OAuth callback URL (e.g. https://app.example.com/auth/callback)

Changes take effect after restarting the backend / compose stack. The web UI reads GOOGLE_CLIENT_ID from /api/config at runtime, so no web rebuild is needed.

Signup Controls (Optional)

Variable Description
ALLOW_SIGNUP Set to false to disable new user signups on a private instance
ALLOWED_EMAIL_DOMAINS Optional comma-separated allowlist of email domains
ALLOWED_EMAILS Optional comma-separated allowlist of exact email addresses
DISABLE_WORKSPACE_CREATION Set to true to make POST /api/workspaces return 403 for every caller — users can only join workspaces they were invited to

Changes take effect after restarting the backend / compose stack. The web UI reads ALLOW_SIGNUP and DISABLE_WORKSPACE_CREATION from /api/config at runtime, so no web rebuild is needed.

Locking down workspace creation

ALLOW_SIGNUP=false blocks new accounts from being created, but it does not block an already-signed-in user from creating another workspace via POST /api/workspaces. On a self-hosted instance where every issue/repo/agent must be visible to the platform admin, set DISABLE_WORKSPACE_CREATION=true to close that gap. The recommended bootstrap sequence is:

  1. Start the instance with DISABLE_WORKSPACE_CREATION=false (the default).
  2. Sign in as the admin and create the shared workspace.
  3. Set DISABLE_WORKSPACE_CREATION=true and restart the backend. Optionally set ALLOW_SIGNUP=false at the same time if you also want to block new account creation.
  4. Going forward, additional users join via invitation only — the "Create workspace" affordance is hidden in the UI and any direct API call returns 403.

Note: setting ALLOW_SIGNUP=false blocks all new account creation, including users who already have a pending invitation. If you need invited users to be able to sign up but not create their own workspaces, keep ALLOW_SIGNUP=true (optionally combined with ALLOWED_EMAIL_DOMAINS / ALLOWED_EMAILS) and only flip DISABLE_WORKSPACE_CREATION=true.

File Storage (Optional)

For file uploads and attachments, configure S3 and (optionally) CloudFront:

Variable Description
S3_BUCKET Bucket name only (e.g. my-bucket). Do not include the .s3.<region>.amazonaws.com suffix — the server constructs the public URL from S3_BUCKET + S3_REGION
S3_REGION AWS region (default: us-west-2). Must match the bucket's actual region — used for both SDK signing and public URLs
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY Static credentials. When both are unset, the AWS SDK default credential chain is used
AWS_ENDPOINT_URL Custom S3-compatible endpoint (e.g. MinIO, R2, B2). Setting this defaults to path-style URLs for backward compatibility
S3_USE_PATH_STYLE Optional S3 addressing mode. Leave empty for the default (true when AWS_ENDPOINT_URL is set, false for AWS S3). Set false for S3-compatible providers that require virtual-hosted-style URLs
ATTACHMENT_DOWNLOAD_MODE Attachment download behavior: auto (default), cloudfront, presign, or proxy. Use proxy for private buckets behind Docker/VPC-only endpoints such as http://rustfs:9000. Avatars follow the same policy — see below
ATTACHMENT_DOWNLOAD_URL_TTL TTL for CloudFront signed URLs and S3 presigned download URLs (default: 30m)
CLOUDFRONT_DOMAIN CloudFront distribution domain — when set, public URLs use this host instead of the S3 host
CLOUDFRONT_KEY_PAIR_ID CloudFront key pair ID for signed URLs
CLOUDFRONT_PRIVATE_KEY CloudFront private key (PEM format)

Avatars on a private bucket

User / agent / squad / workspace avatars are stored as the raw storage object URL. When the bucket is public — a public CLOUDFRONT_DOMAIN, or the default local-disk backend — that URL is served to clients unchanged.

When the bucket is private and no public CDN domain is configured (S3 with Block Public Access, R2, MinIO), the API instead serves avatars from /api/avatars/<signature>/<key> and resolves each request through ATTACHMENT_DOWNLOAD_MODE — a presigned redirect, a CloudFront-signed redirect, or a proxied body. The signature in the path is what authorizes the read: an auth-gated URL cannot be used as an <img src> from the Desktop app or a split-origin frontend, because the session cookie is SameSite=Strict.

Only image objects resolve through this route, and only ones that are avatar-class: a standalone upload not attached to an issue, comment, chat, or task. Pointing an avatar_url at a file someone attached to an issue is rejected when it is set and 404s if it was already stored, so a private image cannot be turned into a public link by way of the avatar field.

No configuration is required, and avatars uploaded before this behavior existed are fixed without a backfill.

Cookies

Variable Description
COOKIE_DOMAIN Optional Domain attribute for session + CloudFront cookies. Leave empty for single-host deployments (localhost, LAN IP, or a single hostname). Only set it when the frontend and backend sit on different subdomains of one registered domain (e.g. .example.com). Do not use an IP literal — RFC 6265 forbids IP addresses in the cookie Domain attribute and browsers will drop such Set-Cookie headers.

The Secure flag on session cookies is derived automatically from the scheme of FRONTEND_ORIGIN: HTTPS origins get Secure cookies; plain-HTTP origins (LAN / private-network self-host) get non-secure cookies so the browser can actually store them.

If the frontend and backend are served from different hostnames, COOKIE_DOMAIN is required, not optional: the browser must be able to read the multica_csrf cookie from the page's own origin to send the X-CSRF-Token header, and without it every write request fails with 403 {"error":"CSRF validation failed"}. Scope it to the narrowest parent domain that covers both hosts — it also shares the multica_auth session cookie with every host under that domain. See Reverse Proxy for the full split-domain configuration and its trust requirements.

Server

Variable Default Description
PORT 8080 Backend port — the one to edit. It is the port the backend process listens on for a local/bare run, and the host port the Compose self-host stack publishes. In Compose the container always listens on 8080 internally, so changing this needs no rebuild.
BACKEND_PORT Value of PORT Optional alias that overrides PORT for the backend. API_PORT and SERVER_PORT are further aliases; the alias order is BACKEND_PORTAPI_PORTSERVER_PORTPORT8080, and it is the same in Makefile, scripts/local-env.sh and docker-compose.selfhost.yml. Leave them unset unless the host port must differ from the port the process listens on.
METRICS_ADDR empty Optional Prometheus metrics listener, for example 127.0.0.1:9090
FRONTEND_PORT 3000 Frontend port. Host port in Compose; the container always listens on 3000 internally.
CORS_ALLOWED_ORIGINS Value of FRONTEND_ORIGIN Comma-separated list of allowed origins. Governs both the HTTP CORS allowlist and the WebSocket Origin check. A browser origin that isn't listed here (and isn't localhost) has its real-time WebSocket upgrade rejected with 403, so live updates stop working until a manual refresh.
LOG_LEVEL info Log level: debug, info, warn, error

Which source wins depends on the entry point, and only the alias order above is shared. Docker Compose lets the calling environment outrank .env (PORT=9100 docker compose … up -d publishes 9100). make is the other way round: it includes the env file, so a value there outranks the same variable from your environment, and a make selfhost PORT=… command-line assignment outranks both. Editing the env file is therefore the one method that behaves the same everywhere. None of this affects whether the reported port is correct: make selfhost and both installers read the published port back from docker compose port, so the health check and the printed URL always match what Compose actually published.

The web development server is one intentional exception to the generic PORT fallback: Next uses PORT for its own frontend listener before it evaluates the rewrite configuration. Its backend fallback therefore accepts BACKEND_PORTAPI_PORTSERVER_PORT8080, while an explicit REMOTE_API_URL or NEXT_PUBLIC_API_URL still takes priority.

CLI / Daemon

These are configured on each user's machine, not on the server:

Variable Default Description
MULTICA_SERVER_URL ws://localhost:8080/ws WebSocket URL for daemon → server connection
MULTICA_APP_URL http://localhost:3000 Frontend URL for CLI login flow
MULTICA_DAEMON_POLL_INTERVAL 3s How often the daemon polls for tasks
MULTICA_DAEMON_HEARTBEAT_INTERVAL 15s Heartbeat frequency

Agent-specific overrides:

Variable Description
MULTICA_CLAUDE_PATH Custom path to the claude binary
MULTICA_CLAUDE_MODEL Override the Claude model used
MULTICA_CODEX_PATH Custom path to the codex binary
MULTICA_CODEX_MODEL Override the Codex model used
MULTICA_COPILOT_PATH Custom path to the copilot (GitHub Copilot CLI) binary
MULTICA_COPILOT_MODEL Override the Copilot model used (note: GitHub Copilot routes models through your account entitlement, so this may not be honoured)
MULTICA_OPENCODE_PATH Custom path to the opencode binary
MULTICA_OPENCODE_MODEL Override the OpenCode model used
MULTICA_OPENCLAW_PATH Custom path to the openclaw binary
MULTICA_OPENCLAW_MODEL Override the OpenClaw model used
MULTICA_HERMES_PATH Custom path to the hermes binary
MULTICA_HERMES_MODEL Override the Hermes model used
MULTICA_PI_PATH Custom path to the pi binary
MULTICA_PI_MODEL Override the Pi model used
MULTICA_CURSOR_PATH Custom path to the cursor-agent binary
MULTICA_CURSOR_MODEL Override the Cursor Agent model used
MULTICA_GROK_PATH Custom path to the grok binary
MULTICA_GROK_MODEL Override the Grok model used (e.g. grok-4.5)

Database Setup

Multica requires PostgreSQL 17 with the pgvector extension.

The docker-compose.selfhost.yml includes PostgreSQL. No separate setup needed.

Using Your Own PostgreSQL

If you prefer to use an existing PostgreSQL instance, ensure the pgvector extension is available:

CREATE EXTENSION IF NOT EXISTS vector;

Set DATABASE_URL in your .env and remove the postgres service from the compose file.

Running Migrations Manually

The Docker Compose setup runs migrations automatically. If you need to run them manually:

# Using the built binary
./server/bin/migrate up

# Or from source
cd server && go run ./cmd/migrate up

Usage Dashboard Rollup

The Usage and Runtime dashboards read from task_usage_hourly, a derived table populated by rollup_task_usage_hourly(). As of MUL-2957 the backend runs this rollup in-process on every replica via a DB-backed scheduler (sys_cron_executions); a fresh self-host install needs no operator action — the bundled pgvector/pgvector:pg17 image works without changes.

How the in-process scheduler works

Every backend replica ticks every 30 seconds and tries to claim the current 5-minute UTC plan in sys_cron_executions. The unique key (job_name, scope_kind, scope_id, plan_time) makes the claim a single-winner contest across all replicas, so multi-instance deployments do not double-write. The handler then calls SELECT rollup_task_usage_hourly(); the SQL function holds advisory lock 4246 internally, so a stray pg_cron job or manual call can run alongside the scheduler without ever colliding on the rollup itself. Inspect the audit table for steady-state operation:

SELECT plan_time, status, attempt, runner_id,
       error_code, error_msg, started_at, finished_at
  FROM sys_cron_executions
 WHERE job_name = 'rollup_task_usage_hourly'
 ORDER BY plan_time DESC
 LIMIT 20;

Compatibility — existing pg_cron registrations

If you previously registered the rollup as a pg_cron job (SELECT cron.schedule('rollup_task_usage_hourly', '*/5 * * * *', …)), it is safe to leave it in place: advisory lock 4246 prevents double-writes, and the loser path no-ops cleanly. To drop the redundant entry once the in-process scheduler is up:

SELECT cron.unschedule('rollup_task_usage_hourly')
  FROM cron.job WHERE jobname = 'rollup_task_usage_hourly';

External cron / systemd / Kubernetes CronJob setups that call SELECT rollup_task_usage_hourly() directly are also still valid — they were the only option before MUL-2957 and remain a supported compatibility path. They are no longer the recommended setup; new deployments should rely on the in-process scheduler.

Standalone backfill command

rollup_task_usage_hourly() only processes new buckets after it starts running. If you already have task_usage rows from before the rollup was claimed for the first time — most commonly when upgrading from v0.3.4 to v0.3.5+ on a database that already has months of usage — you can run backfill_task_usage_hourly to seed historical buckets:

# Docker Compose
docker compose -f docker-compose.selfhost.yml exec backend \
  ./backfill_task_usage_hourly --sleep-between-slices=2s

# Kubernetes
kubectl -n multica exec deploy/multica-backend -- \
  ./backfill_task_usage_hourly --sleep-between-slices=2s

The command walks task_usage's full time range in monthly slices and calls the same idempotent primitive the in-process scheduler uses, so it's safe to re-run, to interrupt with Ctrl-C, and to run concurrently with the scheduler (advisory lock 4246 serialises them). Flags:

Flag Description
--sleep-between-slices Pause between monthly slices to throttle read pressure on busy databases (e.g. 2s). Recommended on production DBs with years of history.
--months-back N Only backfill the last N months. Requires --force-partial because the watermark still advances past the skipped older buckets — those are permanently abandoned.
--dry-run Log slices that would be processed without writing anything.

After backfill completes, the rollup-state watermark is stamped to now() - 5 minutes, so the first scheduled tick after backfill does not redo history.

v0.3.4 → v0.3.5+ upgrade order

Migration 103 adds a fail-closed guard that refuses to drop the legacy daily rollups until task_usage_hourly has caught up. As of MUL-2957 the migrate command runs an idempotent monthly-slice backfill (under advisory lock 4246) automatically immediately before applying migration 103, so v0.3.4 → v0.3.5+ upgrades complete in a single migrate up invocation — no operator step is required.

If you are upgrading from a binary that pre-dates MUL-2957 (or the auto-hook fails for an environmental reason), recovery is the manual path: run backfill_task_usage_hourly against the database, then re-run migrate up (or restart the backend container — migrations run automatically on startup). Fresh installs are exempt — the guard short-circuits when task_usage is empty, and the in-process scheduler picks up new buckets from the first tick.

Manual Setup (Without Docker Compose)

If you prefer to build and run services manually:

Prerequisites: Go 1.26+, Node.js 20+, pnpm 10.28+, PostgreSQL 17 with pgvector.

# Start your PostgreSQL (or use: docker compose up -d postgres)

# Build the backend
make build

# Run database migrations
DATABASE_URL="your-database-url" ./server/bin/migrate up

# Start the backend server
DATABASE_URL="your-database-url" PORT=8080 JWT_SECRET="your-secret" ./server/bin/server

For the frontend:

pnpm install
pnpm build

# Start the frontend (production mode)
cd apps/web
REMOTE_API_URL=http://localhost:8080 pnpm start

Reverse Proxy

In production, put a reverse proxy in front of both the backend and frontend to handle TLS and routing.

Single-domain layout — frontend and backend served on the same hostname (this is what docker-compose.selfhost.yml defaults to):

multica.example.com {
    # WebSocket route — must come before the catch-all
    @multica_ws path /ws /ws/*
    handle @multica_ws {
        reverse_proxy localhost:8080 {
            flush_interval -1
        }
    }

    # Everything else → frontend
    reverse_proxy localhost:3000
}

Even on a single domain, set FRONTEND_ORIGIN / CORS_ALLOWED_ORIGINS to that public origin (e.g. https://multica.example.com) on the backend. The backend's default origin allowlist is localhost only, so without this it rejects the WebSocket upgrade from the public URL with 403 and real-time updates silently stop working. See LAN / Non-localhost Access.

Separate-domain layout — frontend and backend on different hostnames:

app.example.com {
    reverse_proxy localhost:3000
}

api.example.com {
    @multica_ws path /ws /ws/*
    handle @multica_ws {
        reverse_proxy localhost:8080 {
            flush_interval -1
        }
    }

    reverse_proxy localhost:8080
}

Two non-obvious bits inside the /ws block are worth calling out — both are common reasons real-time updates "stop working" on a Caddy-fronted self-host:

  • path /ws /ws/* (not /ws*) — bare handle /ws is an exact match, so future path variants under /ws/ fall through to the frontend block. The obvious shortcut handle /ws* overcorrects in the other direction: Caddy's * is a glob without a path-segment boundary, so it would also catch unrelated paths like /ws-foo, which is a legitimate workspace URL (only the exact slug ws is reserved). Listing /ws and /ws/* explicitly covers both real cases without overreach.
  • flush_interval -1 — disables response buffering so WebSocket frames are forwarded as soon as they arrive. Without it, frames can sit behind Caddy's default flush window, which looks like delayed comments, missing typing indicators, or "comments only appear after a page refresh."

Nginx

# Frontend
server {
    listen 443 ssl;
    server_name app.example.com;

    ssl_certificate     /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;

    location / {
        proxy_pass http://localhost:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

# Backend API
server {
    listen 443 ssl;
    server_name api.example.com;

    ssl_certificate     /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;

    location / {
        proxy_pass http://localhost:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # WebSocket support
    location /ws {
        proxy_pass http://localhost:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_read_timeout 86400;
    }
}

When using separate domains for frontend and backend, set these environment variables accordingly:

# Backend
FRONTEND_ORIGIN=https://app.example.com
CORS_ALLOWED_ORIGINS=https://app.example.com
COOKIE_DOMAIN=.example.com           # narrowest parent covering both hosts — read the scope warning below

# Frontend (only if you are building the web image from source via docker-compose.selfhost.build.yml)
REMOTE_API_URL=https://api.example.com
NEXT_PUBLIC_API_URL=https://api.example.com
NEXT_PUBLIC_WS_URL=wss://api.example.com/ws

COOKIE_DOMAIN is required in this setup — omitting it breaks every write. The web app authenticates with an HttpOnly multica_auth cookie plus a JS-readable multica_csrf cookie, and sends the CSRF value as an X-CSRF-Token header on every non-GET request. Both cookies are host-only unless COOKIE_DOMAIN is set, so a frontend on app.example.com cannot read a cookie issued by api.example.com. The header is then never sent and the backend rejects the request with 403 {"error":"CSRF validation failed"} — while GET requests keep working, so the app renders but nothing can be created or edited.

After changing COOKIE_DOMAIN, delete the existing multica_auth / multica_csrf cookies on both hosts and log in again. Stale host-only cookies otherwise sit alongside the new domain-scoped ones and the browser sends both.

Scope COOKIE_DOMAIN as narrowly as possible. The same value also scopes multica_auth, the session JWT, and the browser then sends it to every host under that domain. HttpOnly only stops page scripts from reading the cookie — it does not stop the server behind a sibling subdomain from receiving it on every request. Use the narrowest parent that still covers both hosts: for agent.example.com + api.agent.example.com that is .agent.example.com, not .example.com, which would also hand your users' session cookie to unrelated hosts such as a separately deployed docs.example.com.

Both of these must hold, or this layout is not safe to use:

  • The frontend and backend share a common parent domain. Unrelated domains (app.com and api.io) cannot be covered by any COOKIE_DOMAIN value.
  • Every host under that parent domain is operated by the same trusted party. If anything in scope is third-party hosted, or other teams can create records under it, a compromise or a rogue service there receives valid sessions for your deployment.

If either condition fails, use the same-origin layout below, which keeps the session cookie host-only.

A separate API domain is not required. Leave NEXT_PUBLIC_API_URL and NEXT_PUBLIC_WS_URL at their defaults and the browser calls /api and /ws on the page's own origin, so no cross-host cookie problem can arise in the first place:

# Backend
FRONTEND_ORIGIN=https://app.example.com
CORS_ALLOWED_ORIGINS=https://app.example.com
COOKIE_DOMAIN=                       # empty: cookies are host-only on app.example.com, which is correct here

# Frontend
NEXT_PUBLIC_API_URL=                 # empty: the client uses relative /api paths on the page origin
NEXT_PUBLIC_WS_URL=                  # empty
REMOTE_API_URL=http://backend:8080   # target the Next.js rewrites proxy /api, /auth and /uploads to

Serve everything from the single app.example.com vhost. HTTP works out of the box, because Next.js rewrites forward /api, /auth and /uploads to REMOTE_API_URL. WebSockets do not go through those rewrites, so add a /ws block to the frontend vhost that reaches the backend directly:

# Add to the app.example.com server block above
location /ws {
    proxy_pass http://localhost:8080;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $host;
    proxy_read_timeout 86400;
}

See WebSocket for LAN / Non-localhost Access for why the rewrites cannot carry the Upgrade handshake.

This keeps cookies, CORS, and the WebSocket origin check on a single origin. It is both the configuration least likely to break and the safer one: the session cookie stays host-only, so no sibling subdomain can ever receive it. An api.example.com vhost can still be kept for CLI and daemon use: those clients authenticate with a mul_ personal access token over Authorization: Bearer, which never goes through the cookie or CSRF path.

LAN / Non-localhost Access

By default, Multica works on localhost. If you access it from another machine on the LAN (e.g. http://192.168.1.100:3000), you need to tell the backend to accept that origin:

# .env — replace with your server's LAN IP
FRONTEND_ORIGIN=http://192.168.1.100:3000
CORS_ALLOWED_ORIGINS=http://192.168.1.100:3000

Then restart the stack:

docker compose -f docker-compose.selfhost.yml up -d

WebSocket for LAN / Non-localhost Access

HTTP requests (issues, comments, uploads) work on LAN out of the box — Next.js rewrites proxy /api, /auth, and /uploads to the backend. WebSockets do not: Next.js rewrites only forward HTTP requests, not the Upgrade handshake a WebSocket needs. If you open the app on http://<lan-ip>:3000, real-time features (chat streaming, live issue updates, notifications) will fail to connect until you do one of the following:

  1. Put a reverse proxy in front of the stack (recommended). Nginx or Caddy terminates the WebSocket upgrade and forwards it to the backend on port 8080. See the Reverse Proxy section above — the Nginx example already includes a location /ws { ... } block with the correct Upgrade / Connection headers. Once a proxy is in place the browser connects directly through it, so no frontend rebuild is needed.

  2. Bake a WebSocket URL into the web image. If you are not running a reverse proxy, rebuild the web image with NEXT_PUBLIC_WS_URL pointing straight at the backend (port 8080 must be reachable from the browser):

    # In .env
    NEXT_PUBLIC_WS_URL=ws://<lan-ip>:8080/ws
    
    # Rebuild the web image so the build-time value is baked in
    docker compose -f docker-compose.selfhost.yml -f docker-compose.selfhost.build.yml up -d --build
    

    NEXT_PUBLIC_WS_URL is a build-time variable (see Dockerfile.web), so setting it only in environment: on the pre-built image has no effect — you must use the selfhost.build.yml override that rebuilds the image.

Also required: allowlist the browser origin. The two options above fix the WebSocket upgrade proxying, but a second, independent setting gates the connection: the backend validates the WebSocket Origin header against an allowlist that defaults to localhost only. When you open Multica from any other origin — a LAN IP or a public domain behind a reverse proxy — set CORS_ALLOWED_ORIGINS (or FRONTEND_ORIGIN) on the backend to that exact origin and restart, exactly as shown under LAN / Non-localhost Access above. Otherwise the upgrade is refused with 403: the backend logs websocket: request origin not allowed by Upgrader.CheckOrigin and the browser console loops disconnected, reconnecting in 3s, while HTTP requests (and manual page refreshes) keep working because they are same-origin to the page. The single value covers both HTTP CORS and the WebSocket origin check.

Note: If you need to hard-code a different public API / WebSocket endpoint into the web image for any other reason, use the same source-build override: docker compose -f docker-compose.selfhost.yml -f docker-compose.selfhost.build.yml up -d --build.

Health Check

The backend exposes public health endpoints:

GET /health
→ {"status":"ok"}

GET /readyz
→ {"status":"ok","checks":{"db":"ok","migrations":"ok"}}

GET /healthz
→ same response as /readyz

Use /health for basic liveness / reachability checks. Use /readyz for dependency-aware readiness probes and external monitoring that should fail when the database is unavailable or migrations are not fully applied. /healthz is kept as an alias for operator familiarity.

Prometheus Metrics

The backend can expose Prometheus metrics on a separate management listener:

METRICS_ADDR=127.0.0.1:9090 ./server/bin/server
curl http://127.0.0.1:9090/metrics

METRICS_ADDR is empty by default, so no metrics listener is started. The public API port does not serve /metrics; keep it that way for internet-facing deployments. HTTP request metrics start accumulating only after the metrics listener is enabled. Metrics can reveal internal routes, traffic volume, dependency state, and runtime health.

For Docker or Kubernetes deployments, prefer a private scrape path: bind the metrics listener to an internal interface and protect it with private networking, allowlists, NetworkPolicy, or proxy authentication. If you bind METRICS_ADDR=0.0.0.0:9090 inside a container, only publish that port to a trusted network, for example a host-local mapping such as 127.0.0.1:9090:9090.

Upgrading

docker compose -f docker-compose.selfhost.yml pull
docker compose -f docker-compose.selfhost.yml up -d

Pin MULTICA_IMAGE_TAG in .env to an exact release like v0.2.4 if you want to stay on a specific version. Migrations run automatically on backend startup. They are idempotent — running them multiple times has no effect. If the selected GHCR tag has not been published yet, fall back to docker compose -f docker-compose.selfhost.yml -f docker-compose.selfhost.build.yml up -d --build.