clarify death loop risk in migration comments and changeset
Be explicit that interrupted index builds do not leave partial progress — each retry starts from scratch. On large tables with short liveness probe timeouts this repeats indefinitely. Recommend running the SQL commands manually before deploying. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Fredrik Adelöw <freben@spotify.com>
This commit is contained in:
@@ -4,9 +4,9 @@
|
||||
|
||||
Added a migration that removes duplicate rows from the `search` table, creates covering indices for improved query performance, and adds a `UNIQUE` constraint on `(entity_id, key, value)`.
|
||||
|
||||
This is a long-running migration on large catalogs. On PostgreSQL with millions of search rows, the index creation may take 5-15 minutes. During this time, other pods running the previous version will continue to serve traffic normally — the index creation does not block reads or writes. If the pod is restarted during the migration, the next startup will clean up and retry automatically.
|
||||
This is a long-running migration on large catalogs. On PostgreSQL with millions of search rows, the index creation may take 5-15 minutes per index. During this time, other pods running the previous version will continue to serve traffic normally — the index creation does not block reads or writes. However, if a Kubernetes liveness probe kills the pod before the index build completes, the build is lost and the next startup will start over. On large tables this can repeat indefinitely.
|
||||
|
||||
**For large installations**, you can run the following SQL commands against your PostgreSQL database before deploying to avoid the startup delay. If these have already completed, the migration will detect the existing indices and skip all work.
|
||||
**For large installations**, it is recommended to run the following SQL commands against your PostgreSQL database **before deploying** this version. Each index build takes a few minutes but does not block reads or writes. If these have already completed, the migration will detect the existing indices and skip all work — startup will be instant.
|
||||
|
||||
```sql
|
||||
-- Step 1: Remove duplicate search rows
|
||||
|
||||
@@ -22,19 +22,21 @@
|
||||
*
|
||||
* On PostgreSQL this uses CREATE INDEX CONCURRENTLY which avoids blocking
|
||||
* reads/writes but can take several minutes on large tables (13M+ rows).
|
||||
* The migration is fully rerun-safe: each step checks the current state
|
||||
* and skips work that is already done or cleans up INVALID indices left by
|
||||
* a previous interrupted attempt.
|
||||
* Each step is idempotent: it checks the current state and skips work
|
||||
* already done, or cleans up INVALID indices left by an interrupted
|
||||
* attempt before retrying. However, an interrupted index build does NOT
|
||||
* leave partial progress — each retry starts from scratch. If the
|
||||
* Kubernetes liveness probe kills the pod before an index build completes,
|
||||
* the next startup will drop the INVALID index and restart the build. On
|
||||
* large tables this can repeat indefinitely. To prevent this, either
|
||||
* increase the liveness probe timeout for this one-time upgrade, or run
|
||||
* the SQL commands below manually before deploying.
|
||||
*
|
||||
* ## Important note for large installations
|
||||
* ## Recommended: run manually before deploying (large installations)
|
||||
*
|
||||
* If your search table has millions of rows, this migration may take
|
||||
* 5-15 minutes. During this time the pod will appear unready. Other pods
|
||||
* running the previous version of Backstage will continue to serve
|
||||
* traffic normally — the index creation does not block reads or writes.
|
||||
*
|
||||
* To avoid the startup delay, you can run the following commands against
|
||||
* your PostgreSQL database BEFORE deploying this version:
|
||||
* For PostgreSQL installations with millions of search rows, run these
|
||||
* commands against your database BEFORE deploying this version. Each
|
||||
* index build takes a few minutes but does not block reads or writes.
|
||||
*
|
||||
* -- 1. Remove duplicate search rows
|
||||
* WITH cte AS (
|
||||
|
||||
Reference in New Issue
Block a user