catalog-backend: drive paginated entity ordering from search-by-key (#34162)

* draft: restructure entities() ordering to drive from search-by-key

When a sort field is specified, build the query so the search table
filtered by that key is the driving relation, instead of left-joining
search onto final_entities and sorting after. The planner can then walk
the (key, value, entity_id) index in already-sorted order and short
circuit on LIMIT.

Measured against a production replica, the catalog UI's default
"first page of components ordered by metadata.name" query goes from
~940 ms to ~8 ms. See PR description for the full numbers.

Known caveats this draft does not yet address:
- Entities lacking the order field are excluded; the previous shape
  put them at the end with NULLS LAST. A UNION ALL pattern can
  preserve the old semantics.
- Multi-field order falls back to the OLD shape (only the first field
  is taken when present today; subsequent fields acted as
  tie-breakers). Restoring tie-breakers needs additional joins or a
  CTE.
- queryEntities (/entities/by-query) is left untouched; the same
  optimization applies but the CTE/cursor structure makes the rewrite
  more involved.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Fredrik Adelöw <freben@spotify.com>

* two-phase entities() ordering: fast path + NULLs-last fallback

When an order field is specified, run a fast path first that drives
from the search-by-key index (excluding entities without the field).
If that path doesn't produce enough rows to cover offset+limit+1, run
a fallback that picks up the no-field entities in entity_id order.
This preserves NULLs-LAST semantics while keeping the fast plan for
the common case where every entity has the order field.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Fredrik Adelöw <freben@spotify.com>

* fix: multi-field order fallback + MySQL quoting in entities()

Fall back to the original LEFT JOIN shape when multiple order fields
are specified, since tie-breaking on secondary fields inherently
requires materialization. The fast INNER-JOIN-driven path is used only
for single-field order (the typical UI case).

Fix the phase 2 NOT EXISTS clause to use knex's ?? identifier escaping
instead of hardcoded double-quotes, which broke on MySQL.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Fredrik Adelöw <freben@spotify.com>

* catalog-backend: fix phase 2 ordering and update changeset

Fix two issues raised in review:

- Phase 2 (entities lacking the sort field) now always sorts by
  entity_id ASC regardless of the primary sort direction, matching the
  original NULLS-LAST behaviour where the NULL group was always
  entity_id ASC.

- Update changeset to describe the actual two-phase behaviour (NULLS
  LAST preserved) instead of the stale description that said entities
  without the field are excluded.

Signed-off-by: Fredrik Adelöw <freben@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

* catalog-backend: apply offset when limit is undefined in runOrderedEntitiesQuery

Previously the fast-path returned the full combined array when no limit
was specified, silently ignoring any pagination offset. The old
implementation always pushed offset to SQL independently of limit.
Restore parity by slicing the combined array by the offset even when
limit is absent.

Signed-off-by: Fredrik Adelöw <freben@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

* catalog-backend: add pagination and phase-boundary tests for entities() ordering

Add two new test cases that cover the parts of runOrderedEntitiesQuery not
exercised before:

- "paginates correctly through single-field ordering": exercises limit, offset,
  and hasNextPage through the fast path (Phase 1 only) across all DB engines.

- "paginates across the Phase 1 / Phase 2 boundary": exercises the case where
  the requested page straddles entities that have the sort field (Phase 1) and
  those that do not (Phase 2), and verifies that Phase 2 entities are always
  ordered ASC by entity_id regardless of the primary sort direction.

Also fixes a double-space caught by prettier in DefaultEntitiesCatalog.ts.

Signed-off-by: Fredrik Adelöw <freben@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

* catalog-backend: treat null sort-field value the same as a missing sort field

buildEntitySearch stores value=NULL for entity fields that are explicitly
null or exceed MAX_VALUE_LENGTH. Previously, Phase 1 included those rows via
the INNER JOIN (key matches, value IS NULL), causing them to sort ahead of
entities that have no row for the key at all — changing semantics vs the old
LEFT JOIN shape where both cases landed in the same NULLS-LAST bucket.

Fix Phase 1 to require order_0.value IS NOT NULL, and fix Phase 2's NOT EXISTS
to check for no non-null value (value IS NOT NULL) so that both null-valued
and missing-key entities are collected in Phase 2 and ordered together by
entity_id ASC.

Adds a regression test covering an entity with spec.b=null alongside one with
no spec.b, asserting both appear after sorted entities regardless of direction.

Signed-off-by: Fredrik Adelöw <freben@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Signed-off-by: Fredrik Adelöw <freben@spotify.com>
Signed-off-by: Fredrik Adelöw <freben@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

.changeset/entities-order-driver.md

@@ -0,0 +1,5 @@
+---
+'@backstage/plugin-catalog-backend': patch
+---
+
+Restructured the entity listing endpoint so that, when a sort field is specified, the search-by-key index drives the query rather than being side-joined onto `final_entities`. This lets PostgreSQL walk the `(key, value, entity_id)` index in already-sorted order and short-circuit on `LIMIT`, reducing typical broad-filter paginated list times from seconds to milliseconds. Entities that lack the sort field still appear at the end of sorted results (NULLS LAST semantics preserved), ordered by `entity_id`.