fix(catalog): processor order not to affect hash generation

sort the arrays such as relations, deferred entities, refresh keys and
parents coming from processors before updating the entity hash. when the
order of the processors is random, the order of the emitted relations
can be also random making unnecessary stitching occur for the entity as
the hash has changed.

this change most probably means that existing hashes for entities do not
match on the first processing round but makes the upcoming processing
rounds much more stable thus making the catalog processing more
performant.

Signed-off-by: Hellgren Heikki <heikki.hellgren@op.fi>

.changeset/moody-clowns-hear.md

@@ -0,0 +1,13 @@
+---
+'@backstage/plugin-catalog-backend': patch
+---
+
+Make the processing hash calculation not care about the order of the processors.
+
+This change does not affect the behavior of the catalog, but it will make the processing
+hash calculation more robust against changes in the order of processors. This should lead to
+more stable processing hashes, which in turn should lead to fewer unnecessary reprocessing
+of entities.
+
+After deploying this fix, you may see a period of increased processing and stitching, but
+this should stabilize over time as the processing hashes become more consistent.

plugins/catalog-backend/src/processing/DefaultCatalogProcessingEngine.ts

@@ -46,6 +46,11 @@ const tracer = trace.getTracer(TRACER_ID);
 
 export type ProgressTracker = ReturnType<typeof progressTracker>;
 
+const stableStringifyArray = (arr: any[]) => {
+  const sorted = arr.map(stableStringify).sort();
+  return `[${sorted.join(',')}]`;
+};
+
 // NOTE(freben): Perhaps surprisingly, this class does not implement the
 // CatalogProcessingEngine type. That type is externally visible and its name is
 // the way it is for historic reasons. This class has no particular reason to
@@ -226,10 +231,10 @@ export class DefaultCatalogProcessingEngine {
 
               hashBuilder = hashBuilder
                 .update(stableStringify({ ...result.completedEntity }))
-                .update(stableStringify([...result.deferredEntities]))
-                .update(stableStringify([...result.relations]))
-                .update(stableStringify([...result.refreshKeys]))
-                .update(stableStringify([...parents]));
+                .update(stableStringifyArray([...result.deferredEntities]))
+                .update(stableStringifyArray([...result.relations]))
+                .update(stableStringifyArray([...result.refreshKeys]))
+                .update(stableStringifyArray([...parents]));
             }
 
             const resultHash = hashBuilder.digest('hex');