Removed unneeded annotation

Signed-off-by: Damon Kaswell <damon.kaswell1@hp.com>
This commit is contained in:
Damon Kaswell
2022-11-29 15:18:20 -08:00
parent 78d5d19721
commit 16af3f6dac
3 changed files with 3 additions and 18 deletions
@@ -13,11 +13,10 @@ Unfortunately, these two kinds of mutations are insufficient for very large data
3. Addressing the above two use case with `full` mutation is not an option on very large datasets because a `full` mutation requires that all entities are in memory to create a diff. If your data source has 100k+ records, this can easily cause your processes to run out of memory.
4. In cases when you can use `full` mutation, committing many entities into the processing pipeline fills up the processing queue and delays the processing of entities from other entity providers.
We created the Incremental Entity Provider to address all of the above issues. The Incremental Entity Provider addresses these issues with a combination of `delta` mutations and a mark-and-sweep mechanism. Instead of doing a single `full` mutation, it performs a series of bursts. At the end of each burst, the Incremental Entity Provider performs the following three operations,
We created the Incremental Entity Provider to address all of the above issues. The Incremental Entity Provider addresses these issues with a combination of `delta` mutations and a mark-and-sweep mechanism. Instead of doing a single `full` mutation, it performs a series of bursts. At the end of each burst, the Incremental Entity Provider performs the following operations,
1. Marks each received entity in the database.
2. Annotates each entity with `backstage/incremental-entity-provider: <entity-provider-id>` annotation.
3. Commits all of the entities with a `delta` mutation.
2. Commits all of the entities with a `delta` mutation.
Incremental Entity Providers will wait a configurable interval before proceeding to the next burst.
@@ -15,11 +15,7 @@
*/
import type { DeferredEntity } from '@backstage/plugin-catalog-backend';
import {
INCREMENTAL_ENTITY_PROVIDER_ANNOTATION,
IterationEngine,
IterationEngineOptions,
} from '../types';
import { IterationEngine, IterationEngineOptions } from '../types';
import { IncrementalIngestionDatabaseManager } from '../database/IncrementalIngestionDatabaseManager';
import type { AbortSignal } from 'node-abort-controller';
import { performance } from 'perf_hooks';
@@ -271,8 +267,6 @@ export class IncrementalIngestionEngine implements IterationEngine {
...deferred.entity.metadata,
annotations: {
...deferred.entity.metadata.annotations,
[INCREMENTAL_ENTITY_PROVIDER_ANNOTATION]:
this.options.provider.getProviderName(),
},
},
},
@@ -32,14 +32,6 @@ import type { DurationObjectUnits } from 'luxon';
import type { Logger } from 'winston';
import { IncrementalIngestionDatabaseManager } from './database/IncrementalIngestionDatabaseManager';
/**
* Entity annotation containing the incremental entity provider.
*
* @public
*/
export const INCREMENTAL_ENTITY_PROVIDER_ANNOTATION =
'backstage.io/incremental-provider-name';
/**
* Ingest entities into the catalog in bite-sized chunks.
*