paulononato/backstage
1
0
Fork 0
Code Issues Pull Requests Actions 42 Packages Projects Releases Wiki Activity

remove some extra content based on sugestions

Browse Source 
Signed-off-by: zeshanziya <zeshan.ziya@axelerant.com>
This commit is contained in:
zeshanziya
2024-08-29 11:39:29 +05:30
parent f99a2620fb
commit 13194a6f93
1 changed files with 18 additions and 63 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/features/software-catalog/external-integrations.md
+18 -63
View File
@@ -1034,44 +1034,9 @@ backend.start();
## Incremental Entity Provider
The Incremental Ingestion catalog backend module provides an Incremental Entity Provider that can be used to ingest data from sources using delta mutations, while retaining the orphan prevention mechanism provided by full mutations.
For large data sources that may not fit into memory but support pagination, the Incremental Entity Provider offers an efficient way to ingest data incrementally, handling deletions and updates seamlessly while minimizing memory usage.
### Why did we create it?
Backstage provides an [Entity Provider mechanism that has two kinds of mutations](https://backstage.io/docs/features/software-catalog/external-integrations#provider-mutations): `delta` and `full`. `delta` mutations tell the Backstage Catalog which entities should be added and removed from the catalog. `full` mutations accept a list of entities and automatically computes which entities must be removed by comparing the provided entities against existing entities to create a diff between the two sets. These two kinds of mutations are convenient for different kinds of data sources. A `delta` mutation can be used with a data source that emits UPDATE and DELETE events for its data. A `full` mutation is useful for APIs that produce fewer entities than can fit in the Backstage processes' memory.
Unfortunately, these two kinds of mutations are insufficient for very large data sources for the following reasons,
1. Even when the API provides DELETE events, we still need a way to create the initial list of entities. For example, if you ingest all repositories from GitHub into Backstage and you use webhooks, you still need the initial list of entities.
2. A `delta` mutation can not guarantee that mutations will not be missed. For example, if your Backstage portal is down while receiving a DELETE event, you might miss the event which leaves your catalog in an unclear state. How can you replay the missed events? Some data sources, like GitHub, provide an API for replaying missed events, but this increases complexity and is not available on all APIs.
3. Addressing the above two use case with `full` mutation is not an option on very large datasets because a `full` mutation requires that all entities are in memory to create a diff. If your data source has 100k+ records, this can easily cause your processes to run out of memory.
4. In cases when you can use `full` mutation, committing many entities into the processing pipeline fills up the processing queue and delays the processing of entities from other entity providers.
We created the Incremental Entity Provider to address all of the above issues. The Incremental Entity Provider addresses these issues with a combination of `delta` mutations and a mark-and-sweep mechanism. Instead of doing a single `full` mutation, it performs a series of bursts. At the end of each burst, the Incremental Entity Provider performs the following operations,
1. Marks each received entity in the database.
2. Commits all of the entities with a `delta` mutation.
Incremental Entity Providers will wait a configurable interval before proceeding to the next burst.
Once the source has no more results, Incremental Entity Provider compares all entities annotated with `@backstage/incremental-entity-provider: <entity-provider-id>` against all marked entities to determine which entities committed by same entity provider were not marked during the last ingestion cycle. All unmarked entities are deleted at the end of the cycle. The Incremental Entity Provider rests for a fixed interval before restarting the ingestion process.
![Diagram of execution of an Incremental Entity Provider](../../assets/software-catalog/software-catalog-incremental-provider-diagram.png)
This approach has the following benefits,
1. Reduced ingestion latency - each burst commits entities which are processed before the entire list is processed.
2. Stable pressure - each period between bursts provides an opportunity for the processing pipeline to settle without overwhelming the pipeline with a large number of unprocessed entities.
3. Built-in retry / back-off - Failed bursts are automatically retried with a built-in back-off interval providing an opportunity for the data source to reset its rate limits before retrying the burst.
4. Prevents orphan entities - Deleted entities are removed as with `full` mutation with a low memory footprint.
### Requirements
The Incremental Entity Provider backend is designed for data sources that provide paginated results. Each burst attempts to handle one or more pages of the query. The plugin will attempt to fetch as many pages as it can within a configurable burst length. At every iteration, it expects to receive the next cursor that will be used to query in the next iteration. Each iteration may happen on a different replica. This has several consequences:
1. The cursor must be serializable to JSON (not an issue for most RESTful or GraphQL based APIs).
2. The client must be stateless - a client is created from scratch for each iteration to allow distributing processing over multiple replicas.
3. There must be sufficient storage in Postgres to handle the additional data. (Presumably, this is also true of sqlite, but it has only been tested with Postgres.)
You can find more details about [why it was created](https://github.com/backstage/backstage/tree/master/plugins/catalog-backend-module-incremental-ingestion#why-did-we-create-it) and its [requirements](https://github.com/backstage/backstage/tree/master/plugins/catalog-backend-module-incremental-ingestion#requirements).
### Installation
@@ -1080,9 +1045,6 @@ The Incremental Entity Provider backend is designed for data sources that provid
2. Add the following code to the `packages/backend/src/index.ts` file:
```ts title="packages/backend/src/index.ts"
/* highlight-add-next-line */
import { catalogModuleCustomIncrementalIngestionProvider } from './extensions/catalogCustomIncrementalIngestion';
const backend = createBackend();
/* highlight-add-start */
@@ -1093,16 +1055,12 @@ backend.add(
);
/* highlight-add-end */
// We have created this in section **Adding an Incremental Entity Provider to the catalog**
/* highlight-add-next-line */
backend.add(catalogModuleCustomIncrementalIngestionProvider);
backend.start();
```
### Writing an Incremental Entity Provider
To create an Incremental Entity Provider, you need to know how to retrieve a single page of the data that you wish to ingest into the Backstage catalog. If the API has pagination and you know how to make a paginated request to that API, you'll be able to implement an Incremental Entity Provider for this API. For more information about compatibility, check out the [requirements](#requirements) section of this page.
To create an Incremental Entity Provider, you need to know how to retrieve a single page of the data that you wish to ingest into the Backstage catalog. If the API has pagination and you know how to make a paginated request to that API, you'll be able to implement an Incremental Entity Provider for this API. For more information about compatibility, check out the [requirements](https://github.com/backstage/backstage/tree/master/plugins/catalog-backend-module-incremental-ingestion#requirements) section.
Here is the type definition for an Incremental Entity Provider.
@@ -1364,27 +1322,24 @@ export const catalogModuleCustomIncrementalIngestionProvider =
});
```
That's it!!!
Add the module to `packages/backend/src/index.ts`
### Administrative Routes
```ts title="packages/backend/src/index.ts"
/* highlight-add-next-line */
import { catalogModuleCustomIncrementalIngestionProvider } from './extensions/catalogCustomIncrementalIngestion';
If you want to manage your incremental entity providers via REST endpoints, the following endpoints are available:
const backend = createBackend();
| Method | Path | Description |
| ------ | ------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------- |
| GET | `/api/catalog/incremental/health` | Checks the health of all incremental providers. Returns array of any unhealthy ones. |
| GET | `/api/catalog/incremental/providers` | Get a list of all known incremental entity providers |
| GET | `/api/catalog/incremental/providers/:provider` | Checks the status of an incremental provider (resting, interstitial, etc). |
| POST | `/api/catalog/incremental/providers/:provider/trigger` | Triggers a provider's next action immediately. E.g., if it's currently interstitial, it will trigger the next burst. |
| POST | `/api/catalog/incremental/providers/:provider/start` | Stop the current ingestion cycle and start a new one immediately. |
| POST | `/api/catalog/incremental/providers/:provider/cancel` | Stop the current ingestion cycle and start a new one in 24 hours. |
| DELETE | `/api/catalog/incremental/providers/:provider` | Completely remove all records for the provider and schedule it to start again in 24 hours. |
| GET | `/api/catalog/incremental/providers/:provider/marks` | Retrieve a list of all ingestion marks for the current ingestion cycle. |
| DELETE | `/api/catalog/incremental/providers/:provider/marks` | Remove all ingestion marks for the current ingestion cycle. |
| POST | `/api/catalog/incremental/cleanup` | Completely remove all records for ALL providers and schedule them to start again in 24 hours. (CAUTION! Can cause orphans!) |
backend.add(
import(
'@backstage/plugin-catalog-backend-module-incremental-ingestion/alpha'
),
);
In all cases, `:provider` is the name of the incremental entity provider.
/* highlight-add-next-line */
backend.add(catalogModuleCustomIncrementalIngestionProvider);
### Error handling
backend.start();
```
If `around` or `next` methods throw an error, the error will show up in logs and it'll trigger the Incremental Entity Provider to try again after a back-off period. It'll keep trying until it reaches the last back-off attempt, at which point it will cancel the current ingestion and start over. You don't need to do anything special to handle the retry logic.
For a deep dive into the technical details of the Incremental Entity Provider, see [the README](https://github.com/backstage/backstage/tree/master/plugins/catalog-backend-module-incremental-ingestion).