paulononato/backstage
1
0
Fork 0
Code Issues Pull Requests Actions 34 Packages Projects Releases Wiki Activity

run prettier

Browse Source 
Signed-off-by: Paul Cowan <paul.cowan@cutting.scot>
This commit is contained in:
Paul Cowan
2023-01-31 14:35:07 +00:00
parent 8c02d89dbe
commit 14985032b5
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
microsite/blog/2023-01-31-incremental-entity-provider.md
+1 -1
View File
@@ -25,7 +25,7 @@ A mutation can be either a full mutation or a delta mutation. A full mutation re
A large organization typically deals with massive datasets. Until recently, ingesting large datasets with entity providers has been problematic because performing a full ingestion resulted in out-of-memory errors, and many data sources don’t provide webhooks or other events-based APIs. At the same time, the datasets were too large to efficiently manage through targeted delta mutations.
This is a problem that [DevEx team at HP](http://hp.com) faced when building their software catalog with Backstage. Damon Kaswell, Senior Application Developer on the DevEx team at HP, shared their experience at [BackstageCon 2022](https://www.youtube.com/watch?v=5qHyZntKXRU&list=PLj6h78yzYM2OKySsTuiip3BqmdYZQRnSf&index=13), detailing the problem and the solution that Frontside created in collaboration with developers on HP’s DevEx team.
This is a problem that [DevEx team at HP](http://hp.com) faced when building their software catalog with Backstage. [Damon Kaswell](https://github.com/dekoding), Senior Application Developer on the DevEx team at HP, shared their experience at [BackstageCon 2022](https://www.youtube.com/watch?v=5qHyZntKXRU&list=PLj6h78yzYM2OKySsTuiip3BqmdYZQRnSf&index=13), detailing the problem and the solution that Frontside created in collaboration with developers on HP’s DevEx team.
The solution HP and Frontside arrived at was to implement an incremental entity provider. An incremental entity provider effectively performs a full mutation using a series of delta mutations combined with a mark and sweep mechanism. It paginates through the dataset, tracking entities retrieved from each page and the cursor of the next page, pausing ingestion every few minutes to give the processing loop time to process existing entities. Once it reaches the end of the dataset, it determines which entities were not ingested during this ingestion cycle and emits a delta mutation to delete unmarked entities.