diff --git a/docs/features/software-catalog/descriptor-format.md b/docs/features/software-catalog/descriptor-format.md index aab2af3145..c8449dd351 100644 --- a/docs/features/software-catalog/descriptor-format.md +++ b/docs/features/software-catalog/descriptor-format.md @@ -399,16 +399,17 @@ well-known / common relations and their semantics. ## Common to All Kinds: Status -The `status` root field is a read-only set of statuses, pertaining to the +The `status` root object is a read-only set of statuses, pertaining to the current state or health of the entity, described in the -[well-known statuses section](well-known-statuses.md). Each status field -contains a specific blob of data that describes some aspect of the state of the -entity, as seen from the point of view of some specific system. Different -systems may contribute to this status object, under their own respective keys. +[well-known statuses section](well-known-statuses.md). + +Currently, the only defined field is the `items` array. Each of its items +contains a specific data structure that describes some aspect of the state of +the entity, as seen from the point of view of some specific system. Different +systems may contribute to this array, under their own respective `type` keys. The current main use case for this field is for the ingestion processes of the -catalog itself to convey information about failures and warnings back to the -user. +catalog itself to convey information about errors and warnings back to the user. A status field as part of a single entity that's read out of the API may look as follows. @@ -417,9 +418,18 @@ follows. { // ... "status": { - "backstage.io/catalog-processing": { - "errors": [] - } + "items": [ + { + "type": "backstage.io/catalog-processing", + "level": "error", + "message": "NotFoundError: File not found", + "error": { + "name": "NotFoundError", + "message": "File not found", + "stack": "..." + } + } + ] }, "spec": { // ... @@ -427,23 +437,27 @@ follows. } ``` -The keys of the `status` object are arbitrary strings. We recommend that any -statuses that are not strictly private within the organization be namespaced to -avoid collisions. Statuses emitted by Backstage core processes will for example -be prefixed with `backstage.io/` as in the example above. +The fields of a status item are: -The values of the `status` object are currently left unrestricted, except that -they must be objects. We reserve the right to extend this model in the future, -such that some fields of those value objects gain standardized meaning. We may -for example want to add a standard concept of "severity" or "level" to these. +| Field | Type | Description | +| --------- | ------ | ------------------------------------------------------------------------------------------------ | +| `type` | String | The type of status as a unique key per source. Each type may appear more than once in the array. | +| `level` | String | The level / severity of the status item: 'info', 'warning, or 'error'. | +| `message` | String | A brief message describing the status, intended for human consumption. | +| `error` | Object | An optional serialized error object related to the status. | -Entity descriptor YAML files are not supposed to contain this field. Instead, -catalog processors analyze the entity descriptor data and its surroundings, and -deduce status entries that are then attached onto the entity as read from the -catalog. +The `type` is an arbitrary string, but we recommend that types that are not +strictly private within the organization be namespaced to avoid collisions. +Types emitted by Backstage core processes will for example be prefixed with +`backstage.io/` as in the example above. + +Entity descriptor YAML files are not supposed to contain a `status` root key. +Instead, catalog processors analyze the entity descriptor data and its +surroundings, and deduce status entries that are then attached onto the entity +as read from the catalog. See the [well-known statuses section](well-known-statuses.md) for a list of -well-known / common relations and their semantics. +well-known / common status types. ## Kind: Component diff --git a/docs/features/software-catalog/extending-the-model.md b/docs/features/software-catalog/extending-the-model.md index 4cc1bc1110..aa5aed0540 100644 --- a/docs/features/software-catalog/extending-the-model.md +++ b/docs/features/software-catalog/extending-the-model.md @@ -353,18 +353,43 @@ relation could be considered for addition to the core. ## Adding a New Status field -Example intents: +Example intent: > "We would like to convey entity statuses through the catalog in a generic way, > as an integration layer. Our monitoring and alerting system has a plugin with > Backstage, and it would be useful if the entity's status field contained the -> current alert state close to the actual entity data for anyone to consume. +> current alert state close to the actual entity data for anyone to consume. We +> find the `status.items` semantics a poor fit, so we would prefer to make our +> own custom field under `status` for these purposes." -While we are considering a mechanism for contributing generic statuses to -entities, no such mechanism has yet been built. If you are interested in that -topic, [this issue](https://github.com/backstage/backstage/issues/2292) contains +We have not yet ventured to define any generic semantics for the `status` +object. We recommend sticking with the `status.items` mechanism where possible +(see below), since third party consumers will not be able to consume your status +information otherwise. Please reach out to the maintainers on Discord or by +making a GitHub issue describing your use case if you are interested in this +topic. + +## Adding a New Status Item Type + +Example intent: + +> "The semantics of the entity `status.items` field are fine for our needs, but +> we want to contribute our own type of status into that array instead of the +> catalog specific one." + +This is a simple, low risk way of adding your own status information to +entities. Consumers will be able to easily track and display the status together +with other types / sources. + +We recommend that any status type that are not strictly private within the +organization be namespaced to avoid collisions. Statuses emitted by Backstage +core processes will for example be prefixed with `backstage.io/`, your +organization may prefix with `my-org.net/`, and `pagerduty.com/active-alerts` +could be a sensible complete status item type for that particular external +system. + +The mechanics for how to emit custom statuses is not in place yet, so if this is +of interest to you, you might consider contacting the maintainers on Discord or +my making a GitHub issue describing your use case. +[This issue](https://github.com/backstage/backstage/issues/2292) also contains more context. - -But in general, errors emitted (and exceptions thrown) by any processor -including custom ones, end up in the [well known key](well-known-statuses.md) -for ingestion status. diff --git a/docs/features/software-catalog/well-known-statuses.md b/docs/features/software-catalog/well-known-statuses.md index 7ddf320d26..d293ae5062 100644 --- a/docs/features/software-catalog/well-known-statuses.md +++ b/docs/features/software-catalog/well-known-statuses.md @@ -6,35 +6,37 @@ sidebar_label: Well-known Statuses description: Lists a number of well known entity statuses, that have defined semantics. They can be attached to catalog entities and consumed by plugins as needed. --- -This section lists a number of well known -[entity status fields](descriptor-format.md#common-to-all-kinds-status), that -have defined semantics. They can be attached to catalog entities and consumed by +This section lists well known +[entity statuses](descriptor-format.md#common-to-all-kinds-status), that have +defined semantics. They can be attached to catalog entities and consumed by plugins as needed. -If you are looking to extend the set of statuses, see +If you are looking to extend the statuses, see [Extending the model](extending-the-model.md). ## Common Fields -The values of statuses are currently left unrestricted, except that they must be -objects. They therefore currently formally have no common fields. +The `status` object of an entity is currently left unrestricted, except for the +`items` field. Its structure is defined in the +[descriptor format](descriptor-format.md#common-to-all-kinds-status) section. -We reserve the right to extend this model in the future, such that some fields -of those value objects gain standardized meaning. We may for example want to add -a standard concept of "severity" or "level" to these. +We reserve the right to extend this model in the future. This status is in +active development and its format will change unexpectedly. Do not consume it in +your own code until such a time that this documentation has been updated. -## Statuses +## Status Item Types -This is a (non-exhaustive) list of statuses that are known to be in active use. +This is a (non-exhaustive) list of `status.items.[].type` values that are known +to be in active use. ### `backstage.io/catalog-processing` -Contains the current status of the catalog's ingestion of this entity. Errors -that may appear here include inability to read from the remote SCM provider, -syntax errors in the YAML file, and similar. +Expresses an aspect of the current status of the catalog's ingestion of this +entity. Errors that may appear here include inability to read from the remote +SCM provider, syntax errors in the YAML file, and similar. Note that the entity data itself may be of an older version, when errors are -present. The ingestion system keeps the old, valid entity data untouched when +present. The ingestion system keeps the old valid entity data untouched when possible, so the errors described in this state may not seem to align with the rest of the entity, because they pertain to a remote that could not be successfully ingested. This is normal. @@ -42,10 +44,12 @@ successfully ingested. This is normal. ```yaml # Example: status: - backstage.io/catalog-processing: - errors: [] + items: + - type: backstage.io/catalog-processing + level: error + message: 'NotFoundError: File not found' + error: + name: NotFoundError + message: File not found + stack: ... ``` - -This status is in active development and its format will change unexpectedly. Do -not consume it in your own code until such a time that this documentation has -been updated. diff --git a/package.json b/package.json index ed404565e4..ff24cafbd0 100644 --- a/package.json +++ b/package.json @@ -49,8 +49,8 @@ }, "version": "1.0.0", "dependencies": { - "@microsoft/api-extractor": "7.13.2-pr1916.0", "@microsoft/api-documenter": "^7.12.16", + "@microsoft/api-extractor": "7.13.2-pr1916.0", "@microsoft/api-extractor-model": "^7.12.5" }, "devDependencies": { diff --git a/packages/catalog-client/api-report.md b/packages/catalog-client/api-report.md index 003e585671..1f911c84e9 100644 --- a/packages/catalog-client/api-report.md +++ b/packages/catalog-client/api-report.md @@ -7,9 +7,6 @@ import { Entity } from '@backstage/catalog-model'; import { EntityName } from '@backstage/catalog-model'; import { Location as Location_2 } from '@backstage/catalog-model'; -import { SerializedError } from '@backstage/errors'; -import { UNSTABLE_EntityStatusLevel } from '@backstage/catalog-model'; -import { UNSTABLE_EntityStatusValue } from '@backstage/catalog-model'; // @public (undocumented) export type AddLocationRequest = { @@ -80,19 +77,7 @@ export type CatalogListResponse = { }; // @public -export const ENTITY_STATUS_CATALOG_PROCESSING_KEY = "backstage.io/catalog-processing"; - -// @public -export type UNSTABLE_CatalogProcessingStatus = UNSTABLE_EntityStatusValue & { - items?: UNSTABLE_CatalogProcessingStatusItem[]; -}; - -// @public (undocumented) -export type UNSTABLE_CatalogProcessingStatusItem = { - status: UNSTABLE_EntityStatusLevel; - message?: string; - error?: SerializedError; -}; +export const ENTITY_STATUS_CATALOG_PROCESSING_TYPE = "backstage.io/catalog-processing"; // (No @packageDocumentation comment for this package) diff --git a/packages/catalog-client/src/types/index.ts b/packages/catalog-client/src/types/index.ts index 16850a0fc0..c1670659ec 100644 --- a/packages/catalog-client/src/types/index.ts +++ b/packages/catalog-client/src/types/index.ts @@ -21,8 +21,4 @@ export type { CatalogEntitiesRequest, CatalogListResponse, } from './api'; -export { ENTITY_STATUS_CATALOG_PROCESSING_KEY } from './status'; -export type { - UNSTABLE_CatalogProcessingStatus, - UNSTABLE_CatalogProcessingStatusItem, -} from './status'; +export { ENTITY_STATUS_CATALOG_PROCESSING_TYPE } from './status'; diff --git a/packages/catalog-client/src/types/status.ts b/packages/catalog-client/src/types/status.ts index 6c39a83ea7..7990c4b121 100644 --- a/packages/catalog-client/src/types/status.ts +++ b/packages/catalog-client/src/types/status.ts @@ -14,50 +14,9 @@ * limitations under the License. */ -import { - UNSTABLE_EntityStatusLevel, - UNSTABLE_EntityStatusValue, -} from '@backstage/catalog-model'; -import { SerializedError } from '@backstage/errors'; - -/* - * This is the entity status field that's emitted by the catalog processing - * engine, to inform about the status of an entity. - * - * Example: - * - * "status": { - * "backstage.io/catalog-processing": { - * "status": "error", - * "items": [ - * { - * "status": "error", - * "error": { - * "name": "InputError", - * "message": "Syntax error: ..." - * } - * } - * ] - * } - * } - */ - /** - * The entity `status` key for the status of the processing engine in regards - * to entity. + * The entity `status.items[].type` for the status of the processing engine in + * regards to an entity. */ -export const ENTITY_STATUS_CATALOG_PROCESSING_KEY = +export const ENTITY_STATUS_CATALOG_PROCESSING_TYPE = 'backstage.io/catalog-processing'; - -/** - * The status value for the `backstage.io/catalog-processing` key. - */ -export type UNSTABLE_CatalogProcessingStatus = UNSTABLE_EntityStatusValue & { - items?: UNSTABLE_CatalogProcessingStatusItem[]; -}; - -export type UNSTABLE_CatalogProcessingStatusItem = { - status: UNSTABLE_EntityStatusLevel; - message?: string; - error?: SerializedError; -}; diff --git a/packages/catalog-model/api-report.md b/packages/catalog-model/api-report.md index c9118ce52c..7b5542fcd6 100644 --- a/packages/catalog-model/api-report.md +++ b/packages/catalog-model/api-report.md @@ -7,6 +7,7 @@ import { JsonObject } from '@backstage/config'; import { JSONSchema7 } from 'json-schema'; import { JsonValue } from '@backstage/config'; +import { SerializedError } from '@backstage/errors'; import * as yup from 'yup'; // @public (undocumented) @@ -112,7 +113,7 @@ export type Entity = { metadata: EntityMeta; spec?: JsonObject; relations?: EntityRelation[]; - status?: Record; + status?: UNSTABLE_EntityStatus; }; // @public @@ -528,15 +529,22 @@ export interface TemplateEntityV1beta2 extends Entity { // @public (undocumented) export const templateEntityV1beta2Validator: KindValidator; -// @public -export type UNSTABLE_EntityStatusLevel = 'ok' | 'info' | 'warning' | 'error'; - -// @public -export type UNSTABLE_EntityStatusValue = { - status: UNSTABLE_EntityStatusLevel; - message?: string; +// @alpha +export type UNSTABLE_EntityStatus = { + items?: UNSTABLE_EntityStatusItem[]; }; +// @alpha +export type UNSTABLE_EntityStatusItem = { + type: string; + level: UNSTABLE_EntityStatusLevel; + message: string; + error?: SerializedError; +}; + +// @alpha +export type UNSTABLE_EntityStatusLevel = 'info' | 'warning' | 'error'; + // @public (undocumented) interface UserEntityV1alpha1 extends Entity { // (undocumented) diff --git a/packages/catalog-model/package.json b/packages/catalog-model/package.json index 7a19be8708..97ce0e6568 100644 --- a/packages/catalog-model/package.json +++ b/packages/catalog-model/package.json @@ -30,6 +30,7 @@ }, "dependencies": { "@backstage/config": "^0.1.5", + "@backstage/errors": "^0.1.1", "@types/json-schema": "^7.0.5", "@types/yup": "^0.29.8", "ajv": "^7.0.3", diff --git a/packages/catalog-model/src/entity/Entity.ts b/packages/catalog-model/src/entity/Entity.ts index 1b5d38dee9..3b5a0621bb 100644 --- a/packages/catalog-model/src/entity/Entity.ts +++ b/packages/catalog-model/src/entity/Entity.ts @@ -16,6 +16,7 @@ import { JsonObject } from '@backstage/config'; import { EntityName } from '../types'; +import { UNSTABLE_EntityStatus } from './EntityStatus'; /** * The format envelope that's common to all versions/kinds of entity. @@ -55,7 +56,7 @@ export type Entity = { * The keys are implementation defined and the values can be any JSON object * with semantics that match that implementation. */ - status?: Record; + status?: UNSTABLE_EntityStatus; }; /** diff --git a/packages/catalog-model/src/entity/EntityStatus.ts b/packages/catalog-model/src/entity/EntityStatus.ts new file mode 100644 index 0000000000..67090f2bc2 --- /dev/null +++ b/packages/catalog-model/src/entity/EntityStatus.ts @@ -0,0 +1,63 @@ +/* + * Copyright 2021 Spotify AB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { SerializedError } from '@backstage/errors'; + +/** + * The current status of the entity, as claimed by various sources. + * @alpha + */ +export type UNSTABLE_EntityStatus = { + /** + * Specific status item on a well known format. + */ + items?: UNSTABLE_EntityStatusItem[]; +}; + +/** + * A specific status item on a well known format. + * @alpha + */ +export type UNSTABLE_EntityStatusItem = { + /** + * The type of status as a unique key per source. + */ + type: string; + /** + * The level / severity of the status item. If the level is "error", the + * processing of the entity may be entirely blocked. In this case the status + * entry may apply to a different, newer version of the data than what is + * being returned in the catalog response. + */ + level: UNSTABLE_EntityStatusLevel; + /** + * A brief message describing the status, intended for human consumption. + */ + message: string; + /** + * An optional serialized error object related to the status. + */ + error?: SerializedError; +}; + +/** + * Each entity status item has a level, describing its severity. + * @alpha + */ +export type UNSTABLE_EntityStatusLevel = + | 'info' // Only informative data + | 'warning' // Warnings were found + | 'error'; // Errors were found diff --git a/packages/catalog-model/src/entity/index.ts b/packages/catalog-model/src/entity/index.ts index 330856125e..2ccd205d85 100644 --- a/packages/catalog-model/src/entity/index.ts +++ b/packages/catalog-model/src/entity/index.ts @@ -27,6 +27,11 @@ export type { EntityRelation, EntityRelationSpec, } from './Entity'; +export type { + UNSTABLE_EntityStatus, + UNSTABLE_EntityStatusItem, + UNSTABLE_EntityStatusLevel, +} from './EntityStatus'; export * from './policies'; export { compareEntityToRef, @@ -36,10 +41,6 @@ export { serializeEntityRef, stringifyEntityRef, } from './ref'; -export type { - UNSTABLE_EntityStatusLevel, - UNSTABLE_EntityStatusValue, -} from './status'; export { entityHasChanges, generateEntityEtag, diff --git a/packages/catalog-model/src/entity/status.ts b/packages/catalog-model/src/entity/status.ts deleted file mode 100644 index a819bea8f6..0000000000 --- a/packages/catalog-model/src/entity/status.ts +++ /dev/null @@ -1,32 +0,0 @@ -/* - * Copyright 2021 Spotify AB - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -/** - * Each entity status entry has a level, describing its severity. - */ -export type UNSTABLE_EntityStatusLevel = - | 'ok' // Everything is OK - | 'info' // Only informative data - | 'warning' // Warnings were found - | 'error'; // Errors were found - -/** - * Reserved root fields in all `status` object values. - */ -export type UNSTABLE_EntityStatusValue = { - status: UNSTABLE_EntityStatusLevel; - message?: string; -}; diff --git a/packages/catalog-model/src/schema/Entity.schema.json b/packages/catalog-model/src/schema/Entity.schema.json index e6b004dfd0..b8c43dda9e 100644 --- a/packages/catalog-model/src/schema/Entity.schema.json +++ b/packages/catalog-model/src/schema/Entity.schema.json @@ -64,13 +64,7 @@ } }, "status": { - "type": "object", - "description": "The current status of the entity, as claimed by various sources.", - "patternProperties": { - "^.+$": { - "$ref": "common#status" - } - } + "$ref": "common#status" } } } diff --git a/packages/catalog-model/src/schema/shared/common.schema.json b/packages/catalog-model/src/schema/shared/common.schema.json index cb4d82cdb9..efb8b6d1bb 100644 --- a/packages/catalog-model/src/schema/shared/common.schema.json +++ b/packages/catalog-model/src/schema/shared/common.schema.json @@ -46,8 +46,53 @@ "status": { "$id": "#status", "type": "object", - "description": "A specific status of an entity.", - "additionalProperties": true + "description": "The current status of the entity, as claimed by various sources.", + "required": [], + "additionalProperties": true, + "properties": { + "items": { + "$ref": "#statusItem" + } + } + }, + "statusItem": { + "$id": "#statusItem", + "type": "object", + "description": "A specific status item on a well known format.", + "required": ["type", "level", "message"], + "additionalProperties": true, + "properties": { + "type": { + "type": "string", + "minLength": 1 + }, + "level": { + "$ref": "#statusLevel", + "description": "The status level / severity of the status item." + }, + "message": { + "type": "string", + "description": "A brief message describing the status, intended for human consumption." + }, + "error": { + "$ref": "#error", + "description": "An optional serialized error object related to the status." + } + } + }, + "statusLevel": { + "$id": "#statusLevel", + "type": "string", + "description": "A status level / severity.", + "enum": ["info", "warning", "error"] + }, + "error": { + "$id": "#error", + "type": "object", + "description": "A serialized error object.", + "required": [], + "additionalProperties": true, + "properties": {} } } } diff --git a/plugins/catalog-backend/src/next/DefaultCatalogProcessingEngine.ts b/plugins/catalog-backend/src/next/DefaultCatalogProcessingEngine.ts index 657c331a25..b7b6dfa297 100644 --- a/plugins/catalog-backend/src/next/DefaultCatalogProcessingEngine.ts +++ b/plugins/catalog-backend/src/next/DefaultCatalogProcessingEngine.ts @@ -119,7 +119,27 @@ export class DefaultCatalogProcessingEngine implements CatalogProcessingEngine { for (const error of result.errors) { this.logger.warn(error.message); } + const errorsString = JSON.stringify( + result.errors.map(e => serializeError(e)), + ); + + // If the result was marked as not OK, it signals that some part of the + // processing pipeline threw an exception. This can happen both as part of + // non-catastrophic things such as due to validation errors, as well as if + // something fatal happens inside the processing for other reasons. In any + // case, this means we can't trust that anything in the output is okay. So + // just store the errors and trigger a stich so that they become visible to + // the outside. if (!result.ok) { + await this.processingDatabase.transaction(async tx => { + await this.processingDatabase.updateProcessedEntityErrors(tx, { + id, + errors: errorsString, + }); + }); + await this.stitcher.stitch( + new Set([stringifyEntityRef(unprocessedEntity)]), + ); return; } @@ -129,7 +149,7 @@ export class DefaultCatalogProcessingEngine implements CatalogProcessingEngine { id, processedEntity: result.completedEntity, state: result.state, - errors: JSON.stringify(result.errors.map(e => serializeError(e))), + errors: errorsString, relations: result.relations, deferredEntities: result.deferredEntities, }); diff --git a/plugins/catalog-backend/src/next/Stitcher.test.ts b/plugins/catalog-backend/src/next/Stitcher.test.ts index 75286743c3..f38af384fa 100644 --- a/plugins/catalog-backend/src/next/Stitcher.test.ts +++ b/plugins/catalog-backend/src/next/Stitcher.test.ts @@ -16,7 +16,6 @@ import { getVoidLogger } from '@backstage/backend-common'; import { Knex } from 'knex'; -import { ENTITY_STATUS_CATALOG_PROCESSING_KEY } from '@backstage/catalog-client'; import { DatabaseManager } from './database/DatabaseManager'; import { DbRefreshStateReferencesRow, @@ -92,9 +91,6 @@ describe('Stitcher', () => { }, }, ], - status: { - [ENTITY_STATUS_CATALOG_PROCESSING_KEY]: { status: 'ok' }, - }, apiVersion: 'a', kind: 'k', metadata: { @@ -175,9 +171,6 @@ describe('Stitcher', () => { }, }, ]), - status: { - [ENTITY_STATUS_CATALOG_PROCESSING_KEY]: { status: 'ok' }, - }, apiVersion: 'a', kind: 'k', metadata: { diff --git a/plugins/catalog-backend/src/next/Stitcher.ts b/plugins/catalog-backend/src/next/Stitcher.ts index 36ead7d817..0281591fb1 100644 --- a/plugins/catalog-backend/src/next/Stitcher.ts +++ b/plugins/catalog-backend/src/next/Stitcher.ts @@ -14,11 +14,12 @@ * limitations under the License. */ +import { ENTITY_STATUS_CATALOG_PROCESSING_TYPE } from '@backstage/catalog-client'; import { - ENTITY_STATUS_CATALOG_PROCESSING_KEY, - UNSTABLE_CatalogProcessingStatusItem, -} from '@backstage/catalog-client'; -import { Entity, parseEntityRef } from '@backstage/catalog-model'; + Entity, + parseEntityRef, + UNSTABLE_EntityStatusItem, +} from '@backstage/catalog-model'; import { ConflictError, SerializedError } from '@backstage/errors'; import { createHash } from 'crypto'; import stableStringify from 'fast-json-stable-stringify'; @@ -138,7 +139,7 @@ export class Stitcher { // it const entity = JSON.parse(processedEntity) as Entity; const isOrphan = Number(incomingReferenceCount) === 0; - let statusItems: UNSTABLE_CatalogProcessingStatusItem[] = []; + let statusItems: UNSTABLE_EntityStatusItem[] = []; if (isOrphan) { this.logger.debug(`${entityRef} is an orphan`); @@ -151,7 +152,9 @@ export class Stitcher { const parsedErrors = JSON.parse(errors) as SerializedError[]; if (Array.isArray(parsedErrors) && parsedErrors.length) { statusItems = parsedErrors.map(e => ({ - status: 'error', + type: ENTITY_STATUS_CATALOG_PROCESSING_TYPE, + level: 'error', + message: e.toString(), error: e, })); } @@ -165,12 +168,12 @@ export class Stitcher { type: row.relationType!, target: parseEntityRef(row.relationTarget!), })); - entity.status = { - ...entity.status, - [ENTITY_STATUS_CATALOG_PROCESSING_KEY]: statusItems.length - ? { status: 'error', items: statusItems } - : { status: 'ok' }, - }; + if (statusItems.length) { + entity.status = { + ...entity.status, + items: [...(entity.status?.items ?? []), ...statusItems], + }; + } // If the output entity was actually not changed, just abort const hash = generateStableHash(entity); diff --git a/plugins/catalog-backend/src/next/database/DefaultProcessingDatabase.ts b/plugins/catalog-backend/src/next/database/DefaultProcessingDatabase.ts index 8588ba5e52..cd3ddf037b 100644 --- a/plugins/catalog-backend/src/next/database/DefaultProcessingDatabase.ts +++ b/plugins/catalog-backend/src/next/database/DefaultProcessingDatabase.ts @@ -123,6 +123,20 @@ export class DefaultProcessingDatabase implements ProcessingDatabase { ); } + async updateProcessedEntityErrors( + txOpaque: Transaction, + options: UpdateProcessedEntityOptions, + ): Promise { + const tx = txOpaque as Knex.Transaction; + const { id, errors } = options; + + await tx('refresh_state') + .update({ + errors, + }) + .where('entity_id', id); + } + private deduplicateRelations(rows: DbRelationsRow[]): DbRelationsRow[] { return lodash.uniqBy( rows, diff --git a/plugins/catalog-backend/src/next/database/types.ts b/plugins/catalog-backend/src/next/database/types.ts index d7a1dcd20b..f65c796a00 100644 --- a/plugins/catalog-backend/src/next/database/types.ts +++ b/plugins/catalog-backend/src/next/database/types.ts @@ -34,6 +34,11 @@ export type UpdateProcessedEntityOptions = { deferredEntities: Entity[]; }; +export type UpdateProcessedEntityErrorsOptions = { + id: string; + errors?: string; +}; + export type RefreshStateItem = { id: string; entityRef: string; @@ -80,10 +85,18 @@ export interface ProcessingDatabase { ): Promise; /** - * Updates the + * Updates a processed entity */ updateProcessedEntity( txOpaque: Transaction, options: UpdateProcessedEntityOptions, ): Promise; + + /** + * Updates only the errors of a processed entity + */ + updateProcessedEntityErrors( + txOpaque: Transaction, + options: UpdateProcessedEntityErrorsOptions, + ): Promise; }