paulononato/backstage
1
0
Fork 0
Code Issues Pull Requests Actions 46 Packages Projects Releases Wiki Activity

Implement suggested changes after discussions

Browse Source 
Signed-off-by: Fredrik Adelöw <freben@gmail.com>
This commit is contained in:
Fredrik Adelöw
2021-05-24 17:03:50 +02:00
parent add62a4552
commit aef9fb2399
20 changed files with 301 additions and 194 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/features/software-catalog/descriptor-format.md
+37 -23
View File
@@ -399,16 +399,17 @@ well-known / common relations and their semantics.
## Common to All Kinds: Status
The `status` root field is a read-only set of statuses, pertaining to the
The `status` root object is a read-only set of statuses, pertaining to the
current state or health of the entity, described in the
[well-known statuses section](well-known-statuses.md). Each status field
contains a specific blob of data that describes some aspect of the state of the
entity, as seen from the point of view of some specific system. Different
systems may contribute to this status object, under their own respective keys.
[well-known statuses section](well-known-statuses.md).
Currently, the only defined field is the `items` array. Each of its items
contains a specific data structure that describes some aspect of the state of
the entity, as seen from the point of view of some specific system. Different
systems may contribute to this array, under their own respective `type` keys.
The current main use case for this field is for the ingestion processes of the
catalog itself to convey information about failures and warnings back to the
user.
catalog itself to convey information about errors and warnings back to the user.
A status field as part of a single entity that's read out of the API may look as
follows.
@@ -417,9 +418,18 @@ follows.
{
// ...
"status": {
"backstage.io/catalog-processing": {
"errors": []
}
"items": [
{
"type": "backstage.io/catalog-processing",
"level": "error",
"message": "NotFoundError: File not found",
"error": {
"name": "NotFoundError",
"message": "File not found",
"stack": "..."
}
}
]
},
"spec": {
// ...
@@ -427,23 +437,27 @@ follows.
}
```
The keys of the `status` object are arbitrary strings. We recommend that any
statuses that are not strictly private within the organization be namespaced to
avoid collisions. Statuses emitted by Backstage core processes will for example
be prefixed with `backstage.io/` as in the example above.
The fields of a status item are:
The values of the `status` object are currently left unrestricted, except that
they must be objects. We reserve the right to extend this model in the future,
such that some fields of those value objects gain standardized meaning. We may
for example want to add a standard concept of "severity" or "level" to these.
| Field | Type | Description |
| --------- | ------ | ------------------------------------------------------------------------------------------------ |
| `type` | String | The type of status as a unique key per source. Each type may appear more than once in the array. |
| `level` | String | The level / severity of the status item: 'info', 'warning, or 'error'. |
| `message` | String | A brief message describing the status, intended for human consumption. |
| `error` | Object | An optional serialized error object related to the status. |
Entity descriptor YAML files are not supposed to contain this field. Instead,
catalog processors analyze the entity descriptor data and its surroundings, and
deduce status entries that are then attached onto the entity as read from the
catalog.
The `type` is an arbitrary string, but we recommend that types that are not
strictly private within the organization be namespaced to avoid collisions.
Types emitted by Backstage core processes will for example be prefixed with
`backstage.io/` as in the example above.
Entity descriptor YAML files are not supposed to contain a `status` root key.
Instead, catalog processors analyze the entity descriptor data and its
surroundings, and deduce status entries that are then attached onto the entity
as read from the catalog.
See the [well-known statuses section](well-known-statuses.md) for a list of
well-known / common relations and their semantics.
well-known / common status types.
## Kind: Component
docs/features/software-catalog/extending-the-model.md
+34 -9
View File
@@ -353,18 +353,43 @@ relation could be considered for addition to the core.
## Adding a New Status field
Example intents:
Example intent:
> "We would like to convey entity statuses through the catalog in a generic way,
> as an integration layer. Our monitoring and alerting system has a plugin with
> Backstage, and it would be useful if the entity's status field contained the
> current alert state close to the actual entity data for anyone to consume.
> current alert state close to the actual entity data for anyone to consume. We
> find the `status.items` semantics a poor fit, so we would prefer to make our
> own custom field under `status` for these purposes."
While we are considering a mechanism for contributing generic statuses to
entities, no such mechanism has yet been built. If you are interested in that
topic, [this issue](https://github.com/backstage/backstage/issues/2292) contains
We have not yet ventured to define any generic semantics for the `status`
object. We recommend sticking with the `status.items` mechanism where possible
(see below), since third party consumers will not be able to consume your status
information otherwise. Please reach out to the maintainers on Discord or by
making a GitHub issue describing your use case if you are interested in this
topic.
## Adding a New Status Item Type
Example intent:
> "The semantics of the entity `status.items` field are fine for our needs, but
> we want to contribute our own type of status into that array instead of the
> catalog specific one."
This is a simple, low risk way of adding your own status information to
entities. Consumers will be able to easily track and display the status together
with other types / sources.
We recommend that any status type that are not strictly private within the
organization be namespaced to avoid collisions. Statuses emitted by Backstage
core processes will for example be prefixed with `backstage.io/`, your
organization may prefix with `my-org.net/`, and `pagerduty.com/active-alerts`
could be a sensible complete status item type for that particular external
system.
The mechanics for how to emit custom statuses is not in place yet, so if this is
of interest to you, you might consider contacting the maintainers on Discord or
my making a GitHub issue describing your use case.
[This issue](https://github.com/backstage/backstage/issues/2292) also contains
more context.
But in general, errors emitted (and exceptions thrown) by any processor
including custom ones, end up in the [well known key](well-known-statuses.md)
for ingestion status.
docs/features/software-catalog/well-known-statuses.md
+25 -21
View File
@@ -6,35 +6,37 @@ sidebar_label: Well-known Statuses
description: Lists a number of well known entity statuses, that have defined semantics. They can be attached to catalog entities and consumed by plugins as needed.
---
This section lists a number of well known
[entity status fields](descriptor-format.md#common-to-all-kinds-status), that
have defined semantics. They can be attached to catalog entities and consumed by
This section lists well known
[entity statuses](descriptor-format.md#common-to-all-kinds-status), that have
defined semantics. They can be attached to catalog entities and consumed by
plugins as needed.
If you are looking to extend the set of statuses, see
If you are looking to extend the statuses, see
[Extending the model](extending-the-model.md).
## Common Fields
The values of statuses are currently left unrestricted, except that they must be
objects. They therefore currently formally have no common fields.
The `status` object of an entity is currently left unrestricted, except for the
`items` field. Its structure is defined in the
[descriptor format](descriptor-format.md#common-to-all-kinds-status) section.
We reserve the right to extend this model in the future, such that some fields
of those value objects gain standardized meaning. We may for example want to add
a standard concept of "severity" or "level" to these.
We reserve the right to extend this model in the future. This status is in
active development and its format will change unexpectedly. Do not consume it in
your own code until such a time that this documentation has been updated.
## Statuses
## Status Item Types
This is a (non-exhaustive) list of statuses that are known to be in active use.
This is a (non-exhaustive) list of `status.items.[].type` values that are known
to be in active use.
### `backstage.io/catalog-processing`
Contains the current status of the catalog's ingestion of this entity. Errors
that may appear here include inability to read from the remote SCM provider,
syntax errors in the YAML file, and similar.
Expresses an aspect of the current status of the catalog's ingestion of this
entity. Errors that may appear here include inability to read from the remote
SCM provider, syntax errors in the YAML file, and similar.
Note that the entity data itself may be of an older version, when errors are
present. The ingestion system keeps the old, valid entity data untouched when
present. The ingestion system keeps the old valid entity data untouched when
possible, so the errors described in this state may not seem to align with the
rest of the entity, because they pertain to a remote that could not be
successfully ingested. This is normal.
@@ -42,10 +44,12 @@ successfully ingested. This is normal.
```yaml
# Example:
status:
backstage.io/catalog-processing:
errors: []
items:
- type: backstage.io/catalog-processing
level: error
message: 'NotFoundError: File not found'
error:
name: NotFoundError
message: File not found
stack: ...
```
This status is in active development and its format will change unexpectedly. Do
not consume it in your own code until such a time that this documentation has
been updated.
package.json
+1 -1
View File
@@ -49,8 +49,8 @@
},
"version": "1.0.0",
"dependencies": {
"@microsoft/api-extractor": "7.13.2-pr1916.0",
"@microsoft/api-documenter": "^7.12.16",
"@microsoft/api-extractor": "7.13.2-pr1916.0",
"@microsoft/api-extractor-model": "^7.12.5"
},
"devDependencies": {
packages/catalog-client/api-report.md
+1 -16
View File
@@ -7,9 +7,6 @@
import { Entity } from '@backstage/catalog-model';
import { EntityName } from '@backstage/catalog-model';
import { Location as Location_2 } from '@backstage/catalog-model';
import { SerializedError } from '@backstage/errors';
import { UNSTABLE_EntityStatusLevel } from '@backstage/catalog-model';
import { UNSTABLE_EntityStatusValue } from '@backstage/catalog-model';
// @public (undocumented)
export type AddLocationRequest = {
@@ -80,19 +77,7 @@ export type CatalogListResponse<T> = {
};
// @public
export const ENTITY_STATUS_CATALOG_PROCESSING_KEY = "backstage.io/catalog-processing";
// @public
export type UNSTABLE_CatalogProcessingStatus = UNSTABLE_EntityStatusValue & {
items?: UNSTABLE_CatalogProcessingStatusItem[];
};
// @public (undocumented)
export type UNSTABLE_CatalogProcessingStatusItem = {
status: UNSTABLE_EntityStatusLevel;
message?: string;
error?: SerializedError;
};
export const ENTITY_STATUS_CATALOG_PROCESSING_TYPE = "backstage.io/catalog-processing";
// (No @packageDocumentation comment for this package)
packages/catalog-client/src/types/index.ts
+1 -5
View File
@@ -21,8 +21,4 @@ export type {
CatalogEntitiesRequest,
CatalogListResponse,
} from './api';
export { ENTITY_STATUS_CATALOG_PROCESSING_KEY } from './status';
export type {
UNSTABLE_CatalogProcessingStatus,
UNSTABLE_CatalogProcessingStatusItem,
} from './status';
export { ENTITY_STATUS_CATALOG_PROCESSING_TYPE } from './status';
packages/catalog-client/src/types/status.ts
+3 -44
View File
@@ -14,50 +14,9 @@
* limitations under the License.
*/
import {
UNSTABLE_EntityStatusLevel,
UNSTABLE_EntityStatusValue,
} from '@backstage/catalog-model';
import { SerializedError } from '@backstage/errors';
/*
* This is the entity status field that's emitted by the catalog processing
* engine, to inform about the status of an entity.
*
* Example:
*
* "status": {
* "backstage.io/catalog-processing": {
* "status": "error",
* "items": [
* {
* "status": "error",
* "error": {
* "name": "InputError",
* "message": "Syntax error: ..."
* }
* }
* ]
* }
* }
*/
/**
* The entity `status` key for the status of the processing engine in regards
* to entity.
* The entity `status.items[].type` for the status of the processing engine in
* regards to an entity.
*/
export const ENTITY_STATUS_CATALOG_PROCESSING_KEY =
export const ENTITY_STATUS_CATALOG_PROCESSING_TYPE =
'backstage.io/catalog-processing';
/**
* The status value for the `backstage.io/catalog-processing` key.
*/
export type UNSTABLE_CatalogProcessingStatus = UNSTABLE_EntityStatusValue & {
items?: UNSTABLE_CatalogProcessingStatusItem[];
};
export type UNSTABLE_CatalogProcessingStatusItem = {
status: UNSTABLE_EntityStatusLevel;
message?: string;
error?: SerializedError;
};
packages/catalog-model/api-report.md
+16 -8
View File
@@ -7,6 +7,7 @@
import { JsonObject } from '@backstage/config';
import { JSONSchema7 } from 'json-schema';
import { JsonValue } from '@backstage/config';
import { SerializedError } from '@backstage/errors';
import * as yup from 'yup';
// @public (undocumented)
@@ -112,7 +113,7 @@ export type Entity = {
metadata: EntityMeta;
spec?: JsonObject;
relations?: EntityRelation[];
status?: Record<string, JsonObject>;
status?: UNSTABLE_EntityStatus;
};
// @public
@@ -528,15 +529,22 @@ export interface TemplateEntityV1beta2 extends Entity {
// @public (undocumented)
export const templateEntityV1beta2Validator: KindValidator;
// @public
export type UNSTABLE_EntityStatusLevel = 'ok' | 'info' | 'warning' | 'error';
// @public
export type UNSTABLE_EntityStatusValue = {
status: UNSTABLE_EntityStatusLevel;
message?: string;
// @alpha
export type UNSTABLE_EntityStatus = {
items?: UNSTABLE_EntityStatusItem[];
};
// @alpha
export type UNSTABLE_EntityStatusItem = {
type: string;
level: UNSTABLE_EntityStatusLevel;
message: string;
error?: SerializedError;
};
// @alpha
export type UNSTABLE_EntityStatusLevel = 'info' | 'warning' | 'error';
// @public (undocumented)
interface UserEntityV1alpha1 extends Entity {
// (undocumented)
packages/catalog-model/package.json
+1
View File
@@ -30,6 +30,7 @@
},
"dependencies": {
"@backstage/config": "^0.1.5",
"@backstage/errors": "^0.1.1",
"@types/json-schema": "^7.0.5",
"@types/yup": "^0.29.8",
"ajv": "^7.0.3",
packages/catalog-model/src/entity/Entity.ts
+2 -1
View File
@@ -16,6 +16,7 @@
import { JsonObject } from '@backstage/config';
import { EntityName } from '../types';
import { UNSTABLE_EntityStatus } from './EntityStatus';
/**
* The format envelope that's common to all versions/kinds of entity.
@@ -55,7 +56,7 @@ export type Entity = {
* The keys are implementation defined and the values can be any JSON object
* with semantics that match that implementation.
*/
status?: Record<string, JsonObject>;
status?: UNSTABLE_EntityStatus;
};
/**
packages/catalog-model/src/entity/EntityStatus.ts
+63
View File
@@ -0,0 +1,63 @@
/*
* Copyright 2021 Spotify AB
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { SerializedError } from '@backstage/errors';
/**
* The current status of the entity, as claimed by various sources.
* @alpha
*/
export type UNSTABLE_EntityStatus = {
/**
* Specific status item on a well known format.
*/
items?: UNSTABLE_EntityStatusItem[];
};
/**
* A specific status item on a well known format.
* @alpha
*/
export type UNSTABLE_EntityStatusItem = {
/**
* The type of status as a unique key per source.
*/
type: string;
/**
* The level / severity of the status item. If the level is "error", the
* processing of the entity may be entirely blocked. In this case the status
* entry may apply to a different, newer version of the data than what is
* being returned in the catalog response.
*/
level: UNSTABLE_EntityStatusLevel;
/**
* A brief message describing the status, intended for human consumption.
*/
message: string;
/**
* An optional serialized error object related to the status.
*/
error?: SerializedError;
};
/**
* Each entity status item has a level, describing its severity.
* @alpha
*/
export type UNSTABLE_EntityStatusLevel =
| 'info' // Only informative data
| 'warning' // Warnings were found
| 'error'; // Errors were found
packages/catalog-model/src/entity/index.ts
+5 -4
View File
@@ -27,6 +27,11 @@ export type {
EntityRelation,
EntityRelationSpec,
} from './Entity';
export type {
UNSTABLE_EntityStatus,
UNSTABLE_EntityStatusItem,
UNSTABLE_EntityStatusLevel,
} from './EntityStatus';
export * from './policies';
export {
compareEntityToRef,
@@ -36,10 +41,6 @@ export {
serializeEntityRef,
stringifyEntityRef,
} from './ref';
export type {
UNSTABLE_EntityStatusLevel,
UNSTABLE_EntityStatusValue,
} from './status';
export {
entityHasChanges,
generateEntityEtag,
packages/catalog-model/src/entity/status.ts
-32
View File
@@ -1,32 +0,0 @@
/*
* Copyright 2021 Spotify AB
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
/**
* Each entity status entry has a level, describing its severity.
*/
export type UNSTABLE_EntityStatusLevel =
| 'ok' // Everything is OK
| 'info' // Only informative data
| 'warning' // Warnings were found
| 'error'; // Errors were found
/**
* Reserved root fields in all `status` object values.
*/
export type UNSTABLE_EntityStatusValue = {
status: UNSTABLE_EntityStatusLevel;
message?: string;
};
packages/catalog-model/src/schema/Entity.schema.json
+1 -7
View File
@@ -64,13 +64,7 @@
}
},
"status": {
"type": "object",
"description": "The current status of the entity, as claimed by various sources.",
"patternProperties": {
"^.+$": {
"$ref": "common#status"
}
}
"$ref": "common#status"
}
}
}
packages/catalog-model/src/schema/shared/common.schema.json
+47 -2
View File
@@ -46,8 +46,53 @@
"status": {
"$id": "#status",
"type": "object",
"description": "A specific status of an entity.",
"additionalProperties": true
"description": "The current status of the entity, as claimed by various sources.",
"required": [],
"additionalProperties": true,
"properties": {
"items": {
"$ref": "#statusItem"
}
}
},
"statusItem": {
"$id": "#statusItem",
"type": "object",
"description": "A specific status item on a well known format.",
"required": ["type", "level", "message"],
"additionalProperties": true,
"properties": {
"type": {
"type": "string",
"minLength": 1
},
"level": {
"$ref": "#statusLevel",
"description": "The status level / severity of the status item."
},
"message": {
"type": "string",
"description": "A brief message describing the status, intended for human consumption."
},
"error": {
"$ref": "#error",
"description": "An optional serialized error object related to the status."
}
}
},
"statusLevel": {
"$id": "#statusLevel",
"type": "string",
"description": "A status level / severity.",
"enum": ["info", "warning", "error"]
},
"error": {
"$id": "#error",
"type": "object",
"description": "A serialized error object.",
"required": [],
"additionalProperties": true,
"properties": {}
}
}
}
plugins/catalog-backend/src/next/DefaultCatalogProcessingEngine.ts
+21 -1
View File
@@ -119,7 +119,27 @@ export class DefaultCatalogProcessingEngine implements CatalogProcessingEngine {
for (const error of result.errors) {
this.logger.warn(error.message);
}
const errorsString = JSON.stringify(
result.errors.map(e => serializeError(e)),
);
// If the result was marked as not OK, it signals that some part of the
// processing pipeline threw an exception. This can happen both as part of
// non-catastrophic things such as due to validation errors, as well as if
// something fatal happens inside the processing for other reasons. In any
// case, this means we can't trust that anything in the output is okay. So
// just store the errors and trigger a stich so that they become visible to
// the outside.
if (!result.ok) {
await this.processingDatabase.transaction(async tx => {
await this.processingDatabase.updateProcessedEntityErrors(tx, {
id,
errors: errorsString,
});
});
await this.stitcher.stitch(
new Set([stringifyEntityRef(unprocessedEntity)]),
);
return;
}
@@ -129,7 +149,7 @@ export class DefaultCatalogProcessingEngine implements CatalogProcessingEngine {
id,
processedEntity: result.completedEntity,
state: result.state,
errors: JSON.stringify(result.errors.map(e => serializeError(e))),
errors: errorsString,
relations: result.relations,
deferredEntities: result.deferredEntities,
});
plugins/catalog-backend/src/next/Stitcher.test.ts
-7
View File
@@ -16,7 +16,6 @@
import { getVoidLogger } from '@backstage/backend-common';
import { Knex } from 'knex';
import { ENTITY_STATUS_CATALOG_PROCESSING_KEY } from '@backstage/catalog-client';
import { DatabaseManager } from './database/DatabaseManager';
import {
DbRefreshStateReferencesRow,
@@ -92,9 +91,6 @@ describe('Stitcher', () => {
},
},
],
status: {
[ENTITY_STATUS_CATALOG_PROCESSING_KEY]: { status: 'ok' },
},
apiVersion: 'a',
kind: 'k',
metadata: {
@@ -175,9 +171,6 @@ describe('Stitcher', () => {
},
},
]),
status: {
[ENTITY_STATUS_CATALOG_PROCESSING_KEY]: { status: 'ok' },
},
apiVersion: 'a',
kind: 'k',
metadata: {
plugins/catalog-backend/src/next/Stitcher.ts
+15 -12
View File
@@ -14,11 +14,12 @@
* limitations under the License.
*/
import { ENTITY_STATUS_CATALOG_PROCESSING_TYPE } from '@backstage/catalog-client';
import {
ENTITY_STATUS_CATALOG_PROCESSING_KEY,
UNSTABLE_CatalogProcessingStatusItem,
} from '@backstage/catalog-client';
import { Entity, parseEntityRef } from '@backstage/catalog-model';
Entity,
parseEntityRef,
UNSTABLE_EntityStatusItem,
} from '@backstage/catalog-model';
import { ConflictError, SerializedError } from '@backstage/errors';
import { createHash } from 'crypto';
import stableStringify from 'fast-json-stable-stringify';
@@ -138,7 +139,7 @@ export class Stitcher {
// it
const entity = JSON.parse(processedEntity) as Entity;
const isOrphan = Number(incomingReferenceCount) === 0;
let statusItems: UNSTABLE_CatalogProcessingStatusItem[] = [];
let statusItems: UNSTABLE_EntityStatusItem[] = [];
if (isOrphan) {
this.logger.debug(`${entityRef} is an orphan`);
@@ -151,7 +152,9 @@ export class Stitcher {
const parsedErrors = JSON.parse(errors) as SerializedError[];
if (Array.isArray(parsedErrors) && parsedErrors.length) {
statusItems = parsedErrors.map(e => ({
status: 'error',
type: ENTITY_STATUS_CATALOG_PROCESSING_TYPE,
level: 'error',
message: e.toString(),
error: e,
}));
}
@@ -165,12 +168,12 @@ export class Stitcher {
type: row.relationType!,
target: parseEntityRef(row.relationTarget!),
}));
entity.status = {
...entity.status,
[ENTITY_STATUS_CATALOG_PROCESSING_KEY]: statusItems.length
? { status: 'error', items: statusItems }
: { status: 'ok' },
};
if (statusItems.length) {
entity.status = {
...entity.status,
items: [...(entity.status?.items ?? []), ...statusItems],
};
}
// If the output entity was actually not changed, just abort
const hash = generateStableHash(entity);
plugins/catalog-backend/src/next/database/DefaultProcessingDatabase.ts
+14
View File
@@ -123,6 +123,20 @@ export class DefaultProcessingDatabase implements ProcessingDatabase {
);
}
async updateProcessedEntityErrors(
txOpaque: Transaction,
options: UpdateProcessedEntityOptions,
): Promise<void> {
const tx = txOpaque as Knex.Transaction;
const { id, errors } = options;
await tx<DbRefreshStateRow>('refresh_state')
.update({
errors,
})
.where('entity_id', id);
}
private deduplicateRelations(rows: DbRelationsRow[]): DbRelationsRow[] {
return lodash.uniqBy(
rows,
plugins/catalog-backend/src/next/database/types.ts
+14 -1
View File
@@ -34,6 +34,11 @@ export type UpdateProcessedEntityOptions = {
deferredEntities: Entity[];
};
export type UpdateProcessedEntityErrorsOptions = {
id: string;
errors?: string;
};
export type RefreshStateItem = {
id: string;
entityRef: string;
@@ -80,10 +85,18 @@ export interface ProcessingDatabase {
): Promise<GetProcessableEntitiesResult>;
/**
* Updates the
* Updates a processed entity
*/
updateProcessedEntity(
txOpaque: Transaction,
options: UpdateProcessedEntityOptions,
): Promise<void>;
/**
* Updates only the errors of a processed entity
*/
updateProcessedEntityErrors(
txOpaque: Transaction,
options: UpdateProcessedEntityErrorsOptions,
): Promise<void>;
}