From 86bb6cc6a8d3424a7f75393a05ebe225f9f68d55 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fredrik=20Adel=C3=B6w?= Date: Sun, 17 May 2020 20:21:17 +0200 Subject: [PATCH 1/3] ADR002: Default Software Catalog File Format --- .../adr001-add-adr-log.md | 4 +- .../adr002-default-catalog-file-format.md | 218 ++++++++++++++++++ 2 files changed, 219 insertions(+), 3 deletions(-) create mode 100644 docs/architecture-decisions/adr002-default-catalog-file-format.md diff --git a/docs/architecture-decisions/adr001-add-adr-log.md b/docs/architecture-decisions/adr001-add-adr-log.md index 23115d43b5..3da9313378 100644 --- a/docs/architecture-decisions/adr001-add-adr-log.md +++ b/docs/architecture-decisions/adr001-add-adr-log.md @@ -1,4 +1,4 @@ -## Date: 26.04.2020 +## Date: 2020-04-26 ## Title: Architecture Decision Record (ADR) log @@ -7,5 +7,3 @@ ## Discussion: There is a need to store big decisions made in a log as a reference point for the team, help with onboarding new members and give context to others interested in the project. ## Risks: People stop adding ADRs to the log and context gets lost - - diff --git a/docs/architecture-decisions/adr002-default-catalog-file-format.md b/docs/architecture-decisions/adr002-default-catalog-file-format.md new file mode 100644 index 0000000000..eead015fbe --- /dev/null +++ b/docs/architecture-decisions/adr002-default-catalog-file-format.md @@ -0,0 +1,218 @@ +# ADR002: Default Software Catalog File Format + +| Created | Status | +| ---------- | ------ | +| 2020-05-17 | Open | + +## Background + +Backstage comes with a software catalog functionality, that you can use to +track all your software components and more. It can be powered by data from +various sources, and one of them that is included with the package, is a +custom database backed catalog. It has the ability to keep itself updated +automatically based on the contents of little descriptor files in your +version control system of choice. Developers create these files and maintain +them side by side with their code, and the catalog system reacts accordingly. + +This ADR describes the default format of these descriptor files. + +### Inspiration + +Internally at Spotify, a home grown software catalog system is used pervasively +and forms a core part of Backstage and other important pieces of the +infrastructure. The user experience, learnings and certain pieces of metadata +from that catalog are being carried over to the open source effort. + +The file format described herein, also draws heavy inspiration from the +[kubernetes object format](https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/). + +### Core Concepts + +There are a number of descriptor files, all of whose locations (e.g. within a +version control system) are registered with the software catalog. The method of +registration is not covered in this document; it could happen either manually +inside Backstage, or by push events from a CI/CD pipelines, or by webhook +triggers from the version control system, etc. + +Each file describes one or more entities in accordance with the +[Backstage System Model](https://github.com/spotify/backstage/issues/390). +All of these entities have a common stucture and nomenclature, and they are +stored in the software catalog from which they then can be queried. + +Entities have distinct names, and they may reference each other by those names. + +## Format + +Descriptor files use the [YAML](https://yaml.org/spec/1.2/spec.html) format. +They may be written by hand, or created using automated tools. Each file may +consist of several YAML documents (separated by `---`), where each document +describes a single entity. + +This is an example entity definition with some mocked data. + +```yaml +--- +apiVersion: backstage.io/v1beta1 +kind: Component +metadata: + name: frobs-awesome + description: | + Backend service that implements the Frobs API, as defined + in [the Frobs RFC](https://example.com/spec/frob.html). + labels: + system: frobs + lifecycle: production + example.com/service-discovery-name: frobsawesome + annotations: + circleci.com/project-slug: gh/example-org/frobs-awesome +spec: + type: service +``` + +The root fields `apiVersion`, `kind`, `metadata`, and `spec` are part of the +_envelope_, defining the overall structure of all kinds of entity. Likewise, +the `name`, `namespace`, `labels`, and `annotations` metadata fields are of +special significance and have reserved purposes and distinct shapes. + +See below for details about these fields. + +## Envelope + +The root envelope object has the following structure. + +### `apiVersion` and `kind` + +The `kind` is the high level entity type being described, typically from the +[Backstage system model](https://github.com/spotify/backstage/issues/390). The +first versions of the catalog will focus on the `Component` kind. + +The `apiVersion`is the version of specification format for that particular +entity that this file is written against. The version is used for being able to +evolve the format, and the tuple of `apiVersion` and `kind` should be enough +for a parser to know how to interpret the contents of the rest of the document. + +Backstage specific entities have an `apiVersion` that is prefixed with +`backstage.io/`, to distinguish them from other types of object that share +the same type of structure. This may be relevant when co-hosting these +specifications with e.g. kubernetes object manifests. + +Early versions of the catalog will be using beta versions, e.g. `backstage.io/v1beta1`, +to signal that the format may still change. After that, we will be using +`backstage.io/v1` and up. + +### `metadata` + +A structure that contains metadata about the entity, i.e. things that aren't directly +part of the entity specification itself. See below for more details about this structure. + +### `spec` + +The actual specification data that describes the entity. + +The precise structure of the `spec` depends on the `apiVersion` and `kind` combination, +and some kinds may not even have a `spec` at all. See further down in this document for +the specification structure of specific kinds. + +## Metadata + +The `metadata` root field has the following nested structure. + +### `name` + +The name of the entity. This name is both meant for human eyes to recognize the entity, +and for machines and other components to reference the entity (e.g. in URLs or from +other entity specification files). + +Names must be unique per kind, within a given namespace (if specified), at any point in +time. Names may be reused at a later time, after an entity is deleted from the registry. + +Names are required to follow a certain format. Entities that do not follow those rules +will not be accepted for registration in the catalog. The ruleset is configurable to fit +your organization's needs, but the default behavior is as follows. + +- Strings of length at least 1, and at most 253 +- Consist only of URL safe characters, e.g. ones that `encodeURIComponent` or similar would + leave unchanged +- Must consist of sequences of `[a-z0-9A-Z]` possibly separated by other characters + +Example: `visits-tracking-service`, `CircleciBuildsDs_avro_gcs` + +In addition to this, names are passed through a normalization function and then compared +to the same normalized form of other entity names and made sure to not collide. This rule +of uniqueness exists to avoid situations where e.g. both `my-component` and `MyComponent` +are registered side by side, which leads to confusion and risk. The normalization function +is also configurable, but the default behavior is as follows. + +- Strip out all characters outside of the set `[a-zA-Z0-9]` +- Convert to lowercase + +Example: `CircleciBuildsDs_avro_gcs` -> `circlecibuildsdsavrogcs` + +### `namespace` + +The `name` of a namespace that the entity belongs to. This field is optional, and currently +has no special semantics apart from bounding the name uniqueness constraint if specified. +It is reserved for future use and may get broader semantic implication. + +Namespaces may also be part of the catalog, and are `v1` / `Namespace` entities, +i.e. not Backstage specific but the same as in Kubernetes. + +### `description` + +A human readable description of the entity, to be shown in Backstage. Should be kept short +and informative, suitable to give an overview of the entity's purpose at a glance. More +detailed explanations and documentation should be placed elsewhere. + +### `labels` + +Labels are optional key/value pairs of that are attached to the entity, and their use is +identical to [kubernetes object labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/). + +Their main purpose is for references to other entities, and for information that is +in one way or another classifying for the current entity. They are often used as values +in queries or filters. + +Both the key and the value are strings, subject to the following restrictions. + +Keys have an optional prefix followed by a slash, and then the name part which is required. +The prefix must be a valid lowercase domain name, at most 253 characters in total. The name +part must be sequences of `[a-zA-Z0-9]` separated by any of `[-_.]`, at most 63 characters +in total. + +The `backstage.io/` prefix is reserved for use by Backstage core components. Some keys such as +`system` also have predefined semantics. + +Values are strings that follow the same restrictions as `name` above. + +### `annotations` + +An object with arbitrary non-identifying metadata attached to the entity, +identical in use to [kubernetes object annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/). + +Their purpose is mainly, but not limited, to reference into external systems. This could +for example be a reference to the git ref the entity was ingested from, to monitoring +and logging systems, to pagerduty schedules, etc. + +The keys are strings and the values are free-form, subject to the following restrictions. + +Keys have an optional prefix followed by a slash, and then the name part which is required. +The prefix must be a valid lowercase domain name, at most 253 characters in total. The name +part must be sequences of `[a-zA-Z0-9]` separated by any of `[-_.]`, at most 63 characters +in total. + +The `backstage.io/` prefix is reserved for use by Backstage core components. + +Values can be of any JSON type. + +## Component + +| Field | Value | +| ------------ | ---------------------- | +| `apiVersion` | `backstage.io/v1beta1` | +| `kind` | `Component` | + +The `spec` object for this kind is as follows: + +| Field | Type | Required | Description | +| ------ | ------ | -------- | -------------------------------------- | +| `type` | String | Yes | The type of component, e.g. `service`. | From 8678a3cd31f619f8d4b8da4ebd6f3cf7591f2a9f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fredrik=20Adel=C3=B6w?= Date: Tue, 19 May 2020 14:08:10 +0200 Subject: [PATCH 2/3] Address comments --- .../adr002-default-catalog-file-format.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/architecture-decisions/adr002-default-catalog-file-format.md b/docs/architecture-decisions/adr002-default-catalog-file-format.md index eead015fbe..d8d5ff4025 100644 --- a/docs/architecture-decisions/adr002-default-catalog-file-format.md +++ b/docs/architecture-decisions/adr002-default-catalog-file-format.md @@ -18,7 +18,7 @@ This ADR describes the default format of these descriptor files. ### Inspiration -Internally at Spotify, a home grown software catalog system is used pervasively +Internally at Spotify, a home grown software catalog system is used heavily and forms a core part of Backstage and other important pieces of the infrastructure. The user experience, learnings and certain pieces of metadata from that catalog are being carried over to the open source effort. @@ -193,7 +193,7 @@ Their purpose is mainly, but not limited, to reference into external systems. Th for example be a reference to the git ref the entity was ingested from, to monitoring and logging systems, to pagerduty schedules, etc. -The keys are strings and the values are free-form, subject to the following restrictions. +Both the key and the value are strings, subject to the following restrictions. Keys have an optional prefix followed by a slash, and then the name part which is required. The prefix must be a valid lowercase domain name, at most 253 characters in total. The name @@ -202,7 +202,7 @@ in total. The `backstage.io/` prefix is reserved for use by Backstage core components. -Values can be of any JSON type. +Values can be of any length, but are limited to being strings. ## Component From b2b3def7041bb4f3178c3ae49ba62f36af663471 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fredrik=20Adel=C3=B6w?= Date: Tue, 19 May 2020 15:03:19 +0200 Subject: [PATCH 3/3] Lied about the name format --- .../adr002-default-catalog-file-format.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/architecture-decisions/adr002-default-catalog-file-format.md b/docs/architecture-decisions/adr002-default-catalog-file-format.md index d8d5ff4025..23db32cc9c 100644 --- a/docs/architecture-decisions/adr002-default-catalog-file-format.md +++ b/docs/architecture-decisions/adr002-default-catalog-file-format.md @@ -130,10 +130,8 @@ Names are required to follow a certain format. Entities that do not follow those will not be accepted for registration in the catalog. The ruleset is configurable to fit your organization's needs, but the default behavior is as follows. -- Strings of length at least 1, and at most 253 -- Consist only of URL safe characters, e.g. ones that `encodeURIComponent` or similar would - leave unchanged -- Must consist of sequences of `[a-z0-9A-Z]` possibly separated by other characters +- Strings of length at least 1, and at most 63 +- Must consist of sequences of `[a-z0-9A-Z]` possibly separated by one of `[-_.]` Example: `visits-tracking-service`, `CircleciBuildsDs_avro_gcs`