diff --git a/docs/README.md b/docs/README.md index cdbfbc5578..b5ec0f1307 100644 --- a/docs/README.md +++ b/docs/README.md @@ -27,7 +27,6 @@ better yet, a pull request. - [Overview](features/software-catalog/index.md) - [System model](features/software-catalog/system-model.md) - [YAML File Format](features/software-catalog/descriptor-format.md) - - [Populating the catalog](features/software-catalog/populating.md) - [Extending the model](features/software-catalog/extending-the-model.md) - [External integrations](features/software-catalog/external-integrations.md) - [API](features/software-catalog/api.md) diff --git a/docs/features/software-catalog/bsc-edit.png b/docs/features/software-catalog/bsc-edit.png new file mode 100644 index 0000000000..d1ac1f52e5 Binary files /dev/null and b/docs/features/software-catalog/bsc-edit.png differ diff --git a/docs/features/software-catalog/bsc-extend.png b/docs/features/software-catalog/bsc-extend.png new file mode 100644 index 0000000000..54d660ea70 Binary files /dev/null and b/docs/features/software-catalog/bsc-extend.png differ diff --git a/docs/features/software-catalog/bsc-register-1.png b/docs/features/software-catalog/bsc-register-1.png new file mode 100644 index 0000000000..117b2ea8ea Binary files /dev/null and b/docs/features/software-catalog/bsc-register-1.png differ diff --git a/docs/features/software-catalog/bsc-register-2.png b/docs/features/software-catalog/bsc-register-2.png new file mode 100644 index 0000000000..fd1ea7b998 Binary files /dev/null and b/docs/features/software-catalog/bsc-register-2.png differ diff --git a/docs/features/software-catalog/bsc-search.png b/docs/features/software-catalog/bsc-search.png new file mode 100644 index 0000000000..8e417cb076 Binary files /dev/null and b/docs/features/software-catalog/bsc-search.png differ diff --git a/docs/features/software-catalog/bsc-starred.png b/docs/features/software-catalog/bsc-starred.png new file mode 100644 index 0000000000..27db19c842 Binary files /dev/null and b/docs/features/software-catalog/bsc-starred.png differ diff --git a/docs/features/software-catalog/extending-the-model.md b/docs/features/software-catalog/extending-the-model.md index e69de29bb2..855334fd09 100644 --- a/docs/features/software-catalog/extending-the-model.md +++ b/docs/features/software-catalog/extending-the-model.md @@ -0,0 +1,38 @@ +# Extending the model + +Backstage natively supports tracking of the following component +[`type`](descriptor-format.md)'s: + +- Services +- Websites +- Libraries +- Documentation +- Other + +![](bsc-extend.png) + +Since these types are likely not the only kind of software you will want to +track in Backstage, it is possible to + +It is possible to add your own software types that fits your organization's data +model. Inside Spotify our model has grown significantly over the years, and now +includes ML models, Apps, data pipelines and many more. + +## Adding a new type + +TODO: Describe what changes are needed to add a new type that shows up in the +catalog. + +## The Other type + +It might be tempting to put software that doesn't fit into any of the existing +types into Other. There are a few reasons why we advice against this; firstly, +we have found that it is preferred to match the conceptual model that your +engineers have when describing your sofware. Secondly, Backstage helps your +engineers manage their software by integrating the infratrucure tooling through +plugins. Different plugins are used for managing different types of components. + +For example, the +[Lighthouse plugin](https://github.com/spotify/backstage/tree/master/plugins/lighthouse) +only makes sense for Websites. The more specific you can be in how you model +your software, the easier it is to provide plugins that are contextual. diff --git a/docs/features/software-catalog/external-integrations.md b/docs/features/software-catalog/external-integrations.md index e69de29bb2..1f2863acf3 100644 --- a/docs/features/software-catalog/external-integrations.md +++ b/docs/features/software-catalog/external-integrations.md @@ -0,0 +1,8 @@ +# External integrations + +Backstage natively supports storing software components in +[metadata YAML files](descriptor-format.md). However, companies that already +have an existing system for keeping track of software and its owners can +integrate such systems with Backstage. + +TODO: Describe the API contract. diff --git a/docs/features/software-catalog/index.md b/docs/features/software-catalog/index.md index d4d9bf1e2d..06785515f6 100644 --- a/docs/features/software-catalog/index.md +++ b/docs/features/software-catalog/index.md @@ -6,25 +6,109 @@ The Backstage Service Catalog — actually, a software catalog, since it include more than just services — is a centralized system that keeps track of ownership and metadata for all the software in your ecosystem (services, websites, libraries, data pipelines, etc). The catalog is built around the concept of -[metadata yaml files](../../architecture-decisions/adr002-default-catalog-file-format.md#format) -stored together with the code, which are then harvested and visualized in -Backstage. +[metadata YAML files](descriptor-format.md) stored together with the code, which +are then harvested and visualized in Backstage. ![service-catalog](https://backstage.io/blog/assets/6/header.png) -We have also found that the service catalog is a great way to organise the -infrastructure tools you use to manage the software as well. This is how -Backstage creates one developer portal for all your tools. Rather than asking -teams to jump between different infrastructure UI’s (and incurring additional -cognitive overhead each time they make a context switch), most of these tools -can be organised around the entities in the catalog. +## How it works -## Using the Service Catalog +Backstage and the Backstage Service Catalog makes it easy for one team to manage +10 services — and makes it possible for your company to manage thousands of +them. -TODO +More specifically, the Service Catalog enables two main use-cases: + +1. Helping teams manage and maintain the software they own. Teams get a uniform + view of all their software; services, libraries, websites, ML models — you + name it, Backstage knows all about it. +2. Makes all the software in your company, and who owns it, discoverable. No + more orphan software hiding in the dark corners of your software ecosystem. ![](service-catalog-home.png) +## Adding components to the catalog + +The source of truth for the components in your service catalog are +[metadata YAML files](descriptor-format.md) stored in source control (GitHub, +GitHub Enterprise, GitLab, ...). + +There are 3 ways to add components to the catalog: + +1. Manually register components +2. Creating new components through Backstage +3. Integrating with and [external source](external-integrations.md) + +### Manually register components + +Users can register new components by going to `/create` and clicking the +**REGSITER EXISTING COMPONENT** button: + +![](bsc-register-1.png) + +Backstage expects the full URL to the YAML in your source control. Example: + +``` +https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/artist-lookup-component.yaml +``` + +_More examples can be found +[here](https://github.com/spotify/backstage/tree/master/packages/catalog-model/examples)._ + +![](bsc-register-2.png) + +It is important to note that any kind of software can be registered in +Backstage. Even if the software is not maintained by your company (SaaS +offering, for example) it is still useful to create components for tracking +ownership. + +### Creating new components through Backstage + +All software created through the +[Backstage Software Templates](../software-templates/index.md) are automatically +registered in the catalog. + +### Updating component metadata + +Teams owning the components are responsible for maintaining the metadata about +them, and do so using their normal Git workflow. + +![](bsc-edit.png) + +Once the change has been merged, Backstage will automatically show the updated +metadata in the service catalog after a short while. + +## Finding software in the catlog + +By default the service catalog shows components owned by the team of the logged +in user. But you can also switch to _All_ to see all the components across your +companie's software ecosystem. Basic inline _search_ and _column filtering_ +makes it easy to browse a big set of components. + +![](bsc-search.png) + +## Starring components + +For easy and quick access to components you visit frequently, Backstage supports +_starring_ of components: + +![](bsc-starred.png) + +## Integrated tooling through plugins + +The service catalog is a great way to organise the infrastructure tools you use +to manage the software. This is how Backstage creates one developer portal for +all your tools. Rather than asking teams to jump between different +infrastructure UI’s (and incurring additional cognitive overhead each time they +make a context switch), most of these tools can be organised around the entities +in the catalog. + +![tools](https://backstage.io/blog/assets/20-05-20/tabs.png) + +The Backstage platform can be customized by incorporating +[existing open source plugins](https://github.com/spotify/backstage/tree/master/plugins), +or by [building your own](../../plugins/index.md). + ## Links - [[Blog post] Backstage Service Catalog released in alpha](https://backstage.io/blog/2020/06/22/backstage-service-catalog-alpha) diff --git a/docs/features/software-catalog/populating.md b/docs/features/software-catalog/populating.md deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/docs/features/software-catalog/system-model.md b/docs/features/software-catalog/system-model.md index e69de29bb2..9485df20d0 100644 --- a/docs/features/software-catalog/system-model.md +++ b/docs/features/software-catalog/system-model.md @@ -0,0 +1,96 @@ +# Backstage System Model + +We believe that a strong shared understanding and terminology around systems, +software and resources leads to a better Backstage experience. + +_This description originates from +[this RFC](https://github.com/spotify/backstage/issues/390). Note that some of +the concpets are not yet supported in Backstage._ + +## Concepts + +We model our technology using these five concepts (further explained below): + +- **Domains** are a high-level grouping of systems +- **Systems** encapsulate the implementation of APIs +- **APIs** are the boundaries between different components and systems +- **Components** are pieces of software +- **Resources** are physical or virtual infrastructure needed to operate a + system + +![Software Ecosystem Model_ Public Github version](https://user-images.githubusercontent.com/24575/77633084-39bcde80-6f4f-11ea-8251-f8df561a3652.png) + +### Domain + +While systems are the basic level of encapsulation for resources, components and +APIs, it is often useful to group a collection of systems that share +terminology, domain models, business purpose, or documentation, i.e. they form a +bounded context. + +For example, it would make sense if the different systems in the “Payments” +domain would come with some documentation on how to accept payments for a new +product or use-case, share the same entity types in their APIs, and integrate +well with each other. + +### System + +With increasing complexity in software, we believe that systems form an +important abstraction level to help us reason about software ecosystems. Systems +are a useful concept in that they allow us to ignore the implementation details +of a certain functionality for consumers, while allowing the owning team to make +changes as they see fit (leading to low coupling). + +A system, in this sense, is a collection of resources and components that +exposes one or several APIs. Components and resources in a system are typically +owned by the same team and are expected to co-evolve. As such, systems usually +consist of at most a handful of components. + +For example, a playlist management system might encapsulate a backend service to +update playlists, a backend service to query them, and a database to store them. +It could expose an RPC API, a daily snapshots dataset, and an event stream of +playlist updates. + +### Component + +A component is a piece of software, for example a mobile feature, web site, +backend service or data pipeline (list not exhaustive). A component can be +tracked in source control, or use some existing open source or commercial +software. + +A component can implement APIs for other components to consume. It might depend +on the resources of the system it belongs to, and APIs from other components or +other systems. All other aspects of the component, e.g. any code dependencies, +must be encapsulated. + +### API + +We believe APIs form an important (maybe the most important) abstraction that +allows large software ecosystems to scale. Thus, APIs are a first class citizen +in the Backstage model and the primary way to discover existing functionality in +the ecosystem. + +APIs are implemented by components and form boundaries between components and +systems. They might be defined using an RPC IDL (eg Protobuf, GraphQL, ...), a +data schema (eg Avro, TFRecord, ...), or as code interfaces. In any case, APIs +exposed by components need to be in a known machine-readable format so we can +build further tooling and analysis on top. + +Some APIs might be exposed by the system, making them available for any other +Spotify component to consume. Those public APIs must be documented and humanly +discoverable in Backstage. + +### Resource + +Resources are the infrastructure a system needs to operate, like BigTable +databases, Pub/Sub topics, S3 buckets or CDNs. Modelling them together with +components and systems will better allow us to visualize resource footprint, and +create tooling around them. + +## Current status + +Backstage currently supports Components and APIs. + +## Links + +- [Original RFC](https://github.com/spotify/backstage/issues/390) +- [YAML file format](../../architecture-decisions/adr002-default-catalog-file-format.md) diff --git a/docs/features/software-templates/index.md b/docs/features/software-templates/index.md index c7358518cd..d1631af70c 100644 --- a/docs/features/software-templates/index.md +++ b/docs/features/software-templates/index.md @@ -5,6 +5,10 @@ Components inside Backstage. It by default has the ability to load skeletons of code, template in some variables and then publish the template to some location like GitHub. + + ### Getting Started The Software Templates are available under `/create`, and if you've followed diff --git a/mkdocs.yml b/mkdocs.yml index 2380001203..adfdf6ae0a 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -25,7 +25,6 @@ nav: - Overview: 'features/software-catalog/index.md' - System model: 'features/software-catalog/system-model.md' - YAML File Format: 'features/software-catalog/descriptor-format.md' - - Populating the catalog: 'features/software-catalog/populating.md' - Extending the model: 'features/software-catalog/extending-the-model.md' - External integrations: 'features/software-catalog/external-integrations.md' - API: 'features/software-catalog/api.md'