Merge branch 'master' into feat/api-catalog

docs/README.md

@@ -27,7 +27,6 @@ better yet, a pull request.
       - [Overview](features/software-catalog/index.md)
       - [System model](features/software-catalog/system-model.md)
       - [YAML File Format](features/software-catalog/descriptor-format.md)
-      - [Populating the catalog](features/software-catalog/populating.md)
       - [Extending the model](features/software-catalog/extending-the-model.md)
       - [External integrations](features/software-catalog/external-integrations.md)
       - [API](features/software-catalog/api.md)

docs/conf/reading.md

@@ -98,7 +98,7 @@ A good pattern for reading optional configuration values is to use the `??`
 operator. For example:
 
 ```ts
-const title = config.getString('my-plugin.title') ?? 'My Plugin';
+const title = config.getOptionalString('my-plugin.title') ?? 'My Plugin';
 ```
 
 To read required configuration, simply use the methods without `Optional`, for

docs/features/software-catalog/extending-the-model.md

@@ -0,0 +1,38 @@
+# Extending the model
+
+Backstage natively supports tracking of the following component
+[`type`](descriptor-format.md)'s:
+
+- Services
+- Websites
+- Libraries
+- Documentation
+- Other
+
+![](bsc-extend.png)
+
+Since these types are likely not the only kind of software you will want to
+track in Backstage, it is possible to
+
+It is possible to add your own software types that fits your organization's data
+model. Inside Spotify our model has grown significantly over the years, and now
+includes ML models, Apps, data pipelines and many more.
+
+## Adding a new type
+
+TODO: Describe what changes are needed to add a new type that shows up in the
+catalog.
+
+## The Other type
+
+It might be tempting to put software that doesn't fit into any of the existing
+types into Other. There are a few reasons why we advice against this; firstly,
+we have found that it is preferred to match the conceptual model that your
+engineers have when describing your sofware. Secondly, Backstage helps your
+engineers manage their software by integrating the infratrucure tooling through
+plugins. Different plugins are used for managing different types of components.
+
+For example, the
+[Lighthouse plugin](https://github.com/spotify/backstage/tree/master/plugins/lighthouse)
+only makes sense for Websites. The more specific you can be in how you model
+your software, the easier it is to provide plugins that are contextual.

docs/features/software-catalog/external-integrations.md

@@ -0,0 +1,8 @@
+# External integrations
+
+Backstage natively supports storing software components in
+[metadata YAML files](descriptor-format.md). However, companies that already
+have an existing system for keeping track of software and its owners can
+integrate such systems with Backstage.
+
+TODO: Describe the API contract.

docs/features/software-catalog/index.md

@@ -6,25 +6,109 @@ The Backstage Service Catalog — actually, a software catalog, since it include
 more than just services — is a centralized system that keeps track of ownership
 and metadata for all the software in your ecosystem (services, websites,
 libraries, data pipelines, etc). The catalog is built around the concept of
-[metadata yaml files](../../architecture-decisions/adr002-default-catalog-file-format.md#format)
-stored together with the code, which are then harvested and visualized in
-Backstage.
+[metadata YAML files](descriptor-format.md) stored together with the code, which
+are then harvested and visualized in Backstage.
 
 ![service-catalog](https://backstage.io/blog/assets/6/header.png)
 
-We have also found that the service catalog is a great way to organise the
-infrastructure tools you use to manage the software as well. This is how
-Backstage creates one developer portal for all your tools. Rather than asking
-teams to jump between different infrastructure UI’s (and incurring additional
-cognitive overhead each time they make a context switch), most of these tools
-can be organised around the entities in the catalog.
+## How it works
 
-## Using the Service Catalog
+Backstage and the Backstage Service Catalog makes it easy for one team to manage
+10 services — and makes it possible for your company to manage thousands of
+them.
 
-TODO
+More specifically, the Service Catalog enables two main use-cases:
+
+1. Helping teams manage and maintain the software they own. Teams get a uniform
+   view of all their software; services, libraries, websites, ML models — you
+   name it, Backstage knows all about it.
+2. Makes all the software in your company, and who owns it, discoverable. No
+   more orphan software hiding in the dark corners of your software ecosystem.
 
 ![](service-catalog-home.png)
 
+## Adding components to the catalog
+
+The source of truth for the components in your service catalog are
+[metadata YAML files](descriptor-format.md) stored in source control (GitHub,
+GitHub Enterprise, GitLab, ...).
+
+There are 3 ways to add components to the catalog:
+
+1. Manually register components
+2. Creating new components through Backstage
+3. Integrating with and [external source](external-integrations.md)
+
+### Manually register components
+
+Users can register new components by going to `/create` and clicking the
+**REGSITER EXISTING COMPONENT** button:
+
+![](bsc-register-1.png)
+
+Backstage expects the full URL to the YAML in your source control. Example:
+
+```
+https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/artist-lookup-component.yaml
+```
+
+_More examples can be found
+[here](https://github.com/spotify/backstage/tree/master/packages/catalog-model/examples)._
+
+![](bsc-register-2.png)
+
+It is important to note that any kind of software can be registered in
+Backstage. Even if the software is not maintained by your company (SaaS
+offering, for example) it is still useful to create components for tracking
+ownership.
+
+### Creating new components through Backstage
+
+All software created through the
+[Backstage Software Templates](../software-templates/index.md) are automatically
+registered in the catalog.
+
+### Updating component metadata
+
+Teams owning the components are responsible for maintaining the metadata about
+them, and do so using their normal Git workflow.
+
+![](bsc-edit.png)
+
+Once the change has been merged, Backstage will automatically show the updated
+metadata in the service catalog after a short while.
+
+## Finding software in the catlog
+
+By default the service catalog shows components owned by the team of the logged
+in user. But you can also switch to _All_ to see all the components across your
+companie's software ecosystem. Basic inline _search_ and _column filtering_
+makes it easy to browse a big set of components.
+
+![](bsc-search.png)
+
+## Starring components
+
+For easy and quick access to components you visit frequently, Backstage supports
+_starring_ of components:
+
+![](bsc-starred.png)
+
+## Integrated tooling through plugins
+
+The service catalog is a great way to organise the infrastructure tools you use
+to manage the software. This is how Backstage creates one developer portal for
+all your tools. Rather than asking teams to jump between different
+infrastructure UI’s (and incurring additional cognitive overhead each time they
+make a context switch), most of these tools can be organised around the entities
+in the catalog.
+
+![tools](https://backstage.io/blog/assets/20-05-20/tabs.png)
+
+The Backstage platform can be customized by incorporating
+[existing open source plugins](https://github.com/spotify/backstage/tree/master/plugins),
+or by [building your own](../../plugins/index.md).
+
 ## Links
 
 - [[Blog post] Backstage Service Catalog released in alpha](https://backstage.io/blog/2020/06/22/backstage-service-catalog-alpha)

docs/features/software-catalog/system-model.md

@@ -0,0 +1,96 @@
+# Backstage System Model
+
+We believe that a strong shared understanding and terminology around systems,
+software and resources leads to a better Backstage experience.
+
+_This description originates from
+[this RFC](https://github.com/spotify/backstage/issues/390). Note that some of
+the concpets are not yet supported in Backstage._
+
+## Concepts
+
+We model our technology using these five concepts (further explained below):
+
+- **Domains** are a high-level grouping of systems
+- **Systems** encapsulate the implementation of APIs
+- **APIs** are the boundaries between different components and systems
+- **Components** are pieces of software
+- **Resources** are physical or virtual infrastructure needed to operate a
+  system
+
+![Software Ecosystem Model_ Public Github version](https://user-images.githubusercontent.com/24575/77633084-39bcde80-6f4f-11ea-8251-f8df561a3652.png)
+
+### Domain
+
+While systems are the basic level of encapsulation for resources, components and
+APIs, it is often useful to group a collection of systems that share
+terminology, domain models, business purpose, or documentation, i.e. they form a
+bounded context.
+
+For example, it would make sense if the different systems in the “Payments”
+domain would come with some documentation on how to accept payments for a new
+product or use-case, share the same entity types in their APIs, and integrate
+well with each other.
+
+### System
+
+With increasing complexity in software, we believe that systems form an
+important abstraction level to help us reason about software ecosystems. Systems
+are a useful concept in that they allow us to ignore the implementation details
+of a certain functionality for consumers, while allowing the owning team to make
+changes as they see fit (leading to low coupling).
+
+A system, in this sense, is a collection of resources and components that
+exposes one or several APIs. Components and resources in a system are typically
+owned by the same team and are expected to co-evolve. As such, systems usually
+consist of at most a handful of components.
+
+For example, a playlist management system might encapsulate a backend service to
+update playlists, a backend service to query them, and a database to store them.
+It could expose an RPC API, a daily snapshots dataset, and an event stream of
+playlist updates.
+
+### Component
+
+A component is a piece of software, for example a mobile feature, web site,
+backend service or data pipeline (list not exhaustive). A component can be
+tracked in source control, or use some existing open source or commercial
+software.
+
+A component can implement APIs for other components to consume. It might depend
+on the resources of the system it belongs to, and APIs from other components or
+other systems. All other aspects of the component, e.g. any code dependencies,
+must be encapsulated.
+
+### API
+
+We believe APIs form an important (maybe the most important) abstraction that
+allows large software ecosystems to scale. Thus, APIs are a first class citizen
+in the Backstage model and the primary way to discover existing functionality in
+the ecosystem.
+
+APIs are implemented by components and form boundaries between components and
+systems. They might be defined using an RPC IDL (eg Protobuf, GraphQL, ...), a
+data schema (eg Avro, TFRecord, ...), or as code interfaces. In any case, APIs
+exposed by components need to be in a known machine-readable format so we can
+build further tooling and analysis on top.
+
+Some APIs might be exposed by the system, making them available for any other
+Spotify component to consume. Those public APIs must be documented and humanly
+discoverable in Backstage.
+
+### Resource
+
+Resources are the infrastructure a system needs to operate, like BigTable
+databases, Pub/Sub topics, S3 buckets or CDNs. Modelling them together with
+components and systems will better allow us to visualize resource footprint, and
+create tooling around them.
+
+## Current status
+
+Backstage currently supports Components and APIs.
+
+## Links
+
+- [Original RFC](https://github.com/spotify/backstage/issues/390)
+- [YAML file format](../../architecture-decisions/adr002-default-catalog-file-format.md)

docs/features/software-templates/index.md

@@ -5,6 +5,10 @@ Components inside Backstage. It by default has the ability to load skeletons of
 code, template in some variables and then publish the template to some location
 like GitHub.
 
+<video width="100%" height="100%" controls>
+  <source src="https://backstage.io/blog/assets/2020-08-05/feature.mp4" type="video/mp4">
+</video>
+
 ### Getting Started
 
 The Software Templates are available under `/create`, and if you've followed

docs/overview/what-is-backstage.md

@@ -14,15 +14,15 @@ to create a streamlined development environment from end to end.
 
 Out of the box, Backstage includes:
 
-- [Backstage Service Catalog](https://github.com/spotify/backstage/blob/master/docs/features/software-catalog/index.md)
-  for managing all your software (microservices, libraries, data pipelines,
+- [Backstage Service Catalog](/docs/features/software-catalog/index.md) for
+  managing all your software (microservices, libraries, data pipelines,
   websites, ML models, etc.)
-- [Backstage Software Templates](https://github.com/spotify/backstage/blob/master/docs/features/software-templates/index.md)
-  for quickly spinning up new projects and standardizing your tooling with your
+- [Backstage Software Templates](/docs/features/software-templates/index.md) for
+  quickly spinning up new projects and standardizing your tooling with your
   organization’s best practices
-- [Backstage TechDocs](https://github.com/spotify/backstage/tree/master/docs/features/techdocs)
-  for making it easy to create, maintain, find, and use technical documentation,
-  using a "docs like code" approach
+- [Backstage TechDocs](/docs/features/techdocs) for making it easy to create,
+  maintain, find, and use technical documentation, using a "docs like code"
+  approach
 - Plus, a growing ecosystem of
   [open source plugins](https://github.com/spotify/backstage/tree/master/plugins)
   that further expand Backstage’s customizability and functionality

docs/verify-links.js

@@ -17,12 +17,21 @@
 
 const { resolve: resolvePath, dirname } = require('path');
 const fs = require('fs-extra');
-const fetch = require('node-fetch');
 const recursive = require('recursive-readdir');
 
 const projectRoot = resolvePath(__dirname, '..');
 
 async function verifyUrl(basePath, url) {
+  // Avoid having absolute URL links within docs/, so that links work on the site
+  if (
+    url.match(
+      /https:\/\/github.com\/spotify\/backstage\/(tree|blob)\/master\/docs\//,
+    ) &&
+    basePath.match(/^(?:docs|microsite)\//)
+  ) {
+    return { url, basePath, problem: 'absolute' };
+  }
+
   url = url.replace(/#.*$/, '');
   url = url.replace(
     /https:\/\/github.com\/spotify\/backstage\/(tree|blob)\/master/,
@@ -39,7 +48,7 @@ async function verifyUrl(basePath, url) {
       : resolvePath(dirname(resolvePath(projectRoot, basePath)), url);
     const exists = await fs.pathExists(path);
     if (!exists) {
-      return { url, basePath };
+      return { url, basePath, problem: 'missing' };
     }
   }
 
@@ -76,8 +85,14 @@ async function main() {
 
   if (badUrls.length) {
     console.log(`Found ${badUrls.length} bad links within repo`);
-    for (const { url, basePath } of badUrls) {
-      console.error(`Unable to reach ${url}, linked from ${basePath}`);
+    for (const { url, basePath, problem } of badUrls) {
+      if (problem === 'missing') {
+        console.error(`Unable to reach ${url}, linked from ${basePath}`);
+      } else if (problem === 'absolute') {
+        console.error(`Link to docs/ should be replaced by a relative URL`);
+        console.error(`  From: ${basePath}`);
+        console.error(`  To: ${url}`);
+      }
     }
     process.exit(1);
   }