Merge branch 'master' into techdocs-common/case-sensitive-migration

Signed-off-by: Eric Peterson <ericpeterson@spotify.com>
2021-08-12 11:27:43 +02:00
parent add5ee5d97 0c58dd73d0
commit 52547371fb
635 changed files with 9792 additions and 6037 deletions
@@ -4,8 +4,6 @@ title: Search Architecture
 description: Documentation on Search Architecture
 ---

-# Search Architecture
-
 > _This architecture has not been fully implemented yet. Find our milestones to
 > follow our progress and help contribute on the
 > [Search Roadmap](./README.md#project-roadmap)._
@@ -4,8 +4,6 @@ title: Search Concepts
 description: Documentation on Backstage Search Concepts
 ---

-# Search Concepts
-
 Backstage Search lets you find the right information you are looking for in the
 Backstage ecosystem.

@@ -4,8 +4,6 @@ title: Getting Started with Search
 description: How to set up and install Backstage Search
 ---

-# Getting Started
-
 Search functions as a plugin to Backstage, so you will need to use Backstage to
 use Search.

@@ -4,8 +4,6 @@ title: Search Engines
 description: Choosing and configuring your search engine for Backstage
 ---

-# Search Engines
-
 Backstage supports 2 search engines by default, an in-memory engine called Lunr
 and ElasticSearch. You can configure your own search engines by implementing the
 provided interface as mentioned in the
@@ -18,7 +16,7 @@ QueryTranslator interface. This modification can be done without touching
 provided search engines by using the exposed setter to set the modified query
 translator into the instance.

-```
+```typescript
 const searchEngine = new LunrSearchEngine({ logger });
 searchEngine.setTranslator(new MyNewAndBetterQueryTranslator());
 ```
@@ -36,6 +34,35 @@ const searchEngine = new LunrSearchEngine({ logger });
 const indexBuilder = new IndexBuilder({ logger, searchEngine });
 ```

+## Postgres
+
+The Postgres based search engine only requires that postgres being configured as
+the database engine for Backstage. Therefore it targets setups that want to
+avoid maintaining another external service like elastic search. The search
+provides decent results and performs well with ten thousands of indexed
+documents. The connection to postgres is established via the database manager
+also used by other plugins.
+
+> **Important**: The search plugin requires at least Postgres 11!
+
+To use the `PgSearchEngine`, make sure that you have a Postgres database
+configured and make the following changes to your backend:
+
+1. Add a dependency on `@backstage/plugin-search-backend-module-pg` to your
+   backend's `package.json`.
+2. Initialize the search engine. It is recommended to initialize it with a
+   fallback to the lunr search engine if you are running Backstage for
+   development locally with SQLite:
+
+```typescript
+// In packages/backend/src/plugins/search.ts
+
+// Initialize a connection to a search engine.
+const searchEngine = (await PgSearchEngine.supported(database))
+  ? await PgSearchEngine.from({ database })
+  : new LunrSearchEngine({ logger });
+```
+
 ## ElasticSearch

 Backstage supports ElasticSearch search engine connections, indexing and
@@ -103,12 +130,9 @@ Other ElasticSearch instances can be connected to by using standard
 ElasticSearch authentication methods and exposed URL, provided that the cluster
 supports that. The configuration options needed are the URL to the node and
 authentication information. Authentication can be handled by either providing
-username/password or an API key or a bearer token. In case both
-username/password combination and one of the tokens are provided, token takes
-precedence. For more information how to create an API key, see
-[Elastic documentation on API keys](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html),
-and how to create a bearer token see
-[Elastic documentation on tokens.](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-service-token.html)
+username/password or an API key. For more information how to create an API key,
+see
+[Elastic documentation on API keys](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html).

 #### Configuration examples

@@ -123,16 +147,6 @@ search:
      password: changeme
 ```

-##### With bearer token
-
-```yaml
-search:
-  elasticsearch:
-    node: http://localhost:9200
-    auth:
-      bearer: token
-```
-
 ##### With API key

 ```yaml
@@ -187,7 +187,7 @@ new `CustomCatalogIndexPage`.
 # packages/app/src/App.tsx
 const routes = (
  <FlatRoutes>
-    <Navigate key="/" to="/catalog" />
+    <Navigate key="/" to="catalog" />
 -    <Route path="/catalog" element={<CatalogIndexPage />} />
 +    <Route path="/catalog" element={<CustomCatalogIndexPage />} />
 ```
@@ -57,6 +57,30 @@ if the original location delegates to another location. A common case is, that a
 location is registered as `bootstrap:bootstrap` which means that it is part of
 the `app-config.yaml` of a Backstage installation.

+### backstage.io/orphan
+
+This annotation is either absent, or present with the exact _string_ value
+`"true"`. It should never be added manually. Instead, the catalog itself injects
+the annotation as part of its processing loops, on entities that are found to
+have no registered locations or config locations that keep them "active" /
+"alive".
+
+For example, suppose that the user first registers a location URL pointing to a
+`Location` kind entity, which in turn refers to two `Component` kind entities in
+two other files nearby. The end result is that the catalog contains those three
+entities. Now suppose that the user edits the original `Location` entity to only
+refer to the first of the `Component` kind entities. This will intentionally
+_not_ lead to the other `Component` entity to be removed from the catalog (for
+safety reasons). Instead, it gains this orphan marker annotation, to make it
+clear that user action is required to completely remove it, if desired.
+
+```yaml
+# Example:
+metadata:
+  annotations:
+    backstage.io/orphan: 'true'
+```
+
 ### backstage.io/techdocs-ref

 ```yaml
@@ -61,7 +61,7 @@ If you do not prefer (3a) and optionally like to use a service account, you can
 follow these steps.

 Create a new Service Account and a key associated with it. In roles of the
-service account, use "Storage Admin".
+service account, use "Storage Object Admin".

 If you want to create a custom role, make sure to include both `get` and
 `create` permissions for both "Objects" and "Buckets". See
@@ -143,6 +143,8 @@ permissions to:

 - `s3:ListBucket` to retrieve bucket metadata
 - `s3:PutObject` to upload files to the bucket
+- `s3:DeleteObject` and `s3:DeleteObjectVersion` to delete stale content during
+  re-publishing

 To _read_ TechDocs from the S3 bucket the IAM policy needs to have at a minimum
 permissions to:
@@ -345,6 +347,10 @@ techdocs:
        accountKey: ${TECHDOCS_AZURE_BLOB_STORAGE_ACCOUNT_KEY}
 ```

+In either case, the account or credentials used to access your container and all
+TechDocs objects underneath it should have the `Storage Blog Data Owner` role
+applied, in order to read, write, and delete objects as needed.
+
 **4. That's it!**

 Your Backstage app is now ready to use Azure Blob Storage for TechDocs, to store