Merge branch 'master' into feature/EKSCatalog

Signed-off-by: Jonah Back <jonah@jonahback.com>
This commit is contained in:
Jonah Back
2022-07-05 10:26:19 -07:00
597 changed files with 17151 additions and 4035 deletions
+1
View File
@@ -350,6 +350,7 @@ router.get('/auth/providerA/handler/frame');
router.post('/auth/providerA/handler/frame');
router.post('/auth/providerA/logout');
router.get('/auth/providerA/refresh'); // if supported
router.post('/auth/providerA/refresh'); // if supported
```
As you can see each endpoint is prefixed with both `/auth` and its provider
+45 -6
View File
@@ -59,10 +59,38 @@ configured and make the following changes to your backend:
// Initialize a connection to a search engine.
const searchEngine = (await PgSearchEngine.supported(env.database))
? await PgSearchEngine.from({ database: env.database })
? await PgSearchEngine.fromConfig(env.config, { database: env.database })
: new LunrSearchEngine({ logger: env.logger });
```
## Optional Configuration
The following is an example of the optional configuration that can be applied when using Postgres as the search backend. Currently this is mostly for just the highlight feature:
```yaml
search:
pg:
highlightOptions:
useHighlight: true # Used to enable to disable the highlight feature. The default value is true
maxWord: 35 # Used to set the longest headlines to output. The default value is 35.
minWord: 15 # Used to set the shortest headlines to output. The default value is 15.
shortWord: 3 # Words of this length or less will be dropped at the start and end of a headline, unless they are query terms. The default value of three (3) eliminates common English articles.
highlightAll: false # If true the whole document will be used as the headline, ignoring the preceding three parameters. The default is false.
maxFragments: 0 # Maximum number of text fragments to display. The default value of zero selects a non-fragment-based headline generation method. A value greater than zero selects fragment-based headline generation (see the linked documentation above for more details).
fragmentDelimiter: ' ... ' # Delimiter string used to concatenate fragments. Defaults to " ... ".
```
**Note:** the highlight search term feature uses `ts_headline` which has been known to potentially impact performance. You only need this minimal config to disable it should you have issues:
```yaml
search:
pg:
highlightOptions:
useHighlight: false
```
The Postgres documentation on [Highlighting Results](https://www.postgresql.org/docs/current/textsearch-controls.html#TEXTSEARCH-HEADLINE) has more details.
## ElasticSearch
Backstage supports ElasticSearch search engine connections, indexing and
@@ -80,15 +108,16 @@ const searchEngine = await ElasticSearchSearchEngine.initialize({
const indexBuilder = new IndexBuilder({ logger: env.logger, searchEngine });
```
For the engine to be available, your backend package needs a dependency into
For the engine to be available, your backend package needs a dependency on
package `@backstage/plugin-search-backend-module-elasticsearch`.
ElasticSearch needs some additional configuration before it is ready to use
within your instance. The configuration options are documented in the
[configuration schema definition file.](https://github.com/backstage/backstage/blob/master/plugins/search-backend-module-elasticsearch/config.d.ts)
The underlying functionality is using official ElasticSearch client version 7.x,
meaning that ElasticSearch version 7 is the only one confirmed to be supported.
The underlying functionality uses either the official ElasticSearch client
version 7.x (meaning that ElasticSearch version 7 is the only one confirmed to
be supported), or the OpenSearch client, when the `aws` provider is configured.
Should you need to create your own bespoke search experiences that require more
than just a query translator (such as faceted search or Relay pagination), you
@@ -97,9 +126,19 @@ search clients. The version of the client need not be the same as one used
internally by the elastic search engine plugin. For example:
```typescript
import { Client } from '@elastic/elastic-search';
import { isOpenSearchCompatible } from '@backstage/plugin-search-backend-module-elasticsearch';
import { Client as ElasticClient } from '@elastic/elastic-search';
import { Client as OpenSearchClient } from '@opensearch-project/opensearch';
const client = searchEngine.newClient(options => new Client(options));
const client = searchEngine.newClient(options => {
// In reality, you would only import / instantiate one of the following, but
// for illustrative purposes, here are both:
if (isOpenSearchCompatible(options)) {
return new OpenSearchClient(options);
} else {
return new ElasticClient(options);
}
});
```
#### Set custom index template
@@ -106,8 +106,7 @@ with [Static Location Configuration](#static-location-configuration) or a
discovery processor like
[GitHub Discovery](../../integrations/github/discovery.md). To enforce usage of
processors to locate entities we can configure the catalog into `readonly` mode.
This configuration disables the mutating backend catalog APIs and disallows
users from registering new entities at run-time.
This configuration disables registering and deleting locations with the catalog APIs.
```yaml
catalog:
@@ -117,6 +116,8 @@ catalog:
> **Note that any plugin relying on the catalog API for creating, updating and
> deleting entities will not work in this mode.**
Deleting an entity by UUID, `DELETE /entities/by-uid/:uid`, is allowed when using this mode. It may be rediscovered as noted in [explicit deletion](life-of-an-entity.md#explicit-deletion).
A common use case for this configuration is when organizations have a remote
source that should be mirrored into Backstage. To make Backstage a mirror of
this remote source, users cannot also register new entities with e.g. the
@@ -86,6 +86,13 @@ contains more information about the required fields.
Once we have a `template.yaml` ready, we can then add it to the software catalog
for use by the scaffolder.
> Note: When you add or modify a template, you will need to refresh the location entity.
> Otherwise, Backstage won't display the template in the available templates,
> or it will keep showing the old template. You can refresh the location instance by
> going into `Catalog` web page, choosing `Locations` instead of `Components`, and selecting the correct location entity.
> From there, you can click on the refresh icon representing "Scheduled entity refresh" action.
> Afterwards, you should see your template updated.
You can add the template files to the catalog through
[static location configuration](../software-catalog/configuration.md#static-location-configuration),
for example:
@@ -97,6 +104,8 @@ catalog:
target: https://github.com/backstage/software-templates/blob/main/scaffolder-templates/react-ssr-template/template.yaml
rules:
- allow: [Template]
- type: file
target: template.yaml # Backstage will expect the file to be in packages/backend/template.yaml
```
Or you can add the template using the `catalog-import` plugin, which unless
@@ -9,8 +9,9 @@ by writing custom actions which can be used along side our
[built-in actions](./builtin-actions.md).
> Note: When adding custom actions, the actions array will **replace the
> built-in actions too**. To ensure you can continue to include the builtin
> actions, see below to include them during registration of your action.
> built-in actions too**. Meaning, you will no longer be able to use them.
> If you want to continue using the builtin actions, include them in the actions
> array when registering your custom actions, as seen below.
## Writing your Custom Action
@@ -122,27 +123,32 @@ will set the available actions that the scaffolder has access to.
```ts
import { createBuiltinActions } from '@backstage/plugin-scaffolder-backend';
import { ScmIntegrations } from '@backstage/integration';
import { createNewFileAction } from './actions/custom';
const integrations = ScmIntegrations.fromConfig(env.config);
export default async function createPlugin(
env: PluginEnvironment,
): Promise<Router> {
const catalogClient = new CatalogClient({ discoveryApi: env.discovery });
const integrations = ScmIntegrations.fromConfig(env.config);
const builtInActions = createBuiltinActions({
containerRunner,
integrations,
catalogClient,
config: env.config,
reader: env.reader,
});
const builtInActions = createBuiltinActions({
integrations,
catalogClient,
config: env.config,
reader: env.reader,
});
const actions = [...builtInActions, createNewFileAction()];
return await createRouter({
containerRunner,
catalogClient,
actions,
logger: env.logger,
config: env.config,
database: env.database,
reader: env.reader,
});
const actions = [...builtInActions, createNewFileAction()];
return createRouter({
actions,
catalogClient: catalogClient,
logger: env.logger,
config: env.config,
database: env.database,
reader: env.reader,
});
}
```
## List of custom action packages
@@ -320,6 +320,30 @@ The `allowedHosts` part should be set to where you wish to enable this template
to publish to. And it can be any host that is listed in your `integrations`
config in `app-config.yaml`.
Besides specifying `allowedHosts` you can also restrict the template to publish to
repositories owned by specific users/groups/namespaces by setting the `allowedOwners`
option. With the `allowedRepos` option you are able to narrow it down further to a
specific set of repository names. A full example could look like this:
```yaml
- title: Choose a location
required:
- repoUrl
properties:
repoUrl:
title: Repository Location
type: string
ui:field: RepoUrlPicker
ui:options:
allowedHosts:
- github.com
allowedOwners:
- backstage
- someGithubUser
allowedRepos:
- backstage
```
The `RepoUrlPicker` is a custom field that we provide part of the
`plugin-scaffolder`. You can provide your own custom fields by
[writing your own Custom Field Extensions](./writing-custom-field-extensions.md)
@@ -485,6 +509,64 @@ the value of `firstName` from the parameters). This is great for passing the
values from the form into different steps and reusing these input variables.
These template strings preserve the type of the parameter.
The `${{ parameters.firstName }}` pattern will work only in the template file.
If you want to start using values provided from the UI in your code, you will have to use
the `${{ values.firstName }}` pattern. Additionally, you have to pass
the parameters from the UI to the input of the `fetch:template` step.
```yaml
apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
name: v1beta3-demo
title: Test Action
description: scaffolder v1beta3 template demo
spec:
owner: backstage/techdocs-core
type: service
parameters:
- title: Fill in some steps
required:
- name
properties:
name:
title: Name
type: string
description: Unique name of your project
urlParameter:
title: URL endpoint
type: string
description: URL endpoint at which the component can be reached
default: 'https://www.example.com'
enabledDB:
title: Enable Database
type: boolean
default: false
...
steps:
- id: fetch-base
name: Fetch Base
action: fetch:template
input:
url: ./template
values:
name: ${{ parameters.name }}
url: ${{ parameters.urlParameter }}
enabledDB: ${{ parameters.enabledDB }}
```
Afterwards, if you are using the builtin templating action, you can start using
the variables in your code. You can use also any other templating functions from
[Nunjucks](https://mozilla.github.io/nunjucks/templating.html#tags) as well.
```bash
#!/bin/bash
echo "Hi my name is ${{ values.name }}, and you can fine me at ${{ values.url }}!"
{% if values.enabledDB %}
echo "You have enabled your database!"
{% endif %}
```
As you can see above in the `Outputs` section, `actions` and `steps` can also
output things. You can grab that output using `steps.$stepId.output.$property`.
+14
View File
@@ -52,6 +52,8 @@ Addons are rendered in the order in which they are registered.
## Installing and using Addons
To start using Addons you need to add the `@backstage/plugin-techdocs-module-addons-contrib` package to your app. You can do that by running this command from the root of your project: `yarn add --cwd packages/app @backstage/plugin-techdocs-module-addons-contrib`
Addons can be installed and configured in much the same way as extensions for
other Backstage plugins: by adding them underneath an extension registry
component (`<TechDocsAddons>`) under the route representing the TechDocs Reader
@@ -74,6 +76,18 @@ import { ReportIssue } from '@backstage/plugin-techdocs-module-addons-contrib';
</Route>;
```
If you are using a custom [TechDocs reader page](./how-to-guides.md#how-to-customize-the-techdocs-reader-page) your setup will be very similar, here's an example:
```ts
<Route path="/docs/:namespace/:kind/:name/*" element={<TechDocsReaderPage />}>
<TechDocsAddons>
<ReportIssue />
{/* Other addons can be added here. */}
</TechDocsAddons>
{techDocsPage} // This is your custom TechDocs reader page
</Route>
```
The process for configuring Addons on the documentation tab on the entity page
is very similar; instead of adding the `<TechDocsAddons>` registry under a
`<Route>`, you'd add it as a child of `<EntityTechdocsContent />`:
+2 -1
View File
@@ -160,7 +160,8 @@ techdocs:
# techdocs.cache is optional, and is only recommended when you've configured
# an external techdocs.publisher.type above. Also requires backend.cache to
# be configured with a valid cache store.
# be configured with a valid cache store. Configure techdocs.cache.ttl to
# enable caching of techdocs assets.
cache:
# Represents the number of milliseconds a statically built asset should
# stay cached. Cache invalidation is handled automatically by the frontend,
+1 -1
View File
@@ -21,7 +21,7 @@ guide to do a repository-based installation.
### Prerequisites
- Access to a Linux-based operating system, such as Linux, MacOS or
- Access to a Unix-based operating system, such as Linux, MacOS or
[Windows Subsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/)
- An account with elevated rights to install the dependencies
- `curl` or `wget` installed
+2 -1
View File
@@ -341,7 +341,8 @@ separate Docker images.
The backend container can be built by running the following command:
```bash
yarn run docker-build
yarn run build
yarn run build-image
```
This will create a container called `example-backend`.
+38 -42
View File
@@ -15,9 +15,9 @@ work"](#future-work). With "next" we mean features planned for release within
the ongoing quarter from April through June 2022. With "future" we mean
features on the radar, but not yet scheduled.
| [What's next](#whats-next) | [Future work](#future-work) |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| [Ease of onboarding](#ease-of-onboarding) <br/> [Backstage Search 1.0](#search-1.0) <br/> [TechDocs Addon Framework](#techdocs-addon-framework) <br/> [Backend Services (initial)](#backend-services-initial) <br/> [Backstage Security Audit](#backstage-security-audit) <br/> [SIGs for contributors](#sigs-for-contributors) | Security Plan (and Strategy) <br/> Composable Homepage 1.0 <br/> GraphQL <br/> Telemetry <br/> Improved UX design |
| [What's next](#whats-next) | [Future work](#future-work) |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| [Backstage Search 1.0](#search-1.0) <br/> [Backend Services (MVP)](#backend-services-mvp) <br/> [Backstage Security Audit](#backstage-security-audit) <br/> [Backstage Threat Model](#backstage-threat-model) <br/> [TechDocs Addon Framework](#techdocs-addon-framework) <br/> [Software Catalog pagination](#software-catalog-pagination) <br/> [More SIGs](#more-sigs) | Ease of onboarding <br/> Composable Homepage 1.0 <br/> Creator experience <br/> GraphQL <br/> Telemetry |
The long-term roadmap (12 - 36 months) is not detailed in the public roadmap.
Third-party contributions are also not currently included in the roadmap. Let us
@@ -30,47 +30,22 @@ The feature set below is planned for the ongoing quarter, and grouped by theme.
The list order doesn't necessarily reflect priority, and the development/release
cycle will vary based on maintainer schedules.
### Ease of onboarding
A faster (with less development) and easier setup of a proof-of-concept
deployment, as part of the onboarding experience, has been a common and loud
suggestion from new adopters as well as analysts looking at Backstage.
With this initiative we plan to start facing this important topic with the most
commonly used and challenging tasks. More in particular we plan to reduce the
effort required to go from zero to production in installing and customizing
Backstage, as well as reducing the effort required to populate the Software
Catalog.
More iterations will be required in the following quarters, but this will be a
good improvement in the onboarding experience, especially for the benefit of new
adopters.
### Backstage Search 1.0
Fix the few remaining issues to get Backstage Search platform up to 1.0. For more information, see the [Backstage Search documentation and roadmap page](https://backstage.io/docs/features/search/search-overview).
### TechDocs Addon Framework
Addons are TechDocs features that are added on top of the base docs-like-code experience. An example would be a feature that showed comments on the page. We plan to add an Addon framework and open source a selection of the Addons that we use internally at Spotify. We encourage the Backstage community to add further Addons.
For more information about the TechDocs Addon Framework, see the documentation page [here](https://backstage.io/docs/features/techdocs/addons)
For general information about TechDocs including roadmap, see [here](https://backstage.io/docs/features/techdocs/techdocs-overview).
### Backend Services (initial)
### Backend Services (MVP)
To better scale and maintain the Backstage instances, a backend services system
is planned to be introduced as part of the software architecture. This layer of
backend services will help in decoupling the various modules (e.g. Catalog and
Scaffolder) from the frontend experience.
In this quarter we plan to start designing the new architecture, together with
the first experimentation and development of the software components.
After the experimentation and design happened in the past quarter, soon we plan to release a first version to start providing the first benefits to adopters and developers.
### Backstage Security Audit
This is the continuation of the initiative started in the previous quarter. This
This is the continuation of the initiative started in the previous quarters. This
quarter will see the publication of the report describing the outcome of the
audit, together the first fixes and the development of some of the changes
required to address the vulnerabilities.
@@ -84,12 +59,34 @@ initiative.
This initiative is done together with, and with the support of, the [Cloud
Native Computing Foundation (CNCF)](https://www.cncf.io/).
### SIGs for contributors
### Backstage Threat Model
The request to better coordinate the increasing number of contributions coming
from the various adopters' developers is loud and clear. We think that the
community is mature enough to start launching the SIGs (Special Interest Groups)
following the successful model of Kubernetes.
This is another (relevant) initiative planned to make Backstage a secure product for the adopters. The goals of this initiative are:
1. Understand where security investment and attention is needed.
2. Guide the upcoming security audit.
3. Communicate expectations to Backstage adopters and inform and attract security researchers.
The planned artifacts are:
- Concise high level threat model that will be included as part of the Backstage security documentation.
- Granular threat model created in conjunction with the security audit to inform further security investment areas for Backstage.
### TechDocs Addon Framework
Addons are TechDocs features that are added on top of the base docs-like-code experience. An example would be a feature that showed comments on the page. We plan to add an Addon framework and open source a selection of the Addons that we use internally at Spotify. We encourage the Backstage community to add further Addons.
For more information about the TechDocs Addon Framework, see the documentation page [here](https://backstage.io/docs/features/techdocs/addons).
For general information about TechDocs including roadmap, see [here](https://backstage.io/docs/features/techdocs/techdocs-overview).
### Software Catalog pagination
Today adopters with a big catalog (with several thousands of software components) might not have an ideal end-user experience when viewing the `/catalog` page. The issue is related to how the entities are fetched by the frontend. In order to provide a better end-user experience the pagination of the catalogs entities needs to be enforced. Some experimentation is already completed but in this quarter we plan to continue, and hopefully complete, this relevant enhancement.
### More SIGs
In the last quarter we launched the [Catalog SIG (Special Interest Group)](https://github.com/backstage/community/tree/main/sigs/sig-catalog) to better coordinate the increasing number of contributions to the project. We think that this is the proper path to follow to engage more with the contributors. For this reason we will launch other SIGs dedicated to the most interesting topics for the community.
## Future work
@@ -97,18 +94,17 @@ The following feature list doesn't represent a commitment to develop, and the
list order doesn't reflect any priority or importance, but these features are on
the maintainers' radar, with clear interest expressed by the community.
- **Security Plan (and Strategy):** The purpose of the Security Strategy is to
move another step along the path to maturing the platform, setting the
expectations of any adopters from a security standpoint.
- **Ease of onboarding:** A faster (with less development) and easier setup of
Backstage and the most relevant/adopted plugins.
- **Composable Homepage 1.0:** Driving this to 1.0 by adding some composable
components.
- **Creator experience:** Provide a better Backstage user experience through
visual guidelines and templates, especially navigation across plug-ins and
portal functionalities.
- **[GraphQL](https://graphql.org/) support:** Introduce the ability to query
Backstage backend services with a standard query language for APIs.
- **Telemetry:** To efficiently generate logging and metrics in such a way that
adopters can get insights so that Backstage can be monitored and improved.
- **Improved UX design:** Provide a better Backstage user experience through
visual guidelines and templates, especially navigation across plug-ins and
portal functionalities.
## How to influence the roadmap
@@ -160,12 +160,12 @@ Create a new `plugins/todo-list-backend/src/conditionExports.ts` file and add th
```typescript
import { TODO_LIST_RESOURCE_TYPE } from '@internal/plugin-todo-list-common';
import { createConditionExports } from '@backstage/plugin-permission-node';
import { permissionRules } from './service/rules';
import { rules } from './service/rules';
const { conditions, createConditionalDecision } = createConditionExports({
pluginId: 'catalog',
resourceType: RESOURCE_TYPE_CATALOG_ENTITY,
rules: permissionRules,
pluginId: 'todolist',
resourceType: TODO_LIST_RESOURCE_TYPE,
rules,
});
export const todoListConditions = conditions;
@@ -173,7 +173,12 @@ export const todoListConditions = conditions;
export const createTodoListConditionalDecision = createConditionalDecision;
```
Make sure `todoListConditions` and `createTodoListConditionalDecision` are exported from the `todo-list-backend` package.
Make sure `todoListConditions` and `createTodoListConditionalDecision` are exported from the `todo-list-backend` package by editing `plugins/todo-list-backend/src/index.ts`:
```diff
export * from './service/router';
+ export * from './conditionExports';
```
## Test the authorized update endpoint
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff