diff --git a/.github/ISSUE_TEMPLATE/feature_template.md b/.github/ISSUE_TEMPLATE/feature_template.md index 012b8d7a06..d70622bf52 100644 --- a/.github/ISSUE_TEMPLATE/feature_template.md +++ b/.github/ISSUE_TEMPLATE/feature_template.md @@ -1,7 +1,7 @@ --- name: 'Feature Request' about: 'Suggest new features and changes' -labels: help wanted +labels: enhancement --- diff --git a/.github/workflows/master.yml b/.github/workflows/master.yml index 04b50c3e60..354838037e 100644 --- a/.github/workflows/master.yml +++ b/.github/workflows/master.yml @@ -74,3 +74,11 @@ jobs: GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}" package_root: "packages/core" tag_prefix: "v" + + - name: Discord notification + if: ${{ failure() }} + uses: Ilshidur/action-discord@0.2.0 + env: + DISCORD_WEBHOOK: ${{ secrets.DISCORD_WEBHOOK }} + with: + args: 'Master build failed https://github.com/{{GITHUB_REPOSITORY}}/actions/runs/{{GITHUB_RUN_ID}}' diff --git a/.gitignore b/.gitignore index 743561c3b6..aee80f4c10 100644 --- a/.gitignore +++ b/.gitignore @@ -92,6 +92,7 @@ dist # Microsite build output microsite/build +microsite/i18n # Gatsby files .cache/ diff --git a/ADOPTERS.md b/ADOPTERS.md index dc02ad3b85..0ba3e3196b 100644 --- a/ADOPTERS.md +++ b/ADOPTERS.md @@ -12,3 +12,4 @@ | [Voi](https://www.voiscooters.com/) | [@K-Phoen](https://github.com/K-Phoen) | Developer portal, main gateway to our infrastructure, documentation and internal tooling. | | [Talkdesk](https://www.talkdesk.com) | [@jaime-talkdesk](https://github.com/jaime-talkdesk) | Initial work for Engineering Portal and Self Provisioning to R&D | | [Wealthsimple](https://www.wealthsimple.com) | [@andrewthauer](https://github.com/andrewthauer) | Developer portal, service catalog, documentation and tooling | +| [Grab](https://www.grab.com) | [@althafh](https://github.com/althafh) | Initial work as a unified interface for all of Grab's internal tooling | diff --git a/CHANGELOG.md b/CHANGELOG.md index ce39a9f8d2..20ae9a414d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,15 @@ If you encounter issues while upgrading to a newer version, don't hesitate to re > Collect changes for the next release below +- The backend plugin + [service builder](https://github.com/spotify/backstage/blob/master/packages/backend-common/src/service/lib/ServiceBuilderImpl.ts) + no longer adds `express.json()` automatically to all routes. While convenient + in a lot of cases, it also led to problems where for example the proxy + middleware could hang because the body had already been altered and could not + be streamed. Also, plugins that rather wanted to handle e.g. form encoded data + still had to cater to that manually. We therefore decided to let plugins add + `express.json()` themselves if they happen to deal with JSON data. + ## v0.1.1-alpha.20 - Includes https://github.com/spotify/backstage/pull/2097 to resolve issues with create-plugin command. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5b8febd1d2..853332f74d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,7 +1,4 @@ ---- -id: CONTRIBUTING -title: Contributing ---- +# Contributing to Backstage Our vision for Backstage is for it to become the trusted standard toolbox (read: UX layer) for the open source infrastructure landscape. Think of it like Kubernetes for developer experience. We realize this is an ambitious goal. We canโ€™t do it alone. @@ -31,7 +28,7 @@ What kind of plugins should/could be created? Some inspiration from the 120+ plu ## Suggesting a plugin -If you start developing a plugin that you aim to release as open source, we suggest that you create a new [new Issue](https://github.com/spotify/backstage/issues/new?template=plugin_template.md). This helps the community know what plugins are in development. +If you start developing a plugin that you aim to release as open source, we suggest that you create a new [new Issue](https://github.com/spotify/backstage/issues/new?labels=plugin&template=plugin_template.md&title=%5BPlugin%5D+THE+PLUGIN+NAME). This helps the community know what plugins are in development. You can also use this process if you have an idea for a good plugin but you hope that someone else will pick up the work. diff --git a/README.md b/README.md index b5e8704548..4009864edc 100644 --- a/README.md +++ b/README.md @@ -61,7 +61,7 @@ Take a look at the [Getting Started](https://backstage.io/docs/getting-started/i ## Documentation -- [Main documentation](https://backstage.io/docs/overview/what-is-backstage) +- [Main documentation](https://backstage.io/docs) - [Service Catalog](https://backstage.io/docs/features/software-catalog/software-catalog-overview) - [Architecture](https://backstage.io/docs/overview/architecture-terminology) ([Decisions](https://backstage.io/docs/architecture-decisions/adrs-overview)) - [Designing for Backstage](https://backstage.io/docs/dls/design) diff --git a/app-config.yaml b/app-config.yaml index 0a85198630..1eca3664b7 100644 --- a/app-config.yaml +++ b/app-config.yaml @@ -14,21 +14,15 @@ backend: client: sqlite3 connection: ':memory:' +# See README.md in the proxy-backend plugin for information on the configuration format proxy: - '/circleci/api': - target: 'https://circleci.com/api/v1.1' - changeOrigin: true - pathRewrite: - '^/proxy/circleci/api/': '/' + '/circleci/api': https://circleci.com/api/v1.1 '/jenkins/api': - target: 'http://localhost:8080' - changeOrigin: true + target: http://localhost:8080 headers: Authorization: $secret: env: JENKINS_BASIC_AUTH_HEADER - pathRewrite: - '^/proxy/jenkins/api/': '/' organization: name: Spotify @@ -50,7 +44,32 @@ newrelic: baseUrl: 'https://api.newrelic.com/v2' key: NEW_RELIC_REST_API_KEY +lighthouse: + baseUrl: http://localhost:3003 + catalog: + rules: + - allow: [Component, API, Group, Template, Location] + processors: + githubApi: + privateToken: + $secret: + env: GITHUB_PRIVATE_TOKEN + bitbucketApi: + username: + $secret: + env: BITBUCKET_USERNAME + appPassword: + $secret: + env: BITBUCKET_APP_PASSWORD + gitlabApi: + privateToken: + $secret: + env: GITLAB_PRIVATE_TOKEN + azureApi: + privateToken: + $secret: + env: AZURE_PRIVATE_TOKEN exampleEntityLocations: github: - https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/artist-lookup-component.yaml @@ -66,6 +85,8 @@ catalog: - https://github.com/spotify/backstage/blob/master/plugins/scaffolder-backend/sample-templates/create-react-app/template.yaml - https://github.com/spotify/cookiecutter-golang/blob/master/template.yaml - https://github.com/spotify/backstage/blob/master/plugins/scaffolder-backend/sample-templates/docs-template/template.yaml + - https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/hello-world-api.yaml + - https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/streetlights-api.yaml auth: providers: diff --git a/docs/features/software-catalog/configuration.md b/docs/features/software-catalog/configuration.md new file mode 100644 index 0000000000..be6314e494 --- /dev/null +++ b/docs/features/software-catalog/configuration.md @@ -0,0 +1,58 @@ +--- +id: software-catalog-configuration +title: Catalog Configuration +--- + +## Static Location Configuration + +To enable declarative catalog setups, it is possible to add locations to the +catalog via [static configuration](../../conf/index.md). Locations are added to +the catalog under the `catalog.locations` key, for example: + +```yaml +catalog: + locations: + - type: github + target: https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/artist-lookup-component.yaml +``` + +The locations added through static configuration can not be removed through the +catalog locations API. To remove the locations, you have to remove them from the +configuration. + +## Catalog Rules + +By default the catalog will only allow ingestion of entities with the kind +`Component`, `API` and `Location`. In order to allow entities of other kinds to +be added, you need to add rules to the catalog. Rules are added either in a +separate `catalog.rules` key, or added to statically configured locations. + +For example, given the following configuration: + +```yaml +catalog: + rules: + - allow: [Component, API, Location, Template] + + locations: + - type: github + target: https://github.com/org/example/blob/master/org-data.yaml + rules: + - allow: [Group] +``` + +We are able to add entities of kind `Component`, `API`, `Location`, or +`Template` from any location, and `Group` entities from the `org-data.yaml`, +which will also be read as statically configured location. + +Note that if the `catalog.rules` key is present it will replace the default +value, meaning that you need to add rules for the default kinds if you want +those to still be allowed. + +The following configuration will reject any kind of entities from being added to +the catalog: + +```yaml +catalog: + rules: [] +``` diff --git a/docs/features/software-catalog/index.md b/docs/features/software-catalog/index.md index db8c183390..bfc3d53d4d 100644 --- a/docs/features/software-catalog/index.md +++ b/docs/features/software-catalog/index.md @@ -79,6 +79,22 @@ All software created through the [Backstage Software Templates](../software-templates/index.md) are automatically registered in the catalog. +### Static catalog configuration + +In addition to manually registering components, it is also possible to register +components though [static configuration](../../conf/index.md). For example, the +above example can be added using the following configuration: + +```yaml +catalog: + locations: + - type: github + target: https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/artist-lookup-component.yaml +``` + +More information about catalog configuration can be found +[here](configuration.md). + ### Updating component metadata Teams owning the components are responsible for maintaining the metadata about diff --git a/docs/features/software-templates/adding-templates.md b/docs/features/software-templates/adding-templates.md index 67e9c23605..4f7af7c015 100644 --- a/docs/features/software-templates/adding-templates.md +++ b/docs/features/software-templates/adding-templates.md @@ -58,7 +58,30 @@ Currently the catalog supports loading definitions from GitHub + Local Files. To load from other places, not only will there need to be another preparer, but the support to load the location will also need to be added to the Catalog. -For loading from a file the following command should work when the backend is +You can add the template files to the catalog through +[static location configuration](../software-catalog/configuration.md#static-location-configuration), +for example + +```yaml +catalog: + locations: + - type: github + target: https://github.com/spotify/cookiecutter-golang/blob/master/template.yaml + rules: + - allow: [Template] +``` + +Templates can also be added by posting the to the catalog directly. Note that if +you're doing this, you need to configure the catalog to allow template entities +to be ingested from any source, for example: + +```yaml +catalog: + rules: + - allow: [Component, API, Template] +``` + +For loading from a file, the following command should work when the backend is running: ```sh @@ -69,7 +92,7 @@ curl \ --data-raw "{\"type\": \"file\", \"target\": \"${YOUR PATH HERE}/template.yaml\"}" ``` -If loading from a git location, you can run the following +If loading from a Git location, you can run the following ```sh curl \ @@ -83,7 +106,7 @@ This should then have added the catalog, and also should now be listed under the create page at http://localhost:3000/create. Alternatively, if you want to get setup with some mock templates that are -already provided for you, you can run the following to load those templates: +already provided, run the following to load those templates: ``` yarn lerna run mock-data diff --git a/docs/features/software-templates/extending/create-your-own-publisher.md b/docs/features/software-templates/extending/create-your-own-publisher.md index 51869dc3ad..391f6f0658 100644 --- a/docs/features/software-templates/extending/create-your-own-publisher.md +++ b/docs/features/software-templates/extending/create-your-own-publisher.md @@ -7,8 +7,8 @@ Publishers are responsible for pushing and storing the templated skeleton after the values have been templated by the `Templater`. See [Create your own templater](./create-your-own-templater.md) for more info. -They recieve a directory or location where the templater has sucessfully run on, -and is now ready to store somewhere. They also get given some other options +They receive a directory or location where the templater has sucessfully run +and is now ready to store somewhere. They also are given some other options which are sent from the frontend, such as the `storePath` which is a string of where the frontend thinks we should save this templated folder. @@ -17,7 +17,7 @@ Currently we provide the following `publishers`: - `github` This publisher is passed through to the `createRouter` function of the -`@spotify/plugin-scaffolder-backend`. Currently only one publisher is supported, +`@spotify/plugin-scaffolder-backend`. Currently, only one publisher is supported, but PR's are always welcome. An full example backend can be found diff --git a/docs/features/software-templates/extending/create-your-own-templater.md b/docs/features/software-templates/extending/create-your-own-templater.md index a39f87924b..9dee66af06 100644 --- a/docs/features/software-templates/extending/create-your-own-templater.md +++ b/docs/features/software-templates/extending/create-your-own-templater.md @@ -8,14 +8,14 @@ returned by the preparers, and then executing the templating command on top of the file and returning the completed template path. This may or may not be the same directory as the input directory. -They also recieve additional values from the frontend, which can be used to +They also receive additional values from the frontend, which can be used to interpolate into the skeleton files. Currently we provide the following templaters: - `cookiecutter` -This templater is added the `TemplaterBuilder` and then passed into the +This templater is added to the `TemplaterBuilder` and then passed into the `createRouter` function of the `@spotify/plugin-scaffolder-backend` An full example backend can be found @@ -48,7 +48,7 @@ This `TemplaterKey` is used to select the correct templater from the `spec.templater` in the [Template Entity](../../software-catalog/descriptor-format.md#kind-template). -If you wish to add a new templater you'll need to register it with the +If you wish to add a new templater, you'll need to register it with the `TemplaterBuilder`. ### Creating your own Templater to add to the `TemplaterBuilder` @@ -83,10 +83,10 @@ follows: - `dockerClient` - a [dockerode](https://github.com/apocas/dockerode) client to be able to run docker containers. -_note_ currently the templaters that we provide are basically docker action +_note_ Currently the templaters that we provide are basically Docker action containers that are run on top of the skeleton folder. This keeps dependencies to a minimal for running backstage scaffolder, but you don't /have/ to use -docker. You could create your own templater that spins up an EC2 instance and +Docker. You could create your own templater that spins up an EC2 instance and downloads the folder and does everything using an AMI if you want. It's entirely up to you! @@ -138,7 +138,7 @@ spec: description: Description of the component ``` -You see that the `spec.templater` is set as `handlebars`, you'll need to +You see that the `spec.templater` is set as `handlebars`, so you'll need to register this with the `TemplaterBuilder` like so: ```ts diff --git a/docs/features/software-templates/extending/index.md b/docs/features/software-templates/extending/index.md index 21d8befda1..02d27a5725 100644 --- a/docs/features/software-templates/extending/index.md +++ b/docs/features/software-templates/extending/index.md @@ -5,11 +5,11 @@ title: Extending the Scaffolder Welcome. Take a seat. You're at the Scaffolder Documentation. -So - You wanna create stuff inside your company from some prebaked templates? +So, you want to create stuff inside your company from some prebaked templates? You're at the right place. -This guide is gonna take you through how the Scaffolder in Backstage works. -We'll dive into some jargon and run through whats going on in the backend to be +This guide is going to take you through how the Scaffolder in Backstage works. +We'll dive into some jargon and run through what's going on in the backend to be able to create these templates. There's also more guides that you might find useful at the bottom of this document. At it's core, theres 3 simple stages. @@ -25,9 +25,9 @@ scaffolder that you will need to know: 3. Publish Each of these steps can be configured for your own use case, but we provide some -sensible defaults too. +sensible defaults, too. -Lets dive a little deeper into these phases. +Let's dive a little deeper into these phases. ### Glossary and Jargon @@ -38,8 +38,8 @@ the router to pick the correct `Preparer` to run for the `Template` entity. **Templater** - The templater is responsible for actually running the chosen templater on top of the previously returned temporary directory from the -**Preprarer**. We advise making these docker containers as it can keep all -dependencies, for example Cookiecutter, self contained and not a dependency on +**Preprarer**. We advise making these Docker containers as it can keep all +dependencies--for example Cookiecutter--self contained and not a dependency on the host machine. **Publisher** - The publisher is responsible for taking the finished directory, @@ -50,11 +50,11 @@ passed through to the scaffolder backend. ### How it works -The main of the heavy lifting is done in the +Most of the heavy lifting is done in the [router.ts](https://github.com/spotify/backstage/blob/master/plugins/scaffolder-backend/src/service/router.ts#L93) file in the `scaffolder-backend` plugin. -There are 2 routes defined in the router. `POST /v1/jobs` and +There are two routes defined in the router: `POST /v1/jobs` and `GET /v1/job/:jobId` To create a scaffolding job, a JSON object containing the @@ -78,7 +78,7 @@ additional templating values must be posted as the post body. The values should represent something that is valid with the `schema` part of the [Template Entity](../../software-catalog/descriptor-format.md#kind-template) -Once that has been posted, a job will be setup with different stages. And the +Once that has been posted, a job will be setup with different stages, and the job processor will complete each stage before moving onto the next stage, whilst collecting logs and mutating the running job. diff --git a/docs/features/software-templates/index.md b/docs/features/software-templates/index.md index 503fbb7a3d..1c67676b3e 100644 --- a/docs/features/software-templates/index.md +++ b/docs/features/software-templates/index.md @@ -4,8 +4,8 @@ title: Software Templates --- The Software Templates part of Backstage is a tool that can help you create -Components inside Backstage. It by default has the ability to load skeletons of -code, template in some variables and then publish the template to some location +Components inside Backstage. By default, it has the ability to load skeletons of +code, template in some variables, and then publish the template to some location like GitHub.