paulononato/backstage
1
0
Fork 0
Code Issues Pull Requests Actions 41 Packages Projects Releases Wiki Activity

Merge branch 'master' into feature/AwsIamAuthForKubernetes

Browse Source
This commit is contained in:
Jonah Back
2021-01-23 16:01:49 -08:00
parent 6811112282 c3ea694e84
commit 7491b7105f
346 changed files with 8073 additions and 2363 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.changeset/bright-icons-repair.md
-5
View File
@@ -1,5 +0,0 @@
---
'@backstage/plugin-kubernetes-backend': patch
---
Support HTTP 400 Bad Request from Kubernetes API
.changeset/bright-radios-eat.md
-5
View File
@@ -1,5 +0,0 @@
---
'@backstage/plugin-github-actions': minor
---
Support GHE
.changeset/calm-turtles-smoke.md
-7
View File
@@ -1,7 +0,0 @@
---
'@backstage/core': minor
---
Removed `InfoCard` variant `height100`, originally deprecated in [#2826](https://github.com/backstage/backstage/pull/2826).
If your component still relies on this variant, simply replace it with `gridItem`.
.changeset/clever-timers-thank.md
-25
View File
@@ -1,25 +0,0 @@
---
'@backstage/backend-common': patch
'@backstage/integration': patch
---
Add support for GitHub Apps authentication for backend plugins.
`GithubCredentialsProvider` requests and caches GitHub credentials based on a repository or organization url.
The `GithubCredentialsProvider` class should be considered stateful since tokens will be cached internally.
Consecutive calls to get credentials will return the same token, tokens older than 50 minutes will be considered expired and reissued.
`GithubCredentialsProvider` will default to the configured access token if no GitHub Apps are configured.
More information on how to create and configure a GitHub App to use with backstage can be found in the documentation.
Usage:
```javascript
const credentialsProvider = new GithubCredentialsProvider(config);
const { token, headers } = await credentialsProvider.getCredentials({
url: 'https://github.com/',
});
```
Updates `GithubUrlReader` to use the `GithubCredentialsProvider`.
.changeset/cost-insights-careless-coins-pretend.md
-5
View File
@@ -1,5 +0,0 @@
---
'@backstage/plugin-cost-insights': patch
---
bug(cost-insights): Remove entity count when none present
.changeset/cost-insights-curvy-dingos-live.md
+5
View File
@@ -0,0 +1,5 @@
---
'@backstage/plugin-cost-insights': patch
---
Allow expand functionality to top panel product chart tooltip.
.changeset/green-coins-confess.md
-40
View File
@@ -1,40 +0,0 @@
---
'@backstage/create-app': patch
---
Migrate to using `FlatRoutes` from `@backstage/core` for the root app routes.
This is the first step in migrating applications as mentioned here: https://backstage.io/docs/plugins/composability#porting-existing-apps.
To apply this change to an existing app, switch out the `Routes` component from `react-router` to `FlatRoutes` from `@backstage/core`.
This also allows you to remove any `/*` suffixes on the route paths. For example:
```diff
import {
OAuthRequestDialog,
SidebarPage,
createRouteRef,
+ FlatRoutes,
} from '@backstage/core';
import { AppSidebar } from './sidebar';
-import { Route, Routes, Navigate } from 'react-router';
+import { Route, Navigate } from 'react-router';
import { Router as CatalogRouter } from '@backstage/plugin-catalog';
...
<AppSidebar />
- <Routes>
+ <FlatRoutes>
...
<Route
- path="/catalog/*"
+ path="/catalog"
element={<CatalogRouter EntityPage={EntityPage} />}
/>
- <Route path="/docs/*" element={<DocsRouter />} />
+ <Route path="/docs" element={<DocsRouter />} />
...
<Route path="/settings" element={<SettingsRouter />} />
- </Routes>
+ </FlatRoutes>
</SidebarPage>
```
.changeset/healthy-comics-drive.md
-6
View File
@@ -1,6 +0,0 @@
---
'@backstage/plugin-catalog': patch
---
Display the owner, system, and domain as links to the entity pages in the about card.
Only display fields in the about card that are applicable to the entity kind.
.changeset/healthy-crews-remember.md
-19
View File
@@ -1,19 +0,0 @@
---
'@backstage/create-app': patch
---
fix routing and config for user-settings plugin
To make the corresponding change in your local app, add the following in your App.tsx
```
import { Router as SettingsRouter } from '@backstage/plugin-user-settings';
...
<Route path="/settings" element={<SettingsRouter />} />
```
and the following to your plugins.ts:
```
export { plugin as UserSettings } from '@backstage/plugin-user-settings';
```
.changeset/honest-jokes-rush.md
+5
View File
@@ -0,0 +1,5 @@
---
'@backstage/plugin-auth-backend': patch
---
Use .text instead of .json for ALB key response
.changeset/loud-kids-dance.md
-5
View File
@@ -1,5 +0,0 @@
---
'@backstage/cli': patch
---
Append `-credentials.yaml` to credentials file generated by `backstage-cli create-github-app` and display warning about sensitive contents.
.changeset/orange-avocados-work.md
-5
View File
@@ -1,5 +0,0 @@
---
'@backstage/plugin-catalog-backend': patch
---
Refuse to remove the bootstrap location
.changeset/rich-turtles-roll.md
-19
View File
@@ -1,19 +0,0 @@
---
'@backstage/backend-common': minor
---
Remove fallback option from `UrlReaders.create` and `UrlReaders.default`, as well as the default fallback reader.
To be able to read data from endpoints outside of the configured integrations, you now need to explicitly allow it by
adding an entry in the `backend.reading.allow` list. For example:
```yml
backend:
baseUrl: ...
reading:
allow:
- host: example.com
- host: '*.examples.org'
```
Apart from adding the above configuration, most projects should not need to take any action to migrate existing code. If you do happen to have your own fallback reader configured, this needs to be replaced with a reader factory that selects a specific set of URLs to work with. If you where wrapping the existing fallback reader, the new one that handles the allow list is created using `FetchUrlReader.factory`.
.changeset/spoon-fork.md
-12
View File
@@ -1,12 +0,0 @@
---
'@backstage/create-app': patch
---
Add `*-credentials.yaml` to gitignore to prevent accidental commits of sensitive credential information.
To apply this change to an existing installation, add these lines to your `.gitignore`
```gitignore
# Sensitive credentials
*-credentials.yaml
```
.changeset/spotty-moons-tap.md
-5
View File
@@ -1,5 +0,0 @@
---
'@backstage/plugin-catalog-backend': patch
---
Reduce log noise on locations refresh
.changeset/strong-ligers-lay.md
-6
View File
@@ -1,6 +0,0 @@
---
'@backstage/plugin-catalog': patch
---
Display systems in catalog table and make both owner and system link to the entity pages.
The owner field is now taken from the relations of the entity instead of its spec.
.changeset/swift-baboons-refuse.md
-53
View File
@@ -1,53 +0,0 @@
---
'@backstage/create-app': patch
---
use `fromConfig` for all scaffolder helpers, and use the url protocol for app-config location entries.
To apply this change to your local installation, replace the contents of your `packages/backend/src/plugins/scaffolder.ts` with the following contents:
```ts
import {
CookieCutter,
createRouter,
Preparers,
Publishers,
CreateReactAppTemplater,
Templaters,
CatalogEntityClient,
} from '@backstage/plugin-scaffolder-backend';
import { SingleHostDiscovery } from '@backstage/backend-common';
import type { PluginEnvironment } from '../types';
import Docker from 'dockerode';
export default async function createPlugin({
logger,
config,
}: PluginEnvironment) {
const cookiecutterTemplater = new CookieCutter();
const craTemplater = new CreateReactAppTemplater();
const templaters = new Templaters();
templaters.register('cookiecutter', cookiecutterTemplater);
templaters.register('cra', craTemplater);
const preparers = await Preparers.fromConfig(config, { logger });
const publishers = await Publishers.fromConfig(config, { logger });
const dockerClient = new Docker();
const discovery = SingleHostDiscovery.fromConfig(config);
const entityClient = new CatalogEntityClient({ discovery });
return await createRouter({
preparers,
templaters,
publishers,
logger,
config,
dockerClient,
entityClient,
});
}
```
This will ensure that the `scaffolder-backend` package can add handlers for the `url` protocol which is becoming the standard when registering entities in the `catalog`
.changeset/techdocs-glasses-wonder.md
-5
View File
@@ -1,5 +0,0 @@
---
'@backstage/techdocs-common': patch
---
TechDocs backend now streams files through from Google Cloud Storage to the browser, improving memory usage.
.changeset/unlucky-cougars-grin.md
-9
View File
@@ -1,9 +0,0 @@
---
'@backstage/create-app': patch
---
Remove the `@types/helmet` dev dependency from the app template. This
dependency is now unused as the package `helmet` brings its own types.
To update your existing app, simply remove the `@types/helmet` dependency from
the `package.json` of your backend package.
.changeset/warm-months-bake.md
-5
View File
@@ -1,5 +0,0 @@
---
'@backstage/plugin-lighthouse': patch
---
Fix display of floating point precision errors in card category scores
.changeset/wild-cats-end.md
-5
View File
@@ -1,5 +0,0 @@
---
'@backstage/plugin-catalog-backend': patch
---
Update the @azure/msal-node dependency to 1.0.0-beta.3.
.changeset/wise-mice-invite.md
-5
View File
@@ -1,5 +0,0 @@
---
'@backstage/backend-common': minor
---
Remove support for HTTPS certificate generation parameters. Use `backend.https = true` instead.
.changeset/yellow-ties-switch.md
-5
View File
@@ -1,5 +0,0 @@
---
'@backstage/catalog-model': minor
---
The catalog no longer attempts to merge old and new annotations, when updating an entity from a remote location. This was a behavior that was copied from kubernetes, and catered to use cases where you wanted to use HTTP POST to update an entity in-place, outside of what the refresh loop does. This has proved to be a mistake, because as a side effect, the refresh loop effectively is unable to ever delete annotations when they are removed from source YAML. This is obviously a breaking change, but we believe that this is not a behavior that is relied upon in the wild, and it has never been an actually supported use flow of the catalog. We therefore choose to break the behavior outright, and instead just store updated annotations verbatim - just like we already do for example for labels
.github/CODEOWNERS
+1
View File
@@ -6,6 +6,7 @@
* @backstage/maintainers
/docs/features/techdocs @backstage/techdocs-core
/docs/features/search @backstage/techdocs-core
/plugins/cost-insights @backstage/silver-lining
/plugins/cloudbuild @trivago/ebarrios
/plugins/search @backstage/techdocs-core
.github/styles/vocab.txt
+4
View File
@@ -67,8 +67,10 @@ Dominik
dtuite
dzolotusky
Ek
etag
env
Env
esbuild
eslint
Expedia
facto
@@ -84,6 +86,7 @@ GitHub
GitLab
Grafana
GraphQL
graphql
graphviz
Gustavsson
Hackathons
@@ -92,6 +95,7 @@ Henneke
Heroku
horizontalpodautoscalers
Hostname
hotspots
html
http
https
.github/workflows/fossa.yml
+44
View File
@@ -0,0 +1,44 @@
name: FOSSA
on:
push:
branches: [master]
pull_request:
branches: [master]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v2.3.4
# We use this to modify the generated .fossa.yml
- name: Install yq
run: sudo snap install yq
- name: Install Fossa
run: "curl -H 'Cache-Control: no-cache' https://raw.githubusercontent.com/fossas/fossa-cli/master/install.sh | bash"
- name: Configure Fossa
# The --option flag for fossa init does not work yet, see https://github.com/fossas/fossa-cli/issues/614
run: |
fossa init
yq eval -i '.analyze.modules[].options.strategy = "yarn-list"' .fossa.yml
# This deletes entries for template and example packages found within packages and plugins
# Seems like yq has a bug that causes only a subset of all matches to be deleted each run
yq eval -i 'del(.analyze.modules[] | select(.path == "*/*/**"))' .fossa.yml
yq eval -i 'del(.analyze.modules[] | select(.path == "*/*/**"))' .fossa.yml
yq eval -i 'del(.analyze.modules[] | select(.path == "*/*/**"))' .fossa.yml
yq eval -i 'del(.analyze.modules[] | select(.path == "*/*/**"))' .fossa.yml
yq eval -i 'del(.analyze.modules[] | select(.path == "*/*/**"))' .fossa.yml
- name: Show config
run: cat .fossa.yml
- name: Fossa Analyze
env:
# FOSSA Push-Only API Token
FOSSA_API_KEY: 9ee7e8893660832a7387dcc32377fb61
run: fossa analyze --branch "$GITHUB_REF"
app-config.yaml
+36 -18
View File
@@ -185,33 +185,46 @@ catalog:
# groupFilter: securityEnabled eq false and mailEnabled eq true and groupTypes/any(c:c+eq+'Unified')
locations:
# Add a location here to ingest it, for example from an URL:
#
# - type: url
# target: https://github.com/backstage/backstage/blob/master/packages/catalog-model/examples/all-components.yaml
#
# For local development you can use a file location instead:
#
# - type: file
# target: ../catalog-model/examples/all-components.yaml
#
# File locations are relative to the current working directory of the
# backend, for example packages/backend/.
# Backstage example components
- type: url
target: https://github.com/backstage/backstage/blob/master/packages/catalog-model/examples/all-components.yaml
- type: file
target: ../catalog-model/examples/all-components.yaml
# Example component for github-actions
- type: url
target: https://github.com/backstage/backstage/blob/master/plugins/github-actions/examples/sample.yaml
- type: file
target: ../../plugins/github-actions/examples/sample.yaml
# Example component for TechDocs
- type: url
target: https://github.com/backstage/backstage/blob/master/plugins/techdocs-backend/examples/documented-component/catalog-info.yaml
- type: file
target: ../../plugins/techdocs-backend/examples/documented-component/catalog-info.yaml
# Backstage example APIs
- type: url
target: https://github.com/backstage/backstage/blob/master/packages/catalog-model/examples/all-apis.yaml
- type: file
target: ../catalog-model/examples/all-apis.yaml
# Backstage example resources
- type: url
target: https://github.com/backstage/backstage/blob/master/packages/catalog-model/examples/all-resources.yaml
- type: file
target: ../catalog-model/examples/all-resources.yaml
# Backstage example systems
- type: url
target: https://github.com/backstage/backstage/blob/master/packages/catalog-model/examples/all-systems.yaml
- type: file
target: ../catalog-model/examples/all-systems.yaml
# Backstage example domains
- type: url
target: https://github.com/backstage/backstage/blob/master/packages/catalog-model/examples/all-domains.yaml
- type: file
target: ../catalog-model/examples/all-domains.yaml
# Backstage example templates
- type: url
target: https://github.com/backstage/backstage/blob/master/plugins/scaffolder-backend/sample-templates/all-templates.yaml
- type: file
target: ../../plugins/scaffolder-backend/sample-templates/all-templates.yaml
# Backstage example groups and users
- type: url
target: https://github.com/backstage/backstage/blob/master/packages/catalog-model/examples/acme-corp.yaml
- type: file
target: ../catalog-model/examples/acme-corp.yaml
scaffolder:
github:
@@ -359,3 +372,8 @@ homepage:
timezone: 'Asia/Tokyo'
pagerduty:
eventsBaseUrl: 'https://events.pagerduty.com/v2'
kafka:
clientId: backstage
brokers:
- localhost:9092
catalog-info.yaml
+1 -1
View File
@@ -6,7 +6,7 @@ metadata:
Backstage is an open-source developer portal that puts the developer experience first.
annotations:
github.com/project-slug: backstage/backstage
backstage.io/techdocs-ref: url:https://github.com/backstage/backstage/tree/master
backstage.io/techdocs-ref: url:https://github.com/backstage/backstage
lighthouse.com/website-url: https://backstage.io
spec:
type: library
contrib/docs/tutorials/authenticate-api-requests.md
+52
View File
@@ -0,0 +1,52 @@
# Authenticate API requests
The Backstage backend APIs are by default available without authentication. To avoid evil-doers from accessing or modifying data, one might use a network protection mechanism such as a firewall or an authenticating reverse proxy. For Backstage instances that are available on the Internet one can instead use the experimental IdentityClient as outlined below.
API requests from frontend plugins include an authorization header with a Backstage identity token acquired when the user logs in. By adding a middleware that verifies said token to be valid and signed by Backstage, non-authenticated requests can be blocked with a 401 Unauthorized response.
Note that this means Backstage will stop working for guests, as no token is issued for them.
Caveat: as of writing this, Backstage does not refresh the identity token so eventually users will get a 401 response on API calls (not on loading the web page as only the API calls are authenticated) and have to logout/login again to get a new token.
```typescript
// packages/backend/src/index.ts from a create-app deployment
import { Request, Response, NextFunction } from 'express';
import { IdentityClient } from '@backstage/plugin-auth-backend';
// ...
async function main() {
// ...
const discovery = SingleHostDiscovery.fromConfig(config);
const identity = new IdentityClient({
discovery,
issuer: await discovery.getExternalBaseUrl('auth'),
});
const authMiddleware = async (
req: Request,
res: Response,
next: NextFunction,
) => {
try {
const token = IdentityClient.getBearerToken(req.headers.authorization);
req.user = await identity.authenticate(token);
next();
} catch (error) {
res.status(401).send(`Unauthorized`);
}
};
const apiRouter = Router();
// The auth route must be publically available as it is used during login
apiRouter.use('/auth', await auth(authEnv));
// Only authenticated requests are allowed to the routes below
apiRouter.use('/catalog', authMiddleware, await catalog(catalogEnv));
apiRouter.use('/techdocs', authMiddleware, await techdocs(techdocsEnv));
apiRouter.use('/proxy', authMiddleware, await proxy(proxyEnv));
apiRouter.use(authMiddleware, notFoundHandler());
// ...
}
```
docs/architecture-decisions/adr006-avoid-react-fc.md
+9
View File
@@ -43,6 +43,15 @@ const GoodComponent = ({ text, children }: GoodProps) => (
{children}
</div>
);
/* Or as a shorthand, if no specifc child type is required */
type GoodProps = PropsWithChildren<{ text: string }>;
const GoodComponent = ({ text, children }: GoodProps) => (
<div>
<div>{text}</div>
{children}
</div>
);
```
## Consequences
docs/features/software-catalog/descriptor-format.md
-2
View File
@@ -977,8 +977,6 @@ metadata:
spec:
owner: artist-relations-team
domain: artists
providesApis:
- artist-api
```
In addition to the [common envelope metadata](#common-to-all-kinds-the-metadata)
docs/features/software-catalog/well-known-annotations.md
+21 -4
View File
@@ -22,7 +22,7 @@ use.
# Example:
metadata:
annotations:
backstage.io/managed-by-location: github:http://github.com/backstage/backstage/catalog-info.yaml
backstage.io/managed-by-location: url:http://github.com/backstage/backstage/blob/master/catalog-info.yaml
```
The value of this annotation is a so called location reference string, that
@@ -30,8 +30,8 @@ points to the source from which the entity was originally fetched. This
annotation is added automatically by the catalog as it fetches the data from a
registered location, and is not meant to normally be written by humans. The
annotation may point to any type of generic location that the catalog supports,
so it cannot be relied on to always be specifically of type `github`, nor that
it even represents a single file. Note also that a single location can be the
so it cannot be relied on to always be specifically of type `url`, nor that it
even represents a single file. Note also that a single location can be the
source of many entities, so it represents a many-to-one relationship.
The format of the value is `<type>:<target>`. Note that the target may also
@@ -40,13 +40,30 @@ expecting a two-item array out of it. The format of the target part is
type-dependent and could conceivably even be an empty string, but the separator
colon is always present.
### backstage.io/managed-by-origin-location
```yaml
# Example:
metadata:
annotations:
backstage.io/managed-by-origin-location: url:http://github.com/backstage/backstage/blob/master/catalog-info.yaml
```
The value of this annotation is a location reference string (see above). It
points to the location, whose registration lead to the creation of the entity.
In most cases, the `backstage.io/managed-by-location` and
`backstage.io/managed-by-origin-location` will be equal. They will be different
if the original location delegates to another location. A common case is, that a
location is registered as `bootstrap:bootstrap` which means that it is part of
the `app-config.yaml` of a Backstage installation.
### backstage.io/techdocs-ref
```yaml
# Example:
metadata:
annotations:
backstage.io/techdocs-ref: github:https://github.com/backstage/backstage.git
backstage.io/techdocs-ref: url:https://github.com/backstage/backstage/tree/master
```
The value of this annotation is a location reference string (see above). If this
docs/features/software-templates/extending/create-your-own-publisher.md
+1 -1
View File
@@ -57,7 +57,7 @@ That type looks like the following:
export type PublisherBase = {
publish(opts: {
entity: TemplateEntityV1alpha1;
values: RequiredTemplateValues & Record<string, JsonValue>;
values: TemplaterValues;
directory: string;
}): Promise<{ remoteUrl: string }>;
};
docs/features/software-templates/extending/create-your-own-templater.md
+1 -1
View File
@@ -61,7 +61,7 @@ That type looks like the following:
```ts
export type TemplaterRunOptions = {
directory: string;
values: RequiredTemplateValues & Record<string, JsonValue>;
values: TemplaterValues;
logStream?: Writable;
dockerClient: Docker;
};
docs/features/techdocs/architecture.md
+14
View File
@@ -123,6 +123,20 @@ a cache for the generated static content. TechDocs is also currently built on
MkDocs which does not allow us to generate docs per-page, so we would have to
build all docs for a entity on every request.
**Q. Can you use the techdocs plugin without the techdocs-backend plugin?**
A: `techdocs` and `techdocs-backend` plugins are designed to be used together,
like any other Backstage plugin with a frontend and its backend (catalog,
scaffolder, etc.). If you set your Backstage instance to generate docs on the
server, `techdocs-backend` will be responsible for managing the whole build
process, making sure it's scalable. It is responsible for securely communicating
with the cloud storage provider, for both fetching static generated sites and
publishing the updates. There are other planned features like an authentication
layer for users to determine whether they have the permission to view a
particular docs site. There are a handful of features which are extremely hard
to develop without a tightly integrated backend in place. Hence, support for
`techdocs` without `techdocs-backend` is limited and challenging to develop.
# Future work
_Ideas here are far fetched and not in the project's milestone for near future
docs/overview/roadmap.md
+4 -2
View File
@@ -57,8 +57,10 @@ guidelines to get started.
see and manage their services running in K8s, regardless if that's locally, in
AWS, GCS, Azure, or elsewhere.
- **Global search** - Extend the basic search functionality currently available
in the Backstage Service Catalog to become a global search experience.
- **[Search platform](../features/search/README.md)** - Evolve the basic search
functionality currently available into a platform that **a)** enables search
across the software catalog, TechDocs, and any other information exposed by
plugins, and **b)** supports a variety of search engine technologies.
- **[Software Templates V2](https://github.com/backstage/backstage/issues/2771)** -
Expand the templates to make the steps more composable by adding the ability
docs/plugins/plugin-development.md
-9
View File
@@ -54,13 +54,4 @@ addRoute(
Component: ComponentType<any>,
options?: RouteOptions,
): void;
/**
* @deprecated See the `addRoute` method
*/
registerRoute(
path: RoutePath,
Component: ComponentType<any>,
options?: RouteOptions,
): void;
```
docs/reference/createPlugin-router.md
-9
View File
@@ -15,15 +15,6 @@ addRoute(
Component: ComponentType<any>,
options?: RouteOptions,
): void;
/**
* @deprecated See the `addRoute` method
*/
registerRoute(
path: RoutePath,
Component: ComponentType<any>,
options?: RouteOptions,
): void;
```
## RouteRef
microsite/core/Components.js
+20
View File
@@ -106,9 +106,29 @@ const Breakpoint = ({ narrow, wide }) => (
</React.Fragment>
);
const Banner = simpleComponent('div', 'Banner', ['hidden']);
Banner.Container = simpleComponent('div', 'Banner__Container');
const BannerDismissButton = simpleComponent(
props => (
<svg {...props} data-banner-dismiss viewBox="0 0 24 24">
<path d="M19 6.41L17.59 5 12 10.59 6.41 5 5 6.41 10.59 12 5 17.59 6.41 19 12 13.41 17.59 19 19 17.59 13.41 12z" />
</svg>
),
'Banner__DismissButton',
);
Banner.Dismissable = ({ storageKey, children }) => (
<Banner hidden data-banner={storageKey}>
{children}
<BannerDismissButton />
</Banner>
);
module.exports = {
Block,
ActionBlock,
Breakpoint,
BulletLine,
Banner,
};
microsite/data/plugins/argo-cd.yaml
+1 -1
View File
@@ -2,7 +2,7 @@
title: Argo CD
author: roadie.io
authorUrl: https://roadie.io
category: CI
category: CI/CD
description: View Argo CD status for your projects in Backstage.
documentation: https://roadie.io/backstage/plugins/argo-cd
iconUrl: https://roadie.io/images/logos/argo.png
microsite/data/plugins/buildkite.yaml
+1 -1
View File
@@ -2,7 +2,7 @@
title: Buildkite
author: roadie.io
authorUrl: https://roadie.io
category: CI
category: CI/CD
description: View Buildkite CI builds for your service in Backstage.
documentation: https://roadie.io/backstage/plugins/buildkite
iconUrl: https://roadie.io/images/logos/buildkite.png
microsite/data/plugins/circleci.yaml
+1 -1
View File
@@ -2,7 +2,7 @@
title: CircleCI
author: Spotify
authorUrl: https://github.com/spotify
category: CI
category: CI/CD
description: Automate your development process with CI hosted in the cloud or on a private server.
documentation: https://github.com/backstage/backstage/tree/master/plugins/circleci
iconUrl: https://www.saaves.com/storage/brochure/logo-circleci-icon1583764538.png
microsite/data/plugins/cloud-build.yaml
+1 -1
View File
@@ -2,7 +2,7 @@
title: Google Cloud Build
author: Trivago
authorUrl: https://www.trivago.com
category: CI
category: CI/CD
description: Build, test, and deploy on Google's serverless CI/CD platform.
documentation: https://github.com/backstage/backstage/tree/master/plugins/cloudbuild
iconUrl: https://avatars2.githubusercontent.com/u/38220399?s=400&v=4
microsite/data/plugins/gcp-projects.yaml
+1 -1
View File
@@ -2,7 +2,7 @@
title: GCP Project Creator
author: Trivago
authorUrl: https://www.trivago.com
category: Cloud
category: Infrastructure
description: Create, list and manage your Google Cloud Projects.
documentation: https://github.com/backstage/backstage/tree/master/plugins/gcp-projects
iconUrl: https://avatars1.githubusercontent.com/u/2810941?s=280&v=4
microsite/data/plugins/github-actions.yaml
+1 -1
View File
@@ -2,7 +2,7 @@
title: GitHub Actions
author: Spotify
authorUrl: https://github.com/spotify
category: CI
category: CI/CD
description: GitHub Actions makes it easy to automate all your software workflows, now with world-class CI/CD. Build, test, and deploy your code right from GitHub.
documentation: https://github.com/backstage/backstage/tree/master/plugins/github-actions
iconUrl: https://avatars2.githubusercontent.com/u/44036562?s=400&v=4
microsite/data/plugins/github-insights.yaml
+1 -1
View File
@@ -2,7 +2,7 @@
title: GitHub Insights
author: roadie.io
authorUrl: https://roadie.io
category: Monitoring
category: Source Control Mgmt
description: View GitHub Insights for your components in Backstage.
documentation: https://roadie.io/backstage/plugins/github-insights
iconUrl: https://roadie.io/images/logos/insights.png
microsite/data/plugins/github-pull-requests.yaml
+1 -1
View File
@@ -2,7 +2,7 @@
title: GitHub Pull Requests
author: roadie.io
authorUrl: https://roadie.io/
category: CI
category: Source Control Mgmt
description: View GitHub pull requests for your service in Backstage.
documentation: https://roadie.io/backstage/plugins/github-pull-requests
iconUrl: https://roadie.io/images/logos/github.png
microsite/data/plugins/jenkins.yaml
+1 -1
View File
@@ -2,7 +2,7 @@
title: Jenkins
author: '@timja'
authorUrl: https://github.com/timja
category: CI
category: CI/CD
description: Jenkins offers a simple way to set up a continuous integration and continuous delivery environment.
documentation: https://github.com/backstage/backstage/tree/master/plugins/jenkins
iconUrl: https://img.icons8.com/color/1600/jenkins.png
microsite/data/plugins/jira.yaml
+1 -1
View File
@@ -2,7 +2,7 @@
title: Jira
author: roadie.io
authorUrl: https://roadie.io
category: Project Management
category: Agile Planning
description: View Jira summary for your projects in Backstage.
documentation: https://roadie.io/backstage/plugins/jira
iconUrl: https://roadie.io/images/logos/jira.png
microsite/data/plugins/kafka.yaml
+11
View File
@@ -0,0 +1,11 @@
---
title: Kafka
author: '@nirga'
authorUrl: https://github.com/nirga
category: Monitoring
description: Observability for Apache Kafka clusters and async API of components.
documentation: https://github.com/backstage/backstage/tree/master/plugins/kafka
iconUrl: https://kafka.apache.org/images/apache-kafka.png
npmPackageName: '@backstage/plugin-kafka'
tags:
- monitoring
microsite/data/plugins/lighthouse.yaml
+1 -1
View File
@@ -2,7 +2,7 @@
title: Lighthouse
author: Spotify
authorUrl: https://github.com/spotify
category: Accessibility
category: Quality
description: Google's Lighthouse tool is a great resource for benchmarking and improving the accessibility, performance, SEO, and best practices of your website.
documentation: https://github.com/backstage/backstage/tree/master/plugins/lighthouse
iconUrl: https://seeklogo.com/images/G/google-lighthouse-logo-1C7FA08580-seeklogo.com.png
microsite/data/plugins/snyk-security.yaml
+9
View File
@@ -0,0 +1,9 @@
---
title: Snyk
author: Snyk Tech Services
authorUrl: https://snyk.io
category: Security
description: View Snyk scanned vulnerabilities and license compliance of your components directly in Backstage.
documentation: https://github.com/snyk-tech-services/backstage-plugin-snyk/blob/main/README.md
iconUrl: https://storage.googleapis.com/snyk-technical-services.appspot.com/snyk-logo-vertical-black.png
npmPackageName: 'backstage-plugin-snyk'
microsite/data/plugins/travis-ci.yaml
+1 -1
View File
@@ -2,7 +2,7 @@
title: Travis CI
author: roadie.io
authorUrl: https://roadie.io/
category: CI
category: CI/CD
description: View Travis CI builds for your service in Backstage.
documentation: https://roadie.io/backstage/plugins/travis-ci
iconUrl: https://roadie.io/images/logos/travis.png
microsite/pages/en/index.js
+10
View File
@@ -11,6 +11,7 @@ const Block = Components.Block;
const ActionBlock = Components.ActionBlock;
const Breakpoint = Components.Breakpoint;
const BulletLine = Components.BulletLine;
const Banner = Components.Banner;
class Index extends React.Component {
render() {
@@ -53,6 +54,15 @@ class Index extends React.Component {
</Block.Container>
</Block>
<Banner.Container>
<Banner.Dismissable storageKey="k8s-launch">
🎉 New feature: Kubernetes for service owners.{' '}
<a href="https://backstage.io/blog/2021/01/12/new-backstage-feature-kubernetes-for-service-owners">
Learn more.
</a>
</Banner.Dismissable>
</Banner.Container>
<Block small className="stripe-top bg-black">
<Block.Container wrapped>
<Block.TextBox>
microsite/pages/en/plugins.js
+4 -4
View File
@@ -97,14 +97,14 @@ const Plugins = () => (
</div>
<Container className="PluginCardFooter">
<p>
See what plugins are already{' '}
See what plugins are already
<a href="https://github.com/backstage/backstage/issues?q=is%3Aissue+is%3Aopen+label%3Aplugin+sort%3Areactions-%2B1-desc">
in progress
</a>{' '}
and 👍. Missing a plugin for your favorite tool? Please{' '}
</a>
and 👍. Missing a plugin for your favorite tool? Please
<a href="https://github.com/backstage/backstage/issues/new?labels=plugin&template=plugin_template.md&title=%5BPlugin%5D+THE+PLUGIN+NAME">
suggest
</a>{' '}
</a>
a new one.
</p>
</Container>
microsite/siteConfig.js
+1
View File
@@ -82,6 +82,7 @@ const siteConfig = {
'https://buttons.github.io/buttons.js',
'https://unpkg.com/medium-zoom@1.0.6/dist/medium-zoom.min.js',
'/js/medium-zoom.js',
'/js/dismissable-banner.js',
],
// On page navigation for the current documentation page.
microsite/static/css/custom.css
+49
View File
@@ -1035,6 +1035,55 @@ code {
}
}
.Banner {
position: relative;
padding: 14px;
margin: 14px 20px;
border-radius: 4px;
background-color: $primaryColor;
font-family: Helvetica Neue, sans-serif;
color: #000;
}
.Banner--hidden {
opacity: 0;
transition: opacity 200ms ease-in-out;
}
.Banner a {
color: #000;
text-decoration: underline;
}
.Banner__Container {
position: relative;
overflow: visible;
z-index: 100;
max-width: 1430px;
height: 0;
margin: -14px auto 14px auto;
}
.Banner__DismissButton {
position: absolute;
display: flex;
right: 8px;
top: 0;
bottom: 0;
margin: auto;
border-radius: 50%;
padding: 6px;
width: 36px;
height: 36px;
cursor: pointer;
}
.Banner__DismissButton:hover {
background: rgba(0, 0, 0, 0.2);
}
.logos-mobile-background {
position: absolute;
width: 200vw;
microsite/static/js/dismissable-banner.js
+18
View File
@@ -0,0 +1,18 @@
window.addEventListener('DOMContentLoaded', () => {
const banners = document.querySelectorAll('[data-banner]');
banners.forEach(banner => {
const storageKey = `hideBanner/${banner.getAttribute('data-banner')}`;
if (!localStorage.getItem(storageKey)) {
banner.classList.remove('Banner--hidden');
}
const dismissButton = banner.querySelector('[data-banner-dismiss]');
if (dismissButton) {
dismissButton.addEventListener('click', () => {
banner.classList.add('Banner--hidden');
localStorage.setItem(storageKey, 'true');
});
}
});
});
package.json
+1 -1
View File
@@ -75,7 +75,7 @@
},
"jest": {
"transformModules": [
"@kyma-project/asyncapi-react"
"@asyncapi/react-component"
]
}
}
packages/app/CHANGELOG.md
+51
View File
@@ -1,5 +1,56 @@
# example-app
## 0.2.12
### Patch Changes
- Updated dependencies [def2307f3]
- Updated dependencies [46bba09ea]
- Updated dependencies [efd6ef753]
- Updated dependencies [593632f07]
- Updated dependencies [8c2437c15]
- Updated dependencies [2b514d532]
- Updated dependencies [33846acfc]
- Updated dependencies [b604a9d41]
- Updated dependencies [d014185db]
- Updated dependencies [a187b8ad0]
- Updated dependencies [8855f61f6]
- Updated dependencies [ed6baab66]
- Updated dependencies [f04db53d7]
- Updated dependencies [a5e27d5c1]
- Updated dependencies [debf359b5]
- Updated dependencies [a93f42213]
- @backstage/catalog-model@0.7.0
- @backstage/plugin-github-actions@0.3.0
- @backstage/core@0.5.0
- @backstage/plugin-catalog@0.2.12
- @backstage/plugin-cost-insights@0.5.7
- @backstage/plugin-catalog-import@0.3.5
- @backstage/cli@0.4.7
- @backstage/plugin-kubernetes@0.3.6
- @backstage/plugin-api-docs@0.4.3
- @backstage/plugin-scaffolder@0.4.0
- @backstage/plugin-techdocs@0.5.4
- @backstage/plugin-lighthouse@0.2.8
- @backstage/plugin-circleci@0.2.6
- @backstage/plugin-cloudbuild@0.2.7
- @backstage/plugin-jenkins@0.3.6
- @backstage/plugin-kafka@0.1.1
- @backstage/plugin-org@0.3.4
- @backstage/plugin-pagerduty@0.2.6
- @backstage/plugin-register-component@0.2.7
- @backstage/plugin-rollbar@0.2.8
- @backstage/plugin-search@0.2.6
- @backstage/plugin-sentry@0.3.3
- @backstage/plugin-explore@0.2.3
- @backstage/plugin-gcp-projects@0.2.3
- @backstage/plugin-gitops-profiles@0.2.3
- @backstage/plugin-graphiql@0.2.6
- @backstage/plugin-newrelic@0.2.3
- @backstage/plugin-tech-radar@0.3.3
- @backstage/plugin-user-settings@0.2.4
- @backstage/plugin-welcome@0.2.4
## 0.2.9
### Patch Changes
packages/app/package.json
+31 -30
View File
@@ -1,38 +1,39 @@
{
"name": "example-app",
"version": "0.2.9",
"version": "0.2.12",
"private": true,
"bundled": true,
"dependencies": {
"@backstage/catalog-model": "^0.6.0",
"@backstage/cli": "^0.4.5",
"@backstage/core": "^0.4.2",
"@backstage/plugin-api-docs": "^0.4.1",
"@backstage/plugin-catalog": "^0.2.10",
"@backstage/plugin-catalog-import": "^0.3.3",
"@backstage/plugin-circleci": "^0.2.5",
"@backstage/plugin-cloudbuild": "^0.2.5",
"@backstage/plugin-cost-insights": "^0.5.5",
"@backstage/plugin-explore": "^0.2.2",
"@backstage/plugin-gcp-projects": "^0.2.2",
"@backstage/plugin-github-actions": "^0.2.6",
"@backstage/plugin-gitops-profiles": "^0.2.2",
"@backstage/plugin-graphiql": "^0.2.3",
"@backstage/plugin-org": "^0.3.2",
"@backstage/plugin-jenkins": "^0.3.4",
"@backstage/plugin-kubernetes": "^0.3.3",
"@backstage/plugin-lighthouse": "^0.2.6",
"@backstage/plugin-newrelic": "^0.2.2",
"@backstage/plugin-pagerduty": "0.2.5",
"@backstage/plugin-register-component": "^0.2.5",
"@backstage/plugin-rollbar": "^0.2.7",
"@backstage/plugin-scaffolder": "^0.3.6",
"@backstage/plugin-sentry": "^0.3.2",
"@backstage/plugin-search": "^0.2.5",
"@backstage/plugin-tech-radar": "^0.3.2",
"@backstage/plugin-techdocs": "^0.5.1",
"@backstage/plugin-user-settings": "^0.2.3",
"@backstage/plugin-welcome": "^0.2.3",
"@backstage/catalog-model": "^0.7.0",
"@backstage/cli": "^0.4.7",
"@backstage/core": "^0.5.0",
"@backstage/plugin-api-docs": "^0.4.3",
"@backstage/plugin-catalog": "^0.2.12",
"@backstage/plugin-catalog-import": "^0.3.5",
"@backstage/plugin-circleci": "^0.2.6",
"@backstage/plugin-cloudbuild": "^0.2.7",
"@backstage/plugin-cost-insights": "^0.5.7",
"@backstage/plugin-explore": "^0.2.3",
"@backstage/plugin-gcp-projects": "^0.2.3",
"@backstage/plugin-github-actions": "^0.3.0",
"@backstage/plugin-gitops-profiles": "^0.2.3",
"@backstage/plugin-graphiql": "^0.2.6",
"@backstage/plugin-org": "^0.3.4",
"@backstage/plugin-jenkins": "^0.3.6",
"@backstage/plugin-kafka": "^0.1.1",
"@backstage/plugin-kubernetes": "^0.3.6",
"@backstage/plugin-lighthouse": "^0.2.8",
"@backstage/plugin-newrelic": "^0.2.3",
"@backstage/plugin-pagerduty": "0.2.6",
"@backstage/plugin-register-component": "^0.2.7",
"@backstage/plugin-rollbar": "^0.2.8",
"@backstage/plugin-scaffolder": "^0.4.0",
"@backstage/plugin-sentry": "^0.3.3",
"@backstage/plugin-search": "^0.2.6",
"@backstage/plugin-tech-radar": "^0.3.3",
"@backstage/plugin-techdocs": "^0.5.4",
"@backstage/plugin-user-settings": "^0.2.4",
"@backstage/plugin-welcome": "^0.2.4",
"@backstage/theme": "^0.2.2",
"@material-ui/core": "^4.11.0",
"@material-ui/icons": "^4.9.1",
packages/app/src/components/catalog/EntityPage.tsx
+6
View File
@@ -63,6 +63,7 @@ import {
UserProfileCard,
} from '@backstage/plugin-org';
import { Router as SentryRouter } from '@backstage/plugin-sentry';
import { Router as KafkaRouter } from '@backstage/plugin-kafka';
import { EmbeddedDocsRouter as DocsRouter } from '@backstage/plugin-techdocs';
import { Button, Grid } from '@material-ui/core';
import {
@@ -243,6 +244,11 @@ const ServiceEntityPage = ({ entity }: { entity: Entity }) => (
title="Code Insights"
element={<GitHubInsightsRouter entity={entity} />}
/>
<EntityPageLayout.Content
path="/kafka/*"
title="Kafka"
element={<KafkaRouter entity={entity} />}
/>
</EntityPageLayout>
);
packages/app/src/plugins.ts
+1
View File
@@ -43,3 +43,4 @@ export { plugin as PagerDuty } from '@backstage/plugin-pagerduty';
export { plugin as Buildkite } from '@roadiehq/backstage-plugin-buildkite';
export { plugin as Search } from '@backstage/plugin-search';
export { plugin as Org } from '@backstage/plugin-org';
export { plugin as Kafka } from '@backstage/plugin-kafka';
packages/backend-common/CHANGELOG.md
+74
View File
@@ -1,5 +1,79 @@
# @backstage/backend-common
## 0.5.0
### Minor Changes
- 5345a1f98: Remove fallback option from `UrlReaders.create` and `UrlReaders.default`, as well as the default fallback reader.
To be able to read data from endpoints outside of the configured integrations, you now need to explicitly allow it by
adding an entry in the `backend.reading.allow` list. For example:
```yml
backend:
baseUrl: ...
reading:
allow:
- host: example.com
- host: '*.examples.org'
```
Apart from adding the above configuration, most projects should not need to take any action to migrate existing code. If you do happen to have your own fallback reader configured, this needs to be replaced with a reader factory that selects a specific set of URLs to work with. If you where wrapping the existing fallback reader, the new one that handles the allow list is created using `FetchUrlReader.factory`.
- 09a370426: Remove support for HTTPS certificate generation parameters. Use `backend.https = true` instead.
### Patch Changes
- 0b135e7e0: Add support for GitHub Apps authentication for backend plugins.
`GithubCredentialsProvider` requests and caches GitHub credentials based on a repository or organization url.
The `GithubCredentialsProvider` class should be considered stateful since tokens will be cached internally.
Consecutive calls to get credentials will return the same token, tokens older than 50 minutes will be considered expired and reissued.
`GithubCredentialsProvider` will default to the configured access token if no GitHub Apps are configured.
More information on how to create and configure a GitHub App to use with backstage can be found in the documentation.
Usage:
```javascript
const credentialsProvider = new GithubCredentialsProvider(config);
const { token, headers } = await credentialsProvider.getCredentials({
url: 'https://github.com/',
});
```
Updates `GithubUrlReader` to use the `GithubCredentialsProvider`.
- 294a70cab: 1. URL Reader's `readTree` method now returns an `etag` in the response along with the blob. The etag is an identifier of the blob and will only change if the blob is modified on the target. Usually it is set to the latest commit SHA on the target.
`readTree` also takes an optional `etag` in its options and throws a `NotModifiedError` if the etag matches with the etag of the resource.
So, the `etag` can be used in building a cache when working with URL Reader.
An example -
```ts
const response = await reader.readTree(
'https://github.com/backstage/backstage',
);
const etag = response.etag;
// Will throw a new NotModifiedError (exported from @backstage/backstage-common)
await reader.readTree('https://github.com/backstage/backstage', {
etag,
});
```
2. URL Reader's readTree method can now detect the default branch. So, `url:https://github.com/org/repo/tree/master` can be replaced with `url:https://github.com/org/repo` in places like `backstage.io/techdocs-ref`.
- 0ea032763: URL Reader: Use API response headers for archive filename in readTree. Fixes bug for users with hosted Bitbucket.
- Updated dependencies [0b135e7e0]
- Updated dependencies [fa8ba330a]
- Updated dependencies [ed6baab66]
- @backstage/integration@0.3.0
## 0.4.3
### Patch Changes
packages/backend-common/config.d.ts
Vendored
-77
View File
@@ -109,81 +109,4 @@ export interface Config {
*/
csp?: { [policyId: string]: string[] | false };
};
/** Configuration for integrations towards various external repository provider systems */
integrations?: {
/** Integration configuration for Azure */
azure?: Array<{
/**
* The hostname of the given Azure instance
*/
host: string;
/**
* Token used to authenticate requests.
* @visibility secret
*/
token?: string;
}>;
/** Integration configuration for BitBucket */
bitbucket?: Array<{
/**
* The hostname of the given Bitbucket instance
*/
host: string;
/**
* Token used to authenticate requests.
* @visibility secret
*/
token?: string;
/**
* The base url for the BitBucket API, for example https://api.bitbucket.org/2.0
*/
apiBaseUrl?: string;
/**
* The username to use for authenticated requests.
* @visibility secret
*/
username?: string;
/**
* BitBucket app password used to authenticate requests.
* @visibility secret
*/
appPassword?: string;
}>;
/** Integration configuration for GitHub */
github?: Array<{
/**
* The hostname of the given GitHub instance
*/
host: string;
/**
* Token used to authenticate requests.
* @visibility secret
*/
token?: string;
/**
* The base url for the GitHub API, for example https://api.github.com
*/
apiBaseUrl?: string;
/**
* The base url for GitHub raw resources, for example https://raw.githubusercontent.com
*/
rawBaseUrl?: string;
}>;
/** Integration configuration for GitLab */
gitlab?: Array<{
/**
* The hostname of the given GitLab instance
*/
host: string;
/**
* Token used to authenticate requests.
* @visibility secret
*/
token?: string;
}>;
};
}
packages/backend-common/package.json
+3 -3
View File
@@ -1,7 +1,7 @@
{
"name": "@backstage/backend-common",
"description": "Common functionality library for Backstage backends",
"version": "0.4.3",
"version": "0.5.0",
"main": "src/index.ts",
"types": "src/index.ts",
"private": false,
@@ -32,7 +32,7 @@
"@backstage/cli-common": "^0.1.1",
"@backstage/config": "^0.1.2",
"@backstage/config-loader": "^0.4.1",
"@backstage/integration": "^0.2.0",
"@backstage/integration": "^0.3.0",
"@types/cors": "^2.8.6",
"@types/express": "^4.17.6",
"archiver": "^5.0.2",
@@ -66,7 +66,7 @@
}
},
"devDependencies": {
"@backstage/cli": "^0.4.6",
"@backstage/cli": "^0.4.7",
"@backstage/test-utils": "^0.1.5",
"@types/archiver": "^5.1.0",
"@types/compression": "^1.7.0",
packages/backend-common/src/errors.ts
+5
View File
@@ -75,3 +75,8 @@ export class NotFoundError extends CustomErrorBase {}
* resource.
*/
export class ConflictError extends CustomErrorBase {}
/**
* The requested resource has not changed since last request.
*/
export class NotModifiedError extends CustomErrorBase {}
packages/backend-common/src/middleware/errorHandler.test.ts
+4
View File
@@ -72,6 +72,9 @@ describe('errorHandler', () => {
it('handles well-known error classes', async () => {
const app = express();
app.use('/NotModifiedError', () => {
throw new errors.NotModifiedError();
});
app.use('/InputError', () => {
throw new errors.InputError();
});
@@ -90,6 +93,7 @@ describe('errorHandler', () => {
app.use(errorHandler());
const r = request(app);
expect((await r.get('/NotModifiedError')).status).toBe(304);
expect((await r.get('/InputError')).status).toBe(400);
expect((await r.get('/AuthenticationError')).status).toBe(401);
expect((await r.get('/NotAllowedError')).status).toBe(403);
packages/backend-common/src/middleware/errorHandler.ts
+2
View File
@@ -101,6 +101,8 @@ function getStatusCode(error: Error): number {
// Handle well-known error types
switch (error.name) {
case errors.NotModifiedError.name:
return 304;
case errors.InputError.name:
return 400;
case errors.AuthenticationError.name:
packages/backend-common/src/reading/AzureUrlReader.test.ts
+60 -8
View File
@@ -23,6 +23,7 @@ import { getVoidLogger } from '../logging';
import { AzureUrlReader } from './AzureUrlReader';
import { msw } from '@backstage/test-utils';
import { ReadTreeResponseFactory } from './tree';
import { NotModifiedError } from '../errors';
const logger = getVoidLogger();
@@ -139,7 +140,12 @@ describe('AzureUrlReader', () => {
describe('readTree', () => {
const repoBuffer = fs.readFileSync(
path.resolve('src', 'reading', '__fixtures__', 'repo.zip'),
path.resolve('src', 'reading', '__fixtures__', 'mock-main.zip'),
);
const processor = new AzureUrlReader(
{ host: 'dev.azure.com' },
{ treeResponseFactory },
);
beforeEach(() => {
@@ -153,24 +159,70 @@ describe('AzureUrlReader', () => {
ctx.body(repoBuffer),
),
),
rest.get(
// https://docs.microsoft.com/en-us/rest/api/azure/devops/git/commits/get%20commits?view=azure-devops-rest-6.0#on-a-branch
'https://dev.azure.com/organization/project/_apis/git/repositories/repository/commits',
(_, res, ctx) =>
res(
ctx.status(200),
ctx.json({
count: 2,
value: [
{
commitId: '123abc2',
comment: 'second commit',
},
{
commitId: '123abc1',
comment: 'first commit',
},
],
}),
),
),
);
});
it('returns the wanted files from an archive', async () => {
const processor = new AzureUrlReader(
{ host: 'dev.azure.com' },
{ treeResponseFactory },
);
const response = await processor.readTree(
'https://dev.azure.com/organization/project/_git/repository',
);
expect(response.etag).toBe('123abc2');
const files = await response.files();
expect(files.length).toBe(2);
const mkDocsFile = await files[1].content();
const indexMarkdownFile = await files[0].content();
const mkDocsFile = await files[0].content();
const indexMarkdownFile = await files[1].content();
expect(mkDocsFile.toString()).toBe('site_name: Test\n');
expect(indexMarkdownFile.toString()).toBe('# Test\n');
});
it('throws a NotModifiedError when given a etag in options', async () => {
const fnAzure = async () => {
await processor.readTree(
'https://dev.azure.com/organization/project/_git/repository',
{ etag: '123abc2' },
);
};
await expect(fnAzure).rejects.toThrow(NotModifiedError);
});
it('should not throw a NotModifiedError when given an outdated etag in options', async () => {
const response = await processor.readTree(
'https://dev.azure.com/organization/project/_git/repository',
{ etag: 'outdated123abc' },
);
expect(response.etag).toBe('123abc2');
const files = await response.files();
expect(files.length).toBe(2);
const mkDocsFile = await files[0].content();
const indexMarkdownFile = await files[1].content();
expect(mkDocsFile.toString()).toBe('site_name: Test\n');
expect(indexMarkdownFile.toString()).toBe('# Test\n');
packages/backend-common/src/reading/AzureUrlReader.ts
+32 -9
View File
@@ -20,10 +20,11 @@ import {
getAzureFileFetchUrl,
getAzureDownloadUrl,
getAzureRequestOptions,
getAzureCommitsUrl,
} from '@backstage/integration';
import fetch from 'cross-fetch';
import { Readable } from 'stream';
import { NotFoundError } from '../errors';
import { NotFoundError, NotModifiedError } from '../errors';
import {
ReaderFactory,
ReadTreeOptions,
@@ -75,20 +76,42 @@ export class AzureUrlReader implements UrlReader {
url: string,
options?: ReadTreeOptions,
): Promise<ReadTreeResponse> {
const response = await fetch(
getAzureDownloadUrl(url),
getAzureRequestOptions(this.options, { Accept: 'application/zip' }),
// TODO: Support filepath based reading tree feature like other providers
// Get latest commit SHA
const commitsAzureResponse = await fetch(
getAzureCommitsUrl(url),
getAzureRequestOptions(this.options),
);
if (!response.ok) {
const message = `Failed to read tree from ${url}, ${response.status} ${response.statusText}`;
if (response.status === 404) {
if (!commitsAzureResponse.ok) {
const message = `Failed to read tree from ${url}, ${commitsAzureResponse.status} ${commitsAzureResponse.statusText}`;
if (commitsAzureResponse.status === 404) {
throw new NotFoundError(message);
}
throw new Error(message);
}
return this.deps.treeResponseFactory.fromZipArchive({
stream: (response.body as unknown) as Readable,
const commitSha = (await commitsAzureResponse.json()).value[0].commitId;
if (options?.etag && options.etag === commitSha) {
throw new NotModifiedError();
}
const archiveAzureResponse = await fetch(
getAzureDownloadUrl(url),
getAzureRequestOptions(this.options, { Accept: 'application/zip' }),
);
if (!archiveAzureResponse.ok) {
const message = `Failed to read tree from ${url}, ${archiveAzureResponse.status} ${archiveAzureResponse.statusText}`;
if (archiveAzureResponse.status === 404) {
throw new NotFoundError(message);
}
throw new Error(message);
}
return await this.deps.treeResponseFactory.fromZipArchive({
stream: (archiveAzureResponse.body as unknown) as Readable,
etag: commitSha,
filter: options?.filter,
});
}
packages/backend-common/src/reading/BitbucketUrlReader.test.ts
+96 -69
View File
@@ -20,6 +20,7 @@ import fs from 'fs';
import { rest } from 'msw';
import { setupServer } from 'msw/node';
import path from 'path';
import { NotModifiedError } from '../errors';
import { BitbucketUrlReader } from './BitbucketUrlReader';
import { ReadTreeResponseFactory } from './tree';
@@ -27,15 +28,24 @@ const treeResponseFactory = ReadTreeResponseFactory.create({
config: new ConfigReader({}),
});
const bitbucketProcessor = new BitbucketUrlReader(
{ host: 'bitbucket.org', apiBaseUrl: 'https://api.bitbucket.org/2.0' },
{ treeResponseFactory },
);
const hostedBitbucketProcessor = new BitbucketUrlReader(
{
host: 'bitbucket.mycompany.net',
apiBaseUrl: 'https://api.bitbucket.mycompany.net/rest/api/1.0',
},
{ treeResponseFactory },
);
describe('BitbucketUrlReader', () => {
describe('implementation', () => {
it('rejects unknown targets', async () => {
const processor = new BitbucketUrlReader(
{ host: 'bitbucket.org', apiBaseUrl: 'https://api.bitbucket.org/2.0' },
{ treeResponseFactory },
);
await expect(
processor.read('https://not.bitbucket.com/apa'),
bitbucketProcessor.read('https://not.bitbucket.com/apa'),
).rejects.toThrow(
'Incorrect URL: https://not.bitbucket.com/apa, Error: Invalid Bitbucket URL or file path',
);
@@ -55,14 +65,40 @@ describe('BitbucketUrlReader', () => {
),
);
it('returns the wanted files from an archive', async () => {
const privateBitbucketRepoBuffer = fs.readFileSync(
path.resolve(
'src',
'reading',
'__fixtures__',
'bitbucket-server-repo.zip',
),
);
beforeEach(() => {
worker.use(
rest.get(
'https://api.bitbucket.org/2.0/repositories/backstage/mock',
(_, res, ctx) =>
res(
ctx.status(200),
ctx.json({
mainbranch: {
type: 'branch',
name: 'master',
},
}),
),
),
rest.get(
'https://bitbucket.org/backstage/mock/get/master.zip',
(_, res, ctx) =>
res(
ctx.status(200),
ctx.set('Content-Type', 'application/zip'),
ctx.set(
'content-disposition',
'attachment; filename=backstage-mock-12ab34cd56ef.zip',
),
ctx.body(repoBuffer),
),
),
@@ -76,17 +112,39 @@ describe('BitbucketUrlReader', () => {
}),
),
),
rest.get(
'https://api.bitbucket.mycompany.net/rest/api/1.0/projects/backstage/repos/mock/archive?format=zip&prefix=mock&path=docs',
(_, res, ctx) =>
res(
ctx.status(200),
ctx.set('Content-Type', 'application/zip'),
ctx.set(
'content-disposition',
'attachment; filename=backstage-mock.zip',
),
ctx.body(privateBitbucketRepoBuffer),
),
),
rest.get(
'https://api.bitbucket.mycompany.net/rest/api/1.0/repositories/backstage/mock/commits/some-branch',
(_, res, ctx) =>
res(
ctx.status(200),
ctx.json({
values: [{ hash: '12ab34cd56ef78gh90ij12kl34mn56op78qr90st' }],
}),
),
),
);
});
const processor = new BitbucketUrlReader(
{ host: 'bitbucket.org', apiBaseUrl: 'https://api.bitbucket.org/2.0' },
{ treeResponseFactory },
);
const response = await processor.readTree(
it('returns the wanted files from an archive', async () => {
const response = await bitbucketProcessor.readTree(
'https://bitbucket.org/backstage/mock/src/master',
);
expect(response.etag).toBe('12ab34cd56ef');
const files = await response.files();
expect(files.length).toBe(2);
@@ -98,38 +156,12 @@ describe('BitbucketUrlReader', () => {
});
it('uses private bitbucket host', async () => {
const privateBitbucketRepoBuffer = fs.readFileSync(
path.resolve(
'src',
'reading',
'__fixtures__',
'bitbucket-server-repo.zip',
),
);
worker.use(
rest.get(
'https://api.bitbucket.mycompany.net/rest/api/1.0/projects/backstage/repos/mock/archive?format=zip&prefix=mock&path=docs',
(_, res, ctx) =>
res(
ctx.status(200),
ctx.set('Content-Type', 'application/zip'),
ctx.body(privateBitbucketRepoBuffer),
),
),
);
const processor = new BitbucketUrlReader(
{
host: 'bitbucket.mycompany.net',
apiBaseUrl: 'https://api.bitbucket.mycompany.net/rest/api/1.0',
},
{ treeResponseFactory },
);
const response = await processor.readTree(
const response = await hostedBitbucketProcessor.readTree(
'https://bitbucket.mycompany.net/projects/backstage/repos/mock/browse/docs?at=some-branch',
);
expect(response.etag).toBe('12ab34cd56ef');
const files = await response.files();
expect(files.length).toBe(1);
@@ -139,37 +171,12 @@ describe('BitbucketUrlReader', () => {
});
it('returns the wanted files from an archive with a subpath', async () => {
worker.use(
rest.get(
'https://bitbucket.org/backstage/mock/get/master.zip',
(_, res, ctx) =>
res(
ctx.status(200),
ctx.set('Content-Type', 'application/zip'),
ctx.body(repoBuffer),
),
),
rest.get(
'https://api.bitbucket.org/2.0/repositories/backstage/mock/commits/master',
(_, res, ctx) =>
res(
ctx.status(200),
ctx.json({
values: [{ hash: '12ab34cd56ef78gh90ij12kl34mn56op78qr90st' }],
}),
),
),
);
const processor = new BitbucketUrlReader(
{ host: 'bitbucket.org', apiBaseUrl: 'https://api.bitbucket.org/2.0' },
{ treeResponseFactory },
);
const response = await processor.readTree(
const response = await bitbucketProcessor.readTree(
'https://bitbucket.org/backstage/mock/src/master/docs',
);
expect(response.etag).toBe('12ab34cd56ef');
const files = await response.files();
expect(files.length).toBe(1);
@@ -177,5 +184,25 @@ describe('BitbucketUrlReader', () => {
expect(indexMarkdownFile.toString()).toBe('# Test\n');
});
it('throws a NotModifiedError when given a etag in options', async () => {
const fnBitbucket = async () => {
await bitbucketProcessor.readTree(
'https://bitbucket.org/backstage/mock',
{ etag: '12ab34cd56ef' },
);
};
await expect(fnBitbucket).rejects.toThrow(NotModifiedError);
});
it('should not throw a NotModifiedError when given an outdated etag in options', async () => {
const response = await bitbucketProcessor.readTree(
'https://bitbucket.org/backstage/mock',
{ etag: 'outdatedetag123abc' },
);
expect(response.etag).toBe('12ab34cd56ef');
});
});
});
packages/backend-common/src/reading/BitbucketUrlReader.ts
+35 -17
View File
@@ -25,7 +25,7 @@ import {
import fetch from 'cross-fetch';
import parseGitUrl from 'git-url-parse';
import { Readable } from 'stream';
import { NotFoundError } from '../errors';
import { NotFoundError, NotModifiedError } from '../errors';
import { ReadTreeResponseFactory } from './tree';
import {
ReaderFactory,
@@ -101,34 +101,52 @@ export class BitbucketUrlReader implements UrlReader {
url: string,
options?: ReadTreeOptions,
): Promise<ReadTreeResponse> {
const { name: repoName, owner: project, resource, filepath } = parseGitUrl(
url,
);
const { filepath } = parseGitUrl(url);
const isHosted = resource === 'bitbucket.org';
const lastCommitShortHash = await this.getLastCommitShortHash(url);
if (options?.etag && options.etag === lastCommitShortHash) {
throw new NotModifiedError();
}
const downloadUrl = await getBitbucketDownloadUrl(url, this.config);
const response = await fetch(
const archiveBitbucketResponse = await fetch(
downloadUrl,
getBitbucketRequestOptions(this.config),
);
if (!response.ok) {
const message = `Failed to read tree from ${url}, ${response.status} ${response.statusText}`;
if (response.status === 404) {
if (!archiveBitbucketResponse.ok) {
const message = `Failed to read tree from ${url}, ${archiveBitbucketResponse.status} ${archiveBitbucketResponse.statusText}`;
if (archiveBitbucketResponse.status === 404) {
throw new NotFoundError(message);
}
throw new Error(message);
}
let folderPath = `${project}-${repoName}`;
if (isHosted) {
const lastCommitShortHash = await this.getLastCommitShortHash(url);
folderPath = `${project}-${repoName}-${lastCommitShortHash}`;
// Get the filename of archive from the header of the response
const contentDispositionHeader = archiveBitbucketResponse.headers.get(
'content-disposition',
) as string;
if (!contentDispositionHeader) {
throw new Error(
`Failed to read tree from ${url}. ` +
'Bitbucket API response for downloading archive does not contain content-disposition header ',
);
}
const fileNameRegEx = new RegExp(
/^attachment; filename=(?<fileName>.*).zip$/,
);
const archiveFileName = contentDispositionHeader.match(fileNameRegEx)
?.groups?.fileName;
if (!archiveFileName) {
throw new Error(
`Failed to read tree from ${url}. Bitbucket API response for downloading archive has an unexpected ` +
`format of content-disposition header ${contentDispositionHeader} `,
);
}
return this.treeResponseFactory.fromZipArchive({
stream: (response.body as unknown) as Readable,
path: `${folderPath}/${filepath}`,
return await this.treeResponseFactory.fromZipArchive({
stream: (archiveBitbucketResponse.body as unknown) as Readable,
path: `${archiveFileName}/${filepath}`,
etag: lastCommitShortHash,
filter: options?.filter,
});
}
@@ -142,7 +160,7 @@ export class BitbucketUrlReader implements UrlReader {
return `bitbucket{host=${host},authed=${authed}}`;
}
private async getLastCommitShortHash(url: string): Promise<String> {
private async getLastCommitShortHash(url: string): Promise<string> {
const { name: repoName, owner: project, ref } = parseGitUrl(url);
let branch = ref;
packages/backend-common/src/reading/GithubUrlReader.test.ts
+184 -105
View File
@@ -21,6 +21,7 @@ import fs from 'fs';
import { rest } from 'msw';
import { setupServer } from 'msw/node';
import path from 'path';
import { NotFoundError, NotModifiedError } from '../errors';
import { GithubUrlReader } from './GithubUrlReader';
import { ReadTreeResponseFactory } from './tree';
@@ -28,11 +29,27 @@ const treeResponseFactory = ReadTreeResponseFactory.create({
config: new ConfigReader({}),
});
describe('GithubUrlReader', () => {
const mockCredentialsProvider = ({
getCredentials: jest.fn().mockResolvedValue({ headers: {} }),
} as unknown) as GithubCredentialsProvider;
const mockCredentialsProvider = ({
getCredentials: jest.fn().mockResolvedValue({ headers: {} }),
} as unknown) as GithubCredentialsProvider;
const githubProcessor = new GithubUrlReader(
{
host: 'github.com',
apiBaseUrl: 'https://api.github.com',
},
{ treeResponseFactory, credentialsProvider: mockCredentialsProvider },
);
const gheProcessor = new GithubUrlReader(
{
host: 'ghe.github.com',
apiBaseUrl: 'https://ghe.github.com/api/v3',
},
{ treeResponseFactory, credentialsProvider: mockCredentialsProvider },
);
describe('GithubUrlReader', () => {
const worker = setupServer();
msw.setupDefaultHandlers(worker);
@@ -43,15 +60,8 @@ describe('GithubUrlReader', () => {
describe('implementation', () => {
it('rejects unknown targets', async () => {
const processor = new GithubUrlReader(
{
host: 'github.com',
apiBaseUrl: 'https://api.github.com',
},
{ treeResponseFactory, credentialsProvider: mockCredentialsProvider },
);
await expect(
processor.read('https://not.github.com/apa'),
githubProcessor.read('https://not.github.com/apa'),
).rejects.toThrow(
'Incorrect URL: https://not.github.com/apa, Error: Invalid GitHub URL or file path',
);
@@ -73,7 +83,7 @@ describe('GithubUrlReader', () => {
worker.use(
rest.get(
'https://api.github.com/repos/backstage/mock/tree/contents/?ref=repo',
'https://api.github.com/repos/backstage/mock/tree/contents/?ref=main',
(req, res, ctx) => {
expect(req.headers.get('authorization')).toBe(
mockHeaders.Authorization,
@@ -90,50 +100,122 @@ describe('GithubUrlReader', () => {
),
);
const processor = new GithubUrlReader(
{
host: 'ghe.github.com',
apiBaseUrl: 'https://api.github.com',
},
{ treeResponseFactory, credentialsProvider: mockCredentialsProvider },
);
await processor.read(
'https://ghe.github.com/backstage/mock/tree/blob/repo',
await githubProcessor.read(
'https://github.com/backstage/mock/tree/blob/main',
);
});
});
describe('readTree', () => {
const repoBuffer = fs.readFileSync(
path.resolve('src', 'reading', '__fixtures__', 'repo.tar.gz'),
path.resolve(
'src',
'reading',
'__fixtures__',
'backstage-mock-etag123.tar.gz',
),
);
const reposGithubApiResponse = {
id: '123',
full_name: 'backstage/mock',
default_branch: 'main',
branches_url:
'https://api.github.com/repos/backstage/mock/branches{/branch}',
archive_url:
'https://api.github.com/repos/backstage/mock/{archive_format}{/ref}',
};
const reposGheApiResponse = {
...reposGithubApiResponse,
branches_url:
'https://ghe.github.com/api/v3/repos/backstage/mock/branches{/branch}',
archive_url:
'https://ghe.github.com/api/v3/repos/backstage/mock/{archive_format}{/ref}',
};
const branchesApiResponse = {
name: 'main',
commit: {
sha: 'etag123abc',
},
};
beforeEach(() => {
worker.use(
rest.get('https://api.github.com/repos/backstage/mock', (_, res, ctx) =>
res(
ctx.status(200),
ctx.set('Content-Type', 'application/json'),
ctx.json(reposGithubApiResponse),
),
),
rest.get(
'https://github.com/backstage/mock/archive/repo.tar.gz',
'https://api.github.com/repos/backstage/mock/branches/main',
(_, res, ctx) =>
res(
ctx.status(200),
ctx.set('Content-Type', 'application/json'),
ctx.json(branchesApiResponse),
),
),
rest.get(
'https://api.github.com/repos/backstage/mock/tarball/etag123abc',
(_, res, ctx) =>
res(
ctx.status(200),
ctx.set('Content-Type', 'application/x-gzip'),
ctx.set(
'content-disposition',
'attachment; filename=backstage-mock-etag123.tar.gz',
),
ctx.body(repoBuffer),
),
),
rest.get(
'https://api.github.com/repos/backstage/mock/branches/branchDoesNotExist',
(_, res, ctx) => res(ctx.status(404)),
),
rest.get(
'https://ghe.github.com/api/v3/repos/backstage/mock/tarball/etag123abc',
(_, res, ctx) =>
res(
ctx.status(200),
ctx.set('Content-Type', 'application/x-gzip'),
ctx.set(
'content-disposition',
'attachment; filename=backstage-mock-etag123.tar.gz',
),
ctx.body(repoBuffer),
),
),
rest.get(
'https://ghe.github.com/api/v3/repos/backstage/mock',
(_, res, ctx) =>
res(
ctx.status(200),
ctx.set('Content-Type', 'application/json'),
ctx.json(reposGheApiResponse),
),
),
rest.get(
'https://ghe.github.com/api/v3/repos/backstage/mock/branches/main',
(_, res, ctx) =>
res(
ctx.status(200),
ctx.set('Content-Type', 'application/json'),
ctx.json(branchesApiResponse),
),
),
);
});
it('returns the wanted files from an archive', async () => {
const processor = new GithubUrlReader(
{
host: 'github.com',
apiBaseUrl: 'https://api.github.com',
},
{ treeResponseFactory, credentialsProvider: mockCredentialsProvider },
const response = await githubProcessor.readTree(
'https://github.com/backstage/mock/tree/main',
);
const response = await processor.readTree(
'https://github.com/backstage/mock/tree/repo',
);
expect(response.etag).toBe('etag123abc');
const files = await response.files();
@@ -145,40 +227,6 @@ describe('GithubUrlReader', () => {
expect(indexMarkdownFile.toString()).toBe('# Test\n');
});
it('includes the subdomain in the github url', async () => {
worker.resetHandlers();
worker.use(
rest.get(
'https://ghe.github.com/backstage/mock/archive/repo.tar.gz',
(_, res, ctx) =>
res(
ctx.status(200),
ctx.set('Content-Type', 'application/x-gzip'),
ctx.body(repoBuffer),
),
),
);
const processor = new GithubUrlReader(
{
host: 'ghe.github.com',
apiBaseUrl: 'https://api.github.com',
},
{ treeResponseFactory, credentialsProvider: mockCredentialsProvider },
);
const response = await processor.readTree(
'https://ghe.github.com/backstage/mock/tree/repo/docs',
);
const files = await response.files();
expect(files.length).toBe(1);
const indexMarkdownFile = await files[0].content();
expect(indexMarkdownFile.toString()).toBe('# Test\n');
});
it('should use the headers from the credentials provider to the fetch request', async () => {
expect.assertions(2);
@@ -193,7 +241,7 @@ describe('GithubUrlReader', () => {
worker.use(
rest.get(
'https://ghe.github.com/backstage/mock/archive/repo.tar.gz',
'https://ghe.github.com/api/v3/repos/backstage/mock/tarball/etag123abc',
(req, res, ctx) => {
expect(req.headers.get('authorization')).toBe(
mockHeaders.Authorization,
@@ -204,52 +252,24 @@ describe('GithubUrlReader', () => {
return res(
ctx.status(200),
ctx.set('Content-Type', 'application/x-gzip'),
ctx.set(
'content-disposition',
'attachment; filename=backstage-mock-etag123.tar.gz',
),
ctx.body(repoBuffer),
);
},
),
);
const processor = new GithubUrlReader(
{
host: 'ghe.github.com',
apiBaseUrl: 'https://api.github.com',
},
{ treeResponseFactory, credentialsProvider: mockCredentialsProvider },
);
await processor.readTree(
'https://ghe.github.com/backstage/mock/tree/repo/docs',
await gheProcessor.readTree(
'https://ghe.github.com/backstage/mock/tree/main',
);
});
it('must specify a branch', async () => {
const processor = new GithubUrlReader(
{
host: 'github.com',
apiBaseUrl: 'https://api.github.com',
},
{ treeResponseFactory, credentialsProvider: mockCredentialsProvider },
);
await expect(
processor.readTree('https://github.com/backstage/mock'),
).rejects.toThrow(
'GitHub URL must contain branch to be able to fetch tree',
);
});
it('returns the wanted files from an archive with a subpath', async () => {
const processor = new GithubUrlReader(
{
host: 'github.com',
apiBaseUrl: 'https://api.github.com',
},
{ treeResponseFactory, credentialsProvider: mockCredentialsProvider },
);
const response = await processor.readTree(
'https://github.com/backstage/mock/tree/repo/docs',
it('includes the subdomain in the github url', async () => {
const response = await gheProcessor.readTree(
'https://ghe.github.com/backstage/mock/tree/main/docs',
);
const files = await response.files();
@@ -259,5 +279,64 @@ describe('GithubUrlReader', () => {
expect(indexMarkdownFile.toString()).toBe('# Test\n');
});
it('returns the wanted files from an archive with a subpath', async () => {
const response = await githubProcessor.readTree(
'https://github.com/backstage/mock/tree/main/docs',
);
const files = await response.files();
expect(files.length).toBe(1);
const indexMarkdownFile = await files[0].content();
expect(indexMarkdownFile.toString()).toBe('# Test\n');
});
it('throws a NotModifiedError when given a etag in options', async () => {
const fnGithub = async () => {
await githubProcessor.readTree('https://github.com/backstage/mock', {
etag: 'etag123abc',
});
};
const fnGhe = async () => {
await gheProcessor.readTree(
'https://ghe.github.com/backstage/mock/tree/main/docs',
{
etag: 'etag123abc',
},
);
};
await expect(fnGithub).rejects.toThrow(NotModifiedError);
await expect(fnGhe).rejects.toThrow(NotModifiedError);
});
it('should not throw error when given an outdated etag in options', async () => {
const response = await githubProcessor.readTree(
'https://github.com/backstage/mock/tree/main',
{
etag: 'outdatedetag123abc',
},
);
expect((await response.files()).length).toBe(2);
});
it('should detect the default branch', async () => {
const response = await githubProcessor.readTree(
'https://github.com/backstage/mock',
);
expect((await response.files()).length).toBe(2);
});
it('should throw error on missing branch', async () => {
const fnGithub = async () => {
await githubProcessor.readTree(
'https://github.com/backstage/mock/tree/branchDoesNotExist',
);
};
await expect(fnGithub).rejects.toThrow(NotFoundError);
});
});
});
packages/backend-common/src/reading/GithubUrlReader.ts
+85 -30
View File
@@ -23,7 +23,7 @@ import {
import fetch from 'cross-fetch';
import parseGitUrl from 'git-url-parse';
import { Readable } from 'stream';
import { InputError, NotFoundError } from '../errors';
import { NotFoundError, NotModifiedError } from '../errors';
import { ReadTreeResponseFactory } from './tree';
import {
ReaderFactory,
@@ -98,51 +98,106 @@ export class GithubUrlReader implements UrlReader {
url: string,
options?: ReadTreeOptions,
): Promise<ReadTreeResponse> {
const {
name: repoName,
ref,
protocol,
resource,
full_name,
filepath,
} = parseGitUrl(url);
if (!ref) {
// TODO(Rugvip): We should add support for defaulting to the default branch
throw new InputError(
'GitHub URL must contain branch to be able to fetch tree',
);
}
const { ref, filepath, full_name } = parseGitUrl(url);
// Caveat: The ref will totally be incorrect if the branch name includes a /
// Thus, readTree can not work on url containing branch name that has a /
const { headers } = await this.deps.credentialsProvider.getCredentials({
url,
});
// TODO(Rugvip): use API to fetch URL instead
const response = await fetch(
new URL(
`${protocol}://${resource}/${full_name}/archive/${ref}.tar.gz`,
).toString(),
// Get GitHub API urls for the repository
const repoGitHubResponse = await fetch(
new URL(`${this.config.apiBaseUrl}/repos/${full_name}`).toString(),
{
headers: {
...headers,
},
headers,
},
);
if (!response.ok) {
const message = `Failed to read tree from ${url}, ${response.status} ${response.statusText}`;
if (response.status === 404) {
if (!repoGitHubResponse.ok) {
const message = `Failed to read tree (repository) from ${url}, ${repoGitHubResponse.status} ${repoGitHubResponse.statusText}`;
if (repoGitHubResponse.status === 404) {
throw new NotFoundError(message);
}
throw new Error(message);
}
const path = `${repoName}-${ref}/${filepath}`;
const repoResponseJson = await repoGitHubResponse.json();
return this.deps.treeResponseFactory.fromTarArchive({
// ref is an empty string if no branch is set in provided url to readTree.
// Use GitHub API to get the default branch of the repository.
const branch = ref || repoResponseJson.default_branch;
const branchesApiUrl = repoResponseJson.branches_url;
const archiveApiUrl = repoResponseJson.archive_url;
// Fetch the latest commit in the provided or default branch to compare against
// the provided sha.
const branchGitHubResponse = await fetch(
// branchesApiUrl looks like "https://api.github.com/repos/owner/repo/branches{/branch}"
branchesApiUrl.replace('{/branch}', `/${branch}`),
{
headers,
},
);
if (!branchGitHubResponse.ok) {
const message = `Failed to read tree (branch) from ${url}, ${branchGitHubResponse.status} ${branchGitHubResponse.statusText}`;
if (branchGitHubResponse.status === 404) {
throw new NotFoundError(message);
}
throw new Error(message);
}
const commitSha = (await branchGitHubResponse.json()).commit.sha;
if (options?.etag && options.etag === commitSha) {
throw new NotModifiedError();
}
const archive = await fetch(
// archiveApiUrl looks like "https://api.github.com/repos/owner/repo/{archive_format}{/ref}"
archiveApiUrl
.replace('{archive_format}', 'tarball')
.replace('{/ref}', `/${commitSha}`),
{ headers },
);
if (!archive.ok) {
const message = `Failed to read tree (archive) from ${url}, ${archive.status} ${archive.statusText}`;
if (archive.status === 404) {
throw new NotFoundError(message);
}
throw new Error(message);
}
// Get the filename of archive from the header of the response
const contentDispositionHeader = archive.headers.get(
'content-disposition',
) as string;
if (!contentDispositionHeader) {
throw new Error(
`Failed to read tree from ${url}. ` +
'GitHub API response for downloading archive does not contain content-disposition header ',
);
}
const fileNameRegEx = new RegExp(
/^attachment; filename=(?<fileName>.*).tar.gz$/,
);
const archiveFileName = contentDispositionHeader.match(fileNameRegEx)
?.groups?.fileName;
if (!archiveFileName) {
throw new Error(
`Failed to read tree from ${url}. GitHub API response for downloading archive has an unexpected ` +
`format of content-disposition header ${contentDispositionHeader} `,
);
}
// The path includes the name of the directory inside the tarball and a sub path
// if requested in readTree.
const path = `${archiveFileName}/${filepath}`;
return await this.deps.treeResponseFactory.fromTarArchive({
// TODO(Rugvip): Underlying implementation of fetch will be node-fetch, we probably want
// to stick to using that in exclusively backend code.
stream: (response.body as unknown) as Readable,
stream: (archive.body as unknown) as Readable,
path,
etag: commitSha,
filter: options?.filter,
});
}
packages/backend-common/src/reading/GitlabUrlReader.test.ts
+149 -42
View File
@@ -23,6 +23,7 @@ import path from 'path';
import { getVoidLogger } from '../logging';
import { GitlabUrlReader } from './GitlabUrlReader';
import { ReadTreeResponseFactory } from './tree';
import { NotModifiedError, NotFoundError } from '../errors';
const logger = getVoidLogger();
@@ -30,6 +31,22 @@ const treeResponseFactory = ReadTreeResponseFactory.create({
config: new ConfigReader({}),
});
const gitlabProcessor = new GitlabUrlReader(
{
host: 'gitlab.com',
apiBaseUrl: 'https://gitlab.com/api/v4',
},
{ treeResponseFactory },
);
const hostedGitlabProcessor = new GitlabUrlReader(
{
host: 'gitlab.mycompany.com',
apiBaseUrl: 'https://gitlab.mycompany.com/api/v4',
},
{ treeResponseFactory },
);
describe('GitlabUrlReader', () => {
const worker = setupServer();
msw.setupDefaultHandlers(worker);
@@ -136,39 +153,102 @@ describe('GitlabUrlReader', () => {
});
describe('readTree', () => {
const repoBuffer = fs.readFileSync(
path.resolve('src', 'reading', '__fixtures__', 'repo.zip'),
const archiveBuffer = fs.readFileSync(
path.resolve('src', 'reading', '__fixtures__', 'gitlab-archive.zip'),
);
const projectGitlabApiResponse = {
id: 11111111,
default_branch: 'main',
};
const branchGitlabApiResponse = {
commit: {
id: 'sha123abc',
},
};
beforeEach(() => {
worker.use(
rest.get(
'https://gitlab.com/backstage/mock/-/archive/repo/mock-repo.zip',
'https://gitlab.com/api/v4/projects/backstage%2Fmock/repository/archive.zip?sha=main',
(_, res, ctx) =>
res(
ctx.status(200),
ctx.set('Content-Type', 'application/zip'),
ctx.body(repoBuffer),
ctx.set(
'content-disposition',
'attachment; filename="mock-main-sha123abc.zip"',
),
ctx.body(archiveBuffer),
),
),
rest.get(
'https://gitlab.com/api/v4/projects/backstage%2Fmock',
(_, res, ctx) =>
res(
ctx.status(200),
ctx.set('Content-Type', 'application/json'),
ctx.json(projectGitlabApiResponse),
),
),
rest.get(
'https://gitlab.com/api/v4/projects/backstage%2Fmock/repository/branches/main',
(_, res, ctx) =>
res(
ctx.status(200),
ctx.set('Content-Type', 'application/json'),
ctx.json(branchGitlabApiResponse),
),
),
rest.get(
'https://gitlab.com/api/v4/projects/backstage%2Fmock/repository/branches/branchDoesNotExist',
(_, res, ctx) => res(ctx.status(404)),
),
rest.get(
'https://gitlab.mycompany.com/api/v4/projects/backstage%2Fmock',
(_, res, ctx) =>
res(
ctx.status(200),
ctx.set('Content-Type', 'application/json'),
ctx.json(projectGitlabApiResponse),
),
),
rest.get(
'https://gitlab.mycompany.com/api/v4/projects/backstage%2Fmock/repository/branches/main',
(_, res, ctx) =>
res(
ctx.status(200),
ctx.set('Content-Type', 'application/json'),
ctx.json(branchGitlabApiResponse),
),
),
rest.get(
'https://gitlab.mycompany.com/api/v4/projects/backstage%2Fmock/repository/archive.zip?sha=main',
(_, res, ctx) =>
res(
ctx.status(200),
ctx.set('Content-Type', 'application/zip'),
ctx.set(
'content-disposition',
'attachment; filename="mock-main-sha123abc.zip"',
),
ctx.body(archiveBuffer),
),
),
);
});
it('returns the wanted files from an archive', async () => {
const processor = new GitlabUrlReader(
{ host: 'gitlab.com' },
{ treeResponseFactory },
);
const response = await processor.readTree(
'https://gitlab.com/backstage/mock/tree/repo',
const response = await gitlabProcessor.readTree(
'https://gitlab.com/backstage/mock/tree/main',
);
const files = await response.files();
expect(files.length).toBe(2);
const indexMarkdownFile = await files[0].content();
const mkDocsFile = await files[1].content();
const mkDocsFile = await files[0].content();
const indexMarkdownFile = await files[1].content();
expect(mkDocsFile.toString()).toBe('site_name: Test\n');
expect(indexMarkdownFile.toString()).toBe('# Test\n');
@@ -177,23 +257,22 @@ describe('GitlabUrlReader', () => {
it('returns the wanted files from hosted gitlab', async () => {
worker.use(
rest.get(
'https://git.mycompany.com/backstage/mock/-/archive/repo/mock-repo.zip',
'https://gitlab.mycompany.com/backstage/mock/-/archive/main.zip',
(_, res, ctx) =>
res(
ctx.status(200),
ctx.set('Content-Type', 'application/zip'),
ctx.body(repoBuffer),
ctx.set(
'content-disposition',
'attachment; filename="mock-main-sha123abc.zip"',
),
ctx.body(archiveBuffer),
),
),
);
const processor = new GitlabUrlReader(
{ host: 'git.mycompany.com' },
{ treeResponseFactory },
);
const response = await processor.readTree(
'https://git.mycompany.com/backstage/mock/tree/repo/docs',
const response = await hostedGitlabProcessor.readTree(
'https://gitlab.mycompany.com/backstage/mock/tree/main/docs',
);
const files = await response.files();
@@ -204,27 +283,9 @@ describe('GitlabUrlReader', () => {
expect(indexMarkdownFile.toString()).toBe('# Test\n');
});
it('throws an error when branch is not specified', async () => {
const processor = new GitlabUrlReader(
{ host: 'gitlab.com' },
{ treeResponseFactory },
);
await expect(
processor.readTree('https://gitlab.com/backstage/mock'),
).rejects.toThrow(
'GitLab URL must contain a branch to be able to fetch its tree',
);
});
it('returns the wanted files from an archive with a subpath', async () => {
const processor = new GitlabUrlReader(
{ host: 'gitlab.com' },
{ treeResponseFactory },
);
const response = await processor.readTree(
'https://gitlab.com/backstage/mock/tree/repo/docs',
const response = await gitlabProcessor.readTree(
'https://gitlab.com/backstage/mock/tree/main/docs',
);
const files = await response.files();
@@ -234,5 +295,51 @@ describe('GitlabUrlReader', () => {
expect(indexMarkdownFile.toString()).toBe('# Test\n');
});
it('throws a NotModifiedError when given a etag in options', async () => {
const fnGitlab = async () => {
await gitlabProcessor.readTree('https://gitlab.com/backstage/mock', {
etag: 'sha123abc',
});
};
const fnHostedGitlab = async () => {
await hostedGitlabProcessor.readTree(
'https://gitlab.mycompany.com/backstage/mock',
{
etag: 'sha123abc',
},
);
};
await expect(fnGitlab).rejects.toThrow(NotModifiedError);
await expect(fnHostedGitlab).rejects.toThrow(NotModifiedError);
});
it('should not throw error when given an outdated etag in options', async () => {
const response = await gitlabProcessor.readTree(
'https://gitlab.com/backstage/mock/tree/main',
{
etag: 'outdatedsha123abc',
},
);
expect((await response.files()).length).toBe(2);
});
it('should detect the default branch', async () => {
const response = await gitlabProcessor.readTree(
'https://gitlab.com/backstage/mock',
);
expect((await response.files()).length).toBe(2);
});
it('should throw error on missing branch', async () => {
const fnGithub = async () => {
await gitlabProcessor.readTree(
'https://gitlab.com/backstage/mock/tree/branchDoesNotExist',
);
};
await expect(fnGithub).rejects.toThrow(NotFoundError);
});
});
});
packages/backend-common/src/reading/GitlabUrlReader.ts
+89 -32
View File
@@ -21,7 +21,7 @@ import {
readGitLabIntegrationConfigs,
} from '@backstage/integration';
import fetch from 'cross-fetch';
import { InputError, NotFoundError } from '../errors';
import { NotFoundError, NotModifiedError } from '../errors';
import { ReadTreeResponseFactory } from './tree';
import {
ReaderFactory,
@@ -39,26 +39,26 @@ export class GitlabUrlReader implements UrlReader {
const configs = readGitLabIntegrationConfigs(
config.getOptionalConfigArray('integrations.gitlab') ?? [],
);
return configs.map(options => {
const reader = new GitlabUrlReader(options, { treeResponseFactory });
const predicate = (url: URL) => url.host === options.host;
return configs.map(provider => {
const reader = new GitlabUrlReader(provider, { treeResponseFactory });
const predicate = (url: URL) => url.host === provider.host;
return { reader, predicate };
});
};
constructor(
private readonly options: GitLabIntegrationConfig,
private readonly config: GitLabIntegrationConfig,
deps: { treeResponseFactory: ReadTreeResponseFactory },
) {
this.treeResponseFactory = deps.treeResponseFactory;
}
async read(url: string): Promise<Buffer> {
const builtUrl = await getGitLabFileFetchUrl(url, this.options);
const builtUrl = await getGitLabFileFetchUrl(url, this.config);
let response: Response;
try {
response = await fetch(builtUrl, getGitLabRequestOptions(this.options));
response = await fetch(builtUrl, getGitLabRequestOptions(this.config));
} catch (e) {
throw new Error(`Unable to read ${url}, ${e}`);
}
@@ -78,45 +78,102 @@ export class GitlabUrlReader implements UrlReader {
url: string,
options?: ReadTreeOptions,
): Promise<ReadTreeResponse> {
const {
name: repoName,
ref,
protocol,
resource,
full_name,
filepath,
} = parseGitUrl(url);
const { ref, full_name, filepath } = parseGitUrl(url);
if (!ref) {
throw new InputError(
'GitLab URL must contain a branch to be able to fetch its tree',
);
}
const archive = `${protocol}://${resource}/${full_name}/-/archive/${ref}/${repoName}-${ref}.zip`;
const response = await fetch(
archive,
getGitLabRequestOptions(this.options),
// Use GitLab API to get the default branch
// encodeURIComponent is required for GitLab API
// https://docs.gitlab.com/ee/api/README.html#namespaced-path-encoding
const projectGitlabResponse = await fetch(
new URL(
`${this.config.apiBaseUrl}/projects/${encodeURIComponent(full_name)}`,
).toString(),
getGitLabRequestOptions(this.config),
);
if (!response.ok) {
const msg = `Failed to read tree from ${url}, ${response.status} ${response.statusText}`;
if (response.status === 404) {
if (!projectGitlabResponse.ok) {
const msg = `Failed to read tree from ${url}, ${projectGitlabResponse.status} ${projectGitlabResponse.statusText}`;
if (projectGitlabResponse.status === 404) {
throw new NotFoundError(msg);
}
throw new Error(msg);
}
const projectGitlabResponseJson = await projectGitlabResponse.json();
const path = filepath ? `${repoName}-${ref}/${filepath}/` : '';
// ref is an empty string if no branch is set in provided url to readTree.
const branch = ref || projectGitlabResponseJson.default_branch;
return this.treeResponseFactory.fromZipArchive({
stream: (response.body as unknown) as Readable,
// Fetch the latest commit in the provided or default branch to compare against
// the provided sha.
const branchGitlabResponse = await fetch(
new URL(
`${this.config.apiBaseUrl}/projects/${encodeURIComponent(
full_name,
)}/repository/branches/${branch}`,
).toString(),
getGitLabRequestOptions(this.config),
);
if (!branchGitlabResponse.ok) {
const message = `Failed to read tree (branch) from ${url}, ${branchGitlabResponse.status} ${branchGitlabResponse.statusText}`;
if (branchGitlabResponse.status === 404) {
throw new NotFoundError(message);
}
throw new Error(message);
}
const commitSha = (await branchGitlabResponse.json()).commit.id;
if (options?.etag && options.etag === commitSha) {
throw new NotModifiedError();
}
// https://docs.gitlab.com/ee/api/repositories.html#get-file-archive
const archiveGitLabResponse = await fetch(
`${this.config.apiBaseUrl}/projects/${encodeURIComponent(
full_name,
)}/repository/archive.zip?sha=${branch}`,
getGitLabRequestOptions(this.config),
);
if (!archiveGitLabResponse.ok) {
const message = `Failed to read tree (archive) from ${url}, ${archiveGitLabResponse.status} ${archiveGitLabResponse.statusText}`;
if (archiveGitLabResponse.status === 404) {
throw new NotFoundError(message);
}
throw new Error(message);
}
// Get the filename of archive from the header of the response
const contentDispositionHeader = archiveGitLabResponse.headers.get(
'content-disposition',
) as string;
if (!contentDispositionHeader) {
throw new Error(
`Failed to read tree from ${url}. ` +
'GitLab API response for downloading archive does not contain content-disposition header ',
);
}
const fileNameRegEx = new RegExp(
/^attachment; filename="(?<fileName>.*).zip"$/,
);
const archiveFileName = contentDispositionHeader.match(fileNameRegEx)
?.groups?.fileName;
if (!archiveFileName) {
throw new Error(
`Failed to read tree from ${url}. GitLab API response for downloading archive has an unexpected ` +
`format of content-disposition header ${contentDispositionHeader} `,
);
}
const path = filepath ? `${archiveFileName}/${filepath}/` : '';
return await this.treeResponseFactory.fromZipArchive({
stream: (archiveGitLabResponse.body as unknown) as Readable,
path,
etag: commitSha,
filter: options?.filter,
});
}
toString() {
const { host, token } = this.options;
const { host, token } = this.config;
return `gitlab{host=${host},authed=${Boolean(token)}}`;
}
}
packages/backend-common/src/reading/UrlReaderPredicateMux.ts
+5 -2
View File
@@ -45,12 +45,15 @@ export class UrlReaderPredicateMux implements UrlReader {
throw new NotAllowedError(`Reading from '${url}' is not allowed`);
}
readTree(url: string, options?: ReadTreeOptions): Promise<ReadTreeResponse> {
async readTree(
url: string,
options?: ReadTreeOptions,
): Promise<ReadTreeResponse> {
const parsed = new URL(url);
for (const { predicate, reader } of this.readers) {
if (predicate(parsed)) {
return reader.readTree(url, options);
return await reader.readTree(url, options);
}
}
packages/backend-common/src/reading/__fixtures__/backstage-mock-etag123.tar.gz
BIN
View File
Binary file not shown.
packages/backend-common/src/reading/__fixtures__/gitlab-archive.zip
BIN
View File
Binary file not shown.
packages/backend-common/src/reading/__fixtures__/mock-main.tar.gz
BIN
View File
Binary file not shown.
packages/backend-common/src/reading/__fixtures__/mock-main.zip
BIN
View File
Binary file not shown.
packages/backend-common/src/reading/__fixtures__/repo.tar.gz
BIN
View File
Binary file not shown.
packages/backend-common/src/reading/__fixtures__/repo.zip
BIN
View File
Binary file not shown.
packages/backend-common/src/reading/tree/ReadTreeResponseFactory.ts
+4
View File
@@ -26,6 +26,8 @@ type FromArchiveOptions = {
stream: Readable;
// If set, the root of the tree will be set to the given directory path.
path?: string;
// etag of the blob
etag: string;
// Filter passed on from the ReadTreeOptions
filter?: (path: string) => boolean;
};
@@ -45,6 +47,7 @@ export class ReadTreeResponseFactory {
options.stream,
options.path ?? '',
this.workDir,
options.etag,
options.filter,
);
}
@@ -54,6 +57,7 @@ export class ReadTreeResponseFactory {
options.stream,
options.path ?? '',
this.workDir,
options.etag,
options.filter,
);
}
packages/backend-common/src/reading/tree/TarArchiveResponse.test.ts
+25 -12
View File
@@ -20,7 +20,7 @@ import { resolve as resolvePath } from 'path';
import { TarArchiveResponse } from './TarArchiveResponse';
const archiveData = fs.readFileSync(
resolvePath(__filename, '../../__fixtures__/repo.tar.gz'),
resolvePath(__filename, '../../__fixtures__/mock-main.tar.gz'),
);
describe('TarArchiveResponse', () => {
@@ -38,7 +38,7 @@ describe('TarArchiveResponse', () => {
it('should read files', async () => {
const stream = fs.createReadStream('/test-archive.tar.gz');
const res = new TarArchiveResponse(stream, 'mock-repo/', '/tmp');
const res = new TarArchiveResponse(stream, 'mock-main/', '/tmp', 'etag');
const files = await res.files();
expect(files).toEqual([
@@ -61,8 +61,12 @@ describe('TarArchiveResponse', () => {
it('should read files with filter', async () => {
const stream = fs.createReadStream('/test-archive.tar.gz');
const res = new TarArchiveResponse(stream, 'mock-repo/', '/tmp', path =>
path.endsWith('.yml'),
const res = new TarArchiveResponse(
stream,
'mock-main/',
'/tmp',
'etag',
path => path.endsWith('.yml'),
);
const files = await res.files();
@@ -79,14 +83,14 @@ describe('TarArchiveResponse', () => {
it('should read as archive and files', async () => {
const stream = fs.createReadStream('/test-archive.tar.gz');
const res = new TarArchiveResponse(stream, 'mock-repo/', '/tmp');
const res = new TarArchiveResponse(stream, 'mock-main/', '/tmp', 'etag');
const buffer = await res.archive();
await expect(res.archive()).rejects.toThrow(
'Response has already been read',
);
const res2 = new TarArchiveResponse(buffer, '', '/tmp');
const res2 = new TarArchiveResponse(buffer, '', '/tmp', 'etag');
const files = await res2.files();
expect(files).toEqual([
@@ -109,21 +113,26 @@ describe('TarArchiveResponse', () => {
it('should extract entire archive into directory', async () => {
const stream = fs.createReadStream('/test-archive.tar.gz');
const res = new TarArchiveResponse(stream, '', '/tmp');
const res = new TarArchiveResponse(stream, '', '/tmp', 'etag');
const dir = await res.dir();
await expect(
fs.readFile(resolvePath(dir, 'mock-repo/mkdocs.yml'), 'utf8'),
fs.readFile(resolvePath(dir, 'mock-main/mkdocs.yml'), 'utf8'),
).resolves.toBe('site_name: Test\n');
await expect(
fs.readFile(resolvePath(dir, 'mock-repo/docs/index.md'), 'utf8'),
fs.readFile(resolvePath(dir, 'mock-main/docs/index.md'), 'utf8'),
).resolves.toBe('# Test\n');
});
it('should extract archive into directory with a subpath', async () => {
const stream = fs.createReadStream('/test-archive.tar.gz');
const res = new TarArchiveResponse(stream, 'mock-repo/docs/', '/tmp');
const res = new TarArchiveResponse(
stream,
'mock-main/docs/',
'/tmp',
'etag',
);
const dir = await res.dir();
expect(dir).toMatch(/^[\/\\]tmp[\/\\].*$/);
@@ -135,8 +144,12 @@ describe('TarArchiveResponse', () => {
it('should extract archive into directory with a subpath and filter', async () => {
const stream = fs.createReadStream('/test-archive.tar.gz');
const res = new TarArchiveResponse(stream, 'mock-repo/', '/tmp', path =>
path.endsWith('.yml'),
const res = new TarArchiveResponse(
stream,
'mock-main/',
'/tmp',
'etag',
path => path.endsWith('.yml'),
);
const dir = await res.dir({ targetDir: '/tmp' });
packages/backend-common/src/reading/tree/TarArchiveResponse.ts
+3
View File
@@ -41,6 +41,7 @@ export class TarArchiveResponse implements ReadTreeResponse {
private readonly stream: Readable,
private readonly subPath: string,
private readonly workDir: string,
public readonly etag: string,
private readonly filter?: (path: string) => boolean,
) {
if (subPath) {
@@ -53,6 +54,8 @@ export class TarArchiveResponse implements ReadTreeResponse {
);
}
}
this.etag = etag;
}
// Make sure the input stream is only read once
packages/backend-common/src/reading/tree/ZipArchiveResponse.test.ts
+31 -18
View File
@@ -20,7 +20,7 @@ import { resolve as resolvePath } from 'path';
import { ZipArchiveResponse } from './ZipArchiveResponse';
const archiveData = fs.readFileSync(
resolvePath(__filename, '../../__fixtures__/repo.zip'),
resolvePath(__filename, '../../__fixtures__/mock-main.zip'),
);
describe('ZipArchiveResponse', () => {
@@ -38,31 +38,35 @@ describe('ZipArchiveResponse', () => {
it('should read files', async () => {
const stream = fs.createReadStream('/test-archive.zip');
const res = new ZipArchiveResponse(stream, 'mock-repo/', '/tmp');
const res = new ZipArchiveResponse(stream, 'mock-main/', '/tmp', 'etag');
const files = await res.files();
expect(files).toEqual([
{
path: 'docs/index.md',
path: 'mkdocs.yml',
content: expect.any(Function),
},
{
path: 'mkdocs.yml',
path: 'docs/index.md',
content: expect.any(Function),
},
]);
const contents = await Promise.all(files.map(f => f.content()));
expect(contents.map(c => c.toString('utf8').trim())).toEqual([
'# Test',
'site_name: Test',
'# Test',
]);
});
it('should read files with filter', async () => {
const stream = fs.createReadStream('/test-archive.zip');
const res = new ZipArchiveResponse(stream, 'mock-repo/', '/tmp', path =>
path.endsWith('.yml'),
const res = new ZipArchiveResponse(
stream,
'mock-main/',
'/tmp',
'etag',
path => path.endsWith('.yml'),
);
const files = await res.files();
@@ -79,51 +83,56 @@ describe('ZipArchiveResponse', () => {
it('should read as archive and files', async () => {
const stream = fs.createReadStream('/test-archive.zip');
const res = new ZipArchiveResponse(stream, 'mock-repo/', '/tmp');
const res = new ZipArchiveResponse(stream, 'mock-main/', '/tmp', 'etag');
const buffer = await res.archive();
await expect(res.archive()).rejects.toThrow(
'Response has already been read',
);
const res2 = new ZipArchiveResponse(buffer, '', '/tmp');
const res2 = new ZipArchiveResponse(buffer, '', '/tmp', 'etag');
const files = await res2.files();
expect(files).toEqual([
{
path: 'docs/index.md',
path: 'mkdocs.yml',
content: expect.any(Function),
},
{
path: 'mkdocs.yml',
path: 'docs/index.md',
content: expect.any(Function),
},
]);
const contents = await Promise.all(files.map(f => f.content()));
expect(contents.map(c => c.toString('utf8').trim())).toEqual([
'# Test',
'site_name: Test',
'# Test',
]);
});
it('should extract entire archive into directory', async () => {
const stream = fs.createReadStream('/test-archive.zip');
const res = new ZipArchiveResponse(stream, '', '/tmp');
const res = new ZipArchiveResponse(stream, '', '/tmp', 'etag');
const dir = await res.dir();
await expect(
fs.readFile(resolvePath(dir, 'mock-repo/mkdocs.yml'), 'utf8'),
fs.readFile(resolvePath(dir, 'mock-main/mkdocs.yml'), 'utf8'),
).resolves.toBe('site_name: Test\n');
await expect(
fs.readFile(resolvePath(dir, 'mock-repo/docs/index.md'), 'utf8'),
fs.readFile(resolvePath(dir, 'mock-main/docs/index.md'), 'utf8'),
).resolves.toBe('# Test\n');
});
it('should extract archive into directory with a subpath', async () => {
const stream = fs.createReadStream('/test-archive.zip');
const res = new ZipArchiveResponse(stream, 'mock-repo/docs/', '/tmp');
const res = new ZipArchiveResponse(
stream,
'mock-main/docs/',
'/tmp',
'etag',
);
const dir = await res.dir();
expect(dir).toMatch(/^[\/\\]tmp[\/\\].*$/);
@@ -135,8 +144,12 @@ describe('ZipArchiveResponse', () => {
it('should extract archive into directory with a subpath and filter', async () => {
const stream = fs.createReadStream('/test-archive.zip');
const res = new ZipArchiveResponse(stream, 'mock-repo/', '/tmp', path =>
path.endsWith('.yml'),
const res = new ZipArchiveResponse(
stream,
'mock-main/',
'/tmp',
'etag',
path => path.endsWith('.yml'),
);
const dir = await res.dir({ targetDir: '/tmp' });
packages/backend-common/src/reading/tree/ZipArchiveResponse.ts
+3
View File
@@ -35,6 +35,7 @@ export class ZipArchiveResponse implements ReadTreeResponse {
private readonly stream: Readable,
private readonly subPath: string,
private readonly workDir: string,
public readonly etag: string,
private readonly filter?: (path: string) => boolean,
) {
if (subPath) {
@@ -47,6 +48,8 @@ export class ZipArchiveResponse implements ReadTreeResponse {
);
}
}
this.etag = etag;
}
// Make sure the input stream is only read once
packages/backend-common/src/reading/types.ts
+22
View File
@@ -32,6 +32,19 @@ export type ReadTreeOptions = {
* If no filter is provided all files are extracted.
*/
filter?(path: string): boolean;
/**
* An etag can be provided to check whether readTree's response has changed from a previous execution.
*
* In the readTree() response, an etag is returned along with the tree blob. The etag is a unique identifer
* of the tree blob, usually the commit SHA or etag from the target.
*
* When a etag is given in ReadTreeOptions, readTree will first compare the etag against the etag
* on the target branch. If they match, readTree will throw a NotModifiedError indicating that the readTree
* response will not differ from the previous response which included this particular etag. If they mismatch,
* readTree will return the rest of ReadTreeResponse along with a new etag.
*/
etag?: string;
};
/**
@@ -70,5 +83,14 @@ export type ReadTreeResponseDirOptions = {
export type ReadTreeResponse = {
files(): Promise<ReadTreeResponseFile[]>;
archive(): Promise<NodeJS.ReadableStream>;
/**
* dir() extracts the tree response into a directory and returns the path of the directory.
*/
dir(options?: ReadTreeResponseDirOptions): Promise<string>;
/**
* A unique identifer of the tree blob, usually the commit SHA or etag from the target.
*/
etag: string;
};
packages/backend/CHANGELOG.md
+34
View File
@@ -1,5 +1,39 @@
# example-backend
## 0.2.12
### Patch Changes
- Updated dependencies [def2307f3]
- Updated dependencies [d54857099]
- Updated dependencies [0b135e7e0]
- Updated dependencies [318a6af9f]
- Updated dependencies [294a70cab]
- Updated dependencies [ac7be581a]
- Updated dependencies [0ea032763]
- Updated dependencies [5345a1f98]
- Updated dependencies [ed6baab66]
- Updated dependencies [ad838c02f]
- Updated dependencies [a5e27d5c1]
- Updated dependencies [0643a3336]
- Updated dependencies [a2291d7cc]
- Updated dependencies [f9ba00a1c]
- Updated dependencies [09a370426]
- Updated dependencies [a93f42213]
- @backstage/catalog-model@0.7.0
- @backstage/plugin-catalog-backend@0.5.4
- @backstage/plugin-kubernetes-backend@0.2.5
- @backstage/backend-common@0.5.0
- @backstage/plugin-scaffolder-backend@0.5.0
- @backstage/plugin-techdocs-backend@0.5.4
- @backstage/plugin-auth-backend@0.2.11
- example-app@0.2.12
- @backstage/plugin-kafka-backend@0.1.1
- @backstage/plugin-app-backend@0.3.4
- @backstage/plugin-graphql-backend@0.1.5
- @backstage/plugin-proxy-backend@0.2.4
- @backstage/plugin-rollbar-backend@0.1.7
## 0.2.11
### Patch Changes
packages/backend/package.json
+15 -14
View File
@@ -1,6 +1,6 @@
{
"name": "example-backend",
"version": "0.2.11",
"version": "0.2.12",
"main": "dist/index.cjs.js",
"types": "src/index.ts",
"license": "Apache-2.0",
@@ -27,23 +27,24 @@
"migrate:create": "knex migrate:make -x ts"
},
"dependencies": {
"@backstage/backend-common": "^0.4.3",
"@backstage/catalog-model": "^0.6.1",
"@backstage/backend-common": "^0.5.0",
"@backstage/catalog-model": "^0.7.0",
"@backstage/config": "^0.1.2",
"@backstage/plugin-app-backend": "^0.3.3",
"@backstage/plugin-auth-backend": "^0.2.10",
"@backstage/plugin-catalog-backend": "^0.5.3",
"@backstage/plugin-graphql-backend": "^0.1.4",
"@backstage/plugin-kubernetes-backend": "^0.2.4",
"@backstage/plugin-proxy-backend": "^0.2.3",
"@backstage/plugin-rollbar-backend": "^0.1.5",
"@backstage/plugin-scaffolder-backend": "^0.4.1",
"@backstage/plugin-techdocs-backend": "^0.5.3",
"@backstage/plugin-app-backend": "^0.3.4",
"@backstage/plugin-auth-backend": "^0.2.11",
"@backstage/plugin-catalog-backend": "^0.5.4",
"@backstage/plugin-graphql-backend": "^0.1.5",
"@backstage/plugin-kubernetes-backend": "^0.2.5",
"@backstage/plugin-kafka-backend": "^0.1.1",
"@backstage/plugin-proxy-backend": "^0.2.4",
"@backstage/plugin-rollbar-backend": "^0.1.7",
"@backstage/plugin-scaffolder-backend": "^0.5.0",
"@backstage/plugin-techdocs-backend": "^0.5.4",
"@gitbeaker/node": "^28.0.2",
"@octokit/rest": "^18.0.12",
"azure-devops-node-api": "^10.1.1",
"dockerode": "^3.2.1",
"example-app": "^0.2.8",
"example-app": "^0.2.12",
"express": "^4.17.1",
"express-promise-router": "^3.0.3",
"knex": "^0.21.6",
@@ -53,7 +54,7 @@
"winston": "^3.2.1"
},
"devDependencies": {
"@backstage/cli": "^0.4.6",
"@backstage/cli": "^0.4.7",
"@types/dockerode": "^3.2.1",
"@types/express": "^4.17.6",
"@types/express-serve-static-core": "^4.17.5"
packages/backend/src/index.ts
+3
View File
@@ -38,6 +38,7 @@ import healthcheck from './plugins/healthcheck';
import auth from './plugins/auth';
import catalog from './plugins/catalog';
import kubernetes from './plugins/kubernetes';
import kafka from './plugins/kafka';
import rollbar from './plugins/rollbar';
import scaffolder from './plugins/scaffolder';
import proxy from './plugins/proxy';
@@ -77,6 +78,7 @@ async function main() {
const rollbarEnv = useHotMemoize(module, () => createEnv('rollbar'));
const techdocsEnv = useHotMemoize(module, () => createEnv('techdocs'));
const kubernetesEnv = useHotMemoize(module, () => createEnv('kubernetes'));
const kafkaEnv = useHotMemoize(module, () => createEnv('kafka'));
const graphqlEnv = useHotMemoize(module, () => createEnv('graphql'));
const appEnv = useHotMemoize(module, () => createEnv('app'));
@@ -87,6 +89,7 @@ async function main() {
apiRouter.use('/auth', await auth(authEnv));
apiRouter.use('/techdocs', await techdocs(techdocsEnv));
apiRouter.use('/kubernetes', await kubernetes(kubernetesEnv));
apiRouter.use('/kafka', await kafka(kafkaEnv));
apiRouter.use('/proxy', await proxy(proxyEnv));
apiRouter.use('/graphql', await graphql(graphqlEnv));
apiRouter.use(notFoundHandler());
packages/backend/src/plugins/kafka.ts
+25
View File
@@ -0,0 +1,25 @@
/*
* Copyright 2020 Spotify AB
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { createRouter } from '@backstage/plugin-kafka-backend';
import { PluginEnvironment } from '../types';
export default async function createPlugin({
logger,
config,
}: PluginEnvironment) {
return await createRouter({ logger, config });
}
packages/backend/src/plugins/scaffolder.ts
+1
View File
@@ -34,6 +34,7 @@ export default async function createPlugin({
const cookiecutterTemplater = new CookieCutter();
const craTemplater = new CreateReactAppTemplater();
const templaters = new Templaters();
templaters.register('cookiecutter', cookiecutterTemplater);
templaters.register('cra', craTemplater);
packages/catalog-client/CHANGELOG.md
+8
View File
@@ -1,5 +1,13 @@
# @backstage/catalog-client
## 0.3.5
### Patch Changes
- Updated dependencies [def2307f3]
- Updated dependencies [a93f42213]
- @backstage/catalog-model@0.7.0
## 0.3.4
### Patch Changes
packages/catalog-client/package.json
+3 -3
View File
@@ -1,6 +1,6 @@
{
"name": "@backstage/catalog-client",
"version": "0.3.4",
"version": "0.3.5",
"main": "src/index.ts",
"types": "src/index.ts",
"license": "Apache-2.0",
@@ -29,12 +29,12 @@
"clean": "backstage-cli clean"
},
"dependencies": {
"@backstage/catalog-model": "^0.6.0",
"@backstage/catalog-model": "^0.7.0",
"@backstage/config": "^0.1.2",
"cross-fetch": "^3.0.6"
},
"devDependencies": {
"@backstage/cli": "^0.4.2",
"@backstage/cli": "^0.4.7",
"@types/jest": "^26.0.7",
"msw": "^0.21.2"
},
packages/catalog-model/CHANGELOG.md
+33
View File
@@ -1,5 +1,38 @@
# @backstage/catalog-model
## 0.7.0
### Minor Changes
- a93f42213: The catalog no longer attempts to merge old and new annotations, when updating an entity from a remote location. This was a behavior that was copied from kubernetes, and catered to use cases where you wanted to use HTTP POST to update an entity in-place, outside of what the refresh loop does. This has proved to be a mistake, because as a side effect, the refresh loop effectively is unable to ever delete annotations when they are removed from source YAML. This is obviously a breaking change, but we believe that this is not a behavior that is relied upon in the wild, and it has never been an actually supported use flow of the catalog. We therefore choose to break the behavior outright, and instead just store updated annotations verbatim - just like we already do for example for labels
### Patch Changes
- def2307f3: Adds a `backstage.io/managed-by-origin-location` annotation to all entities. It links to the
location that was registered to the catalog and which emitted this entity. It has a different
semantic than the existing `backstage.io/managed-by-location` annotation, which tells the direct
parent location that created this entity.
Consider this example: The Backstage operator adds a location of type `github-org` in the
`app-config.yaml`. This setting will be added to a `bootstrap:boostrap` location. The processor
discovers the entities in the following branch
`Location bootstrap:bootstrap -> Location github-org:… -> User xyz`. The user `xyz` will be:
```yaml
apiVersion: backstage.io/v1alpha1
kind: User
metadata:
name: xyz
annotations:
# This entity was added by the 'github-org:…' location
backstage.io/managed-by-location: github-org:…
# The entity was added because the 'bootstrap:boostrap' was added to the catalog
backstage.io/managed-by-origin-location: bootstrap:bootstrap
# ...
spec:
# ...
```
## 0.6.1
### Patch Changes
packages/catalog-model/examples/all-components.yaml
+2
View File
@@ -15,4 +15,6 @@ spec:
- ./components/www-artist-component.yaml
- ./components/shuffle-api-component.yaml
- ./components/wayback-archive-component.yaml
- ./components/wayback-archive-ingestion-component.yaml
- ./components/wayback-archive-storage-component.yaml
- ./components/wayback-search-component.yaml
packages/catalog-model/examples/components/wayback-archive-ingestion-component.yaml
+10
View File
@@ -0,0 +1,10 @@
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
name: wayback-archive-ingestion
description: Ingestion subsystem of the Wayback Archive
spec:
type: service
lifecycle: production
owner: team-d
subcomponentOf: wayback-archive

Some files were not shown because too many files have changed in this diff Show More