Merge pull request #2665 from spotify/rugvip/reading

backend-common: add new common UrlReader to use for reading remote data

docs/features/software-catalog/configuration.md

@@ -11,36 +11,56 @@ tasks, such as reading raw entity data from a remote source, parsing it,
 transforming it, and validating it. These processors are configured under the
 `catalog.processors` key.
 
-### Processor: github
+### Processor: url
 
-The `github` processor is responsible for fetching entity data from files on
-GitHub or GitHub Enterprise. The configuration for this processor lives under
-`catalog.processors.github`. Example:
+The `url` processor is responsible for fetching entity data from files in any
+external provider like GitHub, GitLab, Bitbucket, etc. The configuration of this
+processor lives under the top-level `integrations` key, as it is used by other
+parts of Backstage too.
 
 ```yaml
-catalog:
-  processors:
-    github:
-      providers:
-        - target: https://github.com
-          token:
-            $secret:
-              env: GITHUB_PRIVATE_TOKEN
-        - target: https://ghe.example.net
-          apiBaseUrl: https://ghe.example.net/api/v3
-          rawBaseUrl: https://ghe.example.net/raw
-          token:
-            $secret:
-              env: GHE_PRIVATE_TOKEN
+integrations:
+  github:
+    - host: github.com
+      token:
+        $secret:
+          env: GITHUB_TOKEN
+    - host: ghe.example.net
+      apiBaseUrl: https://ghe.example.net/api/v3
+      rawBaseUrl: https://ghe.example.net/raw
+      token:
+        $secret:
+          env: GHE_TOKEN
+  gitlab:
+    - host: gitlab.com
+      token:
+        $secret:
+          env: GITLAB_TOKEN
+  bitbucket:
+    - host: bitbucket.org
+      username:
+        $secret:
+          env: BITBUCKET_USERNAME
+      appPassword:
+        $secret:
+          env: BITBUCKET_APP_PASSWORD
+  azure:
+    - host: dev.azure.com
+      token:
+        $secret:
+          env: AZURE_TOKEN
 ```
 
-The main subkey is `providers`, where you can list the various GitHub compatible
-providers you want to be able to fetch data from. Each entry is a structure with
-up to four elements:
+Each key under `integrations` is a separate configuration for each external
+provider. The providers each have their own configuration, so let's look at the
+GitHub section as an example.
 
-- `target` (required): The string prefix of the location target that you want to
-  match on, with no trailing slash. For GitHub, it should be exactly
-  `https://github.com`.
+Directly under the `github` key is a list of provider configurations, where you
+can list the various GitHub compatible providers you want to be able to fetch
+data from. Each entry is a structure with up to four elements:
+
+- `host` (optional): The host of the location target that you want to match on.
+  The default host is `github.com`.
 - `token` (optional): An authentication token as expected by GitHub. If
   supplied, it will be passed along with all calls to this provider, both API
   and raw. If it is not supplied, anonymous access will be used.
@@ -74,7 +94,7 @@ the catalog under the `catalog.locations` key, for example:
 ```yaml
 catalog:
   locations:
-    - type: github
+    - type: url
       target: https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/artist-lookup-component.yaml
 ```
 
@@ -97,7 +117,7 @@ catalog:
     - allow: [Component, API, Location, Template]
 
   locations:
-    - type: github
+    - type: url
       target: https://github.com/org/example/blob/master/org-data.yaml
       rules:
         - allow: [Group]

docs/features/software-catalog/index.md

@@ -90,7 +90,7 @@ above example can be added using the following configuration:
 ```yaml
 catalog:
   locations:
-    - type: github
+    - type: url
       target: https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/artist-lookup-component.yaml
 ```
 

docs/features/software-catalog/installation.md

@@ -166,21 +166,21 @@ our example templates through static configuration. Add the following to the
 catalog:
   locations:
     # Backstage Example Component
-    - type: github
+    - type: url
       target: https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/artist-lookup-component.yaml
-    - type: github
+    - type: url
       target: https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/playback-order-component.yaml
-    - type: github
+    - type: url
       target: https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/podcast-api-component.yaml
-    - type: github
+    - type: url
       target: https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/queue-proxy-component.yaml
-    - type: github
+    - type: url
       target: https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/searcher-component.yaml
-    - type: github
+    - type: url
       target: https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/playback-lib-component.yaml
-    - type: github
+    - type: url
       target: https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/www-artist-component.yaml
-    - type: github
+    - type: url
       target: https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/shuffle-api-component.yaml
 ```
 

docs/features/software-templates/adding-templates.md

@@ -66,7 +66,7 @@ for example
 ```yaml
 catalog:
   locations:
-    - type: github
+    - type: url
       target: https://github.com/spotify/cookiecutter-golang/blob/master/template.yaml
       rules:
         - allow: [Template]

docs/features/software-templates/installation.md

@@ -191,13 +191,13 @@ our example templates through static configuration. Add the following to the
 catalog:
   locations:
     # Backstage Example Templates
-    - type: github
+    - type: url
       target: https://github.com/spotify/backstage/blob/master/plugins/scaffolder-backend/sample-templates/react-ssr-template/template.yaml
-    - type: github
+    - type: url
       target: https://github.com/spotify/backstage/blob/master/plugins/scaffolder-backend/sample-templates/springboot-grpc-template/template.yaml
-    - type: github
+    - type: url
       target: https://github.com/spotify/backstage/blob/master/plugins/scaffolder-backend/sample-templates/create-react-app/template.yaml
-    - type: github
+    - type: url
       target: https://github.com/spotify/cookiecutter-golang/blob/master/template.yaml
 ```