Consolidate url reader docs

Signed-off-by: Johan Haals <johan.haals@gmail.com>
2024-08-27 14:30:01 +02:00
parent 6853aac3e0
commit 1d06e061ab
2 changed files with 96 additions and 306 deletions
@@ -45,3 +45,99 @@ createBackendPlugin({
  },
 });
 ```
+
+## Providing custom URL readers
+
+Custom URL readers that are not intended for open sourcing via a custom service factory for that specific reader. The following example shows how to create a custom URL reader and provide it to the backend.
+
+```ts
+// File: packages/backend/src/index.ts
+import { urlReaderFactoriesServiceRef } from '@backstage/backend-defaults/urlReader';
+import {
+  createBackendFeatureLoader,
+  createServiceFactory,
+} from '@backstage/backend-plugin-api';
+
+const customReader = createServiceFactory({
+  service: urlReaderFactoriesServiceRef,
+  deps: {},
+  async factory() {
+    return CustomUrlReader.factory;
+  },
+});
+
+const backend = createBackend();
+// backend.add() of other plugins and modules excluded
+backend.add(customReader);
+```
+
+## Writing URL Readers
+
+We want to make sure all URL Readers behave in the same way. Hence if possible,
+all the methods of the `UrlReaderService` interface should be implemented. However it
+is okay to start by implementing just one of them and create issues for the
+remaining.
+
+You can choose to make new URL Readers open source if the use case is beneficial to other users. Either as its own package or by updating the
+[`default` factory](https://github.com/backstage/backstage/blob/ce2ca68f07ad3334401d3277b989bf145b728a64/packages/backend-defaults/src/entrypoints/urlReader/lib/UrlReaders.ts#L82-L102)
+method of URL Readers. It's recommended to create an issue in the Backstage repository to discuss the use case and get feedback before starting the implementation of a new core URL Reader.
+
+Here are some general guidelines for writing URL Readers
+
+#### `readUrl`
+
+`readUrl` method expects a user-friendly URL, something which can be copied from
+the browser naturally when a person is browsing the provider in their browser.
+
+- ✅ Valid URL :
+  `https://github.com/backstage/backstage/blob/master/ADOPTERS.md`
+- ❌ Not a valid URL :
+  `https://raw.githubusercontent.com/backstage/backstage/master/ADOPTERS.md`
+- ❌ Not a valid URL : `https://github.com/backstage/backstage/ADOPTERS.md`
+
+Upon receiving the URL, `readUrl` converts the user-friendly URL into an API URL
+which can be used to request the provider's API.
+
+`readUrl` then makes an authenticated request to the provider API and returns the response containing the file's contents and ETag(if the provider supports it).
+
+#### `readTree`
+
+`readTree` method also expects user-friendly URLs similar to `read` but the URL
+should point to a tree (could be the root of a repository or even a
+sub-directory).
+
+- ✅ Valid URL : `https://github.com/backstage/backstage`
+- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master`
+- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master/docs`
+
+Using the provider's API documentation, find out an API endpoint which can be
+used to download either a zip or a tarball. You can download the entire tree
+(e.g. a repository) and filter out in case the user is expecting only a
+sub-tree. But some APIs are smart enough to accept a path and return only a
+sub-tree in the downloaded archive.
+
+#### search
+
+`search` method expects a glob pattern of a URL and returns a list of files
+matching the query.
+
+- ✅ Valid URL :
+  `https://github.com/backstage/backstage/blob/master/**/catalog-info.yaml`
+- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master/**/*.md`
+- ✅ Valid URL :
+  `https://github.com/backstage/backstage/blob/master/*/package.json`
+- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master/READM`
+
+The core logic of `readTree` can be used here to extract all the files inside
+the tree and return the files matching the pattern in the `url`.
+
+### Caching
+
+All of the methods above support an ETag based caching. If the method is called
+without an `etag`, the response contains an ETag of the resource (should ideally
+forward the ETag returned by the provider). If the method is called with an
+`etag`, it first compares the ETag and returns a `NotModifiedError` in case the
+resource has not been modified. This approach is very similar to the actual
+[`ETag`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag) and
+[`If-None-Match`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-None-Match)
+HTTP headers.