From f86171027ecf903e65f012825267121f6b8f8b76 Mon Sep 17 00:00:00 2001 From: Eric Peterson Date: Tue, 18 May 2021 14:54:01 +0200 Subject: [PATCH 1/6] WIP documentation for the V1 Search Milestone Signed-off-by: Eric Peterson --- docs/features/search/concepts.md | 88 ++++++++++++++ docs/features/search/getting-started.md | 155 ++++++++++++++++++++++++ 2 files changed, 243 insertions(+) create mode 100644 docs/features/search/concepts.md create mode 100644 docs/features/search/getting-started.md diff --git a/docs/features/search/concepts.md b/docs/features/search/concepts.md new file mode 100644 index 0000000000..6a927d73c0 --- /dev/null +++ b/docs/features/search/concepts.md @@ -0,0 +1,88 @@ +--- +id: concepts +title: Search Concepts +description: Documentation on Backstage Search Concepts +--- + +# Search Concepts + +Backstage Search is _blah_. + +To get started, you should get familiar with these core concepts: + +- Search Engines +- Query Translators +- Documents and Indices +- Collators +- Decorators +- The Scheduler + +### Search Engines + +Backstage Search isn't a search engine itself, rather, it provides an interface +between your Backstage instance and a Search Engine of your choice. More +concretely, a `SearchEngine` is an interface whose concrete implementations +facilitate communication with different search engines (like ElasticSearch, +Solr, Algolia, etc). This abstraction exists in order to support your +organization's needs. + +Out of the box, Backstage Search comes pre-packaged with an in-memory search +engine implementation built on top of Lunr. + +### Query Translators + +Because you can bring your own search engine, and because search engines have +very unique and robust query languages themselves, there needs to be a +translation layer between an abstract search query (containing search terms, +filters, and document types) into a concrete search query that is specific to a +search engine. + +Search Engines come pre-packaged with simple translators that do rudimentary +transformations of search terms and filters, but you may want to provide your +own to help tune seearch results in the context of your organization. + +### Documents and Indices + +"Document" is an abstract concept representing something that can be found by +searching for it. A document can represent a software entity, a TechDocs page, +etc. Documents are made up of metadata fields, at a minimum including a title, +text, and location (as in a URL). + +An index is a collection of such documents of a given type. + +### Collators + +You need to be able to search something! Collators are the way to define what +can be searched. Specifically, they're classes which return documents conforming +to a minimum set of fields (including a document title, location, and text), but +which can contain any other fields as defined by the collator itself. One +collator is responsible for defining and collecting documents of a type. + +Some plugins, like the Catalog Backend, provide so-called "default" collators +which you can use out-of-the-box to start searching across Backstage quickly. + +### Decorators + +Sometimes you want to add extra information to a set of documents in your search +index that the collator may not be aware of. For example, the Software Catalog +knows about software entities, but it may not know about their usage or quality. + +Decorators are classes which can add extra fields to pre-collated documents. +This extra metadata could then be used to bias search results or otherwise +improve the search experience in your Backstage instance. + +### The Scheduler + +There are many ways a search index could be built and maintained, but Backstage +Search chooses to completely rebuild indices on a schedule. Different collators +can be configured to refresh at different intervals, depending on how often the +source information is updated. + +TODO: There should probably be some front-end concepts here too? + +- Search Page +- Search Components +- Search Context + +[Primarily solves for “As an App Integrator, I should know how to add do uments +to the search index”] diff --git a/docs/features/search/getting-started.md b/docs/features/search/getting-started.md new file mode 100644 index 0000000000..867b71c8d4 --- /dev/null +++ b/docs/features/search/getting-started.md @@ -0,0 +1,155 @@ +--- +id: getting-started +title: Getting Started with Search +description: How to set up and install Backstage Search +--- + +# Getting Started + +Search functions as a plugin to Backstage, so you will need to use Backstage to +use Search. + +If you haven't setup Backstage already, start +[here](../../getting-started/index.md). + +> If you used `npx @backstage/create-app`, Search may already be present. +> +> You should skip to [`Customizing Search`](#customizing-search) below. + +## Adding Search to the Frontend + +```bash +# From your Backstage root directory +cd packages/app +yarn add @backstage/plugin-search +``` + +TODO: Add frontend instructions here. + +## Adding Search to the Backend + +Add the following plugins into your backend app: + +```bash +# From your Backstage root directory +cd packages/backend +yarn add @backstage/plugin-search-backend @backstage/plugin-search-backend-node +``` + +Create a `packages/backend/src/plugins/search.ts` file containing the following +code: + +```typescript +import { useHotCleanup } from '@backstage/backend-common'; +import { createRouter } from '@backstage/plugin-search-backend'; +import { + IndexBuilder, + LunrSearchEngine, +} from '@backstage/plugin-search-backend-node'; +import { PluginEnvironment } from '../types'; +import { DefaultCatalogCollator } from '@backstage/plugin-catalog-backend'; + +export default async function createPlugin({ + logger, + discovery, +}: PluginEnvironment) { + const searchEngine = new LunrSearchEngine({ logger }); + const indexBuilder = new IndexBuilder({ logger, searchEngine }); + + indexBuilder.addCollator({ + type: 'software-catalog', + defaultRefreshIntervalSeconds: 600, + collator: new DefaultCatalogCollator({ discovery }), + }); + + const { scheduler } = await indexBuilder.build(); + + scheduler.start(); + useHotCleanup(module, () => scheduler.stop()); + + return await createRouter({ + engine: indexBuilder.getSearchEngine(), + logger, + }); +} +``` + +Make the following modifications to your `packages/backend/src/index.ts` file: + +Import the `plugins/search` file you created above: + +```typescript +import search from './plugins/search'; +``` + +Set up an environment for search: + +```typescript +const searchEnv = useHotMemoize(module, () => createEnv('search')); +``` + +Register the search service with the router: + +```typescript +apiRouter.use('/search', await search(searchEnv)); +``` + +## Customizing Search + +Backstage Search isn't a search engine itself, rather, it provides an interface +between your Backstage instance and a Search Engine of your choice. Currently, +we only support one, in-memory search Engine called Lunr. It can be instantiated +like this: + +```typescript +const searchEngine = new LunrSearchEngine({ logger }); +const indexBuilder = new IndexBuilder({ logger, searchEngine }); +``` + +Backstage Search can be used to power search of anything! Plugins like the +Catalog offer default "collators" which are responsible for providing documents +to be indexed. You can register any number of collators with the `IndexBuilder` +like this: + +```typescript +const indexBuilder = new IndexBuilder({ logger, searchEngine }); + +indexBuilder.addCollator({ + type: 'software-catalog', + defaultRefreshIntervalSeconds: 600, + collator: new DefaultCatalogCollator({ discovery }), +}); + +indexBuilder.addCollator({ + type: 'my-custom-stuff', + defaultRefreshIntervalSeconds: 3600, + collator: new MyCustomCollator(), +}); +``` + +Backstage Search builds and maintains its index on a schedule. You can change +how often the indexes are rebuilt for a given type of document. You may want to +do this if your documents are updated more or less frequently. You can do so by +modifying its `defaultRefreshIntervalSeconds` value, like this: + +```typescript {3} +indexBuilder.addCollator({ + type: 'software-catalog', + defaultRefreshIntervalSeconds: 600, + collator: new DefaultCatalogCollator({ discovery }), +}); +``` + +--- + +[Primarily solves for “As an App Integrator, I should be able to install out-of +the-box Search”] + +Backend: Different for new Backstage instances (created via create-app) vs. +people enabling search on existing Backstage instances (see catalog install for +example). + +Frontend: Search Page Route…? Customizing result components…? + +Conceptually… What’s the difference between “out-of-the-box” or “basic” and the +recommended deployment (which will come) From 62fcca6b9eb5fc3cce05893526076aa528ccbe3c Mon Sep 17 00:00:00 2001 From: Eric Peterson Date: Mon, 24 May 2021 14:57:33 +0200 Subject: [PATCH 2/6] Frontend brain dump. Signed-off-by: Eric Peterson --- docs/features/search/concepts.md | 41 ++++-- docs/features/search/getting-started.md | 176 +++++++++++++++++++++--- 2 files changed, 190 insertions(+), 27 deletions(-) diff --git a/docs/features/search/concepts.md b/docs/features/search/concepts.md index 6a927d73c0..42e0421440 100644 --- a/docs/features/search/concepts.md +++ b/docs/features/search/concepts.md @@ -10,12 +10,16 @@ Backstage Search is _blah_. To get started, you should get familiar with these core concepts: +TODO: Link these like a real TOC + - Search Engines - Query Translators - Documents and Indices - Collators - Decorators - The Scheduler +- The Search Page +- Search Context and Components ### Search Engines @@ -23,8 +27,8 @@ Backstage Search isn't a search engine itself, rather, it provides an interface between your Backstage instance and a Search Engine of your choice. More concretely, a `SearchEngine` is an interface whose concrete implementations facilitate communication with different search engines (like ElasticSearch, -Solr, Algolia, etc). This abstraction exists in order to support your -organization's needs. +Solr, etc). This abstraction exists in order to support your organization's +needs. Out of the box, Backstage Search comes pre-packaged with an in-memory search engine implementation built on top of Lunr. @@ -39,7 +43,7 @@ search engine. Search Engines come pre-packaged with simple translators that do rudimentary transformations of search terms and filters, but you may want to provide your -own to help tune seearch results in the context of your organization. +own to help tune search results in the context of your organization. ### Documents and Indices @@ -78,11 +82,30 @@ Search chooses to completely rebuild indices on a schedule. Different collators can be configured to refresh at different intervals, depending on how often the source information is updated. -TODO: There should probably be some front-end concepts here too? +### The Search Page -- Search Page -- Search Components -- Search Context +Search pages are very custom things. Not every Backstage instance will want the +same interface! In order to allow you to customize your search experience to +your heart's content, the Search Plugin takes care of state management and other +search logic for you, but most of the layout of a search page lives in a search +page component defined in your Backstage App. -[Primarily solves for “As an App Integrator, I should know how to add do uments -to the search index”] +For an example of a simple search page, check [getting started](TODO) + +### Search Context and Components + +A search experience, like a page, is composed of any number of search +components, which are all wired up using a search context. + +Each search experience's context consists of details like a search term, +filters, types, results, and a page cursor for handling pagination. Different +components use this context in different ways, for example the `` +can set the search term, `` components can set filters, and +search results can be displayed using the `` component. + +The `` and `` components are special, in that +they themselves are extensible. For an example of how to extend these +components, [check this out](TODO). + +If you need even more customization, you can use the search context like any +other React context to create custom search components of your own. diff --git a/docs/features/search/getting-started.md b/docs/features/search/getting-started.md index 867b71c8d4..6218afbcb0 100644 --- a/docs/features/search/getting-started.md +++ b/docs/features/search/getting-started.md @@ -12,9 +12,9 @@ use Search. If you haven't setup Backstage already, start [here](../../getting-started/index.md). -> If you used `npx @backstage/create-app`, Search may already be present. -> -> You should skip to [`Customizing Search`](#customizing-search) below. +> If you used `npx @backstage/create-app`, and you have a search page defined in +> `packages/app/src/components/search`, skip to +> [`Customizing Search`](#customizing-search) below. ## Adding Search to the Frontend @@ -24,7 +24,92 @@ cd packages/app yarn add @backstage/plugin-search ``` -TODO: Add frontend instructions here. +Create a new `packages/app/src/components/search/SearchPage.tsx` file in your +Backstage app with the following contents: + +```tsx +import React from 'react'; +import { Content, Header, Page } from '@backstage/core'; +import { Grid, List, Card, CardContent } from '@material-ui/core'; +import { + SearchBar, + SearchResult, + DefaultResultListItem, + SearchFilter, +} from '@backstage/plugin-search'; +import { CatalogResultListItem } from '@backstage/plugin-catalog'; + +export const searchPage = ( + +
+ + + + + + + + + + + + + + + + + + {({ results }) => ( + + {results.map(result => { + switch (result.type) { + case 'software-catalog': + return ( + + ); + default: + return ( + + ); + } + })} + + )} + + + + + +); +``` + +Bind the above Search Page to the `/search` route in your +`packages/app/src/App.tsx` file, like this: + +```tsx +import { SearchPage } from '@backstage/plugin-search'; +import { searchPage } from './components/search/SearchPage'; + +const routes = ( + + }> + {searchPage} + + +); +``` ## Adding Search to the Backend @@ -96,6 +181,75 @@ apiRouter.use('/search', await search(searchEnv)); ## Customizing Search +### Frontend + +The Search Plugin exposes several default filter types as static properties, +including `` and ``. These allow +you to provide values relevant to your Backstage instance that, when selected, +get passed to the backend. + +```tsx {2-5,8-11} + + + + + + +``` + +If you have advanced filter needs, you can specify your own filter component +like this (although contributions are welcome): + +```tsx +import { useSearch, SearchFilter } from '@backstage/plugin-search'; + +const MyCustomFilter = () => { + // Note: filters contain filter data from other filter components. Be sure + // not to clobber other filters' data! + const { filters, setFilters } = useSearch(); + + return (/* ... */); +}; + +// Which could be rendered like this: + +``` + +It's good practice for search results to highlight information that was used to +return it in the first place! The code below highlights how you might specify a +custom result item component, using the `` component as +an example: + +```tsx {7-13} + + {({ results }) => ( + + {results.map(result => { + // result.type is the index type defined by the collator. + switch (result.type) { + case 'software-catalog': + return ( + + ); + // ... + } + })} + + )} + +``` + +### Backend + Backstage Search isn't a search engine itself, rather, it provides an interface between your Backstage instance and a Search Engine of your choice. Currently, we only support one, in-memory search Engine called Lunr. It can be instantiated @@ -139,17 +293,3 @@ indexBuilder.addCollator({ collator: new DefaultCatalogCollator({ discovery }), }); ``` - ---- - -[Primarily solves for “As an App Integrator, I should be able to install out-of -the-box Search”] - -Backend: Different for new Backstage instances (created via create-app) vs. -people enabling search on existing Backstage instances (see catalog install for -example). - -Frontend: Search Page Route…? Customizing result components…? - -Conceptually… What’s the difference between “out-of-the-box” or “basic” and the -recommended deployment (which will come) From 8ad667cd9674d76e36fe9b7e8c081cfb57bcd489 Mon Sep 17 00:00:00 2001 From: Eric Peterson Date: Mon, 24 May 2021 15:01:34 +0200 Subject: [PATCH 3/6] Draft intro statement. Signed-off-by: Eric Peterson --- docs/features/search/concepts.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/features/search/concepts.md b/docs/features/search/concepts.md index 42e0421440..bd115fec4a 100644 --- a/docs/features/search/concepts.md +++ b/docs/features/search/concepts.md @@ -6,7 +6,8 @@ description: Documentation on Backstage Search Concepts # Search Concepts -Backstage Search is _blah_. +Backstage Search lets you find the right information you are looking for in the +Backstage ecosystem. To get started, you should get familiar with these core concepts: From 2714bf9c79bcad6e6315ba78303f32aa4030f95f Mon Sep 17 00:00:00 2001 From: Eric Peterson Date: Mon, 24 May 2021 16:02:03 +0200 Subject: [PATCH 4/6] Cross-linking and menu registration Signed-off-by: Eric Peterson --- docs/features/search/concepts.md | 24 ++++++++++++------------ docs/features/search/getting-started.md | 24 +++++++++++++----------- microsite/sidebars.json | 2 ++ mkdocs.yml | 2 ++ 4 files changed, 29 insertions(+), 23 deletions(-) diff --git a/docs/features/search/concepts.md b/docs/features/search/concepts.md index bd115fec4a..052191bc13 100644 --- a/docs/features/search/concepts.md +++ b/docs/features/search/concepts.md @@ -11,16 +11,14 @@ Backstage ecosystem. To get started, you should get familiar with these core concepts: -TODO: Link these like a real TOC - -- Search Engines -- Query Translators -- Documents and Indices -- Collators -- Decorators -- The Scheduler -- The Search Page -- Search Context and Components +- [Search Engines](#search-engines) +- [Query Translators](#query-translators) +- [Documents and Indices](#documents-and-indices) +- [Collators](#collators) +- [Decorators](#decorators) +- [The Scheduler](#the-scheduler) +- [The Search Page](#the-search-page) +- [Search Context and Components](#search-context-and-components) ### Search Engines @@ -91,7 +89,8 @@ your heart's content, the Search Plugin takes care of state management and other search logic for you, but most of the layout of a search page lives in a search page component defined in your Backstage App. -For an example of a simple search page, check [getting started](TODO) +For an example of a simple search page, check +[getting started](./getting-started.md#adding-search-to-the-frontend) ### Search Context and Components @@ -106,7 +105,8 @@ search results can be displayed using the `` component. The `` and `` components are special, in that they themselves are extensible. For an example of how to extend these -components, [check this out](TODO). +components, check +[getting started](./getting-started.md#adding-search-to-the-frontend). If you need even more customization, you can use the search context like any other React context to create custom search components of your own. diff --git a/docs/features/search/getting-started.md b/docs/features/search/getting-started.md index 6218afbcb0..42b0cdb2c5 100644 --- a/docs/features/search/getting-started.md +++ b/docs/features/search/getting-started.md @@ -204,7 +204,7 @@ get passed to the backend. ``` If you have advanced filter needs, you can specify your own filter component -like this (although contributions are welcome): +like this (although new core filter contributions are welcome): ```tsx import { useSearch, SearchFilter } from '@backstage/plugin-search'; @@ -251,9 +251,10 @@ an example: ### Backend Backstage Search isn't a search engine itself, rather, it provides an interface -between your Backstage instance and a Search Engine of your choice. Currently, -we only support one, in-memory search Engine called Lunr. It can be instantiated -like this: +between your Backstage instance and a +[Search Engine](./concepts.md#search-engines) of your choice. Currently, we only +support one, in-memory search Engine called Lunr. It can be instantiated like +this: ```typescript const searchEngine = new LunrSearchEngine({ logger }); @@ -261,9 +262,9 @@ const indexBuilder = new IndexBuilder({ logger, searchEngine }); ``` Backstage Search can be used to power search of anything! Plugins like the -Catalog offer default "collators" which are responsible for providing documents -to be indexed. You can register any number of collators with the `IndexBuilder` -like this: +Catalog offer default [collators](./concepts.md#collators) which are responsible +for providing documents [to be indexed](./concepts.md#documents-and-indices). +You can register any number of collators with the `IndexBuilder` like this: ```typescript const indexBuilder = new IndexBuilder({ logger, searchEngine }); @@ -281,10 +282,11 @@ indexBuilder.addCollator({ }); ``` -Backstage Search builds and maintains its index on a schedule. You can change -how often the indexes are rebuilt for a given type of document. You may want to -do this if your documents are updated more or less frequently. You can do so by -modifying its `defaultRefreshIntervalSeconds` value, like this: +Backstage Search builds and maintains its index +[on a schedule](./concepts.md#the-scheduler). You can change how often the +indexes are rebuilt for a given type of document. You may want to do this if +your documents are updated more or less frequently. You can do so by modifying +its `defaultRefreshIntervalSeconds` value, like this: ```typescript {3} indexBuilder.addCollator({ diff --git a/microsite/sidebars.json b/microsite/sidebars.json index 6c42614a61..22a47da6d1 100644 --- a/microsite/sidebars.json +++ b/microsite/sidebars.json @@ -75,6 +75,8 @@ "label": "Backstage Search", "ids": [ "features/search/search-overview", + "features/search/getting-started", + "features/search/concepts", "features/search/architecture" ] }, diff --git a/mkdocs.yml b/mkdocs.yml index 574b04f8f7..99c7f332e2 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -58,6 +58,8 @@ nav: - Backstage Search: - Overview: 'features/search/README.md' + - Getting Started: 'features/search/getting-started.md' + - Concepts: 'features/search/concepts.md' - Search Architecture: 'features/search/architecture.md' - TechDocs: - Overview: 'features/techdocs/README.md' From 50873903fb43e3bd18eccb3c2702d15895fc3fa4 Mon Sep 17 00:00:00 2001 From: Emma Indal Date: Sun, 6 Jun 2021 22:00:35 +0200 Subject: [PATCH 5/6] Add Lunr as example Signed-off-by: Emma Indal --- docs/features/search/concepts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/features/search/concepts.md b/docs/features/search/concepts.md index 052191bc13..2af2e7e3a3 100644 --- a/docs/features/search/concepts.md +++ b/docs/features/search/concepts.md @@ -26,7 +26,7 @@ Backstage Search isn't a search engine itself, rather, it provides an interface between your Backstage instance and a Search Engine of your choice. More concretely, a `SearchEngine` is an interface whose concrete implementations facilitate communication with different search engines (like ElasticSearch, -Solr, etc). This abstraction exists in order to support your organization's +Lunr, Solr, etc). This abstraction exists in order to support your organization's needs. Out of the box, Backstage Search comes pre-packaged with an in-memory search From f3a9a063a64d6131f53e268f67a9a305cc725365 Mon Sep 17 00:00:00 2001 From: Emma Indal Date: Sun, 6 Jun 2021 22:04:29 +0200 Subject: [PATCH 6/6] docs fixups Signed-off-by: Emma Indal --- docs/features/search/concepts.md | 6 +++--- docs/features/search/getting-started.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/features/search/concepts.md b/docs/features/search/concepts.md index 2af2e7e3a3..5da7c9c58f 100644 --- a/docs/features/search/concepts.md +++ b/docs/features/search/concepts.md @@ -26,8 +26,8 @@ Backstage Search isn't a search engine itself, rather, it provides an interface between your Backstage instance and a Search Engine of your choice. More concretely, a `SearchEngine` is an interface whose concrete implementations facilitate communication with different search engines (like ElasticSearch, -Lunr, Solr, etc). This abstraction exists in order to support your organization's -needs. +Lunr, Solr, etc). This abstraction exists in order to support your +organization's needs. Out of the box, Backstage Search comes pre-packaged with an in-memory search engine implementation built on top of Lunr. @@ -99,7 +99,7 @@ components, which are all wired up using a search context. Each search experience's context consists of details like a search term, filters, types, results, and a page cursor for handling pagination. Different -components use this context in different ways, for example the `` +components use this context in different ways. For example, the `` can set the search term, `` components can set filters, and search results can be displayed using the `` component. diff --git a/docs/features/search/getting-started.md b/docs/features/search/getting-started.md index 42b0cdb2c5..7408d69193 100644 --- a/docs/features/search/getting-started.md +++ b/docs/features/search/getting-started.md @@ -253,7 +253,7 @@ an example: Backstage Search isn't a search engine itself, rather, it provides an interface between your Backstage instance and a [Search Engine](./concepts.md#search-engines) of your choice. Currently, we only -support one, in-memory search Engine called Lunr. It can be instantiated like +support one, an in-memory search Engine called Lunr. It can be instantiated like this: ```typescript