From d61f0d4fd4ecea434b7dfd5fd9445920a096473a Mon Sep 17 00:00:00 2001 From: Patrik Oldsberg Date: Sun, 3 Oct 2021 14:59:33 +0200 Subject: [PATCH 01/14] docs: initial build system docs Signed-off-by: Patrik Oldsberg --- docs/cli/build-system.md | 72 ++++++++++++++++++++++++++++++++++++++++ microsite/sidebars.json | 2 +- mkdocs.yml | 1 + 3 files changed, 74 insertions(+), 1 deletion(-) create mode 100644 docs/cli/build-system.md diff --git a/docs/cli/build-system.md b/docs/cli/build-system.md new file mode 100644 index 0000000000..fb69aef7d4 --- /dev/null +++ b/docs/cli/build-system.md @@ -0,0 +1,72 @@ +--- +id: builds-system +title: Build System +description: A deep dive into the Backstage build system +--- + +## What is the Backstage build system? + +The Backstage build system is a collection of build and development tools that +help you lint, test, develop and finally release your Backstage projects. The +purpose of it is to provide an out-of-the-box solution lets you focus on +development rather than setting up your own build system. The tooling is shipped +with the [@backstage/cli](https://www.npmjs.com/package/@backstage/cli), and +already included in any project that you create using +[@backstage/create-app](https://www.npmjs.com/package/@backstage/create-app). It +is similar to for example +[react-scripts](https://www.npmjs.com/package/react-scripts), which is the +tooling you get with +[create-react-app](https://github.com/facebook/create-react-app). The Backstage +build system relies heavily on existing open source tools from the JavaScript +and TypeScript ecosystem, such as [Webpack](https://webpack.js.org/), +[Rollup](https://rollupjs.org/), [Jest](https://jestjs.io/), and +[ESLint](https://eslint.org/). + +## Design Considerations + +There are a couple of core beliefs and constraints that guided the design of the +Backstage build system. The first and most important is that we put the +development experience first. If we need to cut corners or add complexity we do +so in other areas, but the experience of firing up an editor and iterating on +some code should always be as smooth as possible. + +In addition, there are a number of hard and soft requirements that we want to +support: + +- Monorepos - The build system should support multi-package setups. +- Publishing - It should be possible to build and publish individual packages. +- Scale - It should scale to hundreds of large packages without excessive wait + times. +- Reloads - The development flow should support quick on-save hot reloads. +- Simple - Usage should simple, just a single command if possible. +- Universal - Development towards both web applications, isomorphic packages, + and Node.js. +- Editors - Things like linting and type checking should work when browsing the + source code. + +During the design of the build system this collection of requirements was not +something that was supported by existing tools like for example `react-scripts`. +Especially the combination of monorepo, publishing and editor support has led to +some specialized setups, as well as the scaling requirements. + +## Architecture + +We can divide the development flow within Backstage into a couple of different +steps: + +- **Formatting** - Applies a consistent formatting to your source code. +- **Linting** - Makes sure your code is free from problems that can be detected + automatically. +- **Type Checking** - Validates your TypeScript type annotations. +- **Testing** - Runs test suites towards your code to catch issues early. +- **Building** (optional) - Compiles the source code in an NPM package to make + it ready for publishing. +- **Bundling** - Compiles and bundles a package and all of its dependencies into + a release bundle. + +These steps are generally kept isolated form each other, with each step focusing +on its specific task. For example, we do not do linting or type checking +together with the building or bundling. This is so that we can provide more +flexibility and avoid duplicate work, improving performance. It is strongly +recommended that as a part of developing withing Backstage you use a code editor +or IDE that has built-in support for formatting, linting, and type checking. diff --git a/microsite/sidebars.json b/microsite/sidebars.json index c0a0b9b24f..6d9f9bde54 100644 --- a/microsite/sidebars.json +++ b/microsite/sidebars.json @@ -34,7 +34,7 @@ { "type": "subcategory", "label": "CLI", - "ids": ["local-dev/cli-overview", "local-dev/cli-commands"] + "ids": ["local-dev/cli-overview", "local-dev/cli-commands", "local-dev/build-system"] }, "local-dev/linking-local-packages" ], diff --git a/mkdocs.yml b/mkdocs.yml index 398863a3c0..074853634c 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -31,6 +31,7 @@ nav: - CLI: - Overview: 'local-dev/cli-overview.md' - Commands: 'local-dev/cli-commands.md' + - Build System: 'cli/build-system.md' - Linking in Local Packages: 'local-dev/linking-local-packages.md' - Core Features: - Software Catalog: From 12174e0b9c8aaedfc58240093fdb6ca29bef7b80 Mon Sep 17 00:00:00 2001 From: Patrik Oldsberg Date: Mon, 4 Oct 2021 17:36:03 +0200 Subject: [PATCH 02/14] docs: more build system docs Signed-off-by: Patrik Oldsberg --- docs/cli/build-system.md | 56 ++++++++++++++++++++++++++++++++++++---- 1 file changed, 51 insertions(+), 5 deletions(-) diff --git a/docs/cli/build-system.md b/docs/cli/build-system.md index fb69aef7d4..dec5d64f5d 100644 --- a/docs/cli/build-system.md +++ b/docs/cli/build-system.md @@ -59,14 +59,60 @@ steps: automatically. - **Type Checking** - Validates your TypeScript type annotations. - **Testing** - Runs test suites towards your code to catch issues early. -- **Building** (optional) - Compiles the source code in an NPM package to make - it ready for publishing. -- **Bundling** - Compiles and bundles a package and all of its dependencies into - a release bundle. +- **Building** - Compiles the source code in an individual package. +- **Bundling** - Compiles and combines a package and all of its dependencies + into a bundle. These steps are generally kept isolated form each other, with each step focusing on its specific task. For example, we do not do linting or type checking together with the building or bundling. This is so that we can provide more flexibility and avoid duplicate work, improving performance. It is strongly recommended that as a part of developing withing Backstage you use a code editor -or IDE that has built-in support for formatting, linting, and type checking. +or IDE that has support for formatting, linting, and type checking. + +Let's dive into a detailed look at each of these steps and how they are +implemented in a typical Backstage app. + +### Formatting + +The formatting setup lives completely within each Backstage application. In an +app created with `@backstage/create-app` the formatting is handled by +[prettier](https://prettier.io/), but each application can those their own +formatting rules and switch to a different formatter if desired. + +### Linting + +The Backstage CLI includes a `lint` command, which is a thin wrapper around +`eslint`. It adds a few options that can't be set through configuration, such as +including the `.ts` and `.tsx` extensions in the set of linted files. The `lint` +command simply provides a sane default and is not intended to be customizable. +If you want to supply more advanced options you can invoke `eslint` directly +instead. + +In addition to the `lint` command, the Backstage CLI also includes a set of base +ESLint configurations, one for frontend and one for backend packages. These lint +configurations in turn build on top of the lint rules from +[@spotify/web-scripts](https://github.com/spotify/web-scripts). + +In a standard Backstage setup, each individual package has its own lint +configuration, along with that there's also a root configuration that applies to +the entire project. Each configuration is initially one that simply extends a +configuration provided by the Backstage CLI, but can be customized to fit the +needs of each package. + +### Type Checking + +Just like formatting, the Backstage CLI does not have its own command for type +checking. It does however have a base configuration with both recommended +defaults as well as some required settings for the build system to work. + +Perhaps the most notable part about the TypeScript setup in Backstage projects +is that the entire project is one big compilation unit. This is due to +performance optimization as well as easy of use, since breaking projects down +into smaller pieces has proven to both lead to a more complicated setup, as well +as type checking of the entire project being an order of magnitude slower. In +order to make this setup work, the entrypoint of each package needs to point to +the TypeScript source files, which in turn cases some complications during +publishing that we'll talk about below. + +The type checking is generally configured to be incremental From aaeda76427424ddb2decae0a7a08f10c63452228 Mon Sep 17 00:00:00 2001 From: Patrik Oldsberg Date: Fri, 10 Dec 2021 17:21:20 +0100 Subject: [PATCH 03/14] docs: move cli/build-system to local-dev Signed-off-by: Patrik Oldsberg --- docs/{cli/build-system.md => local-dev/cli-build-system.md} | 2 +- microsite/sidebars.json | 2 +- mkdocs.yml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) rename docs/{cli/build-system.md => local-dev/cli-build-system.md} (99%) diff --git a/docs/cli/build-system.md b/docs/local-dev/cli-build-system.md similarity index 99% rename from docs/cli/build-system.md rename to docs/local-dev/cli-build-system.md index dec5d64f5d..078bef36f5 100644 --- a/docs/cli/build-system.md +++ b/docs/local-dev/cli-build-system.md @@ -1,5 +1,5 @@ --- -id: builds-system +id: cli-build-system title: Build System description: A deep dive into the Backstage build system --- diff --git a/microsite/sidebars.json b/microsite/sidebars.json index 6d9f9bde54..3ff59b7f9d 100644 --- a/microsite/sidebars.json +++ b/microsite/sidebars.json @@ -34,7 +34,7 @@ { "type": "subcategory", "label": "CLI", - "ids": ["local-dev/cli-overview", "local-dev/cli-commands", "local-dev/build-system"] + "ids": ["local-dev/cli-overview", "local-dev/cli-commands", "local-dev/cli-build-system"] }, "local-dev/linking-local-packages" ], diff --git a/mkdocs.yml b/mkdocs.yml index 074853634c..6974dd6c79 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -31,7 +31,7 @@ nav: - CLI: - Overview: 'local-dev/cli-overview.md' - Commands: 'local-dev/cli-commands.md' - - Build System: 'cli/build-system.md' + - Build System: 'local-dev/cli-build-system.md' - Linking in Local Packages: 'local-dev/linking-local-packages.md' - Core Features: - Software Catalog: From 7a9240a56f8e12c16e2fdd68f2ea4dc792f3291e Mon Sep 17 00:00:00 2001 From: Patrik Oldsberg Date: Fri, 10 Dec 2021 22:10:35 +0100 Subject: [PATCH 04/14] docs/local-dev: build system doc tweaks and more type checking/test Signed-off-by: Patrik Oldsberg --- docs/local-dev/cli-build-system.md | 67 +++++++++++++++++++++++------- 1 file changed, 52 insertions(+), 15 deletions(-) diff --git a/docs/local-dev/cli-build-system.md b/docs/local-dev/cli-build-system.md index 078bef36f5..bbcb4c2bd3 100644 --- a/docs/local-dev/cli-build-system.md +++ b/docs/local-dev/cli-build-system.md @@ -33,16 +33,16 @@ some code should always be as smooth as possible. In addition, there are a number of hard and soft requirements that we want to support: -- Monorepos - The build system should support multi-package setups. -- Publishing - It should be possible to build and publish individual packages. +- Monorepos - The build system should support multi-package setups +- Publishing - It should be possible to build and publish individual packages - Scale - It should scale to hundreds of large packages without excessive wait - times. -- Reloads - The development flow should support quick on-save hot reloads. -- Simple - Usage should simple, just a single command if possible. + times +- Reloads - The development flow should support quick on-save hot reloads +- Simple - Usage should simple, just a single command if possible - Universal - Development towards both web applications, isomorphic packages, - and Node.js. + and Node.js - Editors - Things like linting and type checking should work when browsing the - source code. + source code During the design of the build system this collection of requirements was not something that was supported by existing tools like for example `react-scripts`. @@ -54,14 +54,14 @@ some specialized setups, as well as the scaling requirements. We can divide the development flow within Backstage into a couple of different steps: -- **Formatting** - Applies a consistent formatting to your source code. +- **Formatting** - Applies a consistent formatting to your source code - **Linting** - Makes sure your code is free from problems that can be detected - automatically. -- **Type Checking** - Validates your TypeScript type annotations. -- **Testing** - Runs test suites towards your code to catch issues early. -- **Building** - Compiles the source code in an individual package. -- **Bundling** - Compiles and combines a package and all of its dependencies - into a bundle. + automatically +- **Type Checking** - Verifies that TypeScript types are valid +- **Testing** - Runs test suites towards your code to catch issues early +- **Building** - Compiles the source code in an individual package +- **Bundling** - Combines a package and all of its dependencies into a + production-ready bundle These steps are generally kept isolated form each other, with each step focusing on its specific task. For example, we do not do linting or type checking @@ -115,4 +115,41 @@ order to make this setup work, the entrypoint of each package needs to point to the TypeScript source files, which in turn cases some complications during publishing that we'll talk about below. -The type checking is generally configured to be incremental +The type checking is generally configured to be incremental for local +development, with the output stored in the `dist-types` folder in the repo root. +This provides a significant speedup when running `tsc` multiple times locally, +but it does make the initial run a little bit slower. Because of the slower +initial run we disable incremental type checking in the `tcs:full` Yarn script +that is included by default in any Backstage app and is intended for use in CI. + +Another optimization that is used by default is to skip the checking of library +types, this means that TypeScript will not verify that types within +`node_modules` are sound. Disabling this check significantly speeds up type +checking, but in the end it is still an important check that should not be +completely omitted, it's just very unlikely to catch issues that are introduced +during local development. What we opt for instead is to include the check in CI +through the `tsc:full` script, which will run a full type check, including +`node_modules`. + +For the two reasons mentioned above, it is **highly** recommended to use the +`tsc:full` script to run type checking in CI. + +### Testing + +As mentioned above, the Backstage CLI uses [Jest](https://jestjs.io/), which is +a JavaScript test framework that covers both test execution and assertions. Jest +executes all tests in Node.js, including frontend browser code. The trick it +uses is to execute the tests in a Node.js VM using various predefined +environments, such as one based on [`jsdom`](https://github.com/jsdom/jsdom) +that helps mimic browser APIs and behavior. + +The Backstage CLI has its own command that helps execute tests, +`backstage-cli test`, as well as it's own configuration at +`@backstage/cli/config/jest.js`. The command is a relatively thin wrapper around +running `jest` directly. Its main responsibility is to make sure the included +configuration is used, as well setting the `NODE_ENV` and `TZ` environment +variables, and provided some sane default flags like `--watch` if executed +within a Git repository. + +The by far biggest amount of work is done by the Jest configuration included +with the Backstage CLI. It both takes care of providing a default configuration From 0a052c9be1dfed8b456c9cad0e3c35461614c0a3 Mon Sep 17 00:00:00 2001 From: Patrik Oldsberg Date: Fri, 17 Dec 2021 22:31:13 +0100 Subject: [PATCH 05/14] docs: initial build system build step docs Signed-off-by: Patrik Oldsberg --- docs/local-dev/cli-build-system.md | 45 +++++++++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/docs/local-dev/cli-build-system.md b/docs/local-dev/cli-build-system.md index bbcb4c2bd3..9cc7c1ea8c 100644 --- a/docs/local-dev/cli-build-system.md +++ b/docs/local-dev/cli-build-system.md @@ -152,4 +152,47 @@ variables, and provided some sane default flags like `--watch` if executed within a Git repository. The by far biggest amount of work is done by the Jest configuration included -with the Backstage CLI. It both takes care of providing a default configuration +with the Backstage CLI. It both takes care of providing a default Jest +configuration, as well as allowing for configuration overrides to be defined in +each `package.json`. How this can be done in practice is discussed in the +configuration section below. + +### Building + +The primary purpose of the build process is to prepare packages for publishing, +although it is also used as part of the backend bundling process which we will +discuss below. Since it's only used in these two cases, any Backstage app that +does not use the Backend parts of the project may not need to interact with the +build process at all. It can nevertheless be useful to know how it works, since +all of the published Backstage packages are built using this process. + +The build is currently using [Rollup](https://rollupjs.org/) and executes in +complete isolation for each individual package. There are currently three +different commands in the Backstage CLI that invokes the build process, +`plugin:build`, `backend:build`, and simply `build`. The two former are +pre-configured commands for frontend and backend plugins, while the `build` +command provides more control over the output. It is likely that the two +specialized commands disappear in the future, leaving just the `build` command. + +There are three different possible outputs of the build process: JavaScript in +CommonJS module format, JavaScript in ESM format, and type declarations. Each +invocation of a build command will write one or more of these outputs to the +`dist` folder in the package, and in addition copy any asset files like +stylesheets or images. For more details on what syntax and file formats are +supported by the build process, see the transpilation section below. + +When building CommonJS or ESM output, the build commands will always use +`src/index.ts` as the entrypoint. All dependencies of the package will be marked +as external, meaning that in general it is only the contents of the `src` folder +that ends up being compiled and output to `dist`, any imported external +dependencies, even within the monorepo, will stay intact. + +The build of the type definitions works quite differently. The entrypoint of the +type definition build is the relative location of the package within the +`dist-types` folder in the project root. This means that it is important to run +type checking before building any packages with type definitions, and that +emitting type declarations must be enable in the TypeScript configuration. The +reason for the type definition build step is to strip out all types but the ones +that are exported from the package, leaving a much cleaner type definition file +and making sure that the type definitions are in sync with the generated +JavaScript. From 71c22ae2fc825ad9aba65bddc05cf4aebe9aa0c8 Mon Sep 17 00:00:00 2001 From: Patrik Oldsberg Date: Sun, 19 Dec 2021 13:47:05 +0100 Subject: [PATCH 06/14] docs: halfway bundling docs Signed-off-by: Patrik Oldsberg --- docs/local-dev/cli-build-system.md | 49 ++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/docs/local-dev/cli-build-system.md b/docs/local-dev/cli-build-system.md index 9cc7c1ea8c..45cdeda073 100644 --- a/docs/local-dev/cli-build-system.md +++ b/docs/local-dev/cli-build-system.md @@ -196,3 +196,52 @@ reason for the type definition build step is to strip out all types but the ones that are exported from the package, leaving a much cleaner type definition file and making sure that the type definitions are in sync with the generated JavaScript. + +### Bundling + +The goal of the bundling process is to combine multiple packages together into a +single runtime unit. The way this is done varies between frontend and backend, +and local development versus production deployment. Because of that we will be +covering each of these cases individually. + +#### Frontend Development Bundling + +There are two different commands that starts the frontend development bundling, +`app:serve`, which serves an app and uses `src/index.*` as the entrypoint, and +`plugin:serve`, which serves a plugin and uses `dev/index.*` as the entrypoint. +These are typically invoked via the `yarn start` script, and are intended for +local development only. When running the bundle command, a development server +will be set up that listens to the protocol, host and port set by `app.baseUrl` +in the configuration. If needed it is also possible to override the listening +options through the `app.listen` configuration. + +The frontend development bundling is currently based on +[Webpack](https://webpack.js.org/) and +[Webpack Dev Server](https://webpack.js.org/configuration/dev-server/). The +Webpack configuration itself varies very little between the frontend development +and production bundling, and we'll dive more into the configuration in the +production section below. The main differences are that `process.env.NODE_ENV` +is set to `'development'`, minification is disabled, and there is support for +[React Hot Loader](https://github.com/gaearon/react-hot-loader). + +#### Frontend Production Bundling + +The frontend production bundling creates your typical web content bundle, all +contained within a single folder, ready for static serving. It is invoked using +the `app:build` command, and unlike the development bundling there is no way to +build a production bundle of an individual plugin. The output of the bundling +process is written to the `dist` folder in the package. + +Just like the development bundling, the production bundling is based on +[Webpack](https://webpack.js.org/). It also uses the + +While the output of the production bundling is written to the `dist` folder The +output of the bundling process is split into two types of files. The first is a +set of generic assets with plain names in the root of the `dist/` folder. You +will want to serve these with short-lived caching or no caching at all. The +second is a set of files with hashes in their names in the `dist/static/` folder +that are intended to be cached for a much longer time + +#### Backend Development Bundling + +#### Backend Production Bundling From 525e7aa3d7e70ab12951d6eb860420cb0eb7167c Mon Sep 17 00:00:00 2001 From: Patrik Oldsberg Date: Sun, 16 Jan 2022 12:39:37 +0100 Subject: [PATCH 07/14] docs: frontend bundling docs mostly done Signed-off-by: Patrik Oldsberg --- docs/local-dev/cli-build-system.md | 50 +++++++++++++++++++++++++----- 1 file changed, 42 insertions(+), 8 deletions(-) diff --git a/docs/local-dev/cli-build-system.md b/docs/local-dev/cli-build-system.md index 45cdeda073..247965a6cd 100644 --- a/docs/local-dev/cli-build-system.md +++ b/docs/local-dev/cli-build-system.md @@ -219,11 +219,18 @@ The frontend development bundling is currently based on [Webpack](https://webpack.js.org/) and [Webpack Dev Server](https://webpack.js.org/configuration/dev-server/). The Webpack configuration itself varies very little between the frontend development -and production bundling, and we'll dive more into the configuration in the +and production bundling, so we'll dive more into the configuration in the production section below. The main differences are that `process.env.NODE_ENV` is set to `'development'`, minification is disabled, and there is support for [React Hot Loader](https://github.com/gaearon/react-hot-loader). +If you prefer to run type checking and linting as part of the Webpack process, +you can enable usage of the +[`ForkTsCheckerWebpackPlugin`](https://www.npmjs.com/package/fork-ts-checker-webpack-plugin) +by passing the `--check` flag. However, the recommended way to handle these +checks is to use an editor that has built-in support for them, as well as +running them in CI and the occasional manual run when needed. + #### Frontend Production Bundling The frontend production bundling creates your typical web content bundle, all @@ -233,15 +240,42 @@ build a production bundle of an individual plugin. The output of the bundling process is written to the `dist` folder in the package. Just like the development bundling, the production bundling is based on -[Webpack](https://webpack.js.org/). It also uses the +[Webpack](https://webpack.js.org/). It uses the +[`HtmlWebpackPlugin`](https://webpack.js.org/plugins/html-webpack-plugin/) to +generate the `index.html` entry point, and includes a default template that's +included with the CLI. You can replace the bundled template by adding +`public/index.html` to your package. The template has access to two global +constants, `publicPath` which is the public base path that the bundle is +intended to be served at, as well as `config` which is your regular frontend +scoped configuration from `@backstage/config`. -While the output of the production bundling is written to the `dist` folder The -output of the bundling process is split into two types of files. The first is a -set of generic assets with plain names in the root of the `dist/` folder. You -will want to serve these with short-lived caching or no caching at all. The -second is a set of files with hashes in their names in the `dist/static/` folder -that are intended to be cached for a much longer time +The Webpack configuration also includes a custom plugin for resolving packages +correctly from linked in packages, the `ModuleScopePlugin` from +[`react-dev-utils`](https://www.npmjs.com/package/react-dev-utils) which makes +sure that imports don't react outside the package, a few fallbacks for some +Node.js modules like `'buffer'` and `'events'`, a plugin that writes the +frontend configuration to the bundle as `process.env.APP_CONFIG` and build +information as `process.env.BUILD_INFO`, and lastly minification handled by +[esbuild](https://esbuild.github.io/) using the +[`esbuild-loader`](https://npm.im/esbuild-loader). There are of course also a +set of loaders configured, which is covered in the +[transpilation section](#loaders-and-transpilation). + +The output of the bundling process is split into two types of files. The first +is a set of generic assets with plain names in the root of the `dist/` folder. +You will want to serve these with short-lived caching or no caching at all. The +second is a set of hashed static assets in the `dist/static/`, which you can +configure to be cached for a much longer time. + +The configuration of static assets is optimized for frequent changes and serving +over HTTP 2.0. The assets are aggressively split into small chunks, which means +the browser has to make a lot of small requests to load them. The upside is that +changes to individual plugins and packages will invalidate a smaller number of +files, thereby allowing for rapid development without too much impact on the +page load performance. #### Backend Development Bundling #### Backend Production Bundling + +## Loaders and Transpilation From a2f6b59119a378b117d9b6a4419da3a04ed55edc Mon Sep 17 00:00:00 2001 From: Patrik Oldsberg Date: Sun, 16 Jan 2022 14:02:18 +0100 Subject: [PATCH 08/14] docs: document backend dev and prod bundling setup Signed-off-by: Patrik Oldsberg --- docs/local-dev/cli-build-system.md | 66 ++++++++++++++++++++++++++++++ docs/local-dev/cli-commands.md | 37 ++--------------- 2 files changed, 69 insertions(+), 34 deletions(-) diff --git a/docs/local-dev/cli-build-system.md b/docs/local-dev/cli-build-system.md index 247965a6cd..a769de7587 100644 --- a/docs/local-dev/cli-build-system.md +++ b/docs/local-dev/cli-build-system.md @@ -276,6 +276,72 @@ page load performance. #### Backend Development Bundling +The backend development bundling is also based on Webpack, but rather than +starting up a web server, the backend is started up using the +[`RunScriptWebpackPlugin`](https://www.npmjs.com/package/run-script-webpack-plugin). +The reason for using Webpack for development of the backend is both that it is +convenient way to handle transpilation of a large set of packages, as well us +allowing us to use hot module replacement and maintain state while reloading +individual backend modules. This is particularly useful when running the backend +with in-memory SQLite as the database choice. + +Except for executing in Node.js rather than a web server, the backend +development bundling configuration is quite similar to the frontend one. It +shares most of the Webpack configuration, including the transpilation setup. +Some differences are that it does not inject any environment variables or node +module fallbacks, and it uses +[`webpack-node-externals`](https://www.npmjs.com/package/webpack-node-externals) +to avoid bundling in dependency modules. + +If you want to inspect the running Node.js process, the `--inspect` and +`--inspect-brk` flags can be used, as they will be passed through to the backend +process. + #### Backend Production Bundling +The backend production bundling uses a completely different setup than the other +bundling options. Rather than using Webpack, the backend production bundling +instead collects the backend packages and all of its local dependencies into a +deployment archive. The archive is written to `dist/bundle.tar.gz`, and contains +the packaged version of each of these packages. The layout of the packages in +the archive is the same as the directory layout in the target monorepo, and the +bundle also contains the root `package.json` and `yarn.lock` files. + +Note that before building the production bundle, all of the backend packages +have to have been built first, although when executing the `backend:bundle` +command you can pass the `--build-dependencies` to have this done automatically. +The reason for not using the flag is to avoid duplicate work in case you already +build all of your packages earlier in your build process. + +To use the bundle, extract it into a directory, run `yarn install --production`, +and then start the backend using your backend package as the Node.js entry +point, for example `node packages/backend`. + +The `dist/bundle.tar.gz` is accompanied by a `dist/skeleton.tar.gz`, which has +the same layout, but only contains `package.json` files. This can be used to run +a `yarn install` in environments that will benefit from the caching that this +enables, such as Docker image builds. To use the skeleton archive you copy it +over to the target directory along with the root `package.json` and `yarn.lock`, +extract the archive, and then run `yarn install`. Your target directory now has +all dependencies installed, and as soon as you copy over and extract the +contents of the `bundle.tar.gz` archive, the backend will be ready to run. + +The following is an example of a `Dockerfile` that can be used to package the +output of `backstage-cli backend:bundle` into an image: + +```Dockerfile +FROM node:14-buster-slim +WORKDIR /app + +COPY yarn.lock package.json packages/backend/dist/skeleton.tar.gz ./ +RUN tar xzf skeleton.tar.gz && rm skeleton.tar.gz + +RUN yarn install --production --frozen-lockfile --network-timeout 300000 && rm -rf "$(yarn cache dir)" + +COPY packages/backend/dist/bundle.tar.gz app-config.yaml ./ +RUN tar xzf bundle.tar.gz && rm bundle.tar.gz + +CMD ["node", "packages/backend"] +``` + ## Loaders and Transpilation diff --git a/docs/local-dev/cli-commands.md b/docs/local-dev/cli-commands.md index 21b129e0c2..1168a86e3a 100644 --- a/docs/local-dev/cli-commands.md +++ b/docs/local-dev/cli-commands.md @@ -173,40 +173,9 @@ Options: Scope: `backend` -Bundle the backend and all of its local dependencies into a deployment archive. -The archive is written to `dist/bundle.tar.gz`, and contains the packaged -version of all dependencies of the target package, along with the target package -itself. The layout of the packages in the archive is the same as the directory -layout in the target monorepo, and the bundle also contains the root -`package.json` and `yarn.lock`. - -To use the bundle, extract it into a target directory, run -`yarn install --production`, and then start the target backend package using for -example `node package/backend`. - -The `dist/bundle.tar.gz` is accompanied by a `dist/skeleton.tar.gz`, which has -the same layout, but only contains `package.json` files and `yarn.lock`. This -can be used to run a `yarn install` in environments that will benefit from the -caching that this enables, such as Docker image builds. To use the skeleton -archive, simply extract it first, run install, and then extract the main bundle. - -The following is an example of a `Dockerfile` that can be used to package the -output of `backstage-cli backend:bundle` into an image: - -```Dockerfile -FROM node:14-buster-slim -WORKDIR /app - -COPY yarn.lock package.json packages/backend/dist/skeleton.tar.gz ./ -RUN tar xzf skeleton.tar.gz && rm skeleton.tar.gz - -RUN yarn install --frozen-lockfile --production --network-timeout 300000 && rm -rf "$(yarn cache dir)" - -COPY packages/backend/dist/bundle.tar.gz app-config.yaml ./ -RUN tar xzf bundle.tar.gz && rm bundle.tar.gz - -CMD ["node", "packages/backend"] -``` +Bundles the backend into a `dist/bundle.tar.gz` archive. See the +[backend bundling](./cli-build-system.md#backend-production-bundling) build +systems documentation for more details. ```text Usage: backstage-cli backend:bundle [options] From e3155cbfaaa2036c1e0c073639ed6318eb88c4f3 Mon Sep 17 00:00:00 2001 From: Patrik Oldsberg Date: Sun, 16 Jan 2022 15:05:58 +0100 Subject: [PATCH 09/14] docs: build system transpilers and loaders docs Signed-off-by: Patrik Oldsberg --- .github/styles/vocab.txt | 1 + docs/local-dev/cli-build-system.md | 73 ++++++++++++++++++++++++++++-- 2 files changed, 71 insertions(+), 3 deletions(-) diff --git a/.github/styles/vocab.txt b/.github/styles/vocab.txt index 5a47b720b8..37295d7b26 100644 --- a/.github/styles/vocab.txt +++ b/.github/styles/vocab.txt @@ -299,6 +299,7 @@ tooltips touchpoints transpilation transpiled +transpilers truthy typeahead ui diff --git a/docs/local-dev/cli-build-system.md b/docs/local-dev/cli-build-system.md index a769de7587..ee352af1ec 100644 --- a/docs/local-dev/cli-build-system.md +++ b/docs/local-dev/cli-build-system.md @@ -258,8 +258,8 @@ frontend configuration to the bundle as `process.env.APP_CONFIG` and build information as `process.env.BUILD_INFO`, and lastly minification handled by [esbuild](https://esbuild.github.io/) using the [`esbuild-loader`](https://npm.im/esbuild-loader). There are of course also a -set of loaders configured, which is covered in the -[transpilation section](#loaders-and-transpilation). +set of loaders configured, which you can read more about in the +[transpilation](#transpilation) and [loaders](#loaders) sections. The output of the bundling process is split into two types of files. The first is a set of generic assets with plain names in the root of the `dist/` folder. @@ -344,4 +344,71 @@ RUN tar xzf bundle.tar.gz && rm bundle.tar.gz CMD ["node", "packages/backend"] ``` -## Loaders and Transpilation +## Transpilation + +The transpilers used by the Backstage CLI have been chosen according to the +design considerations that we mentioned above. A few specific requirements are +of course support for TypeScript and JSX, but also React hot reloads or refresh, +and hoisting of Jest mocks. The Backstage CLI also only targets up to date and +modern browsers, so we actually want to keep the transpilation process as +lightweight as possible, and leave most syntax intact. + +Apart from these requirements, we choose transpilers based on their speed. The +build process keeps the integration with the transpilers lightweight, without +additional plugins or such. This enables us to switch out transpilers as new +options and optimizations become available, and use the fastest options that are +available. + +Our current selection of transpilers are [esbuild][https://esbuild.github.io/] +and [Sucrase](https://github.com/alangpierce/sucrase). The reason we choose to +use two transpilers is that esbuild is faster than Sucrase and produces slightly +nicer output, but it does not have the same feature set, and for example does +not support React hot reloading or hoisting of Jest mocks. + +The benchmarking of the various options has been done in the +[ts-build-bench](https://github.com/Rugvip/ts-build-bench) project. It allows +for setups of different shapes and sizes of monorepos, but the setup we consider +the most important during benchmarking is a large monorepo with lots of medium +to large packages that's being bundled with Webpack. Some rough findings have +been that esbuild is the fastest option right now, with Sucrase following +closely after and then [SWC](https://swc.rs/) closely after that. After those +there's a pretty big gap up to the TypeScript compiler run in transpilation only +mode, and lastly another jump up to Babel, being by far the slowest. + +Something to note about these benchmarks is that they take the full Webpack +bundling time into account. This means that even though some transpilation +options may be orders of magnitude faster than others, the total time is not +impacted in the same way as there's a lot of other things that go into the +bundling process. Still, switching from for example Babel to Sucrase is able to +make the bundling anywhere from two to five times faster. + +## Loaders + +The Backstage CLI is configured to support a set of loaders throughout all parts +of the build system, including the bundling, tests, builds, and type checking. +Loaders are always selected based on the file extension. The following is a list +of all supported file extensions: + +| Extension | Exports | Purpose | +| ----------- | --------------- | ----------------------------------------------------------------------------- | +| `.ts` | Script Module | TypeScript | +| `.tsx` | Script Module | TypeScript and XML | +| `.js` | Script Module | JavaScript | +| `.jsx` | Script Module | JavaScript and XML | +| `.mjs` | Script Module | ECMAScript Module | +| `.cjs` | Script Module | CommonJS Module | +| `.json` | JSON Data | JSON Data | +| `.yml` | JSON Data | YAML Data | +| `.yaml` | JSON Data | YAML Data | +| `.css` | classes | Style sheet | +| `.eot` | URL Path | Font | +| `.ttf` | URL Path | Font | +| `.woff2` | URL Path | Font | +| `.woff` | URL Path | Font | +| `.bmp` | URL Path | Image | +| `.gif` | URL Path | Image | +| `.jpeg` | URL Path | Image | +| `.jpg` | URL Path | Image | +| `.png` | URL Path | Image | +| `.svg` | URL Path | Image | +| `.icon.svg` | React Component | SVG converted into a [MUI SvgIcon](https://mui.com/components/icons/#svgicon) | From 035abd52dd7e9967dec768090fa54db6d8493275 Mon Sep 17 00:00:00 2001 From: Patrik Oldsberg Date: Sun, 16 Jan 2022 20:18:54 +0100 Subject: [PATCH 10/14] docs: build system tweaks and corrections Signed-off-by: Patrik Oldsberg --- .github/styles/vocab.txt | 1 + docs/local-dev/cli-build-system.md | 214 +++++++++++++++-------------- 2 files changed, 113 insertions(+), 102 deletions(-) diff --git a/.github/styles/vocab.txt b/.github/styles/vocab.txt index 37295d7b26..3aa1708b14 100644 --- a/.github/styles/vocab.txt +++ b/.github/styles/vocab.txt @@ -299,6 +299,7 @@ tooltips touchpoints transpilation transpiled +transpiler transpilers truthy typeahead diff --git a/docs/local-dev/cli-build-system.md b/docs/local-dev/cli-build-system.md index ee352af1ec..982e5952da 100644 --- a/docs/local-dev/cli-build-system.md +++ b/docs/local-dev/cli-build-system.md @@ -8,10 +8,13 @@ description: A deep dive into the Backstage build system The Backstage build system is a collection of build and development tools that help you lint, test, develop and finally release your Backstage projects. The -purpose of it is to provide an out-of-the-box solution lets you focus on -development rather than setting up your own build system. The tooling is shipped -with the [@backstage/cli](https://www.npmjs.com/package/@backstage/cli), and -already included in any project that you create using +purpose of the build system is to provide an out-of-the-box solution that works +well with Backstage and lets you focus on building an app rather than having to +spend time setting up your own tooling. + +The build system setup is part of the +[@backstage/cli](https://www.npmjs.com/package/@backstage/cli), and already +included in any project that you create using [@backstage/create-app](https://www.npmjs.com/package/@backstage/create-app). It is similar to for example [react-scripts](https://www.npmjs.com/package/react-scripts), which is the @@ -30,24 +33,22 @@ development experience first. If we need to cut corners or add complexity we do so in other areas, but the experience of firing up an editor and iterating on some code should always be as smooth as possible. -In addition, there are a number of hard and soft requirements that we want to -support: +In addition, there are a number of hard and soft requirements: -- Monorepos - The build system should support multi-package setups +- Monorepos - The build system should support multi-package workspaces - Publishing - It should be possible to build and publish individual packages - Scale - It should scale to hundreds of large packages without excessive wait times - Reloads - The development flow should support quick on-save hot reloads -- Simple - Usage should simple, just a single command if possible +- Simple - Usage should simple and configuration should be kept minimal - Universal - Development towards both web applications, isomorphic packages, and Node.js -- Editors - Things like linting and type checking should work when browsing the - source code +- Editors - Type checking and linting should be available within most editors During the design of the build system this collection of requirements was not something that was supported by existing tools like for example `react-scripts`. -Especially the combination of monorepo, publishing and editor support has led to -some specialized setups, as well as the scaling requirements. +The requirements of scaling in combination of monorepo, publishing, and editor +support led us to adopting our own specialized setup. ## Architecture @@ -55,10 +56,9 @@ We can divide the development flow within Backstage into a couple of different steps: - **Formatting** - Applies a consistent formatting to your source code -- **Linting** - Makes sure your code is free from problems that can be detected - automatically +- **Linting** - Analyzes your source code for potential problems - **Type Checking** - Verifies that TypeScript types are valid -- **Testing** - Runs test suites towards your code to catch issues early +- **Testing** - Runs different levels of test suites towards your project - **Building** - Compiles the source code in an individual package - **Bundling** - Combines a package and all of its dependencies into a production-ready bundle @@ -75,10 +75,11 @@ implemented in a typical Backstage app. ### Formatting -The formatting setup lives completely within each Backstage application. In an -app created with `@backstage/create-app` the formatting is handled by -[prettier](https://prettier.io/), but each application can those their own -formatting rules and switch to a different formatter if desired. +The formatting setup lives completely within each Backstage application and is +not part of the CLI. In an app created with `@backstage/create-app` the +formatting is handled by [prettier](https://prettier.io/), but each application +can those their own formatting rules and switch to a different formatter if +desired. ### Linting @@ -97,8 +98,8 @@ configurations in turn build on top of the lint rules from In a standard Backstage setup, each individual package has its own lint configuration, along with that there's also a root configuration that applies to the entire project. Each configuration is initially one that simply extends a -configuration provided by the Backstage CLI, but can be customized to fit the -needs of each package. +base configuration provided by the Backstage CLI, but they can be customized to +fit the needs of each package. ### Type Checking @@ -112,8 +113,8 @@ performance optimization as well as easy of use, since breaking projects down into smaller pieces has proven to both lead to a more complicated setup, as well as type checking of the entire project being an order of magnitude slower. In order to make this setup work, the entrypoint of each package needs to point to -the TypeScript source files, which in turn cases some complications during -publishing that we'll talk about below. +the TypeScript source files, which in turn causes some complications during +publishing that we'll talk about in [that section](#publishing). The type checking is generally configured to be incremental for local development, with the output stored in the `dist-types` folder in the repo root. @@ -126,7 +127,7 @@ Another optimization that is used by default is to skip the checking of library types, this means that TypeScript will not verify that types within `node_modules` are sound. Disabling this check significantly speeds up type checking, but in the end it is still an important check that should not be -completely omitted, it's just very unlikely to catch issues that are introduced +completely omitted, it's simply unlikely to catch issues that are introduced during local development. What we opt for instead is to include the check in CI through the `tsc:full` script, which will run a full type check, including `node_modules`. @@ -155,43 +156,44 @@ The by far biggest amount of work is done by the Jest configuration included with the Backstage CLI. It both takes care of providing a default Jest configuration, as well as allowing for configuration overrides to be defined in each `package.json`. How this can be done in practice is discussed in the -configuration section below. +[Jest configuration](#jest-configuration) section. ### Building The primary purpose of the build process is to prepare packages for publishing, -although it is also used as part of the backend bundling process which we will -discuss below. Since it's only used in these two cases, any Backstage app that -does not use the Backend parts of the project may not need to interact with the -build process at all. It can nevertheless be useful to know how it works, since -all of the published Backstage packages are built using this process. +but it's also used as part of the backend bundling process. Since it's only used +in these two cases, any Backstage app that does not use the Backend parts of the +project may not need to interact with the build process at all. It can +nevertheless be useful to know how it works, since all of the published +Backstage packages are built using this process. The build is currently using [Rollup](https://rollupjs.org/) and executes in -complete isolation for each individual package. There are currently three -different commands in the Backstage CLI that invokes the build process, -`plugin:build`, `backend:build`, and simply `build`. The two former are -pre-configured commands for frontend and backend plugins, while the `build` -command provides more control over the output. It is likely that the two -specialized commands disappear in the future, leaving just the `build` command. +isolation for each individual package. There are currently three different +commands in the Backstage CLI that invokes the build process, `plugin:build`, +`backend:build`, and simply `build`. The two former are pre-configured commands +for frontend and backend plugins, while the `build` command provides more +control over the output. There are three different possible outputs of the build process: JavaScript in -CommonJS module format, JavaScript in ESM format, and type declarations. Each -invocation of a build command will write one or more of these outputs to the -`dist` folder in the package, and in addition copy any asset files like -stylesheets or images. For more details on what syntax and file formats are -supported by the build process, see the transpilation section below. +CommonJS module format, JavaScript in ECMAScript module format, and type +declarations. Each invocation of a build command will write one or more of these +outputs to the `dist` folder in the package, and in addition copy any asset +files like stylesheets or images. For more details on what syntax and file +formats are supported by the build process, see the [loaders section](#loaders). When building CommonJS or ESM output, the build commands will always use `src/index.ts` as the entrypoint. All dependencies of the package will be marked as external, meaning that in general it is only the contents of the `src` folder -that ends up being compiled and output to `dist`, any imported external -dependencies, even within the monorepo, will stay intact. +that ends up being compiled and output to `dist`. All import statements of +external dependencies, even within the same monorepo, will stay intact. The +externalized dependencies are based on dependency information in `package.json`, +which means it's important to keep it up to date. The build of the type definitions works quite differently. The entrypoint of the type definition build is the relative location of the package within the `dist-types` folder in the project root. This means that it is important to run type checking before building any packages with type definitions, and that -emitting type declarations must be enable in the TypeScript configuration. The +emitting type declarations must be enabled in the TypeScript configuration. The reason for the type definition build step is to strip out all types but the ones that are exported from the package, leaving a much cleaner type definition file and making sure that the type definitions are in sync with the generated @@ -201,14 +203,14 @@ JavaScript. The goal of the bundling process is to combine multiple packages together into a single runtime unit. The way this is done varies between frontend and backend, -and local development versus production deployment. Because of that we will be -covering each of these cases individually. +as well as local development versus production deployment. Because of that we +cover each combination of these cases separately. #### Frontend Development Bundling There are two different commands that starts the frontend development bundling, -`app:serve`, which serves an app and uses `src/index.*` as the entrypoint, and -`plugin:serve`, which serves a plugin and uses `dev/index.*` as the entrypoint. +`app:serve`, which serves an app and uses `src/index` as the entrypoint, and +`plugin:serve`, which serves a plugin and uses `dev/index` as the entrypoint. These are typically invoked via the `yarn start` script, and are intended for local development only. When running the bundle command, a development server will be set up that listens to the protocol, host and port set by `app.baseUrl` @@ -221,15 +223,15 @@ The frontend development bundling is currently based on Webpack configuration itself varies very little between the frontend development and production bundling, so we'll dive more into the configuration in the production section below. The main differences are that `process.env.NODE_ENV` -is set to `'development'`, minification is disabled, and there is support for -[React Hot Loader](https://github.com/gaearon/react-hot-loader). +is set to `'development'`, minification is disabled, cheap source maps are used, +and [React Hot Loader](https://github.com/gaearon/react-hot-loader) is enabled. If you prefer to run type checking and linting as part of the Webpack process, you can enable usage of the [`ForkTsCheckerWebpackPlugin`](https://www.npmjs.com/package/fork-ts-checker-webpack-plugin) -by passing the `--check` flag. However, the recommended way to handle these -checks is to use an editor that has built-in support for them, as well as -running them in CI and the occasional manual run when needed. +by passing the `--check` flag. Although as mentioned above, the recommended way +to handle these checks during development is to use an editor that has built-in +support for them instead. #### Frontend Production Bundling @@ -244,7 +246,7 @@ Just like the development bundling, the production bundling is based on [`HtmlWebpackPlugin`](https://webpack.js.org/plugins/html-webpack-plugin/) to generate the `index.html` entry point, and includes a default template that's included with the CLI. You can replace the bundled template by adding -`public/index.html` to your package. The template has access to two global +`public/index.html` to your app package. The template has access to two global constants, `publicPath` which is the public base path that the bundle is intended to be served at, as well as `config` which is your regular frontend scoped configuration from `@backstage/config`. @@ -252,27 +254,28 @@ scoped configuration from `@backstage/config`. The Webpack configuration also includes a custom plugin for resolving packages correctly from linked in packages, the `ModuleScopePlugin` from [`react-dev-utils`](https://www.npmjs.com/package/react-dev-utils) which makes -sure that imports don't react outside the package, a few fallbacks for some +sure that imports don't reach outside the package, a few fallbacks for some Node.js modules like `'buffer'` and `'events'`, a plugin that writes the frontend configuration to the bundle as `process.env.APP_CONFIG` and build information as `process.env.BUILD_INFO`, and lastly minification handled by [esbuild](https://esbuild.github.io/) using the [`esbuild-loader`](https://npm.im/esbuild-loader). There are of course also a set of loaders configured, which you can read more about in the -[transpilation](#transpilation) and [loaders](#loaders) sections. +[loaders](#loaders) and [transpilation](#transpilation) sections. -The output of the bundling process is split into two types of files. The first -is a set of generic assets with plain names in the root of the `dist/` folder. -You will want to serve these with short-lived caching or no caching at all. The -second is a set of hashed static assets in the `dist/static/`, which you can -configure to be cached for a much longer time. +The output of the bundling process is split into two categories of files with +separate caching strategies. The first is a set of generic assets with plain +names in the root of the `dist/` folder. You will want to serve these with +short-lived caching or no caching at all. The second is a set of hashed static +assets in the `dist/static/` folder, which you can configure to be cached for a +much longer time. The configuration of static assets is optimized for frequent changes and serving over HTTP 2.0. The assets are aggressively split into small chunks, which means the browser has to make a lot of small requests to load them. The upside is that changes to individual plugins and packages will invalidate a smaller number of -files, thereby allowing for rapid development without too much impact on the -page load performance. +files, thereby allowing for rapid development without much impact on the page +load performance. #### Backend Development Bundling @@ -294,8 +297,8 @@ module fallbacks, and it uses to avoid bundling in dependency modules. If you want to inspect the running Node.js process, the `--inspect` and -`--inspect-brk` flags can be used, as they will be passed through to the backend -process. +`--inspect-brk` flags can be used, as they will be passed through as options to +`node` execution. #### Backend Production Bundling @@ -304,27 +307,29 @@ bundling options. Rather than using Webpack, the backend production bundling instead collects the backend packages and all of its local dependencies into a deployment archive. The archive is written to `dist/bundle.tar.gz`, and contains the packaged version of each of these packages. The layout of the packages in -the archive is the same as the directory layout in the target monorepo, and the -bundle also contains the root `package.json` and `yarn.lock` files. +the archive is the same as the directory layout in the monorepo, and the bundle +also contains the root `package.json` and `yarn.lock` files. -Note that before building the production bundle, all of the backend packages -have to have been built first, although when executing the `backend:bundle` -command you can pass the `--build-dependencies` to have this done automatically. -The reason for not using the flag is to avoid duplicate work in case you already -build all of your packages earlier in your build process. +Note that before creating a production bundle you must first build all of the +backend packages. This can be done automatically when executing the +`backend:bundle` command by passing the `--build-dependencies` flag. It is an +optional flag since it is quite common that the packages are already build +earlier on in your build process, and building them again would result in +duplicate work. -To use the bundle, extract it into a directory, run `yarn install --production`, -and then start the backend using your backend package as the Node.js entry -point, for example `node packages/backend`. +In order to use the bundle, you extract it into a directory, run +`yarn install --production`, and then start the backend using your backend +package as the Node.js entry point, for example `node packages/backend`. The `dist/bundle.tar.gz` is accompanied by a `dist/skeleton.tar.gz`, which has -the same layout, but only contains `package.json` files. This can be used to run -a `yarn install` in environments that will benefit from the caching that this -enables, such as Docker image builds. To use the skeleton archive you copy it -over to the target directory along with the root `package.json` and `yarn.lock`, -extract the archive, and then run `yarn install`. Your target directory now has -all dependencies installed, and as soon as you copy over and extract the -contents of the `bundle.tar.gz` archive, the backend will be ready to run. +the same layout, but only contains `package.json` files. This skeleton archive +can be used to run a `yarn install` in environments that will benefit from the +caching that this enables, such as Docker image builds. To use the skeleton +archive you copy it over to the target directory along with the root +`package.json` and `yarn.lock`, extract the archive, and then run +`yarn install --production`. Your target directory will then have all +dependencies installed, and as soon as you copy over and extract the contents of +the `bundle.tar.gz` archive on top of it, the backend will be ready to run. The following is an example of a `Dockerfile` that can be used to package the output of `backstage-cli backend:bundle` into an image: @@ -346,39 +351,40 @@ CMD ["node", "packages/backend"] ## Transpilation -The transpilers used by the Backstage CLI have been chosen according to the -design considerations that we mentioned above. A few specific requirements are +The transpilers used by the Backstage CLI have been chosen according to the same +design considerations that were mentioned above. A few specific requirements are of course support for TypeScript and JSX, but also React hot reloads or refresh, and hoisting of Jest mocks. The Backstage CLI also only targets up to date and modern browsers, so we actually want to keep the transpilation process as lightweight as possible, and leave most syntax intact. -Apart from these requirements, we choose transpilers based on their speed. The -build process keeps the integration with the transpilers lightweight, without -additional plugins or such. This enables us to switch out transpilers as new -options and optimizations become available, and use the fastest options that are -available. +Apart from these requirements, the deciding factor for which transpiler to use +is their speed. The build process keeps the integration with the transpilers +lightweight, without additional plugins or such. This enables us to switch out +transpilers as new options and optimizations become available, and keep use the +best options that are available. -Our current selection of transpilers are [esbuild][https://esbuild.github.io/] +Our current selection of transpilers are [esbuild](https://esbuild.github.io/) and [Sucrase](https://github.com/alangpierce/sucrase). The reason we choose to use two transpilers is that esbuild is faster than Sucrase and produces slightly -nicer output, but it does not have the same feature set, and for example does -not support React hot reloading or hoisting of Jest mocks. +nicer output, but it does not have the same set of features, for example it does +not support React hot reloading. -The benchmarking of the various options has been done in the -[ts-build-bench](https://github.com/Rugvip/ts-build-bench) project. It allows -for setups of different shapes and sizes of monorepos, but the setup we consider -the most important during benchmarking is a large monorepo with lots of medium -to large packages that's being bundled with Webpack. Some rough findings have -been that esbuild is the fastest option right now, with Sucrase following -closely after and then [SWC](https://swc.rs/) closely after that. After those -there's a pretty big gap up to the TypeScript compiler run in transpilation only -mode, and lastly another jump up to Babel, being by far the slowest. +The benchmarking of the various options was done in +[ts-build-bench](https://github.com/Rugvip/ts-build-bench). This benchmarking +project allows for setups of different shapes and sizes of monorepos, but the +setup we consider the most important in our case is a large monorepo with lots +of medium to large packages that are being bundled with Webpack. Some rough +findings have been that esbuild is the fastest option right now, with Sucrase +following closely after and then [SWC](https://swc.rs/) closely after that. +After those there's a pretty big gap up to the TypeScript compiler run in +transpilation only mode, and lastly another jump up to Babel, being by far the +slowest out of the transpilers we tested. Something to note about these benchmarks is that they take the full Webpack bundling time into account. This means that even though some transpilation options may be orders of magnitude faster than others, the total time is not -impacted in the same way as there's a lot of other things that go into the +impacted in the same way as there are lots of other things that go into the bundling process. Still, switching from for example Babel to Sucrase is able to make the bundling anywhere from two to five times faster. @@ -412,3 +418,7 @@ of all supported file extensions: | `.png` | URL Path | Image | | `.svg` | URL Path | Image | | `.icon.svg` | React Component | SVG converted into a [MUI SvgIcon](https://mui.com/components/icons/#svgicon) | + +## Publishing + +## Jest Configuration From 6bc3d749e88419e93f9cb1afaf66346a047f4616 Mon Sep 17 00:00:00 2001 From: Patrik Oldsberg Date: Sun, 16 Jan 2022 21:58:37 +0100 Subject: [PATCH 11/14] docs: add build system publishing and jest config sections Signed-off-by: Patrik Oldsberg --- docs/local-dev/cli-build-system.md | 85 ++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) diff --git a/docs/local-dev/cli-build-system.md b/docs/local-dev/cli-build-system.md index 982e5952da..ebcae7069c 100644 --- a/docs/local-dev/cli-build-system.md +++ b/docs/local-dev/cli-build-system.md @@ -421,4 +421,89 @@ of all supported file extensions: ## Publishing +Package publishing is an optional part of the Backstage build system and not +something you well need to worry unless you are publishing packages to a +registry. In order to publish a package, you first need to build it, which will +populate the `dist` folder. Because the Backstage build system is optimized for +local development along with our particular TypeScript and bundling setup, it is +not possible to publish package immediately at this point. This is because the +entry points of the package will still be pointing to `src/index.ts`, but we +want them to point to `dist/` in the published package. + +In order to work around this, the Backstage CLI provides `prepack` and +`postpack` commands that help prepare the package for publishing. These scripts +are automatically run by Yarn before publishing a package. + +The `prepack` command will take entry point fields in `"publishConfig"`, such +as, `"main"` and `"module"`, and move the top level of the `package.json`. This +lets you point at the desired files in the `dist` folder during publishing. The +`postpack` command will simply revert this change in order to leave your project +clean. + +The following is an excerpt of a typical setup of an isomorphic library package: + +```json + "main": "src/index.ts", + "types": "src/index.ts", + "publishConfig": { + "access": "public", + "main": "dist/index.cjs.js", + "module": "dist/index.esm.js", + "types": "dist/index.d.ts" + }, + "scripts": { + "build": "backstage-cli build", + "lint": "backstage-cli lint", + "test": "backstage-cli test", + "prepack": "backstage-cli prepack", + "postpack": "backstage-cli postpack", + "clean": "backstage-cli clean" + }, + "files": ["dist"], +``` + ## Jest Configuration + +The Backstage CLI bundles its own Jest configuration file, which is used +automatically when running `backstage-cli test`. It's available at +`@backstage/cli/config/jest.js` and can be inspected +[here](https://github.com/backstage/backstage/blob/master/packages/cli/config/jest.js). +Usage of this configuration can be overridden either by passing a +`--config ` flag to `backstage-cli test`, or placing a `jest.config.js` or +`jest.config.ts` file in your package. + +The build-in configuration brings a couple of benefits and features. The most +important one being a baseline transformer and module configuration that enables +support for the listed [loaders](#loaders) within tests. It will also +automatically detect and uses `src/setupTests.ts` if it exists, and provide a +coverage configuration that works well with our selected transpilers. + +The configuration also takes a project-wide approach, with the expectation most +if not all packages within a monorepo will use the same base configuration. This +allows for optimizations such as sharing the Jest transform cache across all +packages in a monorepo, avoiding unnecessary transpilation. It also makes it +possible to load in all Jest configurations at once, and with that run +`yarn test ` from the root of a monorepo without having to set the +working directory to the package that the test is in. + +Where small customizations are needed, such as setting coverage thresholds or +support for specific transforms, it is possible to override the Jest +configuration through the `"jest"` in `package.json`. These overrides will be +loaded in from all `package.json` files in the directory ancestry, meaning that +you can place common configuration in the `package.json` at the root of a +monorepo. If multiple overrides are found, they will be merged together with +configuration further down in the directory tree taking precedence. + +The overrides in a single `package.json` may for example look like this: + +```json + "jest": { + "coverageThreshold": { + "global": { + "functions": 100, + "lines": 100, + "statements": 100 + } + } + }, +``` From d5f3d84817aa4b10def59299975b5914a79dceeb Mon Sep 17 00:00:00 2001 From: Patrik Oldsberg Date: Mon, 17 Jan 2022 09:58:36 +0100 Subject: [PATCH 12/14] docs: merge build system docs with cli docs + restructure Signed-off-by: Patrik Oldsberg --- docs/local-dev/cli-build-system.md | 111 ++++++++++++++--------------- docs/local-dev/cli-commands.md | 2 +- docs/local-dev/cli-overview.md | 104 +++++++-------------------- microsite/sidebars.json | 2 +- mkdocs.yml | 2 +- 5 files changed, 82 insertions(+), 139 deletions(-) diff --git a/docs/local-dev/cli-build-system.md b/docs/local-dev/cli-build-system.md index ebcae7069c..1a3a2f511b 100644 --- a/docs/local-dev/cli-build-system.md +++ b/docs/local-dev/cli-build-system.md @@ -4,8 +4,6 @@ title: Build System description: A deep dive into the Backstage build system --- -## What is the Backstage build system? - The Backstage build system is a collection of build and development tools that help you lint, test, develop and finally release your Backstage projects. The purpose of the build system is to provide an out-of-the-box solution that works @@ -43,6 +41,7 @@ In addition, there are a number of hard and soft requirements: - Simple - Usage should simple and configuration should be kept minimal - Universal - Development towards both web applications, isomorphic packages, and Node.js +- Modern - The build system targets modern environments - Editors - Type checking and linting should be available within most editors During the design of the build system this collection of requirements was not @@ -50,7 +49,7 @@ something that was supported by existing tools like for example `react-scripts`. The requirements of scaling in combination of monorepo, publishing, and editor support led us to adopting our own specialized setup. -## Architecture +## Structure We can divide the development flow within Backstage into a couple of different steps: @@ -73,7 +72,7 @@ or IDE that has support for formatting, linting, and type checking. Let's dive into a detailed look at each of these steps and how they are implemented in a typical Backstage app. -### Formatting +## Formatting The formatting setup lives completely within each Backstage application and is not part of the CLI. In an app created with `@backstage/create-app` the @@ -81,7 +80,7 @@ formatting is handled by [prettier](https://prettier.io/), but each application can those their own formatting rules and switch to a different formatter if desired. -### Linting +## Linting The Backstage CLI includes a `lint` command, which is a thin wrapper around `eslint`. It adds a few options that can't be set through configuration, such as @@ -101,7 +100,7 @@ the entire project. Each configuration is initially one that simply extends a base configuration provided by the Backstage CLI, but they can be customized to fit the needs of each package. -### Type Checking +## Type Checking Just like formatting, the Backstage CLI does not have its own command for type checking. It does however have a base configuration with both recommended @@ -135,7 +134,7 @@ through the `tsc:full` script, which will run a full type check, including For the two reasons mentioned above, it is **highly** recommended to use the `tsc:full` script to run type checking in CI. -### Testing +## Testing As mentioned above, the Backstage CLI uses [Jest](https://jestjs.io/), which is a JavaScript test framework that covers both test execution and assertions. Jest @@ -158,7 +157,7 @@ configuration, as well as allowing for configuration overrides to be defined in each `package.json`. How this can be done in practice is discussed in the [Jest configuration](#jest-configuration) section. -### Building +## Building The primary purpose of the build process is to prepare packages for publishing, but it's also used as part of the backend bundling process. Since it's only used @@ -199,14 +198,14 @@ that are exported from the package, leaving a much cleaner type definition file and making sure that the type definitions are in sync with the generated JavaScript. -### Bundling +## Bundling The goal of the bundling process is to combine multiple packages together into a single runtime unit. The way this is done varies between frontend and backend, as well as local development versus production deployment. Because of that we cover each combination of these cases separately. -#### Frontend Development Bundling +### Frontend Development There are two different commands that starts the frontend development bundling, `app:serve`, which serves an app and uses `src/index` as the entrypoint, and @@ -233,7 +232,7 @@ by passing the `--check` flag. Although as mentioned above, the recommended way to handle these checks during development is to use an editor that has built-in support for them instead. -#### Frontend Production Bundling +### Frontend Production The frontend production bundling creates your typical web content bundle, all contained within a single folder, ready for static serving. It is invoked using @@ -277,7 +276,7 @@ changes to individual plugins and packages will invalidate a smaller number of files, thereby allowing for rapid development without much impact on the page load performance. -#### Backend Development Bundling +### Backend Development The backend development bundling is also based on Webpack, but rather than starting up a web server, the backend is started up using the @@ -300,7 +299,7 @@ If you want to inspect the running Node.js process, the `--inspect` and `--inspect-brk` flags can be used, as they will be passed through as options to `node` execution. -#### Backend Production Bundling +### Backend Production The backend production bundling uses a completely different setup than the other bundling options. Rather than using Webpack, the backend production bundling @@ -419,49 +418,6 @@ of all supported file extensions: | `.svg` | URL Path | Image | | `.icon.svg` | React Component | SVG converted into a [MUI SvgIcon](https://mui.com/components/icons/#svgicon) | -## Publishing - -Package publishing is an optional part of the Backstage build system and not -something you well need to worry unless you are publishing packages to a -registry. In order to publish a package, you first need to build it, which will -populate the `dist` folder. Because the Backstage build system is optimized for -local development along with our particular TypeScript and bundling setup, it is -not possible to publish package immediately at this point. This is because the -entry points of the package will still be pointing to `src/index.ts`, but we -want them to point to `dist/` in the published package. - -In order to work around this, the Backstage CLI provides `prepack` and -`postpack` commands that help prepare the package for publishing. These scripts -are automatically run by Yarn before publishing a package. - -The `prepack` command will take entry point fields in `"publishConfig"`, such -as, `"main"` and `"module"`, and move the top level of the `package.json`. This -lets you point at the desired files in the `dist` folder during publishing. The -`postpack` command will simply revert this change in order to leave your project -clean. - -The following is an excerpt of a typical setup of an isomorphic library package: - -```json - "main": "src/index.ts", - "types": "src/index.ts", - "publishConfig": { - "access": "public", - "main": "dist/index.cjs.js", - "module": "dist/index.esm.js", - "types": "dist/index.d.ts" - }, - "scripts": { - "build": "backstage-cli build", - "lint": "backstage-cli lint", - "test": "backstage-cli test", - "prepack": "backstage-cli prepack", - "postpack": "backstage-cli postpack", - "clean": "backstage-cli clean" - }, - "files": ["dist"], -``` - ## Jest Configuration The Backstage CLI bundles its own Jest configuration file, which is used @@ -507,3 +463,46 @@ The overrides in a single `package.json` may for example look like this: } }, ``` + +## Publishing + +Package publishing is an optional part of the Backstage build system and not +something you well need to worry unless you are publishing packages to a +registry. In order to publish a package, you first need to build it, which will +populate the `dist` folder. Because the Backstage build system is optimized for +local development along with our particular TypeScript and bundling setup, it is +not possible to publish package immediately at this point. This is because the +entry points of the package will still be pointing to `src/index.ts`, but we +want them to point to `dist/` in the published package. + +In order to work around this, the Backstage CLI provides `prepack` and +`postpack` commands that help prepare the package for publishing. These scripts +are automatically run by Yarn before publishing a package. + +The `prepack` command will take entry point fields in `"publishConfig"`, such +as, `"main"` and `"module"`, and move the top level of the `package.json`. This +lets you point at the desired files in the `dist` folder during publishing. The +`postpack` command will simply revert this change in order to leave your project +clean. + +The following is an excerpt of a typical setup of an isomorphic library package: + +```json + "main": "src/index.ts", + "types": "src/index.ts", + "publishConfig": { + "access": "public", + "main": "dist/index.cjs.js", + "module": "dist/index.esm.js", + "types": "dist/index.d.ts" + }, + "scripts": { + "build": "backstage-cli build", + "lint": "backstage-cli lint", + "test": "backstage-cli test", + "prepack": "backstage-cli prepack", + "postpack": "backstage-cli postpack", + "clean": "backstage-cli clean" + }, + "files": ["dist"], +``` diff --git a/docs/local-dev/cli-commands.md b/docs/local-dev/cli-commands.md index 1168a86e3a..daf53ac0b2 100644 --- a/docs/local-dev/cli-commands.md +++ b/docs/local-dev/cli-commands.md @@ -1,6 +1,6 @@ --- id: cli-commands -title: CLI Commands +title: Commands description: Descriptions of all commands available in the CLI. --- diff --git a/docs/local-dev/cli-overview.md b/docs/local-dev/cli-overview.md index de9e378632..24dd64d695 100644 --- a/docs/local-dev/cli-overview.md +++ b/docs/local-dev/cli-overview.md @@ -1,97 +1,41 @@ --- id: cli-overview -title: CLI Overview +title: Overview description: Overview of the Backstage CLI --- -## Summary - -Backstage provides an opinionated set of tooling for both frontend and backend -development. It is delivered through the -[`@backstage/cli`](https://www.npmjs.com/package/@backstage/cli) package and -executed either directly through `yarn backstage-cli ` or within -`package.json` scripts. When creating an app using -[`@backstage/create-app`](https://www.npmjs.com/package/@backstage/create-app) -it contains package scripts for executing the most common commands. - -Under the hood the CLI uses [Webpack](https://webpack.js.org/) for bundling, -[Rollup](https://rollupjs.org/) for building packages, -[Jest](https://jestjs.io/) for testing, and [eslint](https://eslint.org/) for -linting. It also includes custom tooling for working within Backstage apps, for -example for keeping the app up to date and verifying static configuration. - -For a full list of CLI commands, see the [commands](./cli-commands.md) page. - ## Introduction A goal of Backstage is to provide a delightful developer experience in and around the project. Creating new apps and plugins should be simple, iteration speed should be fast, and the overhead of maintaining custom tooling should be -minimal. As a part of accomplishing this goal, Backstage provides its own set of -opinionated tooling, delivered primarily through the -[`@backstage/cli`](https://www.npmjs.com/package/@backstage/cli) package. - -The `@backstage/cli` package provides a single executable script, -`backstage-cli`, which you can run directly with `yarn` or within a script in -`package.json`. If you have a Backstage app set up, you can try out the -following command to print the top-level help page of the CLI: - -```text -yarn backstage-cli --help -``` - -If you are familiar with [`create-react-app`](https://create-react-app.dev/) you -may recognize the pattern of bundling tooling up as a CLI, as it uses a package -called [`react-scripts`](https://www.npmjs.com/package/react-scripts) to bring -most of the functionality into the created project. The Backstage equivalent of -`create-react-app` is +minimal. As a part of accomplishing this goal, Backstage provides its own build +system and tooling, delivered primarily through the +[`@backstage/cli`](https://www.npmjs.com/package/@backstage/cli) package. When +creating an app using [`@backstage/create-app`](https://www.npmjs.com/package/@backstage/create-app), -and the equivalent of `react-scripts` is `@backstage/cli`. There are however a -couple of key differences between the two. Most notably, Backstage apps are -monorepos and the CLI is tailored for that environment. It provides tooling both -for bundling and developing full end-user apps, but also for developing, -building and publishing individual packages within the monorepo, as well as -tooling that is more unique to Backstage, such as commands for working with -static configuration. +you receive a project that's already prepared with a typical setup and package +scripts for executing the most common commands. -## Opinionated Tooling +Under the hood the CLI uses [Webpack](https://webpack.js.org/) for bundling, +[Rollup](https://rollupjs.org/) for building packages, +[Jest](https://jestjs.io/) for testing, and [eslint](https://eslint.org/) for +linting. It also includes tooling for working within Backstage apps, for example +for keeping the app up to date and verifying static configuration. For a more +in-depth look info the tooling, see the [build system](./cli-build-system.md) +page, and for a list of commands, see the [commands](./cli-commands.md) page. -The Backstage CLI is highly opinionated in what tools are used and how they are -configured. It is tailored for development in large TypeScript monorepos with -hundreds of separate packages, but with the ability to have edits anywhere in -the codebase reflected within a few seconds. The build output is also optimized -for this setup, and aims to provide an excellent user experience with fast page -load times in modern browsers, rather than a wide range of support. +While the Backstage tooling is opinionated in how it works, it is also possible +to use your own tooling either partially or fully. For example, the CLI provides +a command for building a plugin package for publishing, but the output is a +quite standard combination of transpiled JavaScript and TypeScript type +declarations. The usage of the command from the CLI can therefore be augmented +or replaced with other tools if necessary. -While the Backstage tooling is opinionated in how to develop and build packages, -it is also possible to use your own tooling either partially or fully. For -example, the CLI provides a command for building a plugin package for -publishing, but the output is a quite standard combination of transpiled -JavaScript and TypeScript type declarations. The usage of the command from the -CLI can therefore easily be replaced with other tools if necessary. - -Just like `react-scripts`, the Backstage CLI does not provide many hooks for -overriding or customizing the build process. This is to allow for evolution of -the CLI without having to take a wide API surface into account. This allows us -to quickly iterate and improve the tooling, as well as to more easily keep -dependencies up to date. - -## Opinions & Goals - -In no particular order, this is a list of opinions and goals that guide the -design and development of the Backstage CLI: - -- All you need for development is `yarn start`, there should be no need to - manually build packages or run other separate tasks. -- Development experience comes first. The toolchain is optimized for keeping - development smooth, rather than making it easy to for example build and - publish packages. -- Type checking and linting is left for text editors and Continuous Integration. - Most text editors provide tooling for these checks, and running them a second - time during compilation slows down iteration speed and consumes more system - resources. -- Backstage is run in modern browsers. We keep transpilation lightweight and - rely on modern technologies such as HTTP/2 to optimize frontend speed. +The Backstage CLI intentionally does not provide many hooks for overriding or +customizing the build process. This is to allow for evolution of the CLI without +having to take a wide API surface into account. This allows us to iterate and +improve the tooling, as well as to more easily keep the system up to date. ## Glossary diff --git a/microsite/sidebars.json b/microsite/sidebars.json index 3ff59b7f9d..b1f525937d 100644 --- a/microsite/sidebars.json +++ b/microsite/sidebars.json @@ -34,7 +34,7 @@ { "type": "subcategory", "label": "CLI", - "ids": ["local-dev/cli-overview", "local-dev/cli-commands", "local-dev/cli-build-system"] + "ids": ["local-dev/cli-overview", "local-dev/cli-build-system", "local-dev/cli-commands"] }, "local-dev/linking-local-packages" ], diff --git a/mkdocs.yml b/mkdocs.yml index 6974dd6c79..4dd3f239e8 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -30,8 +30,8 @@ nav: - Local Development: - CLI: - Overview: 'local-dev/cli-overview.md' - - Commands: 'local-dev/cli-commands.md' - Build System: 'local-dev/cli-build-system.md' + - Commands: 'local-dev/cli-commands.md' - Linking in Local Packages: 'local-dev/linking-local-packages.md' - Core Features: - Software Catalog: From 63945e3e85c3daa42c9947d87f2d43dc1df970ac Mon Sep 17 00:00:00 2001 From: Patrik Oldsberg Date: Mon, 17 Jan 2022 10:33:26 +0100 Subject: [PATCH 13/14] microsite: format sidebars.json Signed-off-by: Patrik Oldsberg --- microsite/sidebars.json | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/microsite/sidebars.json b/microsite/sidebars.json index b1f525937d..b3223910ed 100644 --- a/microsite/sidebars.json +++ b/microsite/sidebars.json @@ -34,7 +34,11 @@ { "type": "subcategory", "label": "CLI", - "ids": ["local-dev/cli-overview", "local-dev/cli-build-system", "local-dev/cli-commands"] + "ids": [ + "local-dev/cli-overview", + "local-dev/cli-build-system", + "local-dev/cli-commands" + ] }, "local-dev/linking-local-packages" ], From a6df0ff8bc31435c8f6a642f319bec9d878b449b Mon Sep 17 00:00:00 2001 From: Patrik Oldsberg Date: Mon, 17 Jan 2022 17:58:00 +0100 Subject: [PATCH 14/14] docs: build system review fixes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Fredrik Adelöw Signed-off-by: Patrik Oldsberg --- docs/local-dev/cli-build-system.md | 54 +++++++++++++++--------------- docs/local-dev/cli-overview.md | 2 +- 2 files changed, 28 insertions(+), 28 deletions(-) diff --git a/docs/local-dev/cli-build-system.md b/docs/local-dev/cli-build-system.md index 1a3a2f511b..fdfede6e05 100644 --- a/docs/local-dev/cli-build-system.md +++ b/docs/local-dev/cli-build-system.md @@ -77,7 +77,7 @@ implemented in a typical Backstage app. The formatting setup lives completely within each Backstage application and is not part of the CLI. In an app created with `@backstage/create-app` the formatting is handled by [prettier](https://prettier.io/), but each application -can those their own formatting rules and switch to a different formatter if +can choose their own formatting rules and switch to a different formatter if desired. ## Linting @@ -95,10 +95,10 @@ configurations in turn build on top of the lint rules from [@spotify/web-scripts](https://github.com/spotify/web-scripts). In a standard Backstage setup, each individual package has its own lint -configuration, along with that there's also a root configuration that applies to -the entire project. Each configuration is initially one that simply extends a -base configuration provided by the Backstage CLI, but they can be customized to -fit the needs of each package. +configuration, along with a root configuration that applies to the entire +project. Each configuration is initially one that simply extends a base +configuration provided by the Backstage CLI, but they can be customized to fit +the needs of each package. ## Type Checking @@ -108,7 +108,7 @@ defaults as well as some required settings for the build system to work. Perhaps the most notable part about the TypeScript setup in Backstage projects is that the entire project is one big compilation unit. This is due to -performance optimization as well as easy of use, since breaking projects down +performance optimization as well as ease of use, since breaking projects down into smaller pieces has proven to both lead to a more complicated setup, as well as type checking of the entire project being an order of magnitude slower. In order to make this setup work, the entrypoint of each package needs to point to @@ -144,7 +144,7 @@ environments, such as one based on [`jsdom`](https://github.com/jsdom/jsdom) that helps mimic browser APIs and behavior. The Backstage CLI has its own command that helps execute tests, -`backstage-cli test`, as well as it's own configuration at +`backstage-cli test`, as well as its own configuration at `@backstage/cli/config/jest.js`. The command is a relatively thin wrapper around running `jest` directly. Its main responsibility is to make sure the included configuration is used, as well setting the `NODE_ENV` and `TZ` environment @@ -207,7 +207,7 @@ cover each combination of these cases separately. ### Frontend Development -There are two different commands that starts the frontend development bundling, +There are two different commands that start the frontend development bundling: `app:serve`, which serves an app and uses `src/index` as the entrypoint, and `plugin:serve`, which serves a plugin and uses `dev/index` as the entrypoint. These are typically invoked via the `yarn start` script, and are intended for @@ -281,9 +281,9 @@ load performance. The backend development bundling is also based on Webpack, but rather than starting up a web server, the backend is started up using the [`RunScriptWebpackPlugin`](https://www.npmjs.com/package/run-script-webpack-plugin). -The reason for using Webpack for development of the backend is both that it is +The reason for using Webpack for development of the backend is both that it is a convenient way to handle transpilation of a large set of packages, as well us -allowing us to use hot module replacement and maintain state while reloading +allowing us to use hot module replacement and maintaining state while reloading individual backend modules. This is particularly useful when running the backend with in-memory SQLite as the database choice. @@ -303,7 +303,7 @@ If you want to inspect the running Node.js process, the `--inspect` and The backend production bundling uses a completely different setup than the other bundling options. Rather than using Webpack, the backend production bundling -instead collects the backend packages and all of its local dependencies into a +instead collects the backend packages and all of their local dependencies into a deployment archive. The archive is written to `dist/bundle.tar.gz`, and contains the packaged version of each of these packages. The layout of the packages in the archive is the same as the directory layout in the monorepo, and the bundle @@ -312,7 +312,7 @@ also contains the root `package.json` and `yarn.lock` files. Note that before creating a production bundle you must first build all of the backend packages. This can be done automatically when executing the `backend:bundle` command by passing the `--build-dependencies` flag. It is an -optional flag since it is quite common that the packages are already build +optional flag since it is quite common that the packages are already built earlier on in your build process, and building them again would result in duplicate work. @@ -360,8 +360,8 @@ lightweight as possible, and leave most syntax intact. Apart from these requirements, the deciding factor for which transpiler to use is their speed. The build process keeps the integration with the transpilers lightweight, without additional plugins or such. This enables us to switch out -transpilers as new options and optimizations become available, and keep use the -best options that are available. +transpilers as new options and optimizations become available, and keep on using +the best options that are available. Our current selection of transpilers are [esbuild](https://esbuild.github.io/) and [Sucrase](https://github.com/alangpierce/sucrase). The reason we choose to @@ -428,10 +428,10 @@ Usage of this configuration can be overridden either by passing a `--config ` flag to `backstage-cli test`, or placing a `jest.config.js` or `jest.config.ts` file in your package. -The build-in configuration brings a couple of benefits and features. The most +The built-in configuration brings a couple of benefits and features. The most important one being a baseline transformer and module configuration that enables support for the listed [loaders](#loaders) within tests. It will also -automatically detect and uses `src/setupTests.ts` if it exists, and provide a +automatically detect and use `src/setupTests.ts` if it exists, and provides a coverage configuration that works well with our selected transpilers. The configuration also takes a project-wide approach, with the expectation most @@ -444,9 +444,9 @@ working directory to the package that the test is in. Where small customizations are needed, such as setting coverage thresholds or support for specific transforms, it is possible to override the Jest -configuration through the `"jest"` in `package.json`. These overrides will be -loaded in from all `package.json` files in the directory ancestry, meaning that -you can place common configuration in the `package.json` at the root of a +configuration through the `"jest"` field in `package.json`. These overrides will +be loaded in from all `package.json` files in the directory ancestry, meaning +that you can place common configuration in the `package.json` at the root of a monorepo. If multiple overrides are found, they will be merged together with configuration further down in the directory tree taking precedence. @@ -467,23 +467,23 @@ The overrides in a single `package.json` may for example look like this: ## Publishing Package publishing is an optional part of the Backstage build system and not -something you well need to worry unless you are publishing packages to a +something you will need to worry about unless you are publishing packages to a registry. In order to publish a package, you first need to build it, which will populate the `dist` folder. Because the Backstage build system is optimized for local development along with our particular TypeScript and bundling setup, it is -not possible to publish package immediately at this point. This is because the -entry points of the package will still be pointing to `src/index.ts`, but we +not possible to publish the package immediately at this point. This is because +the entry points of the package will still be pointing to `src/index.ts`, but we want them to point to `dist/` in the published package. In order to work around this, the Backstage CLI provides `prepack` and `postpack` commands that help prepare the package for publishing. These scripts are automatically run by Yarn before publishing a package. -The `prepack` command will take entry point fields in `"publishConfig"`, such -as, `"main"` and `"module"`, and move the top level of the `package.json`. This -lets you point at the desired files in the `dist` folder during publishing. The -`postpack` command will simply revert this change in order to leave your project -clean. +The `prepack` command will take entry point fields in `"publishConfig"`, such as +`"main"` and `"module"`, and move them to the top level of the `package.json`. +This lets you point at the desired files in the `dist` folder during publishing. +The `postpack` command will simply revert this change in order to leave your +project clean. The following is an excerpt of a typical setup of an isomorphic library package: diff --git a/docs/local-dev/cli-overview.md b/docs/local-dev/cli-overview.md index 24dd64d695..9e6884aab0 100644 --- a/docs/local-dev/cli-overview.md +++ b/docs/local-dev/cli-overview.md @@ -22,7 +22,7 @@ Under the hood the CLI uses [Webpack](https://webpack.js.org/) for bundling, [Jest](https://jestjs.io/) for testing, and [eslint](https://eslint.org/) for linting. It also includes tooling for working within Backstage apps, for example for keeping the app up to date and verifying static configuration. For a more -in-depth look info the tooling, see the [build system](./cli-build-system.md) +in-depth look into the tooling, see the [build system](./cli-build-system.md) page, and for a list of commands, see the [commands](./cli-commands.md) page. While the Backstage tooling is opinionated in how it works, it is also possible