paulononato/backstage
1
0
Fork 0
Code Issues Pull Requests Actions 44 Packages Projects Releases Wiki Activity

Merge pull request #27299 from backstage/rugvip/linking

Browse Source 
cli: new solution for local workspace linking
This commit is contained in:
Patrik Oldsberg
2024-10-22 13:41:41 +02:00
committed by GitHub
parent dfd2dc2378 659d28e411
commit 1cb5b012fb
14 changed files with 136 additions and 337 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.changeset/witty-socks-repeat.md
+7
View File
@@ -0,0 +1,7 @@
---
'@backstage/cli': patch
---
Added a new `--link <workspace-path>` option for frontend builds that allow you to override module resolution to link in an external workspace at runtime.
As part of this change the Webpack linked workspace resolution plugin for frontend builds has been removed. It was in place to support the old workspace linking where it was done by Yarn, which is no longer a working option.
docs/tooling/local-dev/linking-local-packages.md
+14 -73
View File
@@ -18,89 +18,30 @@ together:
For example, you might want to make modifications to `@backstage/core-plugin-api` and try them out in your company's
instance of Backstage.
## Linking in Backstage NPM Packages
## External workspace linking
To link in external packages, add them to your root `package.json` and `lerna.json`
`"workspace"` paths. These can be either relative or absolute paths with or without
globs.
:::info
Workspace linking is an experimental feature and may not work in all cases.
:::
For example:
The `backstage-cli package start` command that is used for local development of all packages supports a `--link` flag that can be used to link a single external workspace to the current workspace. It hooks into the module resolution and will override all imports of packages in the linked workspace to be imported from there instead. The only exception are the `react` and `react-dom` packages, which will always be resolved from the target package.
```json title="/lerna.json"
...
"packages": [
"packages/*",
"plugins/*",
"../backstage/packages/core-plugin-api", // New path added to work on @backstage/core-plugin-api
],
...
```
When linking an external workspace, make sure that dependencies are installed and up to date in both workspaces, and that the versions of `react` and `react-dom` are the same in both workspaces.
```json title="/package.json"
...
"workspaces": {
"packages": [
"packages/*",
"plugins/*",
"../backstage/packages/core-plugin-api", // New path added to work on @backstage/core-plugin-api
],
}
...
```
Now reinstall all packages from the root to make yarn set up symlinks from your application to the core Backstage clone:
If you're within the `packages/app` folder inside your `my-backstage-application` workspace in the above example, you can link the `backstage` workspace using the following command:
```bash
yarn install
yarn start --link ../../../backstage
```
## Making Backstage Changes
The path provided to the `--link` option can be a relative or absolute path, and should point to the root of the external workspace.
With this in place you can now modify the `@backstage/core-plugin-api` package
within the main repo, and have those changes be reflected and tested in your
app. Simply run your app using `yarn dev` (or `yarn start` for just frontend) as
normal.
With the `start` command up and running and serving the development version of your frontend app in the browser, you can now make changes to both workspaces and see the changes reflected in the browser.
You can also link backend packages using the exact same process, simply start your backend package with the same `--link <workspace-path>` option.
## Common Problems
### Backend Issues
### React errors
For backend packages you need to make sure that linked packages are
not dependencies of any non-linked package. If you for example want to work on
`@backstage/backend-common`, you need to also link in other backend plugins and
packages that depend on `@backstage/backend-common`, or temporarily disable
those plugins in your backend. This is because the transformation of backend
module tree stops whenever a non-local package is encountered, and from that
point node will `require` packages directly for that entire module subtree.
### Typescript Issues
Type checking can also have issues when linking in external packages, since the
linked in packages will use the types in the external project and dependency
version mismatches between the two projects may cause errors. To fix any of
those errors you need to sync versions of the dependencies in the two projects.
A simple way to do this can be to copy over `yarn.lock` from the external
project and run `yarn install`, although this is quite intrusive and can cause
other issues in existing projects, so use this method with care. It can often be
best to simply ignore the type errors, as app serving will work just fine
anyway.
Another issue with type checking is that the incremental type cache doesn't
invalidate correctly for the linked in packages, causing type checking to not
reflect changes made to types. You can work around this by either setting
`compilerOptions.incremental = false` in `tsconfig.json`, or by deleting the
types cache folder `dist-types` before running `yarn tsc`.
### Version Issues
While `yarn install` might not error, it does not mean that the linking worked properly.
You will know that linking worked properly when:
1. Your Backstage application root `/node_modules/@backstage/[some package]` is a symlink
2. Your Backstage application `/packages/app/node_modules` and `/packages/backend/node_modules` does
not contain the package you are attempting to link!
If you see Yarn continuing to download the package you are trying to link from NPM, you might need to be
explicit in your `package.json` version so that it exactly matches what you have in the cloned Backstage
repository on your machine. For example, if you have cloned `/plugins/catalog` with version
`"version": "1.19.1-next.1"` you will need to be explicit in your application to point to `"1.19.1-next.1"`.
If you are encountering errors related to React, it is likely that the versions of React in the two workspaces are different. Make sure that the versions of `react` and `react-dom` are the same in both workspaces, or at least that they are in sync between the package that you're serving the app from and the external workspace.
packages/cli/cli-report.md
+1
View File
@@ -271,6 +271,7 @@ Options:
--inspect [host]
--inspect-brk [host]
--require <path>
--link <path>
-h, --help
```
packages/cli/config/nodeTransform.cjs
+33
View File
@@ -16,6 +16,39 @@
const { transformSync } = require('@swc/core');
const { addHook } = require('pirates');
const { Module } = require('module');
// This hooks into module resolution and overrides imports of packages that
// exist in the linked workspace to instead be resolved from the linked workspace.
if (process.env.BACKSTAGE_CLI_LINKED_WORKSPACE) {
const { join: joinPath } = require('path');
const { getPackagesSync } = require('@manypkg/get-packages');
const { packages: linkedPackages, root: linkedRoot } = getPackagesSync(
process.env.BACKSTAGE_CLI_LINKED_WORKSPACE,
);
// Matches all packages in the linked workspaces, as well as sub-path exports from them
const replacementRegex = new RegExp(
`^(?:${linkedPackages
.map(pkg => pkg.packageJson.name)
.join('|')})(?:/.*)?$`,
);
const origLoad = Module._load;
Module._load = function requireHook(request, parent) {
if (!replacementRegex.test(request)) {
return origLoad.call(this, request, parent);
}
// The package import that we're overriding will always existing in the root
// node_modules of the linked workspace, so it's enough to override the the
// parent paths with that single entry
return origLoad.call(this, request, {
...parent,
paths: [joinPath(linkedRoot.dir, 'node_modules')],
});
};
}
addHook(
(code, filename) => {
packages/cli/src/commands/index.ts
+1
View File
@@ -138,6 +138,7 @@ export function registerScriptCommand(program: Command) {
'Enable debugger in Node.js environments, breaking before code starts',
)
.option('--require <path>', 'Add a --require argument to the node process')
.option('--link <path>', 'Link an external workspace for module resolution')
.action(lazy(() => import('./start').then(m => m.command)));
command
packages/cli/src/commands/start/command.ts
+1
View File
@@ -26,6 +26,7 @@ export async function command(opts: OptionValues): Promise<void> {
const options = {
configPaths: opts.config as string[],
checksEnabled: Boolean(opts.check),
linkedWorkspace: opts.link,
inspectEnabled: opts.inspect,
inspectBrkEnabled: opts.inspectBrk,
require: opts.require,
packages/cli/src/commands/start/startBackend.ts
+3
View File
@@ -22,6 +22,7 @@ interface StartBackendOptions {
checksEnabled: boolean;
inspectEnabled: boolean;
inspectBrkEnabled: boolean;
linkedWorkspace?: string;
require?: string;
}
@@ -30,6 +31,7 @@ export async function startBackend(options: StartBackendOptions) {
entry: 'src/index',
inspectEnabled: options.inspectEnabled,
inspectBrkEnabled: options.inspectBrkEnabled,
linkedWorkspace: options.linkedWorkspace,
require: options.require,
});
@@ -52,6 +54,7 @@ export async function startBackendPlugin(options: StartBackendOptions) {
inspectEnabled: options.inspectEnabled,
inspectBrkEnabled: options.inspectBrkEnabled,
require: options.require,
linkedWorkspace: options.linkedWorkspace,
});
await waitForExit();
packages/cli/src/commands/start/startFrontend.ts
+2
View File
@@ -26,6 +26,7 @@ interface StartAppOptions {
configPaths: string[];
skipOpenBrowser?: boolean;
isModuleFederationRemote?: boolean;
linkedWorkspace?: string;
}
export async function startFrontend(options: StartAppOptions) {
@@ -37,6 +38,7 @@ export async function startFrontend(options: StartAppOptions) {
configPaths: options.configPaths,
verifyVersions: options.verifyVersions,
skipOpenBrowser: options.skipOpenBrowser,
linkedWorkspace: options.linkedWorkspace,
moduleFederation: getModuleFederationOptions(
name,
options.isModuleFederationRemote,
packages/cli/src/lib/bundler/LinkedPackageResolvePlugin.test.ts
-176
View File
@@ -1,176 +0,0 @@
/*
* Copyright 2020 The Backstage Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import * as os from 'os';
import * as path from 'path';
import { LinkedPackageResolvePlugin } from './LinkedPackageResolvePlugin';
describe('LinkedPackageResolvePlugin', () => {
const root = os.platform() === 'win32' ? 'C:\\root' : '/root';
it('should re-write paths for external packages', () => {
const plugin = new LinkedPackageResolvePlugin(
path.resolve(root, 'repo/node_modules'),
[
{
dir: path.resolve(root, 'external-a'),
packageJson: {
name: 'a',
version: '1.0.0',
},
},
{
dir: path.resolve(root, 'external-b'),
packageJson: {
name: '@s/b',
version: '1.0.0',
},
},
],
);
const tapAsync = jest.fn();
const doResolve = jest.fn();
const resolver = {
hooks: { resolve: { tapAsync } },
doResolve,
};
plugin.apply(resolver);
expect(tapAsync).toHaveBeenCalledTimes(1);
expect(tapAsync).toHaveBeenCalledWith(
'LinkedPackageResolvePlugin',
expect.any(Function),
);
expect(doResolve).toHaveBeenCalledTimes(0);
// Internal module resolution is not affected
const tap = tapAsync.mock.calls[0][1];
const callbackX = jest.fn();
tap(
{
request: path.resolve(root, 'repo/package/x/src/module.ts'),
path: path.resolve(root, 'repo/package/x/src'),
context: {
issuer: path.resolve(root, 'repo/package/x/src/index.ts'),
},
},
'some-context',
callbackX,
);
expect(callbackX).toHaveBeenCalledTimes(1);
expect(callbackX).toHaveBeenCalledWith();
expect(doResolve).toHaveBeenCalledTimes(0);
// Path is sometimes false
const callbackFalse = jest.fn();
tap(
{
request: 'sample',
path: false,
},
'some-context',
callbackFalse,
);
expect(callbackFalse).toHaveBeenCalledTimes(1);
expect(callbackFalse).toHaveBeenCalledWith();
expect(doResolve).toHaveBeenCalledTimes(0);
// Internal modules with a path prefix of an external module
const callbackY = jest.fn();
tap(
{
request: path.resolve(root, 'external-aa/src/module.ts'),
path: path.resolve(root, 'external-aa/src'),
context: {
issuer: path.resolve(root, 'external-aa/src/index.ts'),
},
},
'some-context',
callbackY,
);
expect(callbackY).toHaveBeenCalledTimes(1);
expect(callbackY).toHaveBeenCalledWith();
expect(doResolve).toHaveBeenCalledTimes(0);
// External modules have their path and issuer context rewritten, but not the request
const callbackA = jest.fn();
tap(
{
request: path.resolve(root, 'external-a/src/module.ts'),
path: path.resolve(root, 'external-a/src'),
context: {
issuer: path.resolve(root, 'external-a/src/index.ts'),
},
},
'some-context',
callbackA,
);
expect(callbackA).toHaveBeenCalledTimes(0);
expect(doResolve).toHaveBeenCalledTimes(1);
expect(doResolve).toHaveBeenCalledWith(
resolver.hooks.resolve,
{
request: path.resolve(root, 'external-a/src/module.ts'),
path: path.resolve(root, 'repo/node_modules/a/src'),
context: {
issuer: path.resolve(root, 'repo/node_modules/a/src/index.ts'),
},
},
`resolve ${path.resolve(
root,
'external-a/src/module.ts',
)} in ${path.resolve(root, 'repo/node_modules/a')}`,
'some-context',
callbackA,
);
// Also handles scoped packages correctly, and issuer is not required
const callbackB = jest.fn();
tap(
{
request: path.resolve(root, 'external-b/src/module.ts'),
path: path.resolve(root, 'external-b/src'),
context: {
issuer: false,
},
},
'some-context',
callbackB,
);
expect(callbackB).toHaveBeenCalledTimes(0);
expect(doResolve).toHaveBeenCalledTimes(2);
expect(doResolve).toHaveBeenLastCalledWith(
resolver.hooks.resolve,
{
request: path.resolve(root, 'external-b/src/module.ts'),
path: path.resolve(root, 'repo/node_modules/@s/b/src'),
context: {
issuer: false,
},
},
`resolve ${path.resolve(
root,
'external-b/src/module.ts',
)} in ${path.resolve(root, 'repo/node_modules/@s/b')}`,
'some-context',
callbackB,
);
expect(tapAsync).toHaveBeenCalledTimes(1);
});
});
packages/cli/src/lib/bundler/LinkedPackageResolvePlugin.ts
-82
View File
@@ -1,82 +0,0 @@
/*
* Copyright 2020 The Backstage Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { resolve as resolvePath } from 'path';
import { WebpackPluginInstance } from 'webpack';
import { isChildPath } from '@backstage/cli-common';
import { Package } from '@manypkg/get-packages';
// Enables proper resolution of packages when linking in external packages.
// Without this the packages would depend on dependencies in the node_modules
// of the external packages themselves, leading to module duplication
export class LinkedPackageResolvePlugin implements WebpackPluginInstance {
constructor(
private readonly targetModules: string,
private readonly packages: Package[],
) {}
apply(resolver: any) {
resolver.hooks.resolve.tapAsync(
'LinkedPackageResolvePlugin',
(
data: {
request: string;
path?: false | string;
context?: { issuer?: string };
},
context: unknown,
callback: () => void,
) => {
const pkg = this.packages.find(
pkge => data.path && isChildPath(pkge.dir, data.path),
);
if (!pkg) {
callback();
return;
}
// pkg here is an external package. We rewrite the context of any imports to resolve
// from the location of the package within the node_modules of the target root rather
// than the real location of the external package.
const modulesLocation = resolvePath(
this.targetModules,
pkg.packageJson.name,
);
const newContext = data.context?.issuer
? {
...data.context,
issuer: data.context.issuer.replace(pkg.dir, modulesLocation),
}
: data.context;
// Re-run resolution but this time from the point of view of our target monorepo rather
// than the location of the external package. By resolving modules using this method we avoid
// pulling in e.g. `react` from the external repo, which would otherwise lead to conflicts.
resolver.doResolve(
resolver.hooks.resolve,
{
...data,
context: newContext,
path: data.path && data.path.replace(pkg.dir, modulesLocation),
},
`resolve ${data.request} in ${modulesLocation}`,
context,
callback,
);
},
);
}
}
packages/cli/src/lib/bundler/config.ts
+10 -6
View File
@@ -25,13 +25,10 @@ import ESLintPlugin from 'eslint-webpack-plugin';
import ForkTsCheckerWebpackPlugin from 'fork-ts-checker-webpack-plugin';
import HtmlWebpackPlugin from 'html-webpack-plugin';
import { ModuleFederationPlugin } from '@module-federation/enhanced/webpack';
import { LinkedPackageResolvePlugin } from './LinkedPackageResolvePlugin';
import ModuleScopePlugin from 'react-dev-utils/ModuleScopePlugin';
import ReactRefreshPlugin from '@pmmmwh/react-refresh-webpack-plugin';
import { paths as cliPaths } from '../../lib/paths';
import fs from 'fs-extra';
import { getPackages } from '@manypkg/get-packages';
import { isChildPath } from '@backstage/cli-common';
import { optimization as optimizationConfig } from './optimization';
import pickBy from 'lodash/pickBy';
import { runPlain } from '../run';
@@ -39,6 +36,7 @@ import { transforms } from './transforms';
import { version } from '../../lib/version';
import yn from 'yn';
import { hasReactDomClient } from './hasReactDomClient';
import { createWorkspaceLinkingPlugins } from './linkWorkspaces';
const BUILD_CACHE_ENV_VAR = 'BACKSTAGE_CLI_EXPERIMENTAL_BUILD_CACHE';
@@ -130,8 +128,6 @@ export async function createConfig(
const { plugins, loaders } = transforms(options);
// Any package that is part of the monorepo but outside the monorepo root dir need
// separate resolution logic.
const { packages } = await getPackages(cliPaths.targetDir);
const externalPkgs = packages.filter(p => !isChildPath(paths.root, p.dir));
const validBaseUrl = resolveBaseUrl(frontendConfig, moduleFederation);
let publicPath = validBaseUrl.pathname.replace(/\/$/, '');
@@ -300,6 +296,15 @@ export async function createConfig(
}),
);
if (options.linkedWorkspace) {
plugins.push(
...(await createWorkspaceLinkingPlugins(
bundler,
options.linkedWorkspace,
)),
);
}
// These files are required by the transpiled code when using React Refresh.
// They need to be excluded to the module scope plugin which ensures that files
// that exist in the package are required.
@@ -406,7 +411,6 @@ export async function createConfig(
// FIXME: see also https://github.com/web-infra-dev/rspack/issues/3408
...(!rspack && {
plugins: [
new LinkedPackageResolvePlugin(paths.rootNodeModules, externalPkgs),
new ModuleScopePlugin(
[paths.targetSrc, paths.targetDev],
[paths.targetPackageJson, ...reactRefreshFiles],
packages/cli/src/lib/bundler/linkWorkspaces.ts
+57
View File
@@ -0,0 +1,57 @@
/*
* Copyright 2024 The Backstage Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { relative as relativePath } from 'path';
import { getPackages } from '@manypkg/get-packages';
import webpack from 'webpack';
import { paths } from '../paths';
/**
* This returns of collection of plugins that links a separate workspace into
* the target one. Any packages that are present in the linked workspaces will
* always be used in place of the ones in the target workspace, with the exception
* of react and react-dom which are always resolved from the target workspace.
*/
export async function createWorkspaceLinkingPlugins(
bundler: typeof webpack,
workspace: string,
) {
const { packages: linkedPackages, root: linkedRoot } = await getPackages(
workspace,
);
// Matches all packages in the linked workspaces, as well as sub-path exports from them
const replacementRegex = new RegExp(
`^(?:${linkedPackages
.map(pkg => pkg.packageJson.name)
.join('|')})(?:/.*)?$`,
);
return [
// Any imports of a package that is present in the linked workspace will
// be redirected to be resolved within the context of the linked workspace
new bundler.NormalModuleReplacementPlugin(replacementRegex, resource => {
resource.context = linkedRoot.dir;
}),
// react and react-dom are always resolved from the target directory
// Note: this often requires that the linked and target workspace use the same versions of React
new bundler.NormalModuleReplacementPlugin(/^react(?:-dom)?$/, resource => {
if (!relativePath(linkedRoot.dir, resource.context).startsWith('..')) {
resource.context = paths.targetDir;
}
}),
];
}
packages/cli/src/lib/bundler/types.ts
+4
View File
@@ -36,6 +36,8 @@ export type BundlingOptions = {
publicSubPath?: string;
// Mode that the app is running in, 'protected' or 'public', default is 'public'
appMode?: string;
// An external linked workspace to include in the bundling
linkedWorkspace?: string;
moduleFederation?: ModuleFederationOptions;
rspack?: typeof import('@rspack/core').rspack;
};
@@ -46,6 +48,8 @@ export type ServeOptions = BundlingPathsOptions & {
verifyVersions?: boolean;
skipOpenBrowser?: boolean;
moduleFederation?: ModuleFederationOptions;
// An external linked workspace to include in the bundling
linkedWorkspace?: string;
};
export type BuildOptions = BundlingPathsOptions & {
packages/cli/src/lib/runner/runBackend.ts
+3
View File
@@ -40,6 +40,8 @@ export type RunBackendOptions = {
inspectBrkEnabled: boolean;
/** Additional module to require via the --require flag to the node process */
require?: string;
/** An external linked workspace to override module resolution towards */
linkedWorkspace?: string;
};
export async function runBackend(options: RunBackendOptions) {
@@ -119,6 +121,7 @@ export async function runBackend(options: RunBackendOptions) {
stdio: ['ignore', 'inherit', 'inherit', 'ipc'],
env: {
...process.env,
BACKSTAGE_CLI_LINKED_WORKSPACE: options.linkedWorkspace,
BACKSTAGE_CLI_CHANNEL: '1',
ESBK_TSCONFIG_PATH: paths.resolveTargetRoot('tsconfig.json'),
},