paulononato/backstage
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

cli: remove most lockfile analysis features

Browse Source 
Signed-off-by: Patrik Oldsberg <poldsberg@gmail.com>
This commit is contained in:
Patrik Oldsberg
2024-07-21 13:21:43 +02:00
parent 5c8833cab3
commit 32a38e1f37
14 changed files with 43 additions and 1079 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.changeset/neat-gifts-join.md
+5
View File
@@ -0,0 +1,5 @@
---
'@backstage/plugin-devtools-backend': patch
---
Removed unused code for lockfile analysis.
.changeset/small-spoons-shout.md
+13
View File
@@ -0,0 +1,13 @@
---
'@backstage/cli': minor
---
**BREAKING**: The lockfile (`yarn.lock`) dependency analysis and mutations have been removed from several commands.
The `versions:bump` command will no longer attempt to bump and deduplicate dependencies by modifying the lockfile, it will only update `package.json` files.
The `versions:check` command has been removed, since its only purpose was verification and mutation of the lockfile. We recommend using the `yarn dedupe` command instead, or the `yarn-deduplicate` package if you're using Yarn classic.
The check that was built into the `package start` command has been removed, it will no longer warn about lockfile mismatches.
The packages in the Backstage ecosystem handle package duplications much better now than when these CLI features were first introduced, so the need for these features has diminished. By removing them, we drastically reduce the integration between the Backstage CLI and Yarn, making it much easier to add support for other package managers in the future.
.github/vale/config/vocabularies/Backstage/accept.txt
+2
View File
@@ -85,7 +85,9 @@ debounce
Debounce
debuggability
declaratively
deduplicate
deduplicated
deduplication
deliverables
denormalized
dependabot
docs/getting-started/keeping-backstage-updated.md
+11 -13
View File
@@ -63,18 +63,16 @@ When a given dependency version is the _same_ between different packages, the
dependency is hoisted to the main `node_modules` folder in the monorepo root to
be shared between packages. When _different_ versions of the same dependency are
encountered, Yarn creates a `node_modules` folder within a particular package.
This can lead to multiple versions of the same package being installed and used
in the same app.
This can lead to confusing situations with type definitions, or anything with
global state. React [Context](https://reactjs.org/docs/context.html), for
example, depends on global referential equality. This can cause problems in
Backstage with API lookup, or config loading.
All Backstage core packages are implemented in such as way that package
duplication is **not** a problem. For example, duplicate installations of
packages like `@backstage/core-plugin-api`, `@backstage/core-components`,
`@backstage/plugin-catalog-react`, and `@backstage/backend-plugin-api` are all
acceptable.
To help resolve these situations, the Backstage CLI has
[versions:check](https://backstage.io/docs/tooling/cli/03-commands#versionscheck). This
will validate versions of `@backstage` packages in your app to check for
duplicate definitions:
```bash
# Add --fix to attempt automatic resolution in yarn.lock
yarn backstage-cli versions:check
```
While package duplication might be acceptable in many cases, you might want to
deduplicate packages for the purpose of optimizing bundle size and installation
speed. We recommend using deduplication utilities such as `yarn dedupe` to trim
down the number of duplicate packages.
docs/tooling/cli/03-commands.md
+1 -20
View File
@@ -26,7 +26,6 @@ repo [command] Command that run across an entire
package [command] Lifecycle scripts for individual packages
migrate [command] Migration utilities
versions:bump [options] Bump Backstage packages to the latest versions
versions:check [options] Check Backstage package versioning
clean Delete cache directories [DEPRECATED]
build-workspace <workspace-dir> [packages...] Builds a temporary dist workspace from the provided
packages
@@ -327,8 +326,7 @@ Options:
## versions\:bump
Bump all `@backstage` packages to the latest versions. This checks for updates
in the package registry, and will update entries both in `yarn.lock` and
`package.json` files when necessary.
in the package registry, and will update entries `package.json` files when necessary.
```text
Usage: backstage-cli versions:bump [options]
@@ -339,23 +337,6 @@ Options:
--release <version|next|main> Bump to a specific Backstage release line or version (default: "main")
```
## versions\:check
Validate `@backstage` dependencies within the repo, making sure that there are
no duplicates of packages that might lead to breakages.
By supplying the `--fix` flag the command will attempt to fix any conflict that
can be resolved by editing `yarn.lock`, but will not attempt to search for
remote updates or modify any `package.json` files.
```text
Usage: backstage-cli versions:check [options]
Options:
--fix Fix any auto-fixable versioning problems
-h, --help display help for command
```
## build-workspace
Builds a mirror of the workspace using the packaged production version of each
packages/cli/cli-report.md
-11
View File
@@ -22,7 +22,6 @@ Commands:
package [command]
migrate [command]
versions:bump [options]
versions:check [options]
versions:migrate [options]
clean
build-workspace [options] <workspace-dir> [packages...]
@@ -607,16 +606,6 @@ Options:
-h, --help
```
### `backstage-cli versions:check`
```
Usage: backstage-cli versions:check [options]
Options:
--fix
-h, --help
```
### `backstage-cli versions:migrate`
```
packages/cli/src/commands/index.ts
+1 -1
View File
@@ -396,7 +396,7 @@ export function registerCommands(program: Command) {
.action(lazy(() => import('./versions/bump').then(m => m.default)));
program
.command('versions:check')
.command('versions:check', { hidden: true })
.option('--fix', 'Fix any auto-fixable versioning problems')
.description('Check Backstage package versioning')
.action(lazy(() => import('./versions/lint').then(m => m.default)));
packages/cli/src/commands/versions/bump.test.ts
+5 -169
View File
@@ -23,7 +23,6 @@ import { YarnInfoInspectData } from '../../lib/versioning/packages';
import { setupServer } from 'msw/node';
import { rest } from 'msw';
import { NotFoundError } from '@backstage/errors';
import { Lockfile } from '../../lib/versioning/Lockfile';
import {
MockDirectory,
createMockDirectory,
@@ -117,17 +116,6 @@ const lockfileMock = `${HEADER}
version "1.0.3"
`;
// This is the lockfile that we produce to unlock versions before we run yarn install
const lockfileMockResult = `${HEADER}
"@backstage/core@^1.0.5":
version "1.0.6"
dependencies:
"@backstage/core-api" "^1.0.6"
"@backstage/theme@^1.0.0":
version "1.0.0"
`;
// Avoid flakes by comparing sorted log lines. File system access is async, which leads to the log line order being indeterministic
const expectLogsToMatch = (
recievedLogs: String[],
@@ -204,11 +192,7 @@ describe('bump', () => {
'Using default pattern glob @backstage/*',
'Checking for updates of @backstage/core',
'Checking for updates of @backstage/theme',
'Checking for updates of @backstage/core-api',
'Some packages are outdated, updating',
'unlocking @backstage/core@^1.0.3 ~> 1.0.6',
'unlocking @backstage/core-api@^1.0.6 ~> 1.0.7',
'unlocking @backstage/core-api@^1.0.3 ~> 1.0.7',
'bumping @backstage/core in a to ^1.0.6',
'bumping @backstage/core in b to ^1.0.6',
'bumping @backstage/theme in b to ^2.0.0',
@@ -220,9 +204,8 @@ describe('bump', () => {
'Version bump complete!',
]);
expect(mockFetchPackageInfo).toHaveBeenCalledTimes(3);
expect(mockFetchPackageInfo).toHaveBeenCalledTimes(2);
expect(mockFetchPackageInfo).toHaveBeenCalledWith('@backstage/core');
expect(mockFetchPackageInfo).toHaveBeenCalledWith('@backstage/core-api');
expect(mockFetchPackageInfo).toHaveBeenCalledWith('@backstage/theme');
expect(runObj.run).toHaveBeenCalledTimes(1);
@@ -232,12 +215,6 @@ describe('bump', () => {
expect.any(Object),
);
const lockfileContents = await fs.readFile(
mockDir.resolve('yarn.lock'),
'utf8',
);
expect(lockfileContents).toBe(lockfileMockResult);
const packageA = await fs.readJson(
mockDir.resolve('packages/a/package.json'),
);
@@ -312,11 +289,7 @@ describe('bump', () => {
'Using default pattern glob @backstage/*',
'Checking for updates of @backstage/core',
'Checking for updates of @backstage/theme',
'Checking for updates of @backstage/core-api',
'Some packages are outdated, updating',
'unlocking @backstage/core@^1.0.3 ~> 1.0.6',
'unlocking @backstage/core-api@^1.0.6 ~> 1.0.7',
'unlocking @backstage/core-api@^1.0.3 ~> 1.0.7',
'bumping @backstage/core in a to ^1.0.6',
'bumping @backstage/core in b to ^1.0.6',
'bumping @backstage/theme in b to ^2.0.0',
@@ -328,9 +301,8 @@ describe('bump', () => {
'Version bump complete!',
]);
expect(mockFetchPackageInfo).toHaveBeenCalledTimes(3);
expect(mockFetchPackageInfo).toHaveBeenCalledTimes(2);
expect(mockFetchPackageInfo).toHaveBeenCalledWith('@backstage/core');
expect(mockFetchPackageInfo).toHaveBeenCalledWith('@backstage/core-api');
expect(mockFetchPackageInfo).toHaveBeenCalledWith('@backstage/theme');
expect(runObj.run).not.toHaveBeenCalledWith(
@@ -339,12 +311,6 @@ describe('bump', () => {
expect.any(Object),
);
const lockfileContents = await fs.readFile(
mockDir.resolve('yarn.lock'),
'utf8',
);
expect(lockfileContents).toBe(lockfileMockResult);
const packageA = await fs.readJson(
mockDir.resolve('packages/a/package.json'),
);
@@ -425,12 +391,7 @@ describe('bump', () => {
'Using default pattern glob @backstage/*',
'Checking for updates of @backstage/core',
'Checking for updates of @backstage/theme',
'Checking for updates of @backstage/theme',
'Checking for updates of @backstage/core-api',
'Some packages are outdated, updating',
'unlocking @backstage/core@^1.0.3 ~> 1.0.6',
'unlocking @backstage/core-api@^1.0.6 ~> 1.0.7',
'unlocking @backstage/core-api@^1.0.3 ~> 1.0.7',
'bumping @backstage/theme in b to ^5.0.0',
'bumping @backstage/core in b to ^1.0.6',
'bumping @backstage/core in a to ^1.0.6',
@@ -443,9 +404,8 @@ describe('bump', () => {
'Version bump complete!',
]);
expect(mockFetchPackageInfo).toHaveBeenCalledTimes(2);
expect(mockFetchPackageInfo).toHaveBeenCalledTimes(1);
expect(mockFetchPackageInfo).toHaveBeenCalledWith('@backstage/core');
expect(mockFetchPackageInfo).not.toHaveBeenCalledWith('@backstage/theme');
expect(runObj.run).toHaveBeenCalledTimes(1);
expect(runObj.run).toHaveBeenCalledWith(
@@ -454,12 +414,6 @@ describe('bump', () => {
expect.any(Object),
);
const lockfileContents = await fs.readFile(
mockDir.resolve('yarn.lock'),
'utf8',
);
expect(lockfileContents).toBe(lockfileMockResult);
const packageA = await fs.readJson(
mockDir.resolve('packages/a/package.json'),
);
@@ -629,12 +583,7 @@ describe('bump', () => {
'Using default pattern glob @backstage/*',
'Checking for updates of @backstage/core',
'Checking for updates of @backstage/theme',
'Checking for updates of @backstage/theme',
'Checking for updates of @backstage/core-api',
'Some packages are outdated, updating',
'unlocking @backstage/core@^1.0.3 ~> 1.0.6',
'unlocking @backstage/core-api@^1.0.6 ~> 1.0.7',
'unlocking @backstage/core-api@^1.0.3 ~> 1.0.7',
'bumping @backstage/theme in b to ^5.0.0',
'bumping @backstage/core in b to ^1.0.6',
'bumping @backstage/core in a to ^1.0.6',
@@ -658,21 +607,6 @@ describe('bump', () => {
"@backstage-extra/custom-two@^1.0.0":
version "1.0.0"
`;
const customLockfileMockResult = `${HEADER}
"@backstage-extra/custom-two@^1.0.0":
version "1.0.0"
"@backstage-extra/custom@^1.1.0":
version "1.1.0"
"@backstage/core@^1.0.5":
version "1.0.6"
dependencies:
"@backstage/core-api" "^1.0.6"
"@backstage/theme@^1.0.0":
version "1.0.0"
`;
mockDir.setContent({
'yarn.lock': customLockfileMock,
@@ -731,12 +665,7 @@ describe('bump', () => {
'Checking for updates of @backstage-extra/custom',
'Checking for updates of @backstage-extra/custom-two',
'Checking for updates of @backstage/theme',
'Checking for updates of @backstage/core-api',
'Some packages are outdated, updating',
'unlocking @backstage/core@^1.0.3 ~> 1.0.6',
'unlocking @backstage-extra/custom@^1.0.1 ~> 1.1.0',
'unlocking @backstage/core-api@^1.0.6 ~> 1.0.7',
'unlocking @backstage/core-api@^1.0.3 ~> 1.0.7',
'bumping @backstage/core in a to ^1.0.6',
'bumping @backstage-extra/custom in a to ^1.1.0',
'bumping @backstage-extra/custom-two in a to ^2.0.0',
@@ -754,7 +683,7 @@ describe('bump', () => {
'Version bump complete!',
]);
expect(mockFetchPackageInfo).toHaveBeenCalledTimes(5);
expect(mockFetchPackageInfo).toHaveBeenCalledTimes(4);
expect(mockFetchPackageInfo).toHaveBeenCalledWith('@backstage/core');
expect(mockFetchPackageInfo).toHaveBeenCalledWith('@backstage/theme');
@@ -765,12 +694,6 @@ describe('bump', () => {
expect.any(Object),
);
const lockfileContents = await fs.readFile(
mockDir.resolve('yarn.lock'),
'utf8',
);
expect(lockfileContents).toEqual(customLockfileMockResult);
const packageA = await fs.readJson(
mockDir.resolve('packages/a/package.json'),
);
@@ -798,7 +721,7 @@ describe('bump', () => {
it('should ignore not found packages', async () => {
mockDir.setContent({
'yarn.lock': lockfileMockResult,
'yarn.lock': lockfileMock,
'package.json': JSON.stringify({
workspaces: {
packages: ['packages/*'],
@@ -848,21 +771,11 @@ describe('bump', () => {
'Checking for updates of @backstage/theme',
'Package info not found, ignoring package @backstage/core',
'Package info not found, ignoring package @backstage/theme',
'Checking for updates of @backstage/core',
'Checking for updates of @backstage/theme',
'Package info not found, ignoring package @backstage/core',
'Package info not found, ignoring package @backstage/theme',
'All Backstage packages are up to date!',
]);
expect(runObj.run).toHaveBeenCalledTimes(0);
const lockfileContents = await fs.readFile(
mockDir.resolve('yarn.lock'),
'utf8',
);
expect(lockfileContents).toBe(lockfileMockResult);
const packageA = await fs.readJson(
mockDir.resolve('packages/a/package.json'),
);
@@ -883,83 +796,6 @@ describe('bump', () => {
},
});
});
// eslint-disable-next-line jest/expect-expect
it('should log duplicates', async () => {
jest.spyOn(Lockfile.prototype, 'analyze').mockReturnValue({
invalidRanges: [],
newVersions: [],
newRanges: [
{
name: '@backstage/backend-app-api',
oldRange: '^1.0.0',
newRange: '^2.0.0',
oldVersion: '1.0.0',
newVersion: '2.0.0',
},
],
});
mockDir.setContent({
'yarn.lock': `${HEADER}
"@backstage/backend-app-api@^1.0.0":
version "1.0.0"
`,
'package.json': JSON.stringify({
workspaces: {
packages: ['packages/*'],
},
}),
packages: {
a: {
'package.json': JSON.stringify({
name: 'a',
dependencies: {
'@backstage/backend-app-api': '^1.0.0',
},
}),
},
},
});
jest.spyOn(runObj, 'run').mockResolvedValue(undefined);
worker.use(
rest.get(
'https://versions.backstage.io/v1/tags/main/manifest.json',
(_, res, ctx) =>
res(
ctx.status(200),
ctx.json({
packages: [
{
name: '@backstage/backend-app-api',
version: '2.0.0',
},
],
}),
),
),
);
const { log: logs } = await withLogCollector(['log', 'warn'], async () => {
await bump({ pattern: null, release: 'main' } as unknown as Command);
});
expectLogsToMatch(logs, [
'Using default pattern glob @backstage/*',
'Checking for updates of @backstage/backend-app-api',
'Checking for updates of @backstage/backend-app-api',
'Some packages are outdated, updating',
'bumping @backstage/backend-app-api in a to ^2.0.0',
'Running yarn install to install new versions',
'Checking for moved packages to the @backstage-community namespace...',
'⚠️ The following packages may have breaking changes:',
' @backstage/backend-app-api : 1.0.0 ~> 2.0.0',
' https://github.com/backstage/backstage/blob/master/packages/backend-app-api/CHANGELOG.md',
'Version bump complete!',
' ⚠️ Warning! ⚠️',
' The below package(s) have incompatible duplicate installations, likely due to a bad dependency in a plugin.',
' You can investigate this by running `yarn why <package-name>`, and report the issue to the plugin maintainers.',
' @backstage/backend-app-api',
]);
});
});
describe('bumpBackstageJsonVersion', () => {
packages/cli/src/commands/versions/bump.ts
+1 -97
View File
@@ -24,7 +24,6 @@ import fs from 'fs-extra';
import chalk from 'chalk';
import ora from 'ora';
import semver from 'semver';
import { minimatch } from 'minimatch';
import { OptionValues } from 'commander';
import { isError, NotFoundError } from '@backstage/errors';
import { resolve as resolvePath } from 'path';
@@ -36,7 +35,6 @@ import {
Lockfile,
YarnInfoInspectData,
} from '../../lib/versioning';
import { forbiddenDuplicatesFilter } from './lint';
import { BACKSTAGE_JSON } from '@backstage/cli-common';
import { runParallelWorkers } from '../../lib/parallel';
import {
@@ -44,7 +42,6 @@ import {
getManifestByVersion,
ReleaseManifest,
} from '@backstage/release-manifests';
import { PackageGraph } from '@backstage/cli-node';
import { migrateMovedPackages } from './migrate';
function shouldUseGlobalAgent(): boolean {
@@ -125,8 +122,6 @@ export default async (opts: OptionValues) => {
// Next check with the package registry to see which dependency ranges we need to bump
const versionBumps = new Map<string, PkgVersionInfo[]>();
// Track package versions that we want to remove from yarn.lock in order to trigger a bump
const unlocked = Array<{ name: string; range: string; target: string }>();
await runParallelWorkers({
parallelismFactor: 4,
@@ -157,72 +152,13 @@ export default async (opts: OptionValues) => {
},
});
const filter = (name: string) => minimatch(name, pattern);
// Check for updates of transitive backstage dependencies
await runParallelWorkers({
parallelismFactor: 4,
items: lockfile.keys(),
async worker(name) {
// Only check @backstage packages and friends, we don't want this to do a full update of all deps
if (!filter(name)) {
return;
}
let target: string;
try {
target = await findTargetVersion(name);
} catch (error) {
if (isError(error) && error.name === 'NotFoundError') {
console.log(`Package info not found, ignoring package ${name}`);
return;
}
throw error;
}
for (const entry of lockfile.get(name) ?? []) {
// Ignore lockfile entries that don't satisfy the version range, since
// these can't cause the package to be locked to an older version
if (!semver.satisfies(target, entry.range)) {
continue;
}
// Unlock all entries that are within range but on the old version
unlocked.push({ name, range: entry.range, target });
}
},
});
console.log();
// Write all discovered version bumps to package.json in this repo
if (versionBumps.size === 0 && unlocked.length === 0) {
if (versionBumps.size === 0) {
console.log(chalk.green('All Backstage packages are up to date!'));
} else {
console.log(chalk.yellow('Some packages are outdated, updating'));
console.log();
if (unlocked.length > 0) {
const removed = new Set<string>();
for (const { name, range, target } of unlocked) {
// Don't bother removing lockfile entries if they're already on the correct version
const existingEntry = lockfile.get(name)?.find(e => e.range === range);
if (existingEntry?.version === target) {
continue;
}
const key = JSON.stringify({ name, range });
if (!removed.has(key)) {
removed.add(key);
console.log(
`${chalk.magenta('unlocking')} ${name}@${chalk.yellow(
range,
)} ~> ${chalk.yellow(target)}`,
);
lockfile.remove(name, range);
}
}
await lockfile.save(lockfilePath);
}
const breakingUpdates = new Map<string, { from: string; to: string }>();
await runParallelWorkers({
parallelismFactor: 4,
@@ -334,38 +270,6 @@ export default async (opts: OptionValues) => {
}
console.log();
// Finally we make sure the new lockfile doesn't have any duplicates
const dedupLockfile = await Lockfile.load(lockfilePath);
const result = dedupLockfile.analyze({
filter,
localPackages: PackageGraph.fromPackages(
await PackageGraph.listTargetPackages(),
),
});
const forbiddenNewRanges = result.newRanges.filter(({ name }) =>
forbiddenDuplicatesFilter(name),
);
if (forbiddenNewRanges.length > 0) {
console.log(chalk.yellow(' ⚠️ Warning! ⚠️'));
console.log();
console.log(
chalk.yellow(
' The below package(s) have incompatible duplicate installations, likely due to a bad dependency in a plugin.',
),
);
console.log(
chalk.yellow(
' You can investigate this by running `yarn why <package-name>`, and report the issue to the plugin maintainers.',
),
);
console.log();
for (const { name } of forbiddenNewRanges) {
console.log(chalk.yellow(` ${name}`));
}
}
};
export function createStrictVersionFinder(options: {
packages/cli/src/commands/versions/lint.ts
+3 -111
View File
@@ -14,116 +14,8 @@
* limitations under the License.
*/
import { OptionValues } from 'commander';
import { Lockfile } from '../../lib/versioning';
import { paths } from '../../lib/paths';
import partition from 'lodash/partition';
import { PackageGraph } from '@backstage/cli-node';
// Packages that we try to avoid duplicates for
const INCLUDED = [/^@backstage\//];
export const includedFilter = (name: string) =>
INCLUDED.some(pattern => pattern.test(name));
// Packages that are not allowed to have any duplicates
const FORBID_DUPLICATES = [/^@backstage\/\w+-app-api$/, /^@backstage\/plugin-/];
// There are some packages that ARE explicitly allowed to have duplicates since
// they handle that appropriately. This takes precedence over FORBID_DUPLICATES
// above.
const ALLOW_DUPLICATES = [
/^@backstage\/core-plugin-api$/,
// Duplicates of libraries are OK
// TODO(Rugvip): Check this using package role instead
/^@backstage\/plugin-.*-react$/,
/^@backstage\/plugin-.*-node$/,
/^@backstage\/plugin-.*-common$/,
];
export const forbiddenDuplicatesFilter = (name: string) =>
FORBID_DUPLICATES.some(pattern => pattern.test(name)) &&
!ALLOW_DUPLICATES.some(pattern => pattern.test(name));
export default async (cmd: OptionValues) => {
const fix = Boolean(cmd.fix);
let success = true;
const lockfilePath = paths.resolveTargetRoot('yarn.lock');
const lockfile = await Lockfile.load(lockfilePath);
const result = lockfile.analyze({
filter: includedFilter,
localPackages: PackageGraph.fromPackages(
await PackageGraph.listTargetPackages(),
),
});
logArray(
result.invalidRanges,
"The following packages versions are invalid and can't be analyzed:",
e => ` ${e.name} @ ${e.range}`,
export default async () => {
throw new Error(
'This command has been removed, please consider alternatives such as `yarn dedupe` instead.',
);
if (fix) {
lockfile.replaceVersions(result.newVersions);
await lockfile.save(lockfilePath);
} else {
const [newVersionsForbidden, newVersionsAllowed] = partition(
result.newVersions,
({ name }) => forbiddenDuplicatesFilter(name),
);
if (newVersionsForbidden.length && !fix) {
success = false;
}
logArray(
newVersionsForbidden,
'The following packages must be deduplicated, this can be done automatically with --fix',
e =>
` ${e.name} @ ${e.range} bumped from ${e.oldVersion} to ${e.newVersion}`,
);
logArray(
newVersionsAllowed,
'The following packages can be deduplicated, this can be done automatically with --fix',
e =>
` ${e.name} @ ${e.range} bumped from ${e.oldVersion} to ${e.newVersion}`,
);
}
const [newRangesForbidden, newRangesAllowed] = partition(
result.newRanges,
({ name }) => forbiddenDuplicatesFilter(name),
);
if (newRangesForbidden.length) {
success = false;
}
logArray(
newRangesForbidden,
'The following packages must be deduplicated by updating dependencies in package.json',
e => ` ${e.name} @ ${e.oldRange} should be changed to ${e.newRange}`,
);
logArray(
newRangesAllowed,
'The following packages can be deduplicated by updating dependencies in package.json',
e => ` ${e.name} @ ${e.oldRange} should be changed to ${e.newRange}`,
);
if (!success) {
throw new Error('Failed versioning check');
}
};
function logArray<T>(arr: T[], header: string, each: (item: T) => string) {
if (arr.length === 0) {
return;
}
console.log(header);
console.log();
for (const e of arr) {
console.log(each(e));
}
console.log();
}
packages/cli/src/lib/bundler/server.ts
-39
View File
@@ -14,22 +14,15 @@
* limitations under the License.
*/
import { PackageGraph } from '@backstage/cli-node';
import { AppConfig } from '@backstage/config';
import chalk from 'chalk';
import fs from 'fs-extra';
import uniq from 'lodash/uniq';
import openBrowser from 'react-dev-utils/openBrowser';
import webpack from 'webpack';
import WebpackDevServer from 'webpack-dev-server';
import {
forbiddenDuplicatesFilter,
includedFilter,
} from '../../commands/versions/lint';
import { paths as libPaths } from '../../lib/paths';
import { loadCliConfig } from '../config';
import { Lockfile } from '../versioning';
import { createConfig, resolveBaseUrl, resolveEndpoint } from './config';
import { createDetectedModulesEntryPoint } from './packageDetection';
import { resolveBundlingPaths, resolveOptionalBundlingPaths } from './paths';
@@ -41,38 +34,6 @@ export async function serveBundle(options: ServeOptions) {
const targetPkg = await fs.readJson(paths.targetPackageJson);
if (options.verifyVersions) {
const lockfile = await Lockfile.load(
libPaths.resolveTargetRoot('yarn.lock'),
);
const result = lockfile.analyze({
filter: includedFilter,
localPackages: PackageGraph.fromPackages(
await PackageGraph.listTargetPackages(),
),
});
const problemPackages = [...result.newVersions, ...result.newRanges]
.map(({ name }) => name)
.filter(forbiddenDuplicatesFilter);
if (problemPackages.length > 1) {
console.log(
chalk.yellow(
`⚠️ Some of the following packages may be outdated or have duplicate installations:
${uniq(problemPackages).join(', ')}
`,
),
);
console.log(
chalk.yellow(
`⚠️ This can be resolved using the following command:
yarn backstage-cli versions:check --fix
`,
),
);
}
if (
targetPkg.dependencies?.['react-router']?.includes('beta') ||
targetPkg.dependencies?.['react-router-dom']?.includes('beta')
packages/cli/src/lib/versioning/Lockfile.test.ts
-219
View File
@@ -14,8 +14,6 @@
* limitations under the License.
*/
import fs from 'fs-extra';
import { BackstagePackage } from '@backstage/cli-node';
import { Lockfile } from './Lockfile';
import { createMockDirectory } from '@backstage/backend-test-utils';
@@ -47,34 +45,6 @@ b@^2:
version "2.0.0"
`;
const mockADedup = `${LEGACY_HEADER}
a@^1:
version "1.0.1"
resolved "https://my-registry/a-1.0.01.tgz#abc123"
integrity sha512-xyz
dependencies:
b "^2"
b@2.0.x, b@^2:
version "2.0.1"
`;
const mockB = `${LEGACY_HEADER}
"@s/a@*", "@s/a@1 || 2", "@s/a@^1":
version "1.0.1"
"@s/a@^2.0.x":
version "2.0.0"
`;
const mockBDedup = `${LEGACY_HEADER}
"@s/a@*", "@s/a@1 || 2", "@s/a@^2.0.x":
version "2.0.0"
"@s/a@^1":
version "1.0.1"
`;
describe('Lockfile', () => {
const mockDir = createMockDirectory();
@@ -93,75 +63,6 @@ describe('Lockfile', () => {
]);
expect(lockfile.toString()).toBe(mockA);
});
it('should deduplicate and save mockA', async () => {
mockDir.setContent({
'yarn.lock': mockA,
});
const lockfilePath = mockDir.resolve('yarn.lock');
const lockfile = await Lockfile.load(lockfilePath);
const result = lockfile.analyze({ localPackages: new Map() });
expect(result).toEqual({
invalidRanges: [],
newRanges: [],
newVersions: [
{
name: 'b',
range: '^2',
oldVersion: '2.0.0',
newVersion: '2.0.1',
},
],
});
expect(lockfile.toString()).toBe(mockA);
lockfile.replaceVersions(result.newVersions);
expect(lockfile.toString()).toBe(mockADedup);
await expect(fs.readFile(lockfilePath, 'utf8')).resolves.toBe(mockA);
await expect(lockfile.save(lockfilePath)).resolves.toBeUndefined();
await expect(fs.readFile(lockfilePath, 'utf8')).resolves.toBe(mockADedup);
});
it('should deduplicate mockB', async () => {
mockDir.setContent({
'yarn.lock': mockB,
});
const lockfile = await Lockfile.load(mockDir.resolve('yarn.lock'));
const result = lockfile.analyze({ localPackages: new Map() });
expect(result).toEqual({
invalidRanges: [],
newRanges: [
{
name: '@s/a',
oldRange: '^1',
newRange: '^2.0.x',
oldVersion: '1.0.1',
newVersion: '2.0.0',
},
],
newVersions: [
{
name: '@s/a',
range: '*',
oldVersion: '1.0.1',
newVersion: '2.0.0',
},
{
name: '@s/a',
range: '1 || 2',
oldVersion: '1.0.1',
newVersion: '2.0.0',
},
],
});
expect(lockfile.toString()).toBe(mockB);
lockfile.replaceVersions(result.newVersions);
expect(lockfile.toString()).toBe(mockBDedup);
});
});
const mockANew = `${MODERN_HEADER}
@@ -179,51 +80,6 @@ b@^2:
version: 2.0.0
`;
const mockANewDedup = `${MODERN_HEADER}
a@^1:
version: 1.0.1
dependencies:
b: ^2
integrity: sha512-xyz
resolved: "https://my-registry/a-1.0.01.tgz#abc123"
"b@2.0.x, b@^2.0.1":
version: 2.0.1
b@^2:
version: 2.0.1
`;
const mockANewLocal = `${MODERN_HEADER}
a@^1:
version: 1.0.1
dependencies:
b: ^2
integrity: sha512-xyz
resolved: "https://my-registry/a-1.0.01.tgz#abc123"
"b@2.0.x, b@^2.0.1":
version: 0.0.0-use.local
b@^2:
version: 2.0.0
`;
const mockANewLocalDedup = `${MODERN_HEADER}
a@^1:
version: 1.0.1
dependencies:
b: ^2
integrity: sha512-xyz
resolved: "https://my-registry/a-1.0.01.tgz#abc123"
"b@2.0.x, b@^2.0.1":
version: 0.0.0-use.local
b@^2:
version: 0.0.0-use.local
`;
describe('New Lockfile', () => {
const mockDir = createMockDirectory();
@@ -243,79 +99,4 @@ describe('New Lockfile', () => {
]);
expect(lockfile.toString()).toBe(mockANew);
});
it('should deduplicate and save mockANew', async () => {
mockDir.setContent({
'yarn.lock': mockANew,
});
const lockfilePath = mockDir.resolve('yarn.lock');
const lockfile = await Lockfile.load(lockfilePath);
const result = lockfile.analyze({ localPackages: new Map() });
expect(result).toEqual({
invalidRanges: [],
newRanges: [],
newVersions: [
{
name: 'b',
range: '^2',
oldVersion: '2.0.0',
newVersion: '2.0.1',
},
],
});
expect(lockfile.toString()).toBe(mockANew);
lockfile.replaceVersions(result.newVersions);
expect(lockfile.toString()).toBe(mockANewDedup);
await expect(fs.readFile(lockfilePath, 'utf8')).resolves.toBe(mockANew);
await expect(lockfile.save(lockfilePath)).resolves.toBeUndefined();
await expect(fs.readFile(lockfilePath, 'utf8')).resolves.toBe(
mockANewDedup,
);
});
it('should deduplicate and save mockANewLocal', async () => {
mockDir.setContent({
'yarn.lock': mockANewLocal,
});
const lockfilePath = mockDir.resolve('yarn.lock');
const lockfile = await Lockfile.load(lockfilePath);
const result = lockfile.analyze({
localPackages: new Map([
[
'b',
{
packageJson: { version: '2.0.1' },
} as BackstagePackage,
],
]),
});
expect(result).toEqual({
invalidRanges: [],
newRanges: [],
newVersions: [
{
name: 'b',
range: '^2',
oldVersion: '2.0.0',
newVersion: '0.0.0-use.local',
},
],
});
expect(lockfile.toString()).toBe(mockANewLocal);
lockfile.replaceVersions(result.newVersions);
expect(lockfile.toString()).toBe(mockANewLocalDedup);
await expect(fs.readFile(lockfilePath, 'utf8')).resolves.toBe(
mockANewLocal,
);
await expect(lockfile.save(lockfilePath)).resolves.toBeUndefined();
await expect(fs.readFile(lockfilePath, 'utf8')).resolves.toBe(
mockANewLocalDedup,
);
});
});
packages/cli/src/lib/versioning/Lockfile.ts
-211
View File
@@ -15,10 +15,8 @@
*/
import fs from 'fs-extra';
import semver from 'semver';
import { parseSyml, stringifySyml } from '@yarnpkg/parsers';
import { stringify as legacyStringifyLockfile } from '@yarnpkg/lockfile';
import { BackstagePackage } from '@backstage/cli-node';
const ENTRY_PATTERN = /^((?:@[^/]+\/)?[^@/]+)@(.+)$/;
@@ -39,35 +37,6 @@ type LockfileQueryEntry = {
dataKey: string;
};
/** Entries that have an invalid version range, for example an npm tag */
type AnalyzeResultInvalidRange = {
name: string;
range: string;
};
/** Entries that can be deduplicated by bumping to an existing higher version */
type AnalyzeResultNewVersion = {
name: string;
range: string;
oldVersion: string;
newVersion: string;
};
/** Entries that would need a dependency update in package.json to be deduplicated */
type AnalyzeResultNewRange = {
name: string;
oldRange: string;
newRange: string;
oldVersion: string;
newVersion: string;
};
type AnalyzeResult = {
invalidRanges: AnalyzeResultInvalidRange[];
newVersions: AnalyzeResultNewVersion[];
newRanges: AnalyzeResultNewRange[];
};
// the new yarn header is handled out of band of the parsing
// https://github.com/yarnpkg/berry/blob/0c5974f193a9397630e9aee2b3876cca62611149/packages/yarnpkg-core/sources/Project.ts#L1741-L1746
const NEW_HEADER = `${[
@@ -153,186 +122,6 @@ export class Lockfile {
return this.packages.keys();
}
/** Analyzes the lockfile to identify possible actions and warnings for the entries */
analyze(options: {
filter?: (name: string) => boolean;
localPackages: Map<string, BackstagePackage>;
}): AnalyzeResult {
const { filter, localPackages } = options;
const result: AnalyzeResult = {
invalidRanges: [],
newVersions: [],
newRanges: [],
};
for (const [name, allEntries] of this.packages) {
if (filter && !filter(name)) {
continue;
}
// Get rid of and signal any invalid ranges upfront
const invalid = allEntries.filter(
e => !semver.validRange(e.range) && !e.range.startsWith('workspace:'),
);
result.invalidRanges.push(
...invalid.map(({ range }) => ({ name, range })),
);
// Grab all valid entries, if there aren't at least 2 different valid ones we're done
const entries = allEntries.filter(e => semver.validRange(e.range));
if (entries.length < 2) {
continue;
}
// Find all versions currently in use
const versions = Array.from(new Set(entries.map(e => e.version)))
.map(v => {
// Translate workspace:^ references to the actual version
if (v === '0.0.0-use.local') {
const local = localPackages.get(name);
if (!local) {
throw new Error(`No local package found for ${name}`);
}
if (!local.packageJson.version) {
throw new Error(`No version found for local package ${name}`);
}
return {
entryVersion: v,
actualVersion: local.packageJson.version,
};
}
return { entryVersion: v, actualVersion: v };
})
.sort((v1, v2) => semver.rcompare(v1.actualVersion, v2.actualVersion));
// If we're not using at least 2 different versions we're done
if (versions.length < 2) {
continue;
}
// TODO(Rugvip): Support bumping into workspace ranges too
const acceptedVersions = new Set<string>();
for (const { version, range } of entries) {
// Finds the highest matching version from the the known versions
// TODO(Rugvip): We may want to select the version that satisfies the most ranges rather than the highest one
const acceptedVersion = versions.find(v =>
semver.satisfies(v.actualVersion, range),
);
if (!acceptedVersion) {
throw new Error(
`No existing version was accepted for range ${range}, searching through ${versions}, for package ${name}`,
);
}
if (acceptedVersion.entryVersion !== version) {
result.newVersions.push({
name,
range,
newVersion: acceptedVersion.entryVersion,
oldVersion: version,
});
}
acceptedVersions.add(acceptedVersion.actualVersion);
}
// If all ranges were able to accept the same version, we're done
if (acceptedVersions.size === 1) {
continue;
}
// Find the max version that we may want bump older packages to
const maxVersion = Array.from(acceptedVersions).sort(semver.rcompare)[0];
// Find all existing ranges that satisfy the new max version, and pick the one that
// results in the highest minimum allowed version, usually being the more specific one
const maxEntry = entries
.filter(e => semver.satisfies(maxVersion, e.range))
.map(e => ({ e, min: semver.minVersion(e.range) }))
.filter(p => p.min)
.sort((a, b) => semver.rcompare(a.min!, b.min!))[0]?.e;
if (!maxEntry) {
throw new Error(
`No entry found that satisfies max version '${maxVersion}'`,
);
}
// Find all entries that don't satisfy the max version
for (const { version, range } of entries) {
if (semver.satisfies(maxVersion, range)) {
continue;
}
result.newRanges.push({
name,
oldRange: range,
newRange: maxEntry.range,
oldVersion: version,
newVersion: maxVersion,
});
}
}
return result;
}
remove(name: string, range: string): boolean {
const query = `${name}@${range}`;
const existed = Boolean(this.data[query]);
delete this.data[query];
const newEntries = this.packages.get(name)?.filter(e => e.range !== range);
if (newEntries) {
this.packages.set(name, newEntries);
}
return existed;
}
/** Modifies the lockfile by bumping packages to the suggested versions */
replaceVersions(results: AnalyzeResultNewVersion[]) {
for (const { name, range, oldVersion, newVersion } of results) {
const query = `${name}@${range}`;
// Update the backing data
const entryData = this.data[query];
if (!entryData) {
throw new Error(`No entry data for ${query}`);
}
if (entryData.version !== oldVersion) {
throw new Error(
`Expected existing version data for ${query} to be ${oldVersion}, was ${entryData.version}`,
);
}
// Modifying the data in the entry is not enough, we need to reference an existing version object
const matchingEntry = Object.entries(this.data).find(
([q, e]) => q.startsWith(`${name}@`) && e.version === newVersion,
);
if (!matchingEntry) {
throw new Error(
`No matching entry found for ${name} at version ${newVersion}`,
);
}
this.data[query] = matchingEntry[1];
// Update our internal data structure
const entry = this.packages.get(name)?.find(e => e.range === range);
if (!entry) {
throw new Error(`No entry data for ${query}`);
}
if (entry.version !== oldVersion) {
throw new Error(
`Expected existing version data for ${query} to be ${oldVersion}, was ${entryData.version}`,
);
}
entry.version = newVersion;
}
}
async save(path: string) {
await fs.writeFile(path, this.toString(), 'utf8');
}
toString() {
return this.legacy
? legacyStringifyLockfile(this.data)
plugins/devtools-backend/src/util/Lockfile.ts
+1 -188
View File
@@ -15,7 +15,6 @@
*/
import fs from 'fs-extra';
import semver from 'semver';
import { parseSyml, stringifySyml } from '@yarnpkg/parsers';
import { stringify as legacyStringifyLockfile } from '@yarnpkg/lockfile';
@@ -35,35 +34,6 @@ type LockfileQueryEntry = {
version: string;
};
/** Entries that have an invalid version range, for example an npm tag */
type AnalyzeResultInvalidRange = {
name: string;
range: string;
};
/** Entries that can be deduplicated by bumping to an existing higher version */
type AnalyzeResultNewVersion = {
name: string;
range: string;
oldVersion: string;
newVersion: string;
};
/** Entries that would need a dependency update in package.json to be deduplicated */
type AnalyzeResultNewRange = {
name: string;
oldRange: string;
newRange: string;
oldVersion: string;
newVersion: string;
};
type AnalyzeResult = {
invalidRanges: AnalyzeResultInvalidRange[];
newVersions: AnalyzeResultNewVersion[];
newRanges: AnalyzeResultNewRange[];
};
function parseLockfile(lockfileContents: string) {
try {
return {
@@ -135,11 +105,10 @@ export class Lockfile {
queries.push({ range, version: value.version });
}
return new Lockfile(path, packages, data, legacy);
return new Lockfile(packages, data, legacy);
}
private constructor(
private readonly path: string,
private readonly packages: Map<string, LockfileQueryEntry[]>,
private readonly data: LockfileData,
private readonly legacy: boolean = false,
@@ -155,162 +124,6 @@ export class Lockfile {
return this.packages.keys();
}
/** Analyzes the lockfile to identify possible actions and warnings for the entries */
analyze(options?: { filter?: (name: string) => boolean }): AnalyzeResult {
const { filter } = options ?? {};
const result: AnalyzeResult = {
invalidRanges: [],
newVersions: [],
newRanges: [],
};
for (const [name, allEntries] of this.packages) {
if (filter && !filter(name)) {
continue;
}
// Get rid of and signal any invalid ranges upfront
const invalid = allEntries.filter(e => !semver.validRange(e.range));
result.invalidRanges.push(
...invalid.map(({ range }) => ({ name, range })),
);
// Grab all valid entries, if there aren't at least 2 different valid ones we're done
const entries = allEntries.filter(e => semver.validRange(e.range));
if (entries.length < 2) {
continue;
}
// Find all versions currently in use
const versions = Array.from(new Set(entries.map(e => e.version))).sort(
(v1, v2) => semver.rcompare(v1, v2),
);
// If we're not using at least 2 different versions we're done
if (versions.length < 2) {
continue;
}
const acceptedVersions = new Set<string>();
for (const { version, range } of entries) {
// Finds the highest matching version from the the known versions
// TODO(Rugvip): We may want to select the version that satisfies the most ranges rather than the highest one
const acceptedVersion = versions.find(v => semver.satisfies(v, range));
if (!acceptedVersion) {
throw new Error(
`No existing version was accepted for range ${range}, searching through ${versions}, for package ${name}`,
);
}
if (acceptedVersion !== version) {
result.newVersions.push({
name,
range,
newVersion: acceptedVersion,
oldVersion: version,
});
}
acceptedVersions.add(acceptedVersion);
}
// If all ranges were able to accept the same version, we're done
if (acceptedVersions.size === 1) {
continue;
}
// Find the max version that we may want bump older packages to
const maxVersion = Array.from(acceptedVersions).sort(semver.rcompare)[0];
// Find all existing ranges that satisfy the new max version, and pick the one that
// results in the highest minimum allowed version, usually being the more specific one
const maxEntry = entries
.filter(e => semver.satisfies(maxVersion, e.range))
.map(e => ({ e, min: semver.minVersion(e.range) }))
.filter(p => p.min)
.sort((a, b) => semver.rcompare(a.min!, b.min!))[0]?.e;
if (!maxEntry) {
throw new Error(
`No entry found that satisfies max version '${maxVersion}'`,
);
}
// Find all entries that don't satisfy the max version
for (const { version, range } of entries) {
if (semver.satisfies(maxVersion, range)) {
continue;
}
result.newRanges.push({
name,
oldRange: range,
newRange: maxEntry.range,
oldVersion: version,
newVersion: maxVersion,
});
}
}
return result;
}
remove(name: string, range: string): boolean {
const query = `${name}@${range}`;
const existed = Boolean(this.data[query]);
delete this.data[query];
const newEntries = this.packages.get(name)?.filter(e => e.range !== range);
if (newEntries) {
this.packages.set(name, newEntries);
}
return existed;
}
/** Modifies the lockfile by bumping packages to the suggested versions */
replaceVersions(results: AnalyzeResultNewVersion[]) {
for (const { name, range, oldVersion, newVersion } of results) {
const query = `${name}@${range}`;
// Update the backing data
const entryData = this.data[query];
if (!entryData) {
throw new Error(`No entry data for ${query}`);
}
if (entryData.version !== oldVersion) {
throw new Error(
`Expected existing version data for ${query} to be ${oldVersion}, was ${entryData.version}`,
);
}
// Modifying the data in the entry is not enough, we need to reference an existing version object
const matchingEntry = Object.entries(this.data).find(
([q, e]) => q.startsWith(`${name}@`) && e.version === newVersion,
);
if (!matchingEntry) {
throw new Error(
`No matching entry found for ${name} at version ${newVersion}`,
);
}
this.data[query] = matchingEntry[1];
// Update our internal data structure
const entry = this.packages.get(name)?.find(e => e.range === range);
if (!entry) {
throw new Error(`No entry data for ${query}`);
}
if (entry.version !== oldVersion) {
throw new Error(
`Expected existing version data for ${query} to be ${oldVersion}, was ${entryData.version}`,
);
}
entry.version = newVersion;
}
}
async save() {
await fs.writeFile(this.path, this.toString(), 'utf8');
}
toString() {
return stringifyLockfile(this.data, this.legacy);
}