[rush-lib] Support pnpm lockfile v9 #5009

fzxen · 2024-11-19T11:22:34Z

Summary

pnpm lockfile v9 have some breaking changes on the lockfile format. Rush cannot parse pnpm lockfile v9 correctly with latest version.
After i execute rush install or rush update with pnpm v9, the content of .rush/temp/shrinkwrap-deps.json is not correct. It may break rush cache system.
The expected output should correctly display the hash of each dependency., but actually:

Details

pnpm have some breaking changes on lockfile v9 format.

The packages field has been divided into two parts, packages and snapshots.
In non-workspace mode, the top-level dependencies field is moved into importers['.'].
specifier is not same as lockfile v6

slolution to 1:

Rush will load the lockfile and parse the information in the lockfile by itself.

rushstack/libraries/rush-lib/src/logic/pnpm/PnpmShrinkwrapFile.ts

Lines 318 to 347 in 3c6117e

    
           public static loadFromFile( 
        
             shrinkwrapYamlFilePath: string, 
        
             { withCaching }: ILoadFromFileOptions = {} 
        
           ): PnpmShrinkwrapFile | undefined { 
        
             let loaded: PnpmShrinkwrapFile | undefined; 
        
             if (withCaching) { 
        
               loaded = PnpmShrinkwrapFile._cacheByLockfilePath.get(shrinkwrapYamlFilePath); 
        
             } 
        
             // TODO: Promisify this 
        
             loaded ??= (() => { 
        
               try { 
        
                 const shrinkwrapContent: string = FileSystem.readFile(shrinkwrapYamlFilePath); 
        
                 return PnpmShrinkwrapFile.loadFromString(shrinkwrapContent); 
        
               } catch (error) { 
        
                 if (FileSystem.isNotExistError(error as Error)) { 
        
                   return undefined; // file does not exist 
        
                 } 
        
                 throw new Error(`Error reading "${shrinkwrapYamlFilePath}":\n  ${(error as Error).message}`); 
        
               } 
        
             })(); 
        
             PnpmShrinkwrapFile._cacheByLockfilePath.set(shrinkwrapYamlFilePath, loaded); 
        
             return loaded; 
        
           } 
        
           public static loadFromString(shrinkwrapContent: string): PnpmShrinkwrapFile { 
        
             const parsedData: IPnpmShrinkwrapYaml = yamlModule.safeLoad(shrinkwrapContent); 
        
             return new PnpmShrinkwrapFile(parsedData); 
        
           }

To ensure consistency with pnpm's parsing logic, I copied the relevant logic from @pnpm/lockfile.fs to PnpmshrinwrapFileConverters.ts.
The convertLockfileV9ToLockfileObject method will automatically merge the snapshots field information into the packages field.

public static loadFromString(shrinkwrapContent: string): PnpmShrinkwrapFile {
  const shrinkwrapJson: IPnpmShrinkwrapYaml = yamlModule.safeLoad(shrinkwrapContent);
  if ((shrinkwrapJson as LockfileFileV9).snapshots) {
    const lockfile: IPnpmShrinkwrapYaml | null = convertLockfileV9ToLockfileObject(
      shrinkwrapJson as LockfileFileV9
    );
    if (lockfile) {
      lockfile.dependencies = lockfile.importers['.' as ProjectId]?.dependencies;
    }
    return new PnpmShrinkwrapFile(lockfile);
  }

  return new PnpmShrinkwrapFile(shrinkwrapJson);
}

solution to 2:

In the PnpmShrinkwrapFile.loadFromString method, read importers['.'].dependencies instead of top-level dependencies.

// if ((shrinkwrapJson as LockfileFileV9).snapshots)  {
// ...
if (lockfile) {
  lockfile.dependencies = lockfile.importers['.' as ProjectId]?.dependencies;
  shrinkwrapFileJson = lockfile;
}
// }

solution to 3:

rush try to parse an encoded pnpm dependency key in parsePnpmDependencyKey method. However, the logic here is no longer applicable to lockfile v9. For example, lockfile v9 will not add a / prefix in the specifier field.

rushstack/libraries/rush-lib/src/logic/pnpm/PnpmShrinkwrapFile.ts

Lines 188 to 198 in 3c6117e

	// Example: "path.pkgs.visualstudio.com/@scope/depame/1.4.0" --> 0="@scope/depame" 1="1.4.0"
	// Example: "/isarray/2.0.1" --> 0="isarray" 1="2.0.1"
	// Example: "/sinon-chai/2.8.0/[email protected][email protected]" --> 0="sinon-chai" 1="2.8.0/[email protected][email protected]"
	// Example: "/[email protected]" --> 0=typescript 1="5.1.6"
	// Example: [email protected] --> no match
	// Example: 1.2.3_@[email protected] --> no match
	// Example: 1.2.3([email protected]) --> no match
	// Example: 1.2.3(@scope/[email protected]) --> no match
	const packageNameMatch: RegExpMatchArray \| null = /^[^\/(]\/((?:@[^\/(]+\/)?[^\/(]+)[\/@](.)$/.exec(
	dependencyKey
	);

Summary of changes to specifier in lockfile v9：

remove prefix '/‘: '/@babel/[email protected](@babel/[email protected])' -> @babel/[email protected](@babel/[email protected])
URLs specifier always prefix with https:.
it will prefix with <PACKAGE_NAME>@ if resolved package name is not same as dependency name in package.json

category	specifier	lockfilev6 version	lockfilev9 version
regular	"@babel/plugin-preset-env": "^7.26.0"	7.26.0(@babel/[email protected])	7.26.0(@babel/[email protected])
URLs	"pad-left": "https://github.com/jonschlinkert/pad-left/tarball/2.1.0"	@github.com/jonschlinkert/pad-left/tarball/2.1.0	https://github.com/jonschlinkert/pad-left/tarball/2.1.0
	"pad-left": "https://xxx.xxx.org/pad-left/-/pad-left-2.1.0.tgz"	@xxx.xxx.org/pad-left/-/pad-left-2.1.0.tgz	https://xxx.xxx.org/pad-left/-/pad-left-2.1.0.tgz
	"pad-left": "git://github.com/jonschlinkert/pad-left#2.1.0"	github.com/jonschlinkert/pad-left/7798d648225aa5d879660a37c408ab4675b65ac7	https://codeload.github.com/jonschlinkert/pad-left/tar.gz/7798d648225aa5d879660a37c408ab4675b65ac7
	"pad-left": "git+ssh://[email protected]:jonschlinkert/pad-left.git#2.1.0"	github.com/jonschlinkert/pad-left/7798d648225aa5d879660a37c408ab4675b65ac7	https://codeload.github.com/jonschlinkert/pad-left/tar.gz/7798d648225aa5d879660a37c408ab4675b65ac7
file:	"project1": "file:../pnpm_no_workspace/projects/project1"	file:../pnpm_no_workspace/projects/project1	file:../pnpm_no_workspace/projects/project1
path	"project1": "../pnpm_no_workspace/projects/project1"	link:../pnpm_no_workspace/projects/project1	link:../pnpm_no_workspace/projects/project1
Npm alias	"test-pkg": "npm:@babel/[email protected]"	/@babel/[email protected](@babel/[email protected])	@babel/[email protected](@babel/[email protected])
Alias with URLs	"@scope/myDep1": "https://github.com/jonschlinkert/pad-left/tarball/2.1.0"	@github.com/jonschlinkert/pad-left/tarball/2.1.0	pad-left@https://github.com/jonschlinkert/pad-left/tarball/2.1.0
	"@scope/myDep2": "https://xxx.xxx.org/pad-left/-/pad-left-2.1.0.tgz"	@xxx.xxx.org/pad-left/-/pad-left-2.1.0.tgz	pad-left@https://xxx.xxx.org/pad-left/-/pad-left-2.1.0.tgz
	"@scope/myDep3": "git://github.com/jonschlinkert/pad-left#2.1.0"	github.com/jonschlinkert/pad-left/7798d648225aa5d879660a37c408ab4675b65ac7	pad-left@https://codeload.github.com/jonschlinkert/pad-left/tar.gz/7798d648225aa5d879660a37c408ab4675b65ac7
	"@scope/myDep4": "git+ssh://[email protected]:jonschlinkert/pad-left.git#2.1.0"	github.com/jonschlinkert/pad-left/7798d648225aa5d879660a37c408ab4675b65ac7	pad-left@https://codeload.github.com/jonschlinkert/pad-left/tar.gz/7798d648225aa5d879660a37c408ab4675b65ac7
Alias with file:	"test-pkg": "file:../pnpm_no_workspace/projects/project1"	file:../pnpm_no_workspace/projects/project1	project1@file:../pnpm_no_workspace/projects/project1

It is difficult to maintain those complex regular expressions in the original function. Therefore, I added a new function named as parsePnpm9DependencyKey. It will be called when the expression shrinkwrapFileMajorVersion >= 9 is true in runtime.

const result: DependencySpecifier | undefined =
        this.shrinkwrapFileMajorVersion >= ShrinkwrapFileMajorVersion.V9
          ? parsePnpm9DependencyKey(dependencyName, pnpmDependencyKey)
          : parsePnpmDependencyKey(dependencyName, pnpmDependencyKey);

This MR will not cause any breaking changes. But, pnpm9 and rush subspaces still cannot work together, because pnpm-sync not support pnpm9 yet. tiktok/pnpm-sync#37

I tried use commonly rush command in my own project.

rush install/update -> The content of shrinkwrap-deps.json file is correct.
rush build -> Build cache and Cobuild work as expected.
I added some unit test cases in PnpmShrinkwrapFile.test.ts and ShrinkwrapFile.test.ts.
At the same time, unit tests have also been added for the PnpmShrinkWrapFileConverters.ts file.

fzxen · 2024-11-19T11:34:04Z

@microsoft-github-policy-service agree

iclanton

Thanks for putting this together! I'll take a closer look, hopefully tomorrow.

It'd also be good to get some more eyes on this. @octogonz, @dmichon-msft, @D4N14L?

common/changes/@microsoft/rush/feature-support_pnpmv9_2024-11-19-08-24.json

common/config/rush/browser-approved-packages.json

libraries/rush-lib/package.json

common/changes/@rushstack/lockfile-explorer/feature-support_pnpmv9_2024-11-25-14-05.json

libraries/rush-lib/src/logic/pnpm/PnpmShrinkWrapFileConverters.ts

libraries/rush-lib/src/logic/pnpm/PnpmShrinkwrapFile.ts

jeremymeng · 2024-11-27T22:36:37Z

@fzxen it is great to see support for pnpm v9 coming! I tested the changes against our repo which is not using workspaces. It reported an error:

ERROR: Internal Error: Cannot find tarball path for
"@rush-temp/agrifood-farming" in shrinkwrap.

I ran this github workflow. Do you have any insights?

fzxen · 2024-11-28T09:58:28Z

@fzxen it is great to see support for pnpm v9 coming! I tested the changes against our repo which is not using workspaces. It reported an error:

ERROR: Internal Error: Cannot find tarball path for
"@rush-temp/agrifood-farming" in shrinkwrap.

I ran this github workflow. Do you have any insights?

I took a deep look. Lockfile v9 has a breaking change that I have ignored, namely that importers[x].dependencies[xx].version may is not equal to the key in the packages field.

In the following example, The key of "packages" field is always in the format of "name@version", but the "version" field is missing the "name" information, which causes rush to be unable to obtain package data.

importers:
  .:
    dependencies:
      'project1':
        specifier: file:./projects/project1
        version: file:projects/project1

packages:
  project1@file:projects/project1:
    resolution: {directory: projects/project1, type: directory}

I have fixed this issue on my branch, you can try again.
Additionally, I found that when upgrading to pnpm v9 for the first time, rush may not update common/config/rush/pnpm-lock.yaml. This is because the logic of rush to judge whether shrinkwrapfile is updated does not take into account the change of lockfile version. Therefore, I suggest you use the rush update --recheck command to update the lockfile from v6 to v9.

jeremymeng · 2024-12-02T18:52:53Z

@fzxen Thanks a lot! Yes the latest version of your feature branch and rush update --recheck works on our repo https://github.com/jeremymeng/rushstack/actions/runs/12124656853/job/33802844214. 💯

iclanton · 2024-12-04T00:49:10Z

I've published a prerelease as version 5.143.0-pr5009.0. If that looks good in a few repos, we'll merge this in and publish it as a regular release.

jeremymeng · 2024-12-04T18:12:51Z

I tried 5.143.0-pr5009.1 on Azure/azure-sdk-for-js@main...pnpm-9.14.2. rush update --full works. However, rush install/rush update/rush update --recheck all gave the error of

Updating temp projects in /home/meng/git/jssdk/common/temp/projects

ERROR: Invalid tag name "[email protected]": Tags may not have any characters that encodeURIComponent encodes.

We pin this chai version in common-versions.json https://github.com/Azure/azure-sdk-for-js/blob/c2b7ebafb3919daa306a895d42a910c7fa8530aa/common/config/rush/common-versions.json#L19

iclanton · 2024-12-04T19:09:14Z

@fzxen - can you investigate @jeremymeng's issue?

iclanton · 2024-12-04T19:41:18Z

@fzxen - Seeing issues here too: microsoft/tsdoc#430.

Looks like this breaks non-workspaces projects.

fzxen · 2024-12-09T11:58:34Z

@fzxen - can you investigate @jeremymeng's issue?

Alright, I'll take a look in the next few days.

fzxen · 2024-12-10T12:39:44Z

I tried 5.143.0-pr5009.1 on Azure/[email protected]. rush update --full works. However, rush install/rush update/rush update --recheck all gave the error of

Updating temp projects in /home/meng/git/jssdk/common/temp/projects
ERROR: Invalid tag name "[email protected]": Tags may not have any characters that encodeURIComponent encodes.

We pin this chai version in common-versions.json https://github.com/Azure/azure-sdk-for-js/blob/c2b7ebafb3919daa306a895d42a910c7fa8530aa/common/config/rush/common-versions.json#L19

@jeremymeng I found the cause of this issue and fixed it on my branch. This issue will only affect non-workspaces projects.

fzxen · 2024-12-10T12:46:13Z

@fzxen - Seeing issues here too: microsoft/tsdoc#430.

Looks like this breaks non-workspaces projects.

@iclanton This error seems to be caused by an incorrect pnpm-config.js configuration. Maybe you should set useWorkspaces field as false in common/config/rush/pnpm-config.json.

iclanton · 2024-12-10T19:04:36Z

@fzxen, ah I misremembered. I thought that repo was non-workspaces. @D4N14L thinks the issue is with a cyclic dependency. I'm going to do another prerelease from your branch (this branch).

iclanton · 2024-12-10T19:35:57Z

The latest version of this branch has been published as 5.145.0-pr5009.2

iclanton · 2024-12-10T21:10:21Z

@jeremymeng's repo looks good, so I'm going to go ahead and merge this.

iclanton · 2024-12-10T21:10:50Z

Thanks @fzxen!

jeremymeng · 2024-12-10T21:11:18Z

@jeremymeng I found the cause of this issue and fixed it on my branch. This issue will only affect non-workspaces projects.

Thanks @fzxen!

kenrick95 · 2024-12-11T01:19:25Z

Thanks a lot @fzxen!!

dmichon-msft · 2024-12-11T23:14:58Z

Note that pnpm 9 makes hoist-workspace-packages default to true, so we'll need to update our wrapper to force that back off.

vluoto · 2024-12-13T10:28:17Z

Great to see support being added for pnpm v9! Unfortunately it looks like @rushstack/lockfile-explorer didn't get published yet. Moreover, I tried building, packing, and installing it from a tarball myself, only to discover that it still doesn't seem to support pnpm v9. 🤔 Worth noting that I'm trying it out in a pnpm workspace without rush.

Here are the exact steps I took:

$ nvm use 20
$ npm install --global @microsoft/rush
$ git clone https://github.com/microsoft/rushstack
$ cd rushstack
$ git config --local user.name "Vellu Luoto"
$ git config --local user.email "[email protected]"
$ rush update
$ rush build
$ rush publish --pack --include-all --publish

Global installation from tarball:

$ nvm use 22
$ npm install --global /absolute/path/to/rushstack/common/temp/artifacts/packages/rushstack-lockfile-explorer-1.7.11.tgz

Verifying it's actually the one with @pnpm/dependency-path-lockfile-pre-v9:

$ npm --global list @rushstack/lockfile-explorer
 /Users/vluoto/.nvm/versions/node/v22.11.0/lib
 └── @rushstack/[email protected]
$ cat /Users/vluoto/.nvm/versions/node/v22.11.0/lib/node_modules/@rushstack/lockfile-explorer/package.json | grep @pnpm/dependency-path-lockfile-pre-v9
    "@pnpm/dependency-path-lockfile-pre-v9": "npm:@pnpm/dependency-path@~2.1.2",

And here's the error this resulted in:

$ lockfile-explorer

Rush Lockfile Explorer 1.7.12 - https://lfx.rushstack.io/

App launched on http://localhost:8091
/Users/vluoto/.nvm/versions/node/v22.11.0/lib/node_modules/@rushstack/lockfile-explorer/lib/utils/shrinkwrap.js:65
        throw new Error('The current lockfile version is not supported.');
              ^

Error: The current lockfile version is not supported.
    at getShrinkwrapFileMajorVersion (/Users/vluoto/.nvm/versions/node/v22.11.0/lib/node_modules/@rushstack/lockfile-explorer/lib/utils/shrinkwrap.js:65:15)
    at /Users/vluoto/.nvm/versions/node/v22.11.0/lib/node_modules/@rushstack/lockfile-explorer/lib/cli/explorer/ExplorerCommandLineParser.js:109:95

Node.js v22.11.0

fzxen force-pushed the feature/support_pnpmv9 branch 4 times, most recently from 8111dd3 to 28db14d Compare November 20, 2024 10:59

fzxen marked this pull request as ready for review November 20, 2024 11:32

fzxen requested review from iclanton, octogonz, apostolisms, D4N14L, dmichon-msft and patmill as code owners November 20, 2024 11:32

fzxen mentioned this pull request Nov 21, 2024

[rush] Feature Request: Support PNPM 9 #4841

Closed

iclanton reviewed Nov 25, 2024

View reviewed changes

common/changes/@microsoft/rush/feature-support_pnpmv9_2024-11-19-08-24.json Outdated Show resolved Hide resolved

common/config/rush/browser-approved-packages.json Outdated Show resolved Hide resolved

libraries/rush-lib/package.json Outdated Show resolved Hide resolved

fzxen force-pushed the feature/support_pnpmv9 branch 4 times, most recently from 4d3d410 to 2cd17d5 Compare November 25, 2024 14:15

iclanton reviewed Nov 26, 2024

View reviewed changes

fzxen force-pushed the feature/support_pnpmv9 branch from 2cd17d5 to 21a7a65 Compare November 26, 2024 05:16

fzxen force-pushed the feature/support_pnpmv9 branch from 21a7a65 to 91f2e9f Compare November 28, 2024 07:27

fzxen force-pushed the feature/support_pnpmv9 branch from 91f2e9f to c7bbff2 Compare December 2, 2024 02:26

iclanton force-pushed the feature/support_pnpmv9 branch from c7bbff2 to ff02844 Compare December 4, 2024 03:04

iclanton mentioned this pull request Dec 4, 2024

Bump Rush to 5.143.0-pr5009.1. #5028

Merged

Support pnpm v9

d9efedb

fzxen force-pushed the feature/support_pnpmv9 branch from ff02844 to d9efedb Compare December 10, 2024 12:14

iclanton enabled auto-merge (squash) December 10, 2024 21:10

iclanton approved these changes Dec 10, 2024

View reviewed changes

iclanton merged commit 3f9aa73 into microsoft:main Dec 10, 2024
6 checks passed

octogonz mentioned this pull request Dec 18, 2024

[rush] ERROR: unacceptable kind of an object to dump (undefined) #5053

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[rush-lib] Support pnpm lockfile v9 #5009

[rush-lib] Support pnpm lockfile v9 #5009

fzxen commented Nov 19, 2024 •

edited

Loading

fzxen commented Nov 19, 2024

iclanton left a comment

jeremymeng commented Nov 27, 2024

fzxen commented Nov 28, 2024 •

edited

Loading

jeremymeng commented Dec 2, 2024

iclanton commented Dec 4, 2024

jeremymeng commented Dec 4, 2024

iclanton commented Dec 4, 2024

iclanton commented Dec 4, 2024

fzxen commented Dec 9, 2024 •

edited

Loading

fzxen commented Dec 10, 2024

fzxen commented Dec 10, 2024 •

edited

Loading

iclanton commented Dec 10, 2024

iclanton commented Dec 10, 2024

iclanton commented Dec 10, 2024

iclanton commented Dec 10, 2024

jeremymeng commented Dec 10, 2024

kenrick95 commented Dec 11, 2024

dmichon-msft commented Dec 11, 2024

vluoto commented Dec 13, 2024 •

edited

Loading

	public static loadFromFile(
	shrinkwrapYamlFilePath: string,
	{ withCaching }: ILoadFromFileOptions = {}
	): PnpmShrinkwrapFile \| undefined {
	let loaded: PnpmShrinkwrapFile \| undefined;
	if (withCaching) {
	loaded = PnpmShrinkwrapFile._cacheByLockfilePath.get(shrinkwrapYamlFilePath);
	}

	// TODO: Promisify this
	loaded ??= (() => {
	try {
	const shrinkwrapContent: string = FileSystem.readFile(shrinkwrapYamlFilePath);
	return PnpmShrinkwrapFile.loadFromString(shrinkwrapContent);
	} catch (error) {
	if (FileSystem.isNotExistError(error as Error)) {
	return undefined; // file does not exist
	}
	throw new Error(`Error reading "${shrinkwrapYamlFilePath}":\n ${(error as Error).message}`);
	}
	})();

	PnpmShrinkwrapFile._cacheByLockfilePath.set(shrinkwrapYamlFilePath, loaded);
	return loaded;
	}

	public static loadFromString(shrinkwrapContent: string): PnpmShrinkwrapFile {
	const parsedData: IPnpmShrinkwrapYaml = yamlModule.safeLoad(shrinkwrapContent);
	return new PnpmShrinkwrapFile(parsedData);
	}

[rush-lib] Support pnpm lockfile v9 #5009

[rush-lib] Support pnpm lockfile v9 #5009

Conversation

fzxen commented Nov 19, 2024 • edited Loading

Summary

Details

slolution to 1:

solution to 2:

solution to 3:

fzxen commented Nov 19, 2024

iclanton left a comment

Choose a reason for hiding this comment

jeremymeng commented Nov 27, 2024

fzxen commented Nov 28, 2024 • edited Loading

jeremymeng commented Dec 2, 2024

iclanton commented Dec 4, 2024

jeremymeng commented Dec 4, 2024

iclanton commented Dec 4, 2024

iclanton commented Dec 4, 2024

fzxen commented Dec 9, 2024 • edited Loading

fzxen commented Dec 10, 2024

fzxen commented Dec 10, 2024 • edited Loading

iclanton commented Dec 10, 2024

iclanton commented Dec 10, 2024

iclanton commented Dec 10, 2024

iclanton commented Dec 10, 2024

jeremymeng commented Dec 10, 2024

kenrick95 commented Dec 11, 2024

dmichon-msft commented Dec 11, 2024

vluoto commented Dec 13, 2024 • edited Loading

fzxen commented Nov 19, 2024 •

edited

Loading

fzxen commented Nov 28, 2024 •

edited

Loading

fzxen commented Dec 9, 2024 •

edited

Loading

fzxen commented Dec 10, 2024 •

edited

Loading

vluoto commented Dec 13, 2024 •

edited

Loading