Support grouping all packages together and releasing a single version

[noahnu]

Is your feature request related to a problem? Please describe.

Essentially the opposite of https://github.com/lerna/lerna#independent-mode. The use case is to support a repo such as "jest" where all the packages get the same version.

This is "fixed" mode in Lerna

Describe the solution you'd like

CLI flag + config option.

[noahnu]

Starting by adding the ability to create groups using an arbitrary field in a workspace's manifest.

[markandrus]

I have found a use-case for this. I have a package setup very similar to esbuild (see the npm folder in that project). Specifically, I have

• A native module I build for multiple OS × architectures combinations
• Multiple OS × architecture-specific packages, wrapping the native modules
• An OS × architecture-independent package that lists all the other packages as optionalDependencies and determines which optional native module to invoke at runtime

I would really like to be able to specify workspace:* in the optionalDependencies, and have everything get released together as one version.

---

A related challenge: once I setup the package structure, I don't really need to touch the JavaScript packages again. I only need to iterate on the native code. I continue to use conventional commits for this, but how can I instruct monodeploy to treat changes to files outside of any particular workspace's directory as relevant to the release process? It would be super cool if monodeploy had a solution here.

[noahnu]

how can I instruct monodeploy to treat changes to files outside of any particular workspace's directory as relevant to the release process

@markandrus If you have thoughts on how we could determine the dependencies that live outside of the yarn workspace structure, I'd love to hear it. First thoughts that comes to mind would be supporting per-workspace monodeploy config files and adding an explicit "extraDependencies" option. Pulling it from the package.json is also an option.

Another thought that comes to mind: create a private workspace that houses the native source code. Then list the native workspace as a devDependency of each of the architecture-specific workspaces. When the native code is modified, it'll propagate the change up to the dependent architecture-specific workspaces. The main issue you'll have here is that it'll limit the changes to patch version bumps. We'd need to think of a way to support non-patch bumps with that project structure.

[noahnu]

WIP: #453

[noahnu]

I have a couple questions on my implementation of this that need answers. Will bounce the ideas around with my team but am also curious if there's a preference that would work better with your workflow @markandrus :

Taken from the GitHub PR:

Questions before merging

1. If we have 2 packages belonging to the same group, and only 1 package is modified, do we bump the versions for only the modified package and its dependents? Or all packages in the group?

   • Note that the current implementation in this PR does not touch packages which aren't modified, which can cause drift in the version numbers. However when there is a single package that is the "main" package (e.g. monodeploy here, or "webpack"), almost any other change will trigger a bump to the main package via dependent bump.

2. If we have 2 packages modified in a group, 1 with a patch and 1 with a feat, do we apply the great version strategy to both (i.e. feat here) and then take the max for the largest version to then apply? This is important when the versions for each package don't start off the same, for example when adding groups to an existing monorepo.

   • Note that the current implementation in this PR only updates the target versions but leaves the initial strategies. This results in cases where a feat can cause all packages in a group to end with a patch-like version number, e.g. 4.5.1, as long as one package has a patch strategy and is greater than the other version numbers in the group.

I'm leaning towards leaving the no. 1 to the current implementation, i.e. only update what's been modified, but changing no. 2 so we also update the strategies. Thoughts?

[noahnu]

Will be released in v3 #450

[markandrus]

@noahnu thanks for the feedback request. I'm excited for this feature! I took a look at the updated Markdown and your comments (didn't dive into the implementation). Here are my thoughts:

1. If we have 2 packages belonging to the same group, and only 1 package is modified, do we bump the versions for only the modified package and its dependents? Or all packages in the group?

I would expect we bump versions for all packages in the group. For example, if I had a group "default" with packages "foo" and "bar", both at version 1.0.0, and I add a feature to "foo", I expect both packages to get bumped to 1.1.0. This is what being in a fixed group means to me: everything gets released at the same version. Is there another way to think about it that I am missing? Or a benefit to bumping versions "only for the modified package and its dependents"?

• Note that the current implementation in this PR does not touch packages which aren't modified, which can cause drift in the version numbers. However when there is a single package that is the "main" package (e.g. monodeploy here, or "webpack"), almost any other change will trigger a bump to the main package via dependent bump.

I'm not sure I follow, and I'm curious about the following cases: building on the example above, what happens to the versions if

• "foo" and "bar" are independent, and "foo" gets a new feature?
• "bar" depends on "foo", and "foo" gets a new feature?
• "foo" depends on "bar", and "foo" gets a new feature?

I would expect all packages in a group get touched, irrespective of dependencies.

2. If we have 2 packages modified in a group, 1 with a patch and 1 with a feat, do we apply the great version strategy to both (i.e. feat here) and then take the max for the largest version to then apply? This is important when the versions for each package don't start off the same, for example when adding groups to an existing monorepo.

IIUC, we take the greatest version strategy, apply that to every version in the group, take the greatest resulting version, and then apply that to every package in the group? So when migrating "foo" and "bar" to a group "default", we could imagine the following scenarios:

• Packages in the group already share the same version: "foo" is at 1.0.0 and "bar" is at 1.0.0. "foo" gets a feature, so we increment the minor version for all packages in "default", resulting in 1.1.0 and 1.1.0, respectively. 1.1.0 is greatest, so both "foo" and "bar" get their versions set to 1.1.0.
• The modified package has a higher version number in the group: "foo" is at 2.0.0 and "bar" is at 1.0.0. "foo" gets a feature, so we increment the minor version for all packages in "default", resulting in 2.1.0 and 1.1.0, respectively. 2.1.0 is greatest, so both "foo" and "bar" get their versions set to 2.1.0.
• The modified package has a lower version number in the group: "foo" is at 1.0.0 and "bar" is at 2.0.0. "foo" gets a feature, so we increment the minor version for all packages in "default", resulting in 1.1.0 and 2.1.0, respectively. 2.1.0 is greatest, so both "foo" and "bar" get their versions set to 2.1.0.

I'm assuming all packages in a group get touched, irrespective of dependencies.

• Note that the current implementation in this PR only updates the target versions but leaves the initial strategies. This results in cases where a feat can cause all packages in a group to end with a patch-like version number, e.g. 4.5.1, as long as one package has a patch strategy and is greater than the other version numbers in the group.

OK, and this was changed, as you mentioned ("changing no. 2 according to my previous comment.") — did I represent the behavior accurately above?

[markandrus]

@noahnu regarding your previous comment:

@markandrus If you have thoughts on how we could determine the dependencies that live outside of the yarn workspace structure, I'd love to hear it. First thoughts that comes to mind would be supporting per-workspace monodeploy config files and adding an explicit "extraDependencies" option. Pulling it from the package.json is also an option.

Yes, I could imagine this working. I could imagine passing a glob of source files to check (or globbing them myself). It may make sense to prefer passing directories, since then (I assume) the "pick a strategy" part of monodeploy could do less work; however, I do think this may be asking too much of monodeploy. Especially when your other suggestion (creating a private workspace) would also work at the cost of restructuring my source repository a little bit. Not sure the feature would be worth the maintenance.

Another thought that comes to mind: create a private workspace that houses the native source code. Then list the native workspace as a devDependency of each of the architecture-specific workspaces. When the native code is modified, it'll propagate the change up to the dependent architecture-specific workspaces. The main issue you'll have here is that it'll limit the changes to patch version bumps. We'd need to think of a way to support non-patch bumps with that project structure.

Yes, in fact I was thinking about this again… I am going to file a feature request for discussion. UPDATE: I have filed #456.

[noahnu]

I would expect we bump versions for all packages in the group. For example, if I had a group "default" with packages "foo" and "bar", both at version 1.0.0, and I add a feature to "foo", I expect both packages to get bumped to 1.1.0. This is what being in a fixed group means to me: everything gets released at the same version. Is there another way to think about it that I am missing? Or a benefit to bumping versions "only for the modified package and its dependents"?

I believe lerna only updates modified packages, at least according to their docs

Looking at some other examples out there, if you look at this commit that updates the version for a babel release, @babel/code-frame goes from 7.16.0 to 7.16.7 while @babel/core goes from 7.16.5 to 7.16.7, implying the parent commit had them at different versions. I believe babel uses a custom plugin based on yarn for the releases. I've seen similar behaviour with the react monorepo as well.

@markandrus When we update a dependent via a patch, it's because we're also updating its dependencies in the package.json. So it's never just a version bump. In this case, I think it makes sense to only "release" modified packages and their dependents. Perhaps the alternative behaviour of always applying a bump could somehow be combined into #456, assuming there's a use case for it. Do you have a particular use case where you'd have some value in a non-modified package getting a new version? I suspect in that case it might make sense to merge those packages rather than keep them as distinct workspaces.

[noahnu]

https://github.com/tophat/monodeploy/releases/tag/monodeploy%403.0.0-rc.1

[markandrus]

I believe lerna only updates modified packages, at least according to their docs

Looking at some other examples out there, if you look at this commit that updates the version for a babel release, @babel/code-frame goes from 7.16.0 to 7.16.7 while @babel/core goes from 7.16.5 to 7.16.7, implying the parent commit had them at different versions. I believe babel uses a custom plugin based on yarn for the releases. I've seen similar behaviour with the react monorepo as well.

Ah, interesting — I did not understand how Lerna operates. I think bumping only things that changed works fine (it's clearly been working well for Lerna users).

I'm still confused about Question 2 — if you could share an example, I'd be able to weigh in better.

Do you have a particular use case where you'd have some value in a non-modified package getting a new version? I suspect in that case it might make sense to merge those packages rather than keep them as distinct workspaces.

I have an esbuild-like project where one workspace contains the core code and the other workspaces ship native builds for different OS × arch combinations. The native builds depend on the core code (this could be a dev dependency). Any major-, minor- or patch-level changes in the core code should be directly reflected in the native workspaces' versions. They cannot be merged, because each workspace is specifying its own os and cpu fields in the package.json.

I think it's this use case that had me thinking about versioning in a different way than Lerna manages it. I recognize this may be a very specific use case that monodeploy does not need to support.

[noahnu]

We don't consider devDependencies when propagating updates -- mainly because I haven't dealt with a repo that builds native binaries. I can see the value in specific cases though. Hmm, going to give this more thought. I think there's room for improvement around customizing the version propagation strategy.

It's a 1 line change to consider dev dependencies (few more lines to add the relevant config option). I think listing the "core" package as a devDependency of each "native" package would work for your case. This is the line that needs to be updated behind a config option:

monodeploy/packages/dependencies/src/getDependents.ts

Line 23 in 0f96e0f

for (const key of ['dependencies', 'peerDependencies']) {

The other aspect of this is that I assume the core package is not actually published? If that's the case and it's marked as private, we also ignore private packages in a number of places. Opening a ticket to handle that: #467 -- going to work on this issue next

[noahnu]

@markandrus I believe your use case should now be possible under [email protected]. You'd mark your "core" package as private so it doesn't publish and then list it as a devDependency for all the native workspaces. You'll also want to make sure they're all in the same group.

[noahnu]

Going to mark this issue as closed, but feel free to comment to re-open or create new issues for any offshoots of this change.

https://github.com/tophat/monodeploy/releases/tag/monodeploy%403.0.0-rc.2