add rsync flag option to copy files using rsync #3143

olamilekan000 · 2025-01-23T14:56:37Z

change adds rsync as an option to the copy command

issue #2198

nirs

Do we have rsync in the guest in all cases? Minimal vm images do not have rsync.

Also adding options is bad both for users and for testing. If it is better to copy with rsync, we should detect if rsync is available in the guest and host, and use it without adding new option. Otherwise fall back to scp.

If we always have rsync, better to use it instead of scp and minimize the testing matrix.

cmd/limactl/copy.go

nirs · 2025-01-23T19:29:19Z

cmd/limactl/copy.go

@@ -34,6 +34,7 @@ func newCopyCommand() *cobra.Command {

 	copyCommand.Flags().BoolP("recursive", "r", false, "copy directories recursively")
 	copyCommand.Flags().BoolP("verbose", "v", false, "enable verbose output")
+	copyCommand.Flags().BoolP("rsync", "", false, "use rsync for copying instead of scp")


This is not flexible enough. What if we want to another tool later? If we add an option, adding a "tool" or "backend" option will be better, with possible values "scp", "rsync".

the tool option will default to rsync right?

If we add an option rsync sounds like a better default. But this is trivial to change if needed.

Let's just remove the flag and only support rsync, unless there is a use case where scp is preferable over rsync

rsync isn't available by default on Alpine, Arch Linux, and Debian (see here for pipeline logs). A potential fix for this is to ssh into the guest vm and check if rsync is installed. @jandubois suggested running a command like this within the guest VM:
rsync --rsh="ssh" remote_user@remote_host rsync --version .

However, implementing this approach introduces a limitation: rsync won’t work in scenarios where the instance name is unavailable, as the instance name is needed to fetch the hostname using store.Inspect(instName).

For example, this command would fail since it doesn’t provide the instance name:

limactl copy -r /etc/config /tmp

On the other hand, this would work because it includes the instance name:

limactl copy -r default:/etc/config /tmp

@nirs @AkihiroSuda @jandubois @afbjorklund I’d like to hear your thoughts on this approach and any potential alternatives. Would love to know if there's a better way to handle this while ensuring rsync works reliably.

rsync isn't available by default on Alpine, Arch Linux, and Debian (see here for pipeline logs).

So to support these distros with required rsync, the templates will have to install rsync. Can you check what is the cost in time and space?

A potential fix for this

This is not a fix, but the way to deal with missing rsync.

is to ssh into the guest vm and check if rsync is installed. @jandubois suggested running a command like this within the guest VM: rsync --rsh="ssh" remote_user@remote_host rsync --version .

We can also use ssh user@vm-ip command -v rsync. We already use ssh to preform the initial connection so it should be easier to use it to detect guest capabilities.

However, implementing this approach introduces a limitation: rsync won’t work in scenarios where the instance name is unavailable, as the instance name is needed to fetch the hostname using store.Inspect(instName).

For example, this command would fail since it doesn’t provide the instance name:

limactl copy -r /etc/config /tmp

The online help instructs to prefix the file name with the instance name. It cannot work without this regardless of the tool use to copy.

So the issue about missing instance name does not exist.

On the other hand, this would work because it includes the instance name:

limactl copy -r default:/etc/config /tmp

We can also use ssh user@vm-ip command -v rsync. We already use ssh to preform the initial connection so it should be easier to use it to detect guest capabilities.

@nirs correct me if i'm wrong, is this where you're referring to?

I don't think this it the code checking for initial ssh connection. If you run limactl --debug start ... you wll see the ssh commands used to detect if the guest is accessible.

nirs · 2025-01-23T19:39:06Z

Also, do we have tests for code modified in this PR? if not we should not change it before adding tests. The new rsync flow should also have tests.

jandubois · 2025-01-23T20:05:15Z

@olamilekan000 I've asked before to please test your changes locally instead of force-pushing an update every 10 minutes for 2 hours, triggering potentially costly CI runs all the time.

olamilekan000 · 2025-01-23T21:34:29Z

@olamilekan000 I've asked before to please test your changes locally instead of force-pushing an update every 10 minutes for 2 hours, triggering potentially costly CI runs all the time.

I'd appreciate it if you can help with steps to run locally. I did try but It didn't work as expected.

AkihiroSuda · 2025-01-23T21:47:39Z

If we always have rsync, better to use it instead of scp and minimize the testing matrix.

Yes, rsync is always available on macOS and most Linux distros.
At least it is easily installable.

nirs · 2025-01-24T15:21:36Z

@olamilekan000 I've asked before to please test your changes locally instead of force-pushing an update every 10 minutes for 2 hours, triggering potentially costly CI runs all the time.

I'd appreciate it if you can help with steps to run locally. I did try but It didn't work as expected.

@olamilekan000, Can you describe how did you try to test and why it did not work well?

You can also join the #lima slack channel to get quicker help from other developers.

olamilekan000 · 2025-01-24T20:47:35Z

@olamilekan000 I've asked before to please test your changes locally instead of force-pushing an update every 10 minutes for 2 hours, triggering potentially costly CI runs all the time.

I'd appreciate it if you can help with steps to run locally. I did try but It didn't work as expected.

@olamilekan000, Can you describe how did you try to test and why it did not work well?

You can also join the #lima slack channel to get quicker help from other developers.

@nirs I have joined and shared my error logs on the channel.

cmd/limactl/copy.go

nirs · 2025-01-25T15:24:48Z

cmd/limactl/copy.go

 	sshArgs := sshutil.SSHArgsFromOpts(sshOpts)

-	sshCmd := exec.Command(arg0, append(sshArgs, scpArgs...)...)
+	sshCmd := exec.Command(arg0, createArgs(sshArgs, copyToolArgs, defaultTool)...)
 	sshCmd.Stdin = cmd.InOrStdin()
 	sshCmd.Stdout = cmd.OutOrStdout()
 	sshCmd.Stderr = cmd.ErrOrStderr()


We run rsync via ssh? it does not make sense.

Can you share the complete the scp and rsync commands that you want to run?

The commands can be tested from the shell without lima, using the instance ip. limactl just make this easier for the user by building the right command and running it transparently.

for scp

scp -F ~/.lima/deb/ssh.config "myfile.txt" olalekanodukoya@lima-deb:/dump myfile.txt 100% 7 2.1KB/s 00:00

for rsync

rsync -a -e "ssh -i ~/.lima/deb/ssh.config -p 50337" myfile.txt [email protected]:/dump/

cmd/limactl/copy.go

nirs · 2025-01-26T15:05:04Z

cmd/limactl/copy.go

+		path := strings.Split(arg, ":")
+		switch len(path) {
+		case 1:
+			inst, ok := instances[instName]


How this will work? instances is an empty map created in this function.

I think we have only one case - instance:path, and it is common to both scp and rsync, so this check should be done first before we consider the copy tool. Input validation must always be done first.

nirs · 2025-01-26T15:10:00Z

cmd/limactl/copy.go

+			}
+			sshStr = fmt.Sprintf("ssh -p %s -i %s", fmt.Sprintf("%d", inst.SSHLocalPort), "~/.lima/_config/user")
+			rsyncArgs = append(rsyncArgs, "-avz", "-e", sshStr, path[1])
+			instances[instName] = inst


I see, you add the instance to the map so the next iteration will find it. This is very complicated and hard to follow way to handle the arguments.

We should instead process the argument first, and convert them to internal structure that will be used to construct the command. This should be common to scp and rsync, so it should be done before we consider the tool.

We should instead process the argument first, and convert them to internal structure that will be used to construct the command. This should be common to scp and rsync, so it should be done before we consider the tool.

The argument processing for scp and rsync is quite different, so combining them into a single function could make it complex and harder to maintain. II think it’s better to keep them separate and focus on consistent output structures instead.

cmd/limactl/copy.go

nirs · 2025-01-26T15:32:59Z

pkg/hostagent/hostagent.go

@@ -424,6 +424,41 @@ func (a *HostAgent) Info(_ context.Context) (*hostagentapi.Info, error) {
 	return info, nil
 }

+func (a *HostAgent) installPackage() error {


I would not install the package here. If we depend on rsync in the guest, we can install the package using cloud-init.

In our user-data (part of cidata.iso), we can add:

packages: - rsync

This will install rsync using cloud-init when the instance is provisioned. For distros that do not support this, the provision part of the yaml can install rsync manually.

nirs · 2025-01-26T15:33:42Z

pkg/hostagent/hostagent.go

+			sudo pacman -S --noconfirm rsync
+		else
+			echo "Unsupported Linux distribution. Please install rsync manually."
+		fi


Looks like you are reinventing cloudinit :-)

nirs · 2025-01-26T15:49:12Z

I think we can do this:

Use cloud-init to install rsync
- If cloud-init packages does not work for a distro, it must be done in the lima.yaml provision scripts
Replace scp with rsync

@AkihiroSuda what do you think?

nirs · 2025-01-26T22:27:24Z

hack/test-templates.sh

@@ -192,7 +192,7 @@ if [ "$got" != "$expected" ]; then
 fi

 INFO "Testing limactl copy command"
-tmpfile="$HOME/lima-hostname"
+tmpfile="/var/tmp/lima-hostname"


Not using $HOME/lima-hostname is a very good change - running tests locally should not create files in your home directory.

But this name may clash with another test, or a file created by someone else.

Best to use a temporary directory that can never clash with anything:

% mktemp -d /var/tmp/lima-test-templates.XXXXXX /var/tmp/lima-test-templates.vuRmiE

#3163
I raised a pr for the change @nirs

Thanks! but it seems that this is the only test for limactl copy.

I opened #3170 for improving limactl copy tests. I hope we can improve test coverage before we change the code to avoid regressions.

Okay. That can be done after #3163 gets merged.

AkihiroSuda · 2025-01-26T23:21:01Z

Use cloud-init to install rsync

SGTM

Signed-off-by: olalekan odukoya <[email protected]>

olamilekan000 force-pushed the add-rsycn-as-an-option-for-copying-files-btw-host-and-guest branch 6 times, most recently from a508328 to 4b57bc6 Compare January 23, 2025 17:09

nirs reviewed Jan 23, 2025

View reviewed changes

nirs requested a review from afbjorklund January 24, 2025 15:22

olamilekan000 force-pushed the add-rsycn-as-an-option-for-copying-files-btw-host-and-guest branch from 4b57bc6 to aa5cf22 Compare January 24, 2025 20:45

olamilekan000 force-pushed the add-rsycn-as-an-option-for-copying-files-btw-host-and-guest branch from aa5cf22 to 4c95f10 Compare January 25, 2025 00:27

nirs reviewed Jan 25, 2025

View reviewed changes

olamilekan000 force-pushed the add-rsycn-as-an-option-for-copying-files-btw-host-and-guest branch 5 times, most recently from 73fcc08 to a9bade5 Compare January 26, 2025 15:22

nirs reviewed Jan 26, 2025

View reviewed changes

olamilekan000 force-pushed the add-rsycn-as-an-option-for-copying-files-btw-host-and-guest branch 4 times, most recently from 1d17b78 to 7c55a22 Compare January 26, 2025 22:20

nirs reviewed Jan 26, 2025

View reviewed changes

add rsync flag option to copy files using rsync

54e7507

Signed-off-by: olalekan odukoya <[email protected]>

olamilekan000 force-pushed the add-rsycn-as-an-option-for-copying-files-btw-host-and-guest branch from 7c55a22 to 54e7507 Compare January 29, 2025 02:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add rsync flag option to copy files using rsync #3143

add rsync flag option to copy files using rsync #3143

olamilekan000 commented Jan 23, 2025

nirs left a comment

nirs Jan 23, 2025

olamilekan000 Jan 23, 2025

nirs Jan 23, 2025

AkihiroSuda Jan 23, 2025

olamilekan000 Jan 25, 2025

nirs Jan 25, 2025

olamilekan000 Jan 25, 2025

nirs Jan 26, 2025

nirs commented Jan 23, 2025

jandubois commented Jan 23, 2025

olamilekan000 commented Jan 23, 2025

AkihiroSuda commented Jan 23, 2025

nirs commented Jan 24, 2025

olamilekan000 commented Jan 24, 2025 •

edited

Loading

nirs Jan 25, 2025

olamilekan000 Jan 26, 2025 •

edited

Loading

nirs Jan 26, 2025

nirs Jan 26, 2025

olamilekan000 Jan 26, 2025

nirs Jan 26, 2025

nirs Jan 26, 2025

nirs commented Jan 26, 2025

nirs Jan 26, 2025

olamilekan000 Jan 27, 2025

nirs Jan 28, 2025

olamilekan000 Jan 29, 2025 •

edited

Loading

AkihiroSuda commented Jan 26, 2025

add rsync flag option to copy files using rsync #3143

Are you sure you want to change the base?

add rsync flag option to copy files using rsync #3143

Conversation

olamilekan000 commented Jan 23, 2025

nirs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nirs commented Jan 23, 2025

jandubois commented Jan 23, 2025

olamilekan000 commented Jan 23, 2025

AkihiroSuda commented Jan 23, 2025

nirs commented Jan 24, 2025

olamilekan000 commented Jan 24, 2025 • edited Loading

Choose a reason for hiding this comment

olamilekan000 Jan 26, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nirs commented Jan 26, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

olamilekan000 Jan 29, 2025 • edited Loading

Choose a reason for hiding this comment

AkihiroSuda commented Jan 26, 2025

olamilekan000 commented Jan 24, 2025 •

edited

Loading

olamilekan000 Jan 26, 2025 •

edited

Loading

olamilekan000 Jan 29, 2025 •

edited

Loading