rsync command over sftp is failing with protocol mismatch

[rashmimaharana]

Hi @drakkan & Team,

I have tried to do rsync over sftp, but ended with exception mentioned in last section.
The command I ran : rsync test1.txt [email protected]:/ -e 'ssh -p 2022'
Its on GCP environment(VM) and i am running docker edge image given by you.
I did some some changes to sftpgo.json file to enable some command(ssh, rsync) and also installed "ssh" inside the sftpgo docker container. But still that didn't work.

**Notes: I tried the same setup in my local system(Windows) and it worked fine **

Changes in sftpgo.json file :

"enabled_ssh_commands": [
"md5sum",
"sha1sum",
"cd",
"pwd",
"scp",
"ssh",
"rsync"
]

Docker command to run:

docker run --name sftpgo-edge
-p 8080:8080
-p 2022:2022
-p 10000:10000
--user root
--mount type=bind,source=/root/docker/sftpgo/data,target=/srv/sftpgo
--mount type=bind,source=/root/docker/sftpgo/config,target=/var/lib/sftpgo
--mount type=bind,source=/root/docker/sftpgo/etc,target=/etc/sftpgo
-e SFTPGO_DATA_PROVIDER__DRIVER=postgresql
-e SFTPGO_DATA_PROVIDER__NAME=sftpgo_docker_db
-e SFTPGO_DATA_PROVIDER__HOST=10.11X.X.XX
-e SFTPGO_DATA_PROVIDER__PORT=5432
-e SFTPGO_DATA_PROVIDER__USERNAME=sftpgo
-e SFTPGO_DATA_PROVIDER__PASSWORD=sftpgo
-d "drakkan/sftpgo:edge"

ERROR From Server

{"level":"debug","time":"2021-09-08T09:06:35.247","sender":"SSH","connection_id":"e4256ea6b9a2d92051943bbff49bfb47c9c3766241fbed3a86f520e8d3f5ab89","message":"User "RYAN_MAX", logged in with: "password", from ip: "10.33.XXX.XXX", client version "SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3""}
{"level":"debug","time":"2021-09-08T09:06:35.249","sender":"dataprovider_postgresql","message":"last login updated for user "RYAN_MAX""}
{"level":"debug","time":"2021-09-08T09:06:35.249","sender":"common","connection_id":"e4256ea6b9a2d92051943bbff49bfb47c9c3766241fbed3a86f520e8d3f5ab89","message":"ssh connection added, num open connections: 1"}
{"level":"debug","time":"2021-09-08T09:06:35.800","sender":"sshd_exec","connection_id":"e4256ea6b9a2d92051943bbff49bfb47c9c3766241fbed3a86f520e8d3f5ab89_1","message":"new ssh command: "rsync" args: [--server -e.LsfxC . /] num args: 4 user: RYAN_MAX, error: "}
{"level":"debug","time":"2021-09-08T09:06:35.800","sender":"SSH","connection_id":"SSH_e4256ea6b9a2d92051943bbff49bfb47c9c3766241fbed3a86f520e8d3f5ab89_1","message":"connection added, local address "172.17.0.2:2022", remote address "10.33.XXX.XXX:49260", num open connections: 1"}
{"level":"debug","time":"2021-09-08T09:06:35.800","sender":"SSH","connection_id":"SSH_e4256ea6b9a2d92051943bbff49bfb47c9c3766241fbed3a86f520e8d3f5ab89_1","message":"path separator added to fsPath "transfer/""}
{"level":"debug","time":"2021-09-08T09:06:35.800","sender":"SSH","connection_id":"SSH_e4256ea6b9a2d92051943bbff49bfb47c9c3766241fbed3a86f520e8d3f5ab89_1","message":"new system command "rsync", with args: [--safe-links --server -e.LsfxC . transfer/] fs path "transfer/" quota check path "/fakecontent""}
{"level":"warn","time":"2021-09-08T09:06:35.800","sender":"SSH","connection_id":"SSH_e4256ea6b9a2d92051943bbff49bfb47c9c3766241fbed3a86f520e8d3f5ab89_1","message":"command failed: "rsync" args: [--server -e.LsfxC . /] user: RYAN_MAX err: command unsupported for this configuration"}
{"level":"debug","time":"2021-09-08T09:06:35.800","sender":"SSH","connection_id":"SSH_e4256ea6b9a2d92051943bbff49bfb47c9c3766241fbed3a86f520e8d3f5ab89_1","message":"exit status sent, error: "}
{"level":"debug","time":"2021-09-08T09:06:35.800","sender":"SSH","connection_id":"SSH_e4256ea6b9a2d92051943bbff49bfb47c9c3766241fbed3a86f520e8d3f5ab89_1","message":"connection removed, local address "172.17.0.2:2022", remote address "10.33.XXX.XXX:49260" close fs error: , num open connections: 0"}
{"level":"debug","time":"2021-09-08T09:06:36.075","sender":"common","connection_id":"e4256ea6b9a2d92051943bbff49bfb47c9c3766241fbed3a86f520e8d3f5ab89","message":"ssh connection removed, num open ssh connections: 0"}

Error From Client System (Ubuntu)

$ rsync test1.txt [email protected]:/ -e 'ssh -p 2022'
[email protected]'s password:
protocol version mismatch -- is your shell clean?
(see the rsync man page for an explanation)
rsync error: protocol incompatibility (code 2) at compat.c(178) [sender=3.1.2]

[drakkan]

Hi,

the relevant error is this one command unsupported for this configuration.

System commands like git and rsync work only on local filesystem and they have some other limitations, please find more details here

[rashmimaharana]

Thanks @drakkan

I have checked the link https://github.com/drakkan/sftpgo/blob/main/docs/ssh-commands.md
Is there any possible way of making it work in GCP with GCS as storage.
If we can mount the GCS bucket to VM using fusemount or something else, will that work. But again that may open up some security concerns.

Also are there any other alternatives to rsync ?

[drakkan]

Hi,

no sorry, rsync only support the local filesystem, I should rewrite the rsync protocol in Go as I did for the SCP protocol. This is a huge effort, I don't plan to work on it.

I think you can use rclone. Using rclone you can write directly to GCS but this way rclone don't know about SFTPGo virtual folders, permissions etc. so I suggest to use SFTPGo as rclone backend

https://rclone.org/sftp/

this way rclone sends data to SFTPGo via SFTP.

Another alternative is to use the SFTPGo REST APIs but you have to write a sync program yourself in this case