Implement snapshot replication.

[FransUrbo]

Not actually tested, other than in dry-run mode. But the commands it would run seems to be correct.

Sign-off: Turbo Fredriksson [email protected]
Closes: #5

[wankdanker]

I would just like to point out @mk01's fork of zfs-auto-snapshot. It has an implementation of snapshot replication with some nice options:

    -i, --send-at-once Send more incremental snapshots at once in one package
                       (-i argument is passed to zfs send instead of -I). 
    --send-full=F      Send zfs full backup. F is target filesystem.
    --send-incr=F      Send zfs incremental backup. F is target filesystem.
    -X, --destroy      Destroy remote snapshots to allow --send-full if 
                       destination has snapshots (needed for -F in case 
                       incremental snapshots on local and remote do not match).
                       -f is used automatically.
    -F, --fallback     Allow fallback from --send-incr to --send-full, 
                       if incremental sending is not possible (filesystem 
                       on remote just created or snapshots do not match - 
                       see -X).  
    -R, --replication  Use zfs's replication (zfs send -R) instead of simple 
                       send over newly created snapshots (check man zfs for 
                       details). -f is used automatically. 

Maybe it's not the right fit for the official distribution of zfs-auto-snapshot but thought it is worth mentioning.

[FransUrbo]

Diffing the two, @mk01 basically rewrote the whole thing.. Which makes it impossible to merge.

[wankdanker]

OK. Thank you for taking a look.

[FransUrbo]

Having options to ssh was a nice touch though. Added that. And the possibility to use mbuffer.

I still haven't decided if the fallback is a good idea or not. It's usually only used once (the first time) and this tool seems more towards incrementals (a snapshot IS after all an incremental change - from the previous one). It's not a difficult change, but I just don't know if it's a good idea.

Could @wankdanker and/or @kpande give me some examples where this is used (often)?

[FransUrbo]

Then why don't you use --send-full instead. Because that is essentially the result… ?

[FransUrbo]

Ok, fair enough. It's an easy change anyway.

[wankdanker]

In sending snapshots offsite it's best for us to only send the incremental changes over the internet. But if that fails, we still need to get the latest snapshot sent offsite, so destroy whatever is on the remote side and send full. That's our use case.

[wankdanker]

I agree with @kpande on the option to not create snapshots and just send at least the most recent ones. It would be nice to have a --send-only or --skip-snapshots option that would bypass creating new snapshots and just send the most recently created ones.

[FransUrbo]

As I just said to @kpande on IRC, I'm not sure that zfs-auto-snapshot is the tool for that... It's called '...snapshot' after all. Having simple replication is good, but I don't think it's in the tools ... 'mission statement' to do such advanced replication..

It handles snapshots. That's it's name, it's purpose. Having ANY replication at all might be to much for the tool in the first place. Which is why I've tried to keep it simple. For those that want/need such advanced replication as you request, you should use a tool specifically geared towards replication.

[wankdanker]

I hear you. And I agree with you than any replication at all should technically be out of the scope of the project based on its name. It's just so tempting to want it all in one tool. :)

[FransUrbo]

Note that ALL my commits in this PR is based on what's on the local system. It assumes that the receiving end is identical to localhost. if the remote end deletes and creates snapshots on it's own, my commits will fail anyway.

[FransUrbo]

It's just so tempting to want it all in one tool. :)

How very un-UNIX of you. Shame! :)

[wankdanker]

How very un-UNIX of you. Shame! :)

Yeah, that's the lazy sysadmin in me. Sounds like its time for zfs-auto-replicate. :)

[FransUrbo]

if the name is the issue, change it to zfs-auto-backup and have it do everything it needs to

That is @dajhorn choice. I'm just not directly convinced I want that to happen. I like the simplicity of the tool. It does what it says - manages snapshots. As in, rotates them so I don't have to many of them.

The fact that there is (limited) support for replication is enough (for me).

But if @dajhorn chooses to rename it and make it into a fully fledged backup and replication tool for ZFS, I might help out where needed. But as I've said, I don't really want that to happen. One tool, one job/purpose. That's the UNIX way :)

[FransUrbo]

Having support for multiple receiving host is, as you say, easy. BUT, and it's a big but. It still require that each receiver is identical to the sender...

[FransUrbo]

If you want support for differences between sender and receiver, it's going to be a huge change.

I might consider it if (when! :) @dajhorn have merged these ones. But as you say, he's been 'lost' somewhere...

[FransUrbo]

Yet, your receiver(s) isn't. Because you use a hacked script? I really don't understand why your receiver(s) is so different, but if a snapshot is created on sender/localhost it, and only it, should be sent to the receiver(s)...

[FransUrbo]

I THINK that if the receiver(s) have come out of sync with the sender/localhost/master (whatever one should call it), it should be up to the administrator to manually fix this before using an auto tool.

[FransUrbo]

Ah, ok. I think I get it now. Because you have two receivers, but zfs-auto-snapshot can only send to one at a time, zfs-auto-snapshot creates the inconsistency…

Ok, fair enough. I'll add support for sending to multiple hosts. As I said, that part is fairly easy. Give me a couple of minutes.

[FransUrbo]

What options would --skip-snapshots take? How would it be used?

[wankdanker]

I can see the complication that adding --skip-snapshots would make to the actual code. Right now it looks like the list of snapshots to send is built as the snapshots themselves are created.

The way I see it, when --skip-snapshots is set (no options that I can think of currently) it would behave similarly to how --destroy-only works. But instead of doing destroy logic for each target, it would attempt to send the most recent snapshot that exists on localhost.

I think it might be more clear to call this option --send-only.

[FransUrbo]

At the moment, I can't see --skip-snapshots being a reality. It would require a remote list of snapshots and filesystem and then go through and compare them.

That in addition to the check we're already doing by comparing the local list of snapshots.

[wankdanker]

Example usage might be:

$ zfs-auto-snapshot --send-incr=remote-server:remote-pool --send-only --send-fallback //

Here is what would happen:

• List of target file systems/volumes is generated
• For each target:
  • Determine most recent snapshot
  • Append most recent snapshot to list of snapshots to send
• Execute do_send() which abides by the --send incr and --send-fallback options

So, no new snapshots are created, only the most recent snapshots for each target are sent to remote-server.

[wankdanker]

I just skimmed the code, so I am probably missing some of the additional complications. But, it doesn't seem like it is necessary to get a list of remote snapshots and file systems. For each most recent snapshot on localhost, attempt to send it to the remote. You are checking to see if it exists before sending anyway.

I'll dig deeper into your code and see if I can implement my vision.

[FransUrbo]

Yes, but it should only do that if the sender and receiver is not in sync.

This is what zfs-auto-snapshot do now:

• Get a full list of all local filesystems and snapshots
• Create snapshots
  • Possibly remove old one
• Remember the snapshot being created
• For each of these snaps, find the last (local) one [before this new one]
  • If recursive is called, find all snaps with this date
• Send those snapshots

This is quite simple, and should work. It's not bullet proof, but if there's a failure, I still think it needs manual administrator fixing.

IF one would implement --skip-snaps, this would have to be done:

• Get a full list of all local filesystems and snapshots
• Create snapshots
  • Possibly remove [local] old one
• Remember the [local] snapshot being created
• For each of these [local] snaps, find the last [local] one [before this new one]
  • If recursive is called, find all [local] snaps with this date
• Send those snapshots
  • If this fails:
    • Get a full list of all remote filesystems and snapshots
    • Go through all remote snapshots, finding the last one
      • If recursive is called, find all [remote] snaps with this date
    • Send that snapshot again, with a different incremental base (gathered earlier)

and it might even be more complicated than that. Need to think about this some more.

[FransUrbo]

No it doesn't.

[FransUrbo]

So in practice, the only think not implemented from @mk01's version is the --destroy to destroy/cleanup remote snapshots…

Anything else?

[wankdanker]

See wankdanker@7ad3100 for what I was picturing for --send-only.

It generated the send command as I expected for something like:

$ ./zfs-auto-snapshot.sh -v -n --send-incr=bu01:test --send-only --send-fallback --label=hourly ms03-ssd/vm01-disk

[FransUrbo]

@wankdanker Thanx! Looks good. I've merged that into this PR if you don't mind. If you do, let me know and I'll remove it (it's still marked as yours).

[wankdanker]

@FransUrbo that's great, thank you.

I didn't get the impression that @mk01's --destroy option actually cleaned up anything. I thought it would just destroy the remote target if it existed but couldn't receive because there were existing snapshots that didn't work out with the send stream.

It looks like you handle that with the $remote -F $opt_recv_pool in the case where fallback is allowed.

The one nice addition using the --send-fallback option would be if the incremental send fails because the remote filesystems don't have the correct snapshots then it should try again with a full send.

[FransUrbo]

@wankdanker I broke out the part that gets the last snapshot into it's own function (see FransUrbo@3b0e383#diff-e7f2480c37404caadf77e766166e1b3aR155) and then your addition was simple (see FransUrbo@705ec33#diff-e7f2480c37404caadf77e766166e1b3aR270).

[wankdanker]

Awesome work, @FransUrbo. 👍 🍻

[colttt]

how can it be used to send backups via cron automatically?