• Backup Nightmares @ 2oFB..

    From paulie420@21:2/150 to All on Thu Jun 29 19:49:40 2023
    I'm bad at running at the mouth in posts, so I'll try to stay on track:

    I have a BBS VM. I mount some NFS share drives on the BBS VM. I also have a PBS backup server.

    Last week some users mentioned they were having DL issues; I thought it was a separate issue with filenames - but I've been upgrading many systems to Debian 12 this past week and finally got around to my PBS machine. I noticed in the PBS dashboard that only 2TB of a 16TB drive (thats usually close to full) was being used. Wait, whaaa??? So I checked its NFS backup's DIRs and they were all empty!! HUH?

    The rsync command I was using to backup had the --delete tag. I thought that if I modified my BBS VM and removed some /mystic files that I'd want that to be replicated on my backups....

    Well; come to find out, my NAS went down sometime last week... and, my BBS VM and PBS servers NFS mount points went down. Historically, I thought this fixed itself - I'm almost sure of it, but...

    The weekly rsync backup ran - and... --delete made sure ALL the backup data was - deleted. And, users weren't able to download this past week and the bug reports started flowing in.

    When I found it, a simple reboot and remount fixed things... I'm currently backing up some 32TB of data; great for the drive, right? Delete all the data and then re-write it. Awesome.

    I've removed the --delete tag, and am rethinking how I'm doing things... guess I need to put in REDUNDANT checks on the mountpoints, and maybe check in my backup script that if /WHATEVER/MOUNTPOINT is empty, don't backup.

    Ugh; stupid, Paulie...



    |07p|15AULIE|1142|07o
    |08.........

    --- Mystic BBS v1.12 A48 (Linux/64)
    * Origin: 2o fOr beeRS bbs>>>20ForBeers.com:1337 (21:2/150)
  • From fusion@21:1/616 to paulie420 on Fri Jun 30 17:28:08 2023
    On 29 Jun 2023, paulie420 said the following...

    The weekly rsync backup ran - and... --delete made sure ALL the backup data was - deleted. And, users weren't able to download this past week
    and the bug reports started flowing in.

    ouch, saw it coming the second i saw 'rsync' ..

    synced, incremental, etc backups always creeped me out because of that type of thing.. even if they save space.

    --- Mystic BBS v1.12 A47 2021/12/25 (Windows/32)
    * Origin: cold fusion - cfbbs.net - grand rapids, mi (21:1/616)
  • From tassiebob@21:3/169 to paulie420 on Sat Jul 1 09:24:36 2023
    The rsync command I was using to backup had the --delete tag. I thought

    Does '--max-delete=NUM' help your use case at all?

    --- Mystic BBS v1.12 A47 2021/12/24 (Linux/64)
    * Origin: TassieBob BBS, Hobart, Tasmania (21:3/169)
  • From deon@21:2/116 to paulie420 on Sat Jul 1 10:59:36 2023
    Re: Backup Nightmares @ 2oFB..
    By: paulie420 to All on Thu Jun 29 2023 07:49 pm

    Hey...

    The weekly rsync backup ran - and... --delete made sure ALL the backup data was - deleted. And, users weren't able to download this past week and the bug reports started flowing in.

    Sorry for your luck dude...

    Reminds me of a time that I worked for an airline (IT admin) long time ago, and we had a system that kept core dumping (it was a new app), and the core files were so big, that 2 or 3 of them filled some important mount points.

    The core files were important for the developers, so I had to keep them, so I wrote a script to find them each night and compress them (they compressed really well if I recall).

    I tested, tested, tested - since my script tralled through the whole system looking for core files and I was confident it was working.

    I deployed the script to another important system (starting with the airlines network management system), and got a call at 1am - the system wasnt working, and I couldnt remote it to look at it. When I got to the office, my script had compressed every file on the host. Argh! - That was a reinstall and a couple of days I couldnt of spent doing something better...

    Anyway, wanted to share, if you want a different approach to backup, take a look at restic - IMHO, it does a better job than a rsync approach to backup.


    ...δεσ∩
    --- SBBSecho 3.20-Linux
    * Origin: I'm playing with ANSI+videotex - wanna play too? (21:2/116)
  • From Tracker1@21:3/149 to fusion on Tue Jul 4 15:53:16 2023
    The weekly rsync backup ran - and... --delete made sure ALL the

    ouch, saw it coming the second i saw 'rsync' ..

    Yeah, I don't use --delete with rsync, ever... even if it's annoying cleaning up the extra files on the backup side if I ever want/need it. Worst is when some doors crash and create a 4gb data file in their wake.


    --
    Michael J. Ryan
    +o roughneckbbs.com
    tracker1@roughneckbbs.com
    --- SBBSecho 3.15-Linux
    * Origin: Roughneck BBS - roughneckbbs.com (21:3/149)
  • From paulie420@21:2/150 to tassiebob on Sat Jul 1 19:48:53 2023
    Does '--max-delete=NUM' help your use case at all?

    COULD have; but I'm switching to a more modern backup utility... currently enamored with borg-backup, and thinking of some borg add-ons for a web-gui; dunno if I'll land here, tho.

    It's interesting, b/c I found borg-import that allows me to import rsync backups directly into my borg-backups...

    I might also switch to something in my NAS running openmediaserver... they have to have a backup add-on, right?? :P

    I use PBS (proxmox backup server) for my PVE (proxmox) VMs, and love it - but I wish they had some simple add-on to handle basic non-PVE backups. There is a CLI command, but when I tested it a while ago I wasn't super impressed.



    |07p|15AULIE|1142|07o
    |08.........

    --- Mystic BBS v1.12 A48 (Linux/64)
    * Origin: 2o fOr beeRS bbs>>>20ForBeers.com:1337 (21:2/150)
  • From paulie420@21:2/150 to deon on Sat Jul 1 19:53:29 2023
    The weekly rsync backup ran - and... --delete made sure ALL the backup was - deleted. And, users weren't able to download this past week and t bug reports started flowing in.

    Sorry for your luck dude...

    Well... I guess the title should have been 'backup issues' as I didn't lose any data; that would have been the 'nightmare'. :P I'll get there... sometimes intermediate users shouldn't get over-happy using basic tools - we think we know everything, only to have the software (working correctly) to hurt us b/c we didn't think of all the things...

    Reminds me of a time that I worked for an airline (IT admin) long time ago, and we had a system that kept core dumping (it was a new app), and the core files were so big, that 2 or 3 of them filled some important mount points.

    The core files were important for the developers, so I had to keep them, so I wrote a script to find them each night and compress them (they compressed really well if I recall).

    I tested, tested, tested - since my script tralled through the whole system looking for core files and I was confident it was working.

    I deployed the script to another important system (starting with the airlines network management system), and got a call at 1am - the system wasnt working, and I couldnt remote it to look at it. When I got to the office, my script had compressed every file on the host. Argh! - That
    was a reinstall and a couple of days I couldnt of spent doing something better...

    :P I suppose these issues help us learn in the end... as stated earlier, my issue wasn't as hurtful as even yours - I didn't LOSE anything; its just knowing that I could have... I didn't even know for a few days. SMFH!

    Anyway, wanted to share, if you want a different approach to backup,
    take a look at restic - IMHO, it does a better job than a rsync approach to backup.

    I *did* want to look at restic - I think I will, before I choose to permanently land @ borg-backup; I know that both of those tools have various add-ons that add more function... something will work out, and I'll be better off for it - trust me, I still love rsync - I just realize now that you need to think of all the things - and what actual use case yer trying to get to... some 'solutions' (--delete to keep ~/mystic up to date) really aren't good choices.



    |07p|15AULIE|1142|07o
    |08.........

    --- Mystic BBS v1.12 A48 (Linux/64)
    * Origin: 2o fOr beeRS bbs>>>20ForBeers.com:1337 (21:2/150)