Good. This is just Thursday, December the 20 first, 2023. This is the St.
Louis Linux user group. It's our monthly meeting. And, tonight our speaker is going to be, Sean Freehouse and We are definitely looking forward to his presentation on CFS.
Doing 3 2 one backups and bit route protection. And on on a small at home style system so this isn't necessarily you know big corporate world so it's something we can probably all make some good use of.
So, so yeah, sounds like this should be very interesting. Are you usual thing for the Thursday night meetings is to open up with announcements and brief Q&A is also time for Q&A after the presentation.
But for those who need to perhaps run out early, put a little bit of QA here at the beginning.
As far as announcements, today is the winter solstice. It is the shortest day of the year.
And last week in St. Louis was the earliest. Sunset. And so those are the important things.
Right. So, anybody else have any announcements they would care to make?
Don't be shy.
There were some things in the discuss list, this week. So, I think you were involved in some of that.
Something about the backups.
That back up some patches. I thought you had some postings and discuss. Oh,
I asked some questions. I forgot what they are. I think they were a little bit deep for most of folks.
I didn't catch anybody but Okay, so it's. Yeah, sometimes you have Oh, let's see.
Other things we, Our mailing list and our archives website all that. We run it on a machine named Bach and that's of course maintained by our members who doing this is admin on it where a couple of releases out of date And so we're planning to do a upgrade from Debian 10.
And we're hoping to get all with Debian 12. And, we've done this before as a nice team effort.
Rather than just a couple of guys, doing it on their own it's good opportunity for, some of the rest of us who maybe don't do a lot of.
Upgrades particularly of systems or something other than just a personal box. You know if you're doing something that's a for work or something where there's multiple users.
Yeah, sometimes you wanna be a little bit more careful about, you know, what files are involved. How long it's going to be down.
So this is a good opportunity to kind of practice being a sis admin for a multi-user system.
So, if anybody would care to kind of volunteer. You know the people trying to put it together will you know try and organize some some meetings online or physical whichever and looking for a date to do it so sort, of, let Stan or Lee or myself know that you're interested and, we'll try and get you lined up.
5 days from now, fourth Tuesday of the month. Is the new Linux meeting. And the new Linux user group is basically it's the thing of whoever whoever wants to claim that they're the newest user of Linux they get to ask their questions till they're satisfied with their We want the next person.
So it's a good meeting. But that doesn't mean you have to be a newbie.
That's the whole idea to get some people with a little bit more experience. I've gotten into some really interesting discussions there.
Somebody asked what Seems like a newbie or new user question, then it turns into a great technical discussion.
Between some experienced people, how to best. Pursue the issue. So to please your experience also comes to a new Linux meeting.
Next meeting is the Slack meeting. It follows on the first Thursday of January. It's more of a hardware.
Oriented or multiple different operating systems doesn't have to be Linux. And See any other announcements anybody else have anything they put out and discuss they want to bring up at this point
Any follow ups?
Anything else coming up on the calendar that anybody? Wants to acknowledge his thing called New Year's coming up soon.
I'm also gonna put in a plug. There's a, and there's a listing of our different special interest groups.
It's a meeting that I didn't mention before, that's the steering committee meeting.
And that's usually the first Monday of the month. We've been trying to do is try to make the sausage there rather than here at this meeting.
But that does mean we need some people who want to be involved in organizing things, organizing meetings.
So, anybody who wants to make sausage, you're more than welcome to come to this very committee meeting.
Isn't a fancy membership thing, so. Please consider coming helping out and if you don't like meetings fine we can You can be one of the guys who does this admin work or other updates on.
Different pieces of software we have. Get a lot of lot of need for volunteers. Robert Lovets been doing a wonderful job being proactive on trying to get out some of the publicity.
And Robert probably could use a few a few extra people helping them doing some publicity stuff so Anyway, moving on, calendar, We're trying to keep a calendar of all the technical events that our type of people might be interested in.
And we used to do it all just for St. Louis meetings, but of course in the days of Zoom, we've been getting into other things as well.
So there's the calendar up there.
See, Stan, anything particular attract your attention?
And there's the new Linux user group showing up next Tuesday.
Steering Committee Steering Committee. Follow on Tuesday.
Hacker, public radio, New Year's Eve show. I remember that from, last year, the year before that I think I went to that.
That was a fun time.
Okay. Yeah, moving on to the next thing.
Oh, by the way, the one the events on the calorie which are in red. Are the events that this organization actually does the other events or or other organizations.
We're just recognizing they may be of interest.
Question calendar.
Go ahead.
Are they anything on the calendar like the monthly meetings, etc? Any place you can subscribe to and see them or is it you get nothing on the calendar or everything on a calendar?
Good question. I think right now it's if you get Well, there there are a couple of divisions.
We you can just get the sloop meetings and not the other ones. That's one option.
There's also an option and we kind of just been working on it. Stan may. Be willing to say it's ready to go, but we're trying to make it so that there's a division between the actual St.
Louis meetings. And the meetings that may be of interest but may not be in St. Louis, but you can get them by going to.
Zoom or something else, but Stan saying, that's not really ready yet. So, we're working towards that, but, so right now the main distinction is just whether it's an actual sloop meeting.
Or a.
A non, you know, something that we don't run that we're just putting on the calendar.
So you can turn either of those off and on.
Okay, thank you.
There I think the way that's done is. In the upper right hand corner. Is that where it is?
And saying, wait a second.
And there we go. Up there in that right hand corner.
And you can click. Click on it and it'll look. Give you the options of which ones you want to turn on.
See now now stands got just the the slug sponsored events, the ones that we actually run.
Right, that's display within that web portal. I was actually thinking about like I could. Subscribe my personal calendar too.
So as things are added there, I would see them on my devices.
Yeah, yes, you can do that.
Okay, yeah, that's what I was looking for.
Feature request fulfilled. Thank you.
Fantastic.
And Stan is counting down.
2, 3.
Okay.
And budding heads breaking. Sounds like. Never mind, okay, okay. All right, we're gonna move on.
Okay. Discuss. There's some mailing list we maintain.
And kind of a variation of what you were just asking. Is there a way of just subscribing to certain meetings?
That's what our announced list has been. And we tried to put it in the past of, make it an announcement.
You could call it this week or something. It just says what things might be of interest for you this week.
To be honest, we just haven't had enough volunteers to. Make it into a newsletter or a long list.
So the last year it's kind of just becomes Sent out on the week of our meetings and usually it's just announced. Okay, the St.
Louis log meeting is tonight. The new user log is tonight. So, you know, it's kind of Usually not more than once a week to discuss.
As Lee was just saying, you know, hey, you can put anything in discuss, you know, but.
The security breach this week or you know what you wanna Start a whole discussion on sometimes you get into great long discussions that people respond to in 10 min Other times, yeah, something goes out there and just nobody knows.
Let me try and see what you get.
Moving on. We drew try and maintain an archive. So the, the Wednesday night general meetings loop meeting the Thursday night STL Linux meeting we try and record if everybody agrees And, and then we try and put that out our archives so people can actually go.
And listen to the meetings afterwards. That's Let's see, there it is, the table of contents in the presentation archive.
And they're organized by year so you can click on this particular year or you can just start scrolling down that list.
I'll scroll you back to the ears.
It's been one nice thing about the age of COVID. Once we you know started doing everything on zoom our or any other remote media remote.
Broadcast media. We've managed to record almost everything. So, we had some recordings that were older than that, but yeah, for the last 4 years, we just about recorded every meeting.
Okay, anybody have any other questions?
I've grown on long enough. As I say, there'll be more time for more questions.
General questions after our presentation. And, I'm gonna turn it over to Sean. Shawn did a wonderful tutorial for us last week.
And tonight we're really happy he's gonna do a presentation on on the ZFS file system and how he's using it on the system he's working on.
I'll turn it over to Sean.
And thanks, Gary. Yeah, back to back this time. It'll be interesting. Let's see how it goes.
So I'm gonna share my screen and I have some slides and they're numbered and everything.
This over, alright. Hey, everybody can hear me and see the slides. All right.
Can I assume that's a yes? Alright, so tonight I'd like to talk about 3 2 one backups with a limited hardware and I've seen some.
Presentations. Sorry, just checking something where they talk about RTO and RPO and these other backup type of things and they don't really present a use case and some of those are kind of hard to latch onto.
And so I thought it might be a little bit more interesting tonight if I presented a use case to go along with the concepts and things.
And so I tried to pick a pretty general. Use case and that is like. Some documents that have too much Personally, identifiable information for you to feel comfortable saving them on Google or Dropbox or something like that.
And obviously the level of. Comfort that you'll have is gonna be different from every person. But I imagine most of us here have some level of personal information in a digital form that we don't feel comfortable saving.
Unencrypted on a place like Google or Dropbox. And so the solution that I'm presenting here could be used.
To help you store and securely, save and a resilient manner those documents.
So for me, it's gonna be things like tax returns, marriage certificate, stuff like that that has personal information.
I don't feel comfortable saving that on Google or Dropbox in an unencrypted manner.
And so I'm going to want to keep those under my control and but I also want to make sure they're backed up and but I also want to make sure they're backed up and secured.
So I want to start with a disclaimer. So I want to start with a disclaimer.
I'm gonna start with a disclaimer. I'm speaking out of school on a lot of the stuff here tonight.
I'm much closer to a disclaimer. I'm speaking at a school on a lot of the stuff here tonight.
I'm much closer to a ZFS ZFS expert. And, but I think that also goes to show how good the software is.
Is that you don't need to be an expert to make some things. Work with the software, right?
You shouldn't need to be an expert just to use. Piece of software and I think CFS is excellently designed and to where you can use it without being a total expert, you know, unlike something like get, which you pretty much need to be an expert to use it at all.
So, so yeah, so that's my disclaimer, you know, please bear with me and I'm definitely speaking at a layman's level here.
Okay.
Alright, so the first thing when you're talking about backups, it usually comes up as RTO.
Does anybody wanna help me out with what RTO might stand for what it means?
Our EO brings back memories. Hmm. Sorry, Sean. Okay.
All good, all good. So RTO, recovery time objective. Basically, you're asking the question, how long can the server be down before the wife starts?
Yelling at you or or some kind of hardship is caused. Okay. And here we're talking about personal documents, tax records and things.
And I feel like, you know, these are things you're not accessing every single day of the week.
And so for this, I for our use case, I feel like our RTO would be like on the order of weeks.
Not days or hours. And I put some dollar signs down here to signify that, you know, lowering this number can be really expensive.
So let's say we wanted an RTO of like 2 h. Instead of 2 weeks. Well, in order to maintain something like that, we'd almost have to maintain something like that we'd almost have to have like a hot site, we'd almost have to have like a hot site, right?
Like we'd have to have a whole identical duplicate of all of our equipment, all of our hardware on some other power grid, right, to, to achieve a 2 h RTO.
And you can imagine how you know, how much more expensive that is than something on the order of weeks.
And so, of course, everyone would like an RTO 5 min. But are you willing to pay the cost for that?
And in this use case, I think we definitely don't need to pay. Any cost for that so we can do definitely weeks not hours or days.
And then the companion to our To is our PO, which stands for Recovery Point objective. And this is kind of where we asked the question, how many hours of data are you willing to lose and that might sound a bit strange to some folks because it's like wait we measure data and like bytes and kilobytes.
What are you talking about? Well, you're talking about hours of data. But really this is how frequently are you taking your snapshots.
So if you take snapshots every day, one PM. And then let's say you take the snapshot at one p.
M. And you work until 5 PM. Well that's 4 h of work that is not included in any snapshot.
So if your hard drive goes pop. You've just lost 4 h worth of work or 4 h worth of data.
Activity 4 h worth of activity. However, if you would have been doing snapshots every 15 min, then the most You know the most work you can lose is just 15 min right because any more than that is covered with your backups.
And, for this use case, like private, you know, household documents, we're talking about probably an RPO of one day, but this number assuming you have a low volume of documents and a low volume of churn on those documents.
This number is pretty cheap. To lower, you know, I think going from 1 h, sorry, from 24 h down to 12 h down to 6 h.
Wouldn't actually cost that much. So if this is something that you would want to lower. You know, probably wouldn't cost much to do that.
So feel free to lower that number in your own. If you want to. Alright, the next like, the next, pillar and the triumvirate of backups is retention time.
Okay.
And this basically means how much time you have to realize you shouldn't have deleted a file. Okay, so If you delete a file today and you have a 30 day retention time, that means you have 30 days to change your mind about deleting that file.
Cause on the 30 first day it's gone you're not getting it back anymore after that with the 30 day of retention time.
And this like the RPO number is gonna be relatively cheap to. To increase right going from 30 days to 60 if you're not talking about a lot of churn and a lot of new data would be relatively inexpensive.
Okay.
So. You know, willing to spend just a little bit more if you get a lot more bang for your buck in this dimension as well.
And I guess the acronym for retention time is whoops.
Yeah. Yeah. Well, here's a good question then for the group. So what is the retention time of a rate array?
Yeah.
0.
That's right. Yeah, if you delete a file off the rate array, it's gone from all the discs at the same time.
So it's absolutely 0. And this is why they say rate is not a backup, right?
Because you don't have any retention time with raid.
Well, that's why you have to keep all the snapshots on a rate.
Yeah.
Oh yeah, I want to talk about that later too. I agree. Okay, and then there's bit route protection.
This is something I mostly just wanted to play around with, but so, okay, so I forgot to tell you the, the kind of hardware that I'm.
So I'm kind of expecting here. And so what I really wanted to target this talk at is like you have like an 8 year old laptop.
Or Gary's newest laptop, you know, however we wanna say that.
Hmm.
And then you have like a USB drive. Connected. To it. So that's the kind of hardware that I'm for aiming at, right?
Really old laptop and then USB drive. Connected to it. And, you know, there's this thing that, the, you know, bit rot is the thing, you know, we got phantom rights, it can be caused by bugs in some hard drive firmware.
It can be caused by bit flips. I've heard it can be caused by like aging hard drives and so when we're talking about this Oh, limited hardware scenario.
I just have a gut feeling that you know This kind of corruption could happen more likely than if we were using all brand new, you know.
Mid range or top of the lion gear right and so perhaps bit rot protection is something that you might want to consider and enabling.
In this use case right but it is expensive i put $2 signs here because it is pretty expensive because in order to have bit rot correction, which is where it's a text the bit route and automatically fixes it using a backup you have to have 2 copies or you have to have some redundancy so you have to have a least 2 copies.
Of all your data. So what are to get this bitrock correction. You need to have.
2 copies which effectively have your storage capacity right and so that's a big cost for something that might never.
What's that?
I'm back to that. I'd like to come back to that later because I have a different understanding with
Yeah, I would love to and yeah, there's a slide later that we can talk about that I'd love to hear you because I'm a ZFS new so I'd love to hear any insights that you have.
So. Okay. So here's our summary. So RTO, how long can the server be down? 2 weeks, right?
This is not important, not critical. It's critical that we have this data and it's we know how to get to it.
It's not critical that I see something in the chat.
It's not critical that it's up all the time. Stan asked what about spin right maintenance software Oh, you know, I don't know anything about spin, right?
So maybe we can have a discussion about that. A little bit later.
Put my name down for that.
Alrighty. Okay, so back to the summer. So 2 weeks for the RTO.
RPO is like how many hours of data are we willing to use? You know, we're gonna say about one day, retention time, about one month.
And we want bit route protection on all of our files. Okay. And so this is what we want.
This is like our policy that we want to implement one popular implementation of Those policies are one way to achieve that is to do what we what people call 3 2 one backups and there's different interpretations of what this means but one popular interpretation that I've seen online is you want 3 copies of your data.
You want 2 copies on different storage devices and you want one copy off site. So that's 3, 2, one.
And so this is, you know, as far as I'm, understand it, this is the minimum hardware requirements to do a 3 2 one backup you need to least 2 hard drives 2 storage devices and one cloud provider or off site storage pool or whatever you're gonna do there.
So this is what the onsite setup that we're gonna do looks like. So let's say you have a laptop, it only has one hard drive in it.
Okay, so Grant just put in the chat 2 different types of media and I've heard that I've heard that is one, interpretation of 3, 2, one.
Okay. But I've also heard 2 storage devices and what do, what do you think?
What do you think the benefit is of 2 different? What do you think the distinction is there?
That the intention behind that is like. You have. CDs versus hard drive or floppy disk or something.
It's If something could impact one type of media. It might not impact the other. God forbid you had a fire, it might.
Irreparably damage CDs. But a hard drive might. Still contain data and be recoverable.
Yeah, and unfortunately if you're trying to recover data from a drive, you're talking. Extreme costs so It's possible, but.
You have to have a used case to justify.
And this goes back to zip discs and various things. Well, it depends like if I'm referring to there's a chance.
If you plug the drive in and it works. Great. Sending off to data recovery is something else, but.
Sorry, didn't mean to sign track you, Stan.
No, I think it's a good discussion. I actually remember a case where Somebody was trying to do a restore.
And it was a defective Hard drive. That you can take removal media. And, so put the first one in and it It didn't work, but it also destroyed the disc.
Oh, it must be something wrong here. So we put in the next backup there so he put in the next backup.
This tried the second copy of the backup. This tried the second copy of the backup. To so yeah sometimes it's nice to have it on optical or something non-magnetic to just have a chance of maybe getting something else not to break it.
In other words, verify your physical condition before you start swapping drives if they're removal.
Totally. Yeah.
I tried to store all of my, all of my stuff on clay tablets and did take centuries literally centuries.
Chip, you only get one bit on a tablet. Anyway, sorry, go on, please, John.
That's good. I like the discussion. I appreciate that. Yeah.
So, I don't know. Yeah. So we're gonna do that too devices.
And interpretation for the rest of this talk. And I, yeah, I think it would be better to have.
The 2 different media types, but I don't think that's realistic in the scenario where somebody just has a laptop and a USB hard drive kind of thing.
So, so, okay, so you have this, you have the hard drive in your laptop, right?
And I'm hoping, what I'm hoping for is that it's big enough where you can carve off some like a third partition, right?
So you have your normal Linux partitions, you have boot and root. And maybe some swap if you're doing swap.
And then I'm hoping you have enough where you can carve off a couple gigabytes at the end and we can call that a SRV partition.
I don't know, it's just some name I picked. But we can make that a ZFS partition. Okay.
And, and then if you have your backup drive, which is like a USB hard drive of some sort, you know, we can also partition that with ZFS and we could do some ZFS snapshot like replication stuff that way.
And so this is kind of what I'm what I'm hoping for is that you have a laptop hard drive that you can, that you can do that in a backup USB or something equivalent.
You know, it doesn't have to be exactly that, but something along those lines is what I was thinking and then also you know there's plenty of cloud providers now that provide S 3 compatible storage many of them have Have a free tier offerings.
Back blaze is one we're gonna talk about, but also, you know, rustic, which is what I plan on discussing tonight.
It works with any, with our clone as well. So anything you can connect to with our clone you can use with Restik for your offsite.
You're off-site backups and I think this would you know we were talking about the different types of media and like fire and all that kind of stuff and through me I mean yeah maybe different types of media.
Survive different environmental conditions, but I think just having an off-site backup. Is in my opinion a more reliable way to survive those kind of local, you know.
Site wide disasters that may happen. And so here's the full picture. So for the 3 2 one that I'm talking about here, we're gonna do, we're gonna do the OS drive, we're gonna do ZF snapshots to a USB and then we'll do resting snapshots up to a cloud provider.
The reason why I like doing, I could just do rustic snapshots up to a cloud provider.
The reason why I like doing, because I could just do rustic, right? I could do. It's up to a cloud provider.
The reason why I like doing, cause I could just do rustic, right? I could do rustic to the USB and rustic to the cloud and it would be slightly less mental overhead, slightly less setup.
But I don't know, like what if I mess up the the rustic, command or something?
What if I do something and rest it gets borked, you know? It there's a chance that I could like with one mistake.
Grew up both the on-prem and the cloud, you know, and not realize it or something.
Same way with ZFS, right? You can do a ZFS snapshot to like our sync.
Dot net or something and you could do ZFS both to the local and to on cloud. But then what if you mess up your ZFS somehow and then you you have a possibility of wiping out All your backups with one mistake, right?
Or something like that. And so that's kinda why I like doing the 2 different backup technologies for the 2 different backups.
It's just because I feel like.
I feel like it's a, Just more resilient that way. You know, I feel like it's hard for you to mess up a rustic repository and a ZFS.
File system with like one mistake, you know, because it's just 2 different technologies. So that's kind of what my thought process was when I was coming up with this.
Alright, so Stan asks.
I think that's a great idea. Absolutely great idea. And you explained it perfectly. I mean, Definitely good.
And I see stands in the chat asking how hard is it to test your backups? Yeah, that's, you know, that's one thing I have slide on later too, you know, backups always succeed.
That's the thing about backups. They always succeed, but you know it doesn't always succeed.
Restore. Restore doesn't always succeed. But, okay. Alright, let's talk about ZFS.
So ZFS is a file system and a logical volume manager kind of 2 and one originally developed by some micro systems in 2,001 the code was released in 2,005 and it was released under the common development and distribution license, the CDL, and you know over the decades there's been some heartbre about is the CDL compatible with the G the GPLV 2 which is what the Linux kernel
is developed using. And there's been some back and forth on, well, these are incompatible licenses, are they not?
And so as such, ZFS has never really been treated as a first-class citizen under Linux and you know one example would be like, Fedora, right?
Fedora is a very they are very adverse to any legal liability, right? Like, they will not take any legal risk at all.
It feels like And so, ora absolutely will not ship. The ZFS. Headers in their default kernel because you know, there could be some licensing compatibility here.
Even though both these licenses are copy left. There's still some nuance incompatibility, you know, or that's what Fedora probably thinks.
And you know, also Fedora bumps their kernel version like mid release, right? So you could be like on Fedora 39.
One day you're on Linux kernel 6 3 and the next day you're on Linux kernel 6 4 but you're still on fedora 39 right and so if you were to try to use zFS on fedora you'd have to you know fiddle around with DKMS which I don't know you know, probably a lot of you have had.
More luck than me, but I've had some serious problems with DKMS in the past because the kernel upgrade, get updated and then is your.
Module is ZFS ready for that new kernel version or you know, whatever DKMS module your, using is their compatibility there or is it not up to date yet?
All this kind of stuff. And so that's why if you're gonna use the FS on Linux, I would definitely recommend to go with a bunch.
Because in 2016 canonical conducted a legal review and they found in their opinion that actually the CDL and the GPLV 2 are not incompatible.
They say that both licenses talk about derivative works. And, and they, and their opinion, ZFS is not a derivative work of the Linux kernel.
And the Linux kernel is not a derivative work of ZFS and so for that reason they don't feel like there's any licensing compatibility here.
And so what they do is they include the ZFS, headers or whatever in their default Linux kernel.
So if you just have default a bunch of installed You already have the ZFS headers. And so if you install, you can install and use.
ZFS without doing any DKMS or KMOT or anything like that. You just literally, it's already there it's already handled by the kernel team at Canonical.
And so for this example, I'll definitely be using a bunch of. And this is something that I'm starting to appreciate more and more about a bunch of over some other distributions is their willingness.
To take on slight legal liability, right? Because I don't know if you guys saw that story a couple, maybe it was a couple of years ago now, but.
Somebody a red hat legal tapped fedora on the shoulder and says hey you're including this A one Kodak, in your Mesa drivers and we don't have the right license for you to distribute that we could get sued and they yanked it.
They yanked it so fast, you know, and they just kind of said, well, users, you're kind of SOL because we can't, you know, there's legal liability if we provide this to you.
And so all the users had to kind of scramble to find new ways to get that Kodak.
But you know, a bunch of didn't, as far as I know, a bunch of didn't hold that Kodak out and a bunch is willing to take this minor legal risk as well.
And I think over decades, you know, these type of things add up to, you know, a better or worse.
User experience. But that's just my, I guess that's just my rant and kind of opinionated.
No.
So maybe I shouldn't have. I said all that but I've been talking for a while so let's let my voice rest I wanna turn the i wanna turn it over to the group let's have a discussion about a relevant or a recent topic and open source.
When did you say that? decision or opinion about the about the legality. When was that?
It was 2016 I believe. Let's see.
7, yeah. So 7 years ago, yeah, February, 2016.
And it says the GPLV 2 applies to derivative works. And this is their whole, this is their key argument here, is that ZFS is not a derivative of Linux and Linux is not a derivative of ZFS and so To them, there's no licensing compatibility.
You bumped the found a couple times once.
Yeah, I don't wanna, I'm not gonna take into that. Okay.
Okay, back to my slides. Okay. So anyway, something that's been in the news lately as Hashikorp has moved all of their stuff to
To this thing called a business source license and I was wondering what anybody else thinks about this right so it's basically the code is still like you can still read the code right they're still accepting Development contributions from the community, but basically they're adding this extra clause to their license to say that you basically can't use the code to directly compete with Hashicorp.
And so, just to give my voice a break, what do we think about this, licenses?
Because Hashi Corp is not the only one doing stuff like this. So what do you all think about it?
Isn't that somewhere in what Red Hat's doing?
You know, they've, I'm not familiar with it, but they've basically locked up their source code except for their paying customers way understand it.
Yeah, but that's a bit different because they didn't pay the still GPL here.
They're actually relicensing their source code under a new license from that like from now forward.
Okay.
Oh, okay, I get you. I get you. Okay.
I have a lot of sympathy for the BSL. People even though I'm not sure I actually like the license but you know, producing a useful piece of open source software, getting funding for any type of open source software is really hard.
You want to try to charge enough to pay the developers somebody might just fork it and give it away for free So it's a really tough position And we've seen things where, I think going back it was.
I don't know if it was Mongol or Cassandra, one of these no SQL document storage databases.
Where, you know, these guys did all the work, develop the product that was really popular, very successful.
And Amazon said, hey, this is great. We're going to just, take it and charge people to use it on our platform and none of that money went back to the original authors and Amazon got all the money for all the clients.
That were using it. That's a that's gotta be tough to take.
Yeah, I completely agree, especially Amazon, right? Like they don't contribute anything back to Mongo, right?
And yeah, you're right. Mongol went with that. Server-side public license, right, which basically says we're still like the code is still open source.
Everything's still open source except if you're in a, basically if your name is Google, Amazon, or Microsoft, you can't host Mongodb and sell it as a service right that's but the license is still you know mostly open source besides that one clause right.
I have.
Yeah, yeah, and again, it's a similar thing with the Red Hat recent situation I'm not a fan of what they did.
But when you see someone like Oracle, basically say thank you for the multi-millions of dollars worth of work you've done on this.
I'm just going to take it for free. I'm gonna do a search and replace on everything that says Red Hat and switch it to Oracle and now I'm going to sell it.
And no especially Oracle of all people. That's gotta really hurt to see them do that to your stuff.
I have a couple of questions. Is the Amazon stuff? I absolutely agree with taking. Somebody else's product and selling at a profit.
That is one thing clear and simple. Take somebody else's product. Using it to host and offer a service.
Using their product. Is completely different. A real maybe thorny analogy is. IBM sold computers to Germany.
During the World Wars. That does not make IBM the company. Remotely responsible for atrocities.
Their product or tool. Was used maliciously or for something else. Analogy being Amazon could take somebody's product.
And host and provide a service and charge. For the power for hosting it for the connectivity to it for the storage to it.
I don't have a problem with that. Reselling saying thank you I'm going to copy your product and sell it as what I got for free from you turn around and sell.
Copies for more than just distribution. I have a problem with. A side comment is my understanding is that the red hat code that is GPL.
Requires you to not use the Red Hat name, Ergo, you are obligated to at a minimum.
Do a find and replace.
Well, that's the trademark, right? Red Hat owns the trademark on Red Hat.
And so you can't put Red Hat in your stuff because you don't own the Red Hat train bar.
So that's different than copyright. That's trademark. Why you have to do that? But I really think the red hat is like a separate thing than what like Hashi Corp and Mongo dB and all these are doing right because like for the hashi corp one you know vault is a very popular product and IBM was basically doing an oracle with that right where they were basically taking
all of the vault code, stripping out any mention of Hashi Corp and putting some IBM badge on it and then selling I don't know if they were selling the code or if they were selling you know service level like access to the code they were like, like we're running vault and then you can use it for if you give me $5 type of service or if they were like selling it as a
product. I don't know which one, but IBM is very upset with this move by Hash Corp and so they are a couple of IBM employees have forked it into open tofu or some terrible, like that's how you know the fork is gonna fail because it's got like a really terrible name.
Okay.
But, but I mean, I sympathize like it, you know, you don't really think of IBM as like great open source stewards that give back to Hashicorp and you know I'm sure it's been a very one-way relationship just like route.
I mean, Yeah, and I guess in that way this sort of like the real thing where it's like, you know, rocky, you know, how much are they really giving back to Real?
You know, they're taking a lot, but are they really giving anything back or adding any value to the equation?
It's up in the air, right?
Okay, well that was a good discussion. I appreciate it. But giving my voice a rest for a little while.
Let's get back to ZFS. So ZFS, the way that it works is you have multiple disks that can be or one or more disks can be added to a storage pool and then on top of a storage pool you can create one or more data sets and the data sets are the things.
At the very top and the storage pools in the middle and at the bottom we have the hard drives.
And so let's see some. Some of this in action and I need to close some of these things.
Alright, so here, so here on the screen what I have is I have just a very standard, a to server.
Install And here we have, so I have the OS disk here that we had in the pictures before and it's a hundred 30 gigs, you know, just whatever.
And I only have a 50 gig route partition on here. So I.
So there's about 75 gigs, you know, left over or whatever, right? And so that's what I'm gonna.
Partition off and put ZFS on. And then I also have this Dev SDB here that I'm going to also partition with.
ZFS and this is what we're gonna use, you know, simulating a kind of USB drive or whatever that we're gonna put our backup side So the first thing that I need to do is I need to I'm gonna use F-disc.
To partition the OS drive which is Dev S. So I'm going to do a new partition and then I'm going to do just the defaults for everything else.
So partition number 3, we're going to start at the first sector, we're going to end at the last sector.
I'm gonna hit W to write this to the disk. And now if I do another LSBLK.
Then we see, okay, now we have The third partition, like I said, is about 75 gigs or whatever and we're gonna put ZFS on there.
So first. I need to install. Let me do, oops, update first just to make sure.
That I have the package cache.
Okay. And now I'm gonna install, ZFS, which on a bunch of is called ZFS,
And what we'll see is. When this is installing, you see how.
See it ran some ins MoD and once again I'm speaking out of school here. I'm not a Colonel anything.
But you see how it's just running insert MoD here. We didn't have to build any headers or anything and you see how, you know, the running kernels up to date.
We didn't like do any a new, and at RAM disk or anything, right? And so And I think that, you know, this to me means that, you know, all the kernel headers are already built in like I was saying before and we just basically had to insert them, turn them on, you know, whatever like that.
That's my understanding. And that's because we're using a Okay, go ahead.
Okay.
Been building kernels. From scratch for. 25 years. What that's doing is it's a kernel module.
That it originates completely outside the kernel tree. And it needs to have.
API, ABI. And compatibility with the kernel version. So if your kernel version moves forward.
The ZFS kernel module. Me to be able to. Doc with the kernel version. So if something significant changes.
In the kernel version. Presentation. Colonel module, which originates completely outside. Has to be updated to match.
It's when you say including headers that to me is more. Like dot H files or something else that's actually already in the kernel.
That's not the case. What's happening is the kernel modules. Are using standard kernel.
Okay. Okay. The last one.
APIs. Ubuntu is distributing. The proper modules that mate with their kernel version.
And the utilities at the same time.
Cool. Yeah, I didn't know any of that. So that's all great information to have.
So how should I say, so what do you think it is that a bunch is doing then that we didn't have to do any DKMS here.
If it's not the headers. And the always why I said headers is because that fedora package like Linux dash editors, but I really don't know anything about this, so.
Zkms, I believe is an abstraction layer. To account for abstraction slash translation layer.
To account for the differences in the Linux kernel. Versus the Solaris kernel. Where came from. A lot of ZFS code is still Open sluris specific.
So. I believe it's ZKMS that does that translation. So they both have to.
Be compatible with the kernel you're running.
I was, oh, I was thinking this thing right here. Dynamic kernel module support.
DKMS or am I thinking of the whole wrong thing entirely?
No, it's probably close and it's been too long of a week, it's too late in the year.
To differentiate but There's some abstraction. You can create Linux kernel modules. Without. Any Linux source code if you happen to know.
Okay. And.
What you're supposed to match that's easiest by looking at the code. So.
Yeah. I think metric versus English sockets. You're gonna have a hard time using metric on English and English on metric.
You really need them to match and As long as it's adhering to this published standards. Something else can.
Write to those standards. And they should mesh. And there's 2 or 3 or 4 pieces that are required.
Or ZFS on Linux. One is ZFS itself. In kernel module. The or modules that are put into the kernel space.
That originate confirmed completely outside the kernel. There's the user space utilities to control. What ZFS is doing.
And there's an abstraction layer which. DK M or ZK, I'd have to look things up.
But Ubuntu is Distributing things. That others are saying, no, you can't do for license and compatibility.
Anyone too saying, yes, but. And the article you talked about earlier. They are Ubuntu is providing what is necessary.
That could be done on any Linux, they're just making it exceedingly easy and readyly available for you to do.
Right. Right. Is that yeah. And then as you know.
Yeah, there's there's Oh, yeah, I agree with everything you said and I appreciate that context about all that, but yeah, it's something about distributing, right?
Like, Like, you know, like, whenever you install WSL on Windows, you have to go click a download button on a web page somewhere as one step of it.
And I think it's something about you downloading something versus Microsoft giving it to you like distributing it to you.
I don't know, something weird, like, you know, I think it's something like that.
That's just my, hunch.
Okay. Agreed.
Alright, Gary, see you putting a lot of stuff in the chat. Do you want to talk about that now or you want to talk about it a little bit or?
There's others that have been more prolific than me.
No, I said Gary, sorry.
Oh, sorry.
I was interspersed there with Grant's, entry and the chance. Is it a couple of URLs there?
One is an opinion from Linus Torvalds. On why he doesn't want Colonel developers to be incorporating any of the FS stuff because as you were just saying whether there's legal entanglements with it.
And the other one is similar and it basically was. Complimenting a bunch on being the first one to kind of wrestle with this.
Try and and leave the hooks in there that This could be dealt with more easily. Ubuntu's opinion was that the GPL of the the CDL, CD DL.
Or close enough that So, whether that's true or not, but anyway, if you wanna get into the minutiae of arguing.
About the trademarks and copyrights and the intellectual property. There's a couple of URLs of articles that we're talking about.
I'll throw out the real quick. There are other things. That are drivers or features that are not in a Linux kernel proper.
That are integrated via modules. Developed completely outside of. Virtual box, which is also from Oracle now, has some things that are outside the Linux kernel that are modules that plug in.
And I believe. Fancier video cards. 10 to be done. Completely outside the kernel, but have modules that plug in.
So, kernel module is.
Kind of ice. Standard per. Colonel White Linux kernel has them. BSD Colonel has, Solaris Colonel has them.
It's the idea of. We publish a specification of here's how you do things inside the kernel to load it dynamically.
If you adhere to that, you're welcome to play.
Right, but I've heard of people having a bad time running ZFS on arch because Arch will update the kernel right away and then ZFS will not be something about the new ABI will not work with the ZFS and CFS won't be ready and so then you have a problem and you got a PIN.
Your kernel version and wrestle with all that and so you know that's why you know we said a bunch who's making it easy yeah they're basically doing all that for you one by just locking down the kernel right they don't they don't bump the kernel version number after their release date, but then too they're including these things in the kernel to make it easier on you and so That's kind
of what I was trying to. Say it before without really knowing how to say it.
But okay, let me get back to my demo pretty, if I can. So, okay, so just to remind everybody where we're at, so I created this third partition here.
This is where we're going to store our data. I installed the ZFS utilities and now I'm gonna create the Z pool and the data set.
So here we go. Is the name of the data set that I'm gonna create and I'm gonna create and I'm gonna give it the disk the partition that we just created. Dev SDA 3.
So I'm gonna create that and so now we do z pool of status. Will be able to see that there's one pool called SRV.
The device is SDA 3, known errors, everything's online. And with. With ZFS, you don't need to worry about like fs tab or anything.
You can just, it literally mounts it for you. So it's like already, mounted here.
I didn't have to do any FS tab stuff. Alright, so I'm gonna go ahead and create a file.
With today's date on it. In the, just to prove that. That we're actually there and it's working.
So there's the file that I just created the 5 MB. File that I just created.
And so this is on our OS drive. Let's go ahead and do the same. Or a similar thing on.
Our.
Okay, Grant says automatic amount is a per data set option and it's on by default. Okay, cool.
So let's do the same thing on Dev SDB. This is gonna be like our USB drives.
Is our backup.
We like.
Thank you. Alright, so, I'm formatting SDB. I'm gonna do a new partition.
That was you. Of the
I'm gonna accept all the defaults. It's gonna be a primary partition, tradition number one, gonna start at the first sector and in the last sector I'm gonna hit W to write those changes to the disc.
And then I'm gonna do LSBLK and we're gonna see, okay, now Dev S DV one has this partition here.
So let's go ahead and add that partition. Let's create a Z pool and add that partition to it.
So I'm gonna create a Z pool right here. I'm gonna name the Z pool backup and I'm gonna give it that partition, SDV one.
Alright, so now once again if I do z pool status we'll see that We have the backup pool with Dev STV.
We have the SRV pool with SDA 3 and. Those are online good to go.
So now, gonna go back to the slides. Really quickly. Alright, so this is where we can talk about bit wrap protection a little bit more.
So, So like I said, in order for ZFS to you know detect and correct bit rot it uses a z fs scrub.
And when you install like, the, that ZFS package on a BUN 2, it adds a Kron entry that will run CFS scrub on the first Sunday of every month.
And if you only have like how we have it configured right now with only one copy of all of our data, it will.
Put in the logs if it detects any. Corruption.
So the ZFS Scrabble check all the check sums of all the blocks or something like that and it will put in the logs if it finds corruption, but it won't be able to fix it.
So what I've read and the documentation is a little bit sparse on this. From what I've been able to find, but apparently if we do this set copies equal to 2 then ZFS will actually store 2 copies of everything we save on that data set and it will.
Use that second copy. To. To.
For the bit right, correction. So automatically repair any bit route that defines using the second copy.
So let's go ahead. And give that a shot right now.
Let me know when you want to.
Yeah, go ahead. Let's do that now.
You mentioned I check some on a block. And my understanding from experience. As I understood it and talking with multiple people.
Is ZFS. Actually check sums. It's metadata. And user data.
That it is storing. So if a single bit or few enough bits are flipped. Not only it will detect it But if it's small enough.
Corruption, it will actually correct it. As the data is being read. And then it will automatically. Rewrite the corrected data back to disc.
Yeah.
I have had pools in the past, you know, I'm the cheap and reused hard drives that should have been disposed of long ago.
And I have failures from that, but I've had drives. That we're starting to fail.
And ZFS. When I would read something I hadn't read in a long time or when I would do a scrub.
It would detect the bitrod. And there was subsequently a message saying it had corrected it and if I reread what I read that detected it.
There was no corruption. And these are on pools that had a single copy. To me, 2 copies.
Is if you want much larger protection. So if too much. For the checksum. You have the other copy and it can get it.
It's kind of like the difference between Parody detection? Versus ECC memory that can correct.
A certain number of corruptions.
The other place number of copies comes into play. Is when you have multiple pool or multiple drives in the pool.
Right, yeah, cause I know that, so from what I read, whenever you have a, or, you know, a one equivalent, Z pool.
It can do the bitrock detection and correction. Using the mirror, right? But I thought if you had a single disk Z pool like we have here in this example, that's where you use the copies because it needs Another you know another copy of the data in order to know what to replace the the rotted part with right
That's where the.
The check some. That is included. My understanding is the check, some that is included. Is more than just a checksum.
It's actually. Error detection and correction. If the amount of error is too small. Or is sufficiently small.
Okay, cool. So maybe we don't need that at all then.
It won't hurt other than the fact. It will make it slower. And consume space faster.
Right Ron asked if copies 2 is different from hearing. Yeah, so I mean copies to so mirroring is like 2 separate storage devices like you're guaranteed that obviously 2 or more storage devices, but the copies even if you have, 2 devices and your z pool and you put copies according to the documentation, which I could pull up, it's gonna try to put the copy on the other.
Disk but it doesn't guarantee it So if you have 2 disks and you put copies equal to 2 and you think that you're.
You have like the same redundancy as a mirror, not technically true because it might end up putting the copy on the same disc as the original.
So that's what I've read online at least.
And that's also evident by the fact that you can set copies to. On a single physical device back in a pool.
Yeah. And so here's the command. I think we've mentioned scrub a couple of times.
So you could run a Z pool scrub SRV like obviously this VM that I'm messing with here has been online for a Sunday so it hasn't done it automatically on the crime but I can run it manually like that and since there's nothing here on these disks it's very fast.
But if I run Z pool status again, you can see that. It. It prints out here.
When the last scrub was performed and how many what was repaired and all that. As part of the output.
But anyway, I'll just show how to set copies to 2 and then maybe if anybody wants to do more research on their own or whatever then or maybe we don't need it I don't know but well I'll just show how to do it just in case some people might not know so yeah you just do ZFS set copies equal to and then you put the name of the data set and then to check and see what that
did, you can run get copies and you see on the backup our copies is only set to one that's the default but then on SRV it set the 2.
Which is what we just changed right here. And so, yeah. Alright, so now what I want to do is I want to add a couple more files.
To our data set, our SRV. So let me. That someone do 2 more 5 MB files And then.
Alright, so now where are we at? So where we're at is we've now all the stuff in green we've done.
So we have the SRV partition, we put CFS on it. We have the backup partition on our USB drive, put CFS on that.
I'm not sure.
What we still need to do is all the stuff in red. So we still need to take a ZFS snapshot.
What? You'll get him and it'll get right. Okay.
One. From one tool to the other. So let's try to.
Let's try to do that.
Oh wait, so I'm gonna talk about how we do that. So ZFS is a copy on right file system and in my opinion the main thing that that enables is snapshots.
So ZFS snapshots are read only. They take up no space initially, but then as you make changes to the disc, that, to, yeah, to the data, that's when they begin to take up space, you know, roughly equivalent to the difference.
Based on the changes you made. And so let's see that in action.
Okay. Okay.
So I created a couple of extra files here. So if we look at. .
Hello, Jeff. When it got here, then it got right.
You can see a few files there. And now what I'm gonna do is I'm gonna run ZFS snapshot.
I don't think I need to create, do I? No, okay, that was the typo.
Zfs snapshot and, and then you put the, this is the data set name, then there's the at symbol and though I'm pretty sure those are required and then to the right of the at symbol is anything you want any name that you want and all I'm boring so I just did year month day.
But over here you do anything you want. Any name that you desire. But let's just pretend like this is our first basically daily snapshot.
If we're doing, you know, back to our use case, we're doing like 30 daily snapshots is basically our backup policy.
And so there's our first one. We can view the snapshot by running this command.
And we see that here's our snapshot, it's using 0 Byte right now.
Alright, so.
Okay, yes I do. I have a single slide. Alright, and so, okay, so now that we've created a snapshot, how do we use it?
How could we use that snapshot to restore data if something bad happens? Well, there's a ZFS rollback command and that will reset an entire data set back to how it was when you took the snapshot.
There's also The ZFS mount command, which can you can mount a ZFS snapshot and then use that to recover.
Specific files. And actually on Linux, I don't think you can use the FS mount.
You have to use mount. And then you, you pass in the minus t, ZFS option.
So let's take a look at that. Happening in real life. So. The first thing I'm gonna do is I'm gonna remove all those files from SRV.
Now if we look at SRV. There's nothing in there, okay? Completely empty.
And now.
Take a look at this. And now we see that there is some used space in the snapshot now, right?
So even though I deleted some files, you know, the amount of like the disk, the amount of used on the disk did not get any smaller, right?
Because those files are still included in this backup. And so that data still needs to be on the disk.
And so, our snapshot was using 0 Byte. Now I deleted all those files and now our snapshot is taking up some space because It contains files that are like working copy or current data set or however you wanna say it doesn't have.
Okay, so let's do a roll back. Using the ZFS roll back command. And it's really simple.
You just do ZFS roll back and then the name of the, the snap, the data set, the outside and the name of the snapshot.
And then once I do that. We can see all of our files are back in our SRV directory.
Just like they were at the time we took that snapshot. And so now, so that's how you restore an entire data set using ZFS roll back.
Now let me show you how to do a single file. So I'm gonna remove one single file out of the SRV.
And then what I'm gonna do is I'm gonna do a mount command. And I'm gonna do mount minus t, that's just last amount knows the ZFS file system and then we just put the snapshot and then we put where we want to mount it to, which I'm just gonna put it on slash MNT.
And once I do this, we can do an LS dash L on MT and we see all of our files are there.
And so if I just wanna restore this one file that I deleted, maybe it was on accident. Then I can copy that out back to.
Sorry. I can copy that back out into this SRV directory and there it is. So now we've restored a single file, by mounting a ZFS snapshot.
Let me. Unmount that before I forget. Okay, so that's how you can do that.
Let me pull this back over.
Okay. And so now, where are we now? So now we've done the ZFS snapshot thing, right?
So we have a snapshot. But it's on the OS drive, right? Our snapshot is over here on the OS drive, which is the same place that our originals are.
So yeah, the snapshot gives us some retention time, but the snapshot that I wouldn't actually call it a backup, right?
Because it's on the same disk as the original data. So if the original data go, or so if this hard drive goes pop.
We lose the original and we lose all the snapshots. So in order to actually have a backup, we need to copy the snapshot over to the backup drive.
And the way that we do that with CFS is called ZFS send and receive. Then this allows you to send a snapshot or I think a data set or a snapshot.
To another Z pool. And so let's see that in action. So, it's kinda, kinda cool kinda funny the way this works.
So you do ZFS send, you do the name of the snapshot and that's just going to splat it to standard out.
And so then you pipe it. To ZFS receive. And you pass in where you want to receive the snapshot too.
And so I'm doing the backup Z pool. And then I'm putting it under slash SRV.
So let's go ahead and run this. And then.
And sorry, just reading chat. And then I'm going to, okay, so now if we look at our snapshots we see we have one snapshot on SRV and then we have one snapshot on backup.
So we have a successfully copied. The snapshot over and now we finally have a backup.
We've made a backup.
Okay.
What's the referral on there, Sean?
So the refer, it's my understanding the refer is how much, Space.
How much data that snapshot is referring to. So for example.
One of the articles I read said, you need to make sure whatever did, data set is receiving has at least the refer amount of space.
So like if you refer was like 3 gigs, but you're Trying to send that to a disk that's only got 2 gigs of free space, that's gonna fail.
This is my understanding.
Now why these 2 numbers don't match up here? You know, I don't know. They matches this one.
So maybe I, I don't know. But, about it just to prove that it works. So we could do.
Go ahead. Somebody talking?
It gets to the. Yeah. Yeah, it gets into the minutia of how ZFS works in some ways.
Is a massive layering violation. It's Almost 3 layers in one opaque thing. Part of the volume management and the way it works is it breaks data up into, I'm gonna say, 4 K.
For the sake of discussion or eightk. Here. Discrete units.
Those are just opaque. Things like object stores. Which are in vogue. Then there's another layer that associates.
Metadata between those and presents it. As a final system. So that's kind of volume management.
Snapshots take. That metadata structure of the. These blocks make up these files. Stuff and saves it.
And when you create a snapshot, you're just basically duplicating that metadata and changing.
To which one is live. But the data still lives on the back end. And is consuming space and the refer refers to is actually the amount of on disk space.
That is used. One of the things I've been struggling with during this presentation is I thought you created 5 MB files.
Yet reference has been around a hundred 4 K for most of it. I don't know is ZFS compression turned on by DFS.
Install and when your data, your 5 meg files. Have compressed that small.
I assumed it was compression, but I mean, if I did this right, it says it's off.
Okay, that's something I question, but my understanding is the reference is in relation to The amount of metadata.
And the amount of disk space. That is dedicated to like exists in that snapshot that doesn't exist elsewhere.
It almost looks like it's ours file.
I think it's also something to do with Yeah, I think there's some sparse files in there as well, so.
It would be interesting if the If non blank data were piped into one of those files to see.
Oh yeah. Yeah, Sparse will make this more interesting. That is if Sparse files on Linux are effectively using less disk space.
Yes. Go up and said sparse there on the
Hello.
Yeah, sparse extended extended part. Yeah.
Well, there's our answer. Maybe I should. Some kind of DDE command to create the files and set as far as.
Oh yeah.
Probably, you know, for the next iteration of your talk. Yes.
Well, since it's quite sensitive. Since it's copy on right. What would happen if you did something from Dev random into one of the files that has a snapshot of it.
How would the snapshot? Change as the Target with a copy I'm writing stuff.
The snapshot. Won't because the snapshot is literally a picture. Of that data structure pointing to the data blocks at the time it was taken.
Right.
What's gonna change is the current. Version.
Why DD is notorious for not using. Minus signs. I would think BS equals the size you want for your block size.
I don't want to go forever.
And Do account of like.
Account.
Like this.
Sure.
Data Destroyer.
Okay. This 1 MB smaller now.
Oh yeah, cause 4 K blocks for a thousand would be roughly 4 MB, right?
I guess now what you want to do is take snapshot, right?
Yep. Do a list your snapshots. DFS list T snapshot I think.
Yeah, I really wish that would come up with a better command for that. That's okay. Alright, so the snapshot didn't change, right?
Because they read only, they won't change, right? So. I would have to take another snapshot, right?
And we'd see this, used go up probably on
Change snapshot. To all. On your previous command. There's a way you can get, yeah.
So.
So you're using referrals, 100 fourk. On both. Are you?
You're on the serve backup. So you're now referencing. 8 point some Meg.
Whereas your snapshot. Itself. Is only the 104 K.
So if you do another snapshot.
It would be 7.
And you. The file like you did before. The snapshot would stay at that 8.1 5 ish.
And the new current. Would get smaller. Based on the truncated file. Or the
Okay, we gotta test it now.
Okay.
I'm gonna just increment the day by one, simulate the next day. Let me do all again.
And so. Yeah, so see we see this here, which matches this here.
Now truncate your file or recreate it the way you did with So truncate.
Which one was that that I do?
The that file.
Alright, thank you.
Yeah.
So. Now you get into some interesting things of snapshots. Are kind of a pseudo data set.
So your serve data set.
Is consuming. Space or things dependent on it. There actually ways to take a snapshot and split it out into its own data set.
Make dependent data sets. It can get really, really complicated. But I think that's way beyond the scope of what you're presenting here.
Yeah, yeah, yeah. And let me, okay. So let me go on to, cause I'm only halfway done, sorry, 8 o'clock, so we're gonna have to cut a lot of this stuff that I.
Prepared. Okay, so let me do.
Oh
Oh no, no cuts, We swear we won't give you the hook. Yeah.
Alright, so.
Actually, I will say, Shawn, that if If you see a reasonable space that you know this could be an interesting or a good place to pause We'll let you get away with cutting a shark tonight then on the promise that you'll do a final version or part 2 on a later date.
Sure. I mean, I, yeah, that's fine. I don't know if I have, I mean.
Yeah, well, let's just see how far we can get. I'm not ready to stop yet.
Okay.
Let's just go a little bit more. So, all right, so now if we wanted to send a second.
Snapshot from the serve to back up. So now we've created. 2 more of these serves snapshots, okay, with all this testing.
And so if we wanted to send these, we can send them incrementally, right? That's what this dash capital I flag does is an incremental send.
And so I'm saying. Increment from this snapshot to the snapshot.
And when I do that, And yeah, that's how you have to do it for the subsequent snapshots.
So I do that and I list them all again and now we see that we have our second snapshot here.
And we can do the same thing for. The next day as well, we increment off of 22 and we send 23 and now we're sending the next.
Daily. Increment. As well. And so now we have basically Sent them all and so basically this is what you could do every day right like every day you come in you ask new documents whatever you take a snapshot you send it over to your backup right and then okay let's pretend like I have 31 snapshots right now.
Okay, because our retention time is only 30, so we don't want 31 dailies, we only want 30.
Okay.
So the way that you do that is you use destroy. So what you'd want to do if you have 31 is you'd want to destroy the oldest one.
Which for us. Is on the 20 first So we're gonna run destroy on the 20 first to get rid of our 30, you know, our oldest one number 31 and then we're gonna have to also run the same thing on our backup to destroy the oldest one.
Data set is busy. Oh, we broke it.
Literally never seen that before.
That means this. Oh, I have that one mounted, don't I? I mounted that one when I was messing around.
Okay. There we go. Now we destroy it and I listen to snapshots again. Okay.
And so, yeah, so you could do this every day. You could literally take a snapshot, send it, delete the old one.
Snapshot, send it, delete the old one. And I'm sure a lot of you are like yelling, cry, Oh, totally could. Right.
Yeah.
Totally could write a script. To do this but you know Hi, I've written a lot of bash scripts and I've always, you know, I'm not the best at it, but I've always had a lot of problems.
Doing date comparisons with Bash. Maybe there's some magic way to do it, but like at some point you're gonna get to like January first and you're gonna have to convince Mash that that's Newer than 1231, you know, and I don't know.
Let's have a discussion about that on discuss. I've got a lot of experience in dealing with that.
And overcoming the problems.
Oh, well, I'm gonna make the argument that don't. Then we don't need to solve that problem because You know, open source is great and wonderful and somebody's already solved that for us.
Multiple people have, but my current favorite implementation of that solve problem is called ZFS auto backup.
This is an open source application written in Python. It's good for backup snapshots and replication and it can easily handle those, you know, couple of commands we're running with a snapshot, send it, delete the old one, it can easily handle, that capability.
And the way that it works is you have to set this auto backup property on the data sets that you want it to back up.
And so I'm kind of rushing here. Let's see. What we can get done.
So it is a Python utility. The.
Author of the software recommends that we install it with PIP on their GitHub.
So let's do that. So let me, Copy in this. So first I got install pip on my new Ubuntu system here.
And then we'll install ZFS auto backup.
Good. Oh, yeah.
Yeah, Okay.
Are you gonna touch on anything other than local send and receive?
I'll throw out there you can pretty easily do a ZFS send on one system. Pipe it through SSH.
And have a remote system do a ZFS receive. Or you can send to a file. Transfer that file.
And do a receive from that file. And a remote system. Don't treat that file as a backup.
It is rather fragile. And very specific to your data set and pool at that time. But it is a decent transport.
Yep.
A lot of examples online like this where you're the CSS and then you pipe that into SSH.
And then you run the ZFS receive on some remote host. I saw a lot of examples like that online.
Is that the dash I is had an incremental? I look like there were 2 sources there. I didn't quite catch that.
Yeah, incremental. So you're saying the dash I is like, what are you incrementing on top of and then this is just the snapshot name that you want to send.
Got it. Nice.
It's typically sending a delta from the 1221 to the 1222. It's not sending the entire, 1222 snapshot.
Okay, so like I said the ZFS auto backup thing. It works by setting these properties on the data sets that you want to back up.
And so I ran ZFS set auto backup colon SRV equals true and then I. And then I just ran this to get the property and we can see right here it's set to true on this one.
And this is what the ZFS auto backup uses to know. I guess which data sets the backup and then the command to actually use ZFS auto backup looks like this.
And okay, I'm gonna walk through this. So we have, we're just calling the CFS auto backup.
We're asking for, output. I forget what this does. But then we're telling it to keep.
30 snapshots on the source. Keep 30 snapshots on the target. And then we're telling it the snapshot format that we want.
So like the name of the snapshot. And the source is specified right here, SRV.
And then the target is specified right here, backup. And the way this works is it goes based on the name, right?
So if you have snapshots on your system, that don't match this name, this format, it'll ignore them.
So if you take your own snapshots like manually or whatever for whatever reason, it's gonna ignore it.
And, you know, I have this on like the, on the system D timer to run every day.
And so all this snapshots are daily. And so it's just a really simple keep 30 of them.
BAM, that's 30 dailies. But it can do much more complicated things, right?
You can tell like you can do a keep source like And then you can put like how many daily should it keep?
How many weeklies should it keep? How many monthly is and yearly is all like you can do a lot a lot more advanced stuff that I'm showing here.
But this is like a very basic. This is basically just enough to do what we were just doing manually.
Automatically. And I like the verbose output because it basically tells you like, okay, this is the format.
And then it can tell you which, data set was selected for the source and it's keeping the last 30 snapshots.
And then it says it created the snapshot. And then it says it sent it. An incremental over to the backup.
And all operations completed successfully. So if we look at our snapshots now.
We can see it's It's this one. It puts this little, so I guess it does a slightly different name that I was doing.
It puts this data set name again over here. But either way, that's not a big deal.
And either way, it's this one. And this one. These are the ones that were created by ZFS auto backup.
Oh, Tony, I see your hand.
Thanks, I was one of somebody's gonna know us. Can you hear me? Hello? Okay, good.
Yep.
Yes.
So my understanding of the snapshot is from LVM. It's really not the data is not on there twice.
It's only like an accounting method. Like it's just copying the data if it's if it's changed like it'll keep You know the the lower layer, you know, they're once once you create the stash out the layer
You know, it's just it's just creates another like an accounting sheet where where if something something is written to one of those files that are that they're existing it creates a copy and rights to change and then.
The original, I mean, you know, that's the only copy. So is, is it different with ZFS or?
No, it's very similar. You could describe it as I've got. Files that are taking up.
Pigeon holes 1 2 3 4 5 And I have a list that says my file is in, And if I make it take a snapshot, it's like a photocopy of that.
My file is HMOs, whatever. You make a change. And it changes. What's in pick them hole 3.
It actually writes it. If they can hold 6. And update my file is now. Did you hold 1 2?
6. And that's a new list. If you want the old file. It looks at the previous listing.
Okay.
Of what constituted the file. My understanding is LVM and ZFS are both very similar in that that they only take up the additional space to store the change.
When a change is made.
Okay.
Yeah, we kind of saw that earlier. Like I took a snapshot in the minute that I took it, it wasn't taking up any space, right?
But then once I deleted some files, then all of a sudden it was the snapshot was taking up some space because the snapshot had some extra files that my current working set didn't, right?
And it would be the same if instead of deleting files, if I just change some files, it would be the same thing, right?
Whichever blocks were changed wouldn't actually be changed on disk, they would be copied to a new location and that's kind of where the copy on right thing comes from.
I don't know how LVM does it. Cause I, I don't know if that does copy on right or whatever, but I know Butter FS and CFS.
You know, I think the copy on right part is a big reason why they're able to do these snapshots so well.
Looks like Ron put some more information in the chat there too about that.
Okay. Well, let's, so I wanna have a discussion here and this is something that I've been wondering myself because I have an O joint H 3 plus and I think this is what I'm going to do but what do you think so I've talked about how we talked about how rate is not a backup because the retention time is 0.
I've also talked about how snapshots are not a backup too because they live on the same device is the original, right?
But that was in the context of a single, single device. Z pool. But what if you have a mirrored Z pool, which is equivalent of Ra one.
And you're taking snapshots. Now all of a sudden you have 2 storage devices, right?
2 hard drives. And you have snapshots for retention time. So let's have a little discussion about this.
What is that, would you, is that considered a backup? Would you feel safe having a mere z pool with snapshots, be 2 of your 3 copies in a 3 2 1.
I would argue a snapshot. Is a backup. For human error. Like if I'm working on a file.
I copy the file to dot back.
That is a backup. But. So the snapshot is as well.
It's not protection against hardware failure. Or corruption of the pool, but it is a backup from me messing up my file.
What does anybody else think? Anybody else got an opinion on that?
Depends where the snapshot is stored.
Alright, well it is getting late so let's move on. So we have the
Okay.
Sorry about that.
Oh, and we can hear you typing too, so sound like a nice key word. Oh good, I good.
Allergies for nice things.
Snapchats being back. So I have an O. Okay, I guess I'm gonna go off on this tangent.
I have OD. So I about one of these. And this has.
It has an M dot 2 slot and it also has 2 SATA ports down here. And so when I was Well, I'm thinking about doing now.
After doing this presentation is putting the OS on the in that 2 and then doing the to say the ports is a rate, mirrored, and then just doing snapshots on that and not sending them anywhere.
And then I'm just gonna use the mirror, z pool as my 2. 2 different storage devices and then still send.
One copy off site, to back blaze. But this is what I'm thinking but this Yeah, that is what I'm gonna do now.
Yeah, you got a redundancy. You're gonna have one off site. That sounds reasonably a reasonable backup solution.
Yeah, cool. Thanks. We'll see. Alright, so, alright, so anyway back to this so we have our ZFS snapshots going to our backup drive so we got 2 copies on site.
Now just gotta worry about the off-site backup. And so, rustic is what I've, I've been using rustic for a very long time and it's fast, efficient, secure backups.
It's another open source program written in Go. It supports several well known operating systems and BSD.
Okay, never going to call me out on that one. All right.
Yeah.
The backups are encrypted by default. So when you create a restic repository, you're going to put a password on it.
And, if you forget that password, you're up a creek. Nobody else, nobody else has that password if you lose it your day that you will never recover your data because it's encrypted by default.
Restic also does some de duplication and like I said before it supports many different backends like our clone and anything S 3 compatible.
The ton of stuff. The back end the the back end that I've been using is back place this company has been around for you know over a decade at least probably and as far as I know they've maintained a really good reputation over that long time.
Their storage solution is called B 2. It's an S 3 compatible object storage. They're free tiers pretty good.
You get 10 GB free tier. They have pretty good documentation for how to integrate with Restik.
And then I think if you go above their free tier, I think it's $6, per month per terabyte.
I think. I don't know. I don't plan on going over the free tier, so.
Okay, so let's see some rustic in action here.
Yeah.
And alright, and I'm not gonna do. I thought about doing the whole book back place thing, on this demo, but I don't know, I'd have to create like an application key and be really hard to like keep it hidden and stuff.
So for this demo, I'm just gonna do local.
Repository but the steps are very similar to if you were doing an actual remote repository. And I'll let you know where the differences are.
So the first thing I'm doing is I'm just installing. Which is in the Ubuntu repository.
And then rest it comes with a self update command so I'm gonna run that to update rustic to the latest version Cause the Ubuntu repo probably doesn't have the latest version.
Next. I'm gonna do a couple different configuration files for Restick. So I'm gonna make a directory in Etsy called
The first configuration file I'm gonna make is called, And this step right here. Would be different if you were using a cloud or like a remote storage.
So this is one of the steps that would be different. If you're doing cloud here, I'm just gonna do a local repository on my backup.
Folder but if you were doing yeah this would be different if you were doing an off-site backup.
But like I said they have very good documentation so you could just look there and see What you need to put based on what provider you're doing.
The other thing is specify here is the password file. Like I said, if you forget the password, you're up a creek.
So putting it in this file and putting it in your password manager and all that probably. Probably a decent way to go, especially if you lock this file down with really strict.
Permissions.
There's our password. Okay, so now all we need to do is we need to source that environment file that we created that has those 2, environment variables.
Once we do that, we should be able to see the restic repository. There we go.
Okay, and now Because I'm doing a local, I need to make the directory where the rest of repository is gonna be.
You don't have to do this if you're using some kind of cloud provider.
Okay, that's pretty much all the setup. And now we're gonna do rustic and knit.
And this is gonna initialize. The restic repository.
Using and this uses the configuration files.
That we had in there. So, The repository is slash backup slash restic.
And here's what I've been saying. If you lose your password, it's gone.
Your data is gone.
Alright, so now that we have Restick installed and initialized, let's go ahead and do, backup, take a snapshot.
So I'm gonna do a restic backup of the slash SRV folder.
And we can see the it did 3 files in one directory. Along with some other metadata And if we, if we look inside of.
Our repository. We're not gonna see anything of value, right? If I do like, let's do a recursive LS through our backup repository and all we're gonna see is like Rastic data FF, right?
And like. And like index files and stuff and all this stuff is just like encrypted and And you know, and so this is why I feel comfortable putting this on back blaze.
Or, you know, some other cloud provider because it's all. Encrypted kind of data nonsense here, you know, it's not like.
It's not like there's a file. A folder on here called SRV and then a folder inside of our files inside of that that have anything.
This is all just kind of. You know, data, block level, encrypted stuff here.
And so, yeah, so that's why I like using Restick because It's encrypted and so I don't have to worry about trusting my cloud storage provider.
Okay, so let's talk a couple more slides.
So, with ZFS, we did the rollback and then we did the mount or rustic has very same thing very similar thing restic has a restore command which is similar to zFS roll back you can also do a rustic mount which is similar to what we did with mount dash t, space CFS.
And so let's see that. How do we recover files? Using rustic. So the first thing I'm gonna do is I'm going to delete all the files in SRV.
Yeah, they're gone. Then I'm going to look at my snapshots.
Snapshots. And then I'm gonna restore.
This snapshot to the target directory. And the reason why I'm doing root is because it will add the SIV folder there for me.
And so I'm doing it like this. And then it says it restored. For file slash directory.
So if you notice when we took this snapshot and said 3 files one directory but here in the restore output it kind of adds those 2 numbers together to say for file slash.
And so if we do LS dash L and SRV, we see there are all of our files.
And this is similar to the ZFS roll back where it kind of did them all, right? This was a, rustic restore of the entire snapshot.
Alright, now let's do a single file. So I'm gonna remove a single file from SRV.
And then I'm going to do a, I don't, have this in my notes, on MNT.
Oh, whoops, I messed that up. Hold on, I gotta do a restic amount and I gotta I gotta put this in the background because as long as the processes running that's how long it stays mounted so i'm gonna put it in the background And now if I do an LS dash L on MNT, you can see here this is different than when we look in our repository.
Okay, this is all like unencrypted and put back together when we do the mount.
So for example, if I look inside of.
M. Alexander's asking why not also encrypt the ZFS pools. Yeah, probably a good idea.
Probably a good idea to do that. I guess I didn't think about it because they're staying in my house.
I guess I feel pretty secure in my house, but yeah, I mean, encryption probably a good idea.
Okay, but anyway, I've used the rustic mount.
And now if we go look in here, so one thing we saw was there was a host, directory in here and that lists all the hosts that have added, snapshots to the repository and 3 2 one demo that's the name of this host that I'm on right now and then I can go to the latest snapshot.
This is like a pseudo. Snapshot ID kind of thing, like a fake snapshot ID.
And then here we're actually seeing like, okay, this is the SRV folder. If I go in there, I'll see actual files, right?
And from here, I can do what,
What I did with the ZFS when it was mounted, I can just copy. The file that we care about back to the place that I wanted to be in.
And then if we do LS dash L for SRV, we see BAM. There's a single file restored using Restic.
Okay, so let me, cancel this mount before I forget.
Oh, I need a CD out of there first.
Okay, I guess it's fine. Alright, alright, so that's it.
Had a question there in chat.
Yeah, I guess it is using fuse because I got this air about fuser mount. When I messed it up just now.
So it seems like it is using fuse, I'm guessing.
Yeah, but this also works on Windows. So it must be able to, I don't know if there's a Windows port of.
But oh, and here's something I'm not gonna be able to demo, but You can use a command called rustic, forget, and you can pass it, keep last and then some number.
And this is very similar to when I was using ZFS auto backup. And I was passing it, keep target equals 30.
It's basically the same thing. The only difference is the restic forget command doesn't actually remove any data for some reason.
And then you need to follow that up with a rest. And when you run rest, that actually removes the data from the repository.
I don't understand why it's like 2 steps like this, but I don't know, I guess it's just a design decision they made at some point.
To do that. So, and the documentation says to run rustic check after you do the prun.
I guess I've never done that. I guess I should have read the docs closer.
Last time
If you do a forget.
And you mount. The restic, whatever you call it. Is the data still accessible there?
Let's see.
Because I feel like this is playing with a bunch of things along the lines wrapper around in C FS or whatever it's called.
Where it's managing metadata and it's using. Standard components. And I could see how forget would update an index.
And then a prune would go back through and remove any encrypted pieces that were no longer referenced in the end.
Kind of like a compact on a database after you delete. Entries.
Doesn't look like there's anything in there.
No, everything's empty. So I did the forget like you said. So here I forgot the only snapshot.
Okay.
And then I mounted it and now it's saying host is empty, snapshot is empty.
So, so yeah, I guess, okay, but here's the question, right, if we look in the repository.
Yeah, see there's still this data FFE, all this data nonsense is still there.
And so What's that?
Well, those are empty directories. Those the FF, EE, whatever those are empty directories.
What's up?
Oh, and can I do, do you.
And what is it like? And, or something.
To see just the directories right beside you.
How much the rustic directory is without recursively going down.
I habitually used D. HSC m. And give it the item you wanna look at. So just the restic directory.
There's those options do more, but it'll give you what you want. So that's 5 meg and the M is megabyte.
You can change it to K or whatever.
Thank you.
Okay. So you still have data in there though. I assume the repository is not 5 meg when you first initialize it.
I don't know if sure though.
Yeah.
Oh, wait, I need to unmounted first.
Only you need battle.
A bunch of stuff. There's some stuff right now if I do this. One.
Did you do the, Do you. Okay, so yeah, they're still encrypted data on disk is no longer referenced keys for it or probably gone.
Brune actually removes it from the back end store.
Yeah, it was still very confusing why it's not on one, but what do I know?
Okay, so next I was planning on doing data loss stories, but we're running a little bit over so why don't we skip that for next time.
Okay, do we wanna talk about failure scenarios? I feel like, might be beneficial or feel like this group might already know.
But we have 3 separate pieces here that all could fail independently or all the same time or any combination in between. So what does that mean?
We have like 9 or 10 different failure scenarios or something. I'm not sure. But you know the simple one is like what if your originals die, right?
Well, since we have like a 2 week RTO, we have plenty of time to go on your favorite retailer.
Buy a new drive, casually install a one too and set everything back up. And then just plug your USB drive in.
Restore the latest snapshot. Or mounted and copy, you know, whatever you wanna do.
And that would be the scenario here. And then one of your backup drive dies. Here you would buy a new USB, plug it in and you'd have to do Well, I don't know.
One way would be like to do 30. Zen and receives right to restore all the all the different snapshots
No, you too. Okay.
You'll need to. You can actually either drive dies when you're ready to bring something back.
You can, if none of the data sets are mounted, you can tell ZFS to recursively send the entire pool.
So like when your OS drive died. And you replaced it and you were ready to pull things back from your local backup.
You would do a ZFS send from backup. 2 and then a ZFS receive your SRV or whatever.
And it will copy the entire pool when you give it the proper flags.
Right, yeah, I remember seeing that in Christ supply. Now that you mentioned that's a good point.
Third family scenario if your cloud storage provider, goes to put. You know, I still feel pretty confident here because we still have 2 copies locally.
So we just find another cloud storage provider and upload our snapshots. Back to it. So I guess what you could do.
And this case is like you could mount your, well, I don't know. Maybe that's like mount your oldest snapshot and then send that up and then mount your second older snapshot and send that up and and do all that if you really wanted to like restore fully.
Another way you could just be lazy and just like wait 30 days and eventually it would be caught back up with the new stuff.
So kind of doesn't how. Lazy you are
Yeah.
Then there's this scenario here. This is like what we were talking about earlier like fire, flood, something like that.
And in this scenario, you just apply the the, you'd have to like, you know, download the older snapshot, take a ZFS snapshot, download the second all the snapshot, take it ZFS.
Like that's if you wanted to build back up all of your dailies, locally.
Or if you like said you lazy just recover the latest and then just Over the next 30 days, you'll just build back your attention time back up.
And then there's this failure scenario number 5 and this is when you just completely screwed right if everything goes out all the same time you just you're done your date is lost right So.
Okay.
This is the one that you hope never happens. And then like I said earlier, backups always succeed, but restores don't.
So you should frequently test your backups. And you know, we have this meeting every month. And so, you know, if you spend the first 10 min of this meeting.
Testing your backups. You'd feel really confident about backup so you test every month like you'd feel really good about that and so you know maybe the first 5 10 min of this meeting is when you should Should do you restore testing.
Also, monitoring is very important, you know, and I don't really have time to go into it now.
I haven't really figured out what I'm gonna do for that yet, but what I think I'm gonna do.
Is have these Chron jobs or these system D timers. I'm gonna have them, you know how like in Bash, we do like and and.
It will only run the second command if the first command exits with a 0. So I think I would.
Do like the backup command and and touch a file somewhere, right? And then what I can have is I can have some monitoring software.
SSH in, make sure that file is there every day. And if it's not, start shooting off alarms.
And the reason why I like that approach is because You're SSH in so that means the box has to be up.
So like the whole server is down, you know, that's gonna start setting off alarms. And then if the servers up, but the backups aren't working, like they're not touching that file, then that's also gonna set off alarms.
But that does kind of complicate the setup because then you need something. External to SSH.
And so you like another at home server thing, Raspberry Pi, something something. But I just want to say monitoring is important and so probably want to add some monitoring to your backups as well.
And that's it. We've made it through the whole thing and it's not even 9 o'clock yet.
How about that? If you want, we could end with some story time. If anybody has any. Good, bad, terrifying, disastrous.
A comment on the monitoring. I noticed restex message did something that I very much so like.
It included that it's process was successful. I am a very strong advocate of collecting that in logs but also Sending that to me as email.
And I even want the emails where it is successful. I then configure. Message filters, rules, what have you.
To file the message away and if it is successful because the text same that is in there I mark the messages red.
Whereas if there's some sort of failure, it comes in and it gets filed away, but it is left as unread.
So it is as I'm scanning email, hey, why does backup. Older, have an unread message.
Go look at it and you see it. Make the information available. But make it be quiet. Most of the time, but you want the success messages so you can go and say, okay, it backed up last night.
Instead of I haven't heard from this in a while, oh crap, it hasn't run in 2 quarters.
Yeah
One comment on that. All our backup jobs run through Kron. I've got crowded up to email me after everyone.
So each backup job, emails. You know the common email the result of that job. And all you have to do is, you know.
Lands at them and you can tell whether they were successful or not. Saving them in a permanent log would be as I think would be as simple as a T at some point.
But I'd have to think that through.
Agreed. I'm We have been using Zimon. It's an older monitoring package.
At day job and I implemented it here at home and for a friend. All 3 places due to various circumstances.
Chron stopped. On at least one system. So I, yeah.
Really?
You know, one at work was a Solaris box roots password expired. So root crime job stop.
So I have an hourly cron job that just uses I'm on to the client to send a message to the server.
Hey, Chron ran at date and time on host. And I said a lifetime of 65 min.
And the lifetime is important. Because if there is no update within that lifetime. Zimon considers, there's a problem.
Notify and let people go check it out. So we actually had another system, a different system at work.
Roots password expired. We noticed it 2 h after it happened. And it was corrected and we manually ran the jobs that would have run in that time and moved on.
Instead of the 3 weeks later, why aren't we getting these things? Why data backing up?
Monitoring is very important.
Good story. Really?
Why, sorry, that I have. So, and this is kind of what this is part of the reason of what sparked all this, why I've been trying to look for a place to store my documents and my projects.
Because, you know, I was distro hopping as I do all way too often, right?
And I, I'm like, I gotta back everything up. So I'm gonna wipe this drive.
And you know Everything's so scattered all over the place that backing up everything is like 20 folders and who knows if I got everything or not and so I really want like a central place to store a bunch of stuff and and have it be backed up and that's kind of what I'm doing with this project.
And, yeah, so one time I thought I had everything backed up, but what I didn't have backed up was the copy or the code for that website.
Redesign for this loop.org. So yeah, I went to show hop and I thought I had that back.
Up where I thought I had it on GitHub and I didn't and now it's gone and so That's one day the last story I have.
Got a hardware data loss story. 3 drive, hardware rate 5. It's guys he drives probably 20 years ago.
Had a spare drive at the shop. That had been temporarily used in another system. But the intention was, you know.
Scratch, erase it, whatever. So when the drive died in the raid.
The raid went into degraded mode, everything was okay. Replace the drive. When we and it was not.
A hot swamp chassis. So when we booted back up. The raid card. Got very cranky.
And we apparently hit. A bug in the firmware on the card. That instead of writing data to.
The new drive? It copied data from the new drive to the 2 surviving drives, which promptly, immediately corrupted all the data.
Oh my god.
That was a 48 h emergency. Build a Windows 2,000 domain controller. Pull things back over, restore applications and.
Oh, I was ultimately a week of my life.
Well Well, Jim, just as an option. And it's in the scenario that Shawn has been describing.
15 years ago I set up an own cloud box with 4 drives and a rate 6. Because I have been bit more than once.
With 2 dry failure.
And actually I can I can speak from experience that I actually had 3 drives fail in a rate 6. But there was 24 drives to start with.
And so I just picked one on it was failed and forced to rebuild. And I got to, I got the ray back, then I could swap on the other 2 drives.
But anyway, where I was going is a central storage. And it doesn't have to be local.
It can be out in the cloud. Shut up your own own cloud or next cloud server on it And set up a login for each type of data you want to say.
For example, I've got. A main one here in the shop. Probably got 50 gig worth a day worth of data that I can consider critical.
I've got another one for my for my phone. I've got I don't know maybe 2 or 300 meg worth of data that I want to take with me.
But that is another option and you never have to worry about it.
Because once you install the web. DAF client, which is what. Most of these, old cloud styles.
Tools run on. Everything runs under the covers. You know, you don't have to worry about it.
You change your file. Within a minute, it synchronized back to the server. And then of course I back up the server.
Itself. So I've always got a year's worth of backups there.
It's a question in. In chat. That which is more stable, a 4 drive.
Rate 0 one or 1 0 combination. Rage 6 Great 6 is much more fault.
Grade one or rate 10 are for performance only they provide no data redundancy.
Rate one provides redundancy for physical failure.
Yeah, red ones a mirror.
There's a lot of people that will put. 4 drives. And a mirror across stripes or stripe across a pair of mirrors.
And they think they're safe.
But
Well, that, yeah, that's right. That's rate 10, right? 10 gives you performance.
It does not give you any redundancy.
It gives you some redundancy against a single drive failure. Right.
Compared compared to an actual rate array. Especially if it's built right. I would argue that the redundancy it provides is is a false assurance.
No, it is not. If you have a 4 drive.
Raid your calling rate 10 array. It will protect. Against a single. Hard drive failure.
It may. Protect against a second. Drive failure.
Raid 6 will. Protect against drive failures.
It depends what you're protecting against hardware failure. But rate 6 is better than rate 10 for the same number of drives.
Which is true.
Gary's give me the evil eye here.
Just thinking through just thinking through.
Okay.
Okay
The reason. Rate 6 itself inherently protects against. Who drive failures. The thing that bites people with grade 10.
Is you either have
Stripe across 2 mirrors. Or you have a mirror across 2 stripes. So if it's a mirror across 2 stripes.
If you lose one disk. In each stripe. The entire mirror is gone. Or conversely, if you're striping.
Across mirrors. If you lose both disks in a mirrorplex. Your stripe is broken and your data is gone.
Read 6, any 2 drives. Of 4 driver 8 6. Any 2 drives. Then die.
And your data is still accessible.
What's that?
Of course, there's another factor there. I haven't built except for. Amir for an OS I haven't built a rate around 10 years with less than 7 or 8 drives Sometimes up to 24.
ZFS will happily do this.
Zfs, you can. Mirror across stripes or you can stripe across multiple mirrors.
And with traditional raid. Hardware cards and traditional software raid. It is difficult. ZFS makes this almost trivial to the point of encouraging it.
But you know a lot of people that I used to work with had a one U rack server that had 4 3 and a half inch drive slots on the front.
You're not, you don't have the luxury of more drives so that they would go, well I have 8 10 across my 4 drives and I'd say.
I can be mean and I can pull 2 drives. Your box is dead. No, you can't. Yes, I can.
Are you willing to bet your box's survivability? But if you have the luxury of many drives.
Possibilities open up. Exponential.
And you can't mirror our run rate 10 on them. Without a major nightmare. Any and if you need a server Go up to orange and you can you know, for like 1,500 bucks you can get 4 50 TB.
Actually, the the last one I got up there a couple of years ago. It was at it was actually halfway quiet.
Yeah, you could almost put in an office.
And where is this you talk about? I'm new to the St. Louis area.
Would you?
Orange Computer of a Maryland Heights. They, they, they take off lease server hardware.
Okay.
And clean it, refurbish it, rebuild it.
Would you?
Very, very similar to EPC out in St. Charles except EPC does what deals primarily desktop stuff and oranges all server stuff.
Sure.
Would you please put their names? And if you have a URLs in the chat or mailing list.
Sure.
Thank you.
We're talking about backups, bare metal backups are a whole nother type. Well, bare metal recoveries are a whole nother type.
And if you want to go to EPC. A open source friend of ours, Tony Levasco.
And he'll help you with whatever you want. And if you tell them you're an Omni tech customer or client, they'll give you 10% discount.
And if you want to have fun. Watch the tech calendar. How is actually out at.
Couple months ago and they brought their disc hard disk shredder.
Awesome drive in about 15 s. It is toast. No, no piece bigger than a pencil.
Yep. But it makes some noise and some heat.
Oh yeah.
And they're actually pretty good. You know, I buy systems from them all the time, bear.
Just Tony, Tony can pull one out of stock and pull the, pull the windows drive out of it and that way you don't have to pay for a Microsoft license.
Using the OEM sticker on the chassis.
Yeah, but usually the stickers old enough that. You, you can't do that kind of upgrade without the OEM remanufacturing license.
Yeah.
So it's kind of a moot point. Because the stickers like Windows 7 and the machine was Windows 10.
But anyway, I've. Sticker on the box is different from the sticker on the OS.
Yeah.
It's retail versus. Oh, Well, anybody else got any comments, questions? Observations.
Gonna make 2 comments first off a really really big thanks to Sean Tweet house the in a real tour de forest presentation here of, using ZFS.
Ins and outs. Just Just really, if it wasn't so much fun and I'd say, you know, it hurt our brains just trying to keep up with it.
But it certainly hurts a lot more when something where something breaks breaks. I second the motion, Sean.
Thanks a lot. Good job. Thank you very, very much, Sean. And, so yeah, we hope, hope to hear more from you again soon.
Certainly if there are any more questions that people think of, I'm sure that if you post it out and, discuss mailing list.
I'm sure Reha Shawn will pick it up or one of us will not shown to, look at discuss and there may be some other people who would comment on it.
But be posted out to discuss. So. Please do. Also big thanks to a few of the just attendees tonight we've got some really good audience participation, some great, verbal comments and entries in the chat.
So, thanks to those who, who participated. It's really appreciated. Any comments?
Well first off, any additional comments on tonight's presentation?
Hearing none write off any Any other comments about something other than tonight's presentation and I'll lead off with that of I should have mentioned it early when we were talking about the the general calendar.
And see if I can pull my calendar up here real fast. And let me see if I can do it.
Good.
Stay in if you can pull the calendar up
Either way. I believe you can pull it up. I can find the right. Browser wouldn't know.
Okay.
And.
Share.
Alright, Stan, you beat me. Good find the freaking tank. Okay, look at look at tomorrow
And look at the. Is it 50 fifth anniversary there?
Yeah, I've just always been intrigued talking to a number of people who've gotten their MBAs, a couple of the, of fine universities here in town or other places for that matter and they literally joked that yes, besides all these fancy classes that when they actually go out to get hired.
But people want to know is, yeah, but what can you do for me in Excel? You know, can you handle the spreadsheet?
That's all we really need. And it turns out that the beginning spreadsheet, the first spreadsheet was actually written on GCOS, the grandmother.
Of Unix. And a couple of you may have know we have little chat here couple weeks ago about the history of unix and where it came from so general electric had a their main facility was out.
Phoenix there and at 1 point they were doing some of the early development along with Bell Labs.
Of Unix and so some of the guys working out there in Phoenix actually Just because they're working on a project and had a lot of stuff that would today we would go, oh yeah, that's a spreadsheet.
Didn't exist back then, but they put it together, made software out of it. And it ran on GCOS and So the early predecessor of Unix and it was actually a marketable thing.
Most people when you talk about the history of spreadsheets, they want to say, oh, well, before it was on.
IBM PCs it was on Macintoshes. Lotus 1 2 3 and but nope it actually.
Was on the mainframes before that and then was available on Unix. And, that rings well, rings well with me because St.
Louis, you had a, a Gicose. Mainframe and yes, I put the spreadsheet out on it.
Completely bamboozled the guy at the business school. Like, huh? You got a spreadsheet on the mainframe?
And so anyway, that's the trivia for tonight.
Anybody else got anything? Of either phone trivia or last minute.
I found a podcast on where the author talked about how it works. And as follow up to recent presentation.
It actually does hook into the PTY drivers and records the output. From the computer. That was something I had asked and we've discussed briefly.
During that presentation, but I got. The author talked about that and it is indeed. Recording the output, not what you input, which is why it won't get passwords.
Nice.
Got a piece of trivia that, Gary will appreciate.
Goes back to the 1980, s mid 19 eighties. Running deck as. Mainframes back then.
But people wanted Windows type. Spreadsheets. Well, there was a company. I can't remember who they were anymore.
They built an add-on box. You could attach to a deck. And supply out through the old 2 20 deck terminals.
Spreadsheets. To anybody that was connected to the deck. A
Oh!
Talk about a bizarre combination.
Yeah.
Okay, back in that same time frame. BSD, 4.3, I guess it was, came out on a Motorola, 68,000.
And a company called Fortune. Computer systems took that motororola microprocessor. Build it in a white box with character terminals.
And they had a weighing style word processor and a spreadsheet and all kinds of and then business basic for the company.
Because I was selling them back in the back in the eighties. Hello.
Hey, you remember the wang then back in those days. Okay.
Yeah. I was pretty good on that word processor under the. I could bang those keys. And, in fact, that was, that was the first time I actually figured out how to do a 2 column document.
Here to 4, it had been impossible.
At that point. Wow.
Head document formatting is its own specialty.
Oh yeah. And you remember back in those days on certain word processors, you could actually play with the meta characters that did the formatting.
Hmm.
They were embedded in the document. And certain. Word processes allowed you to get at them.
Yeah, that was work perfect or it was original. They'll let you do that.
Yeah, you're right. Word perfect was one of them.
Well, well, it sounds like things are winding down. We've appreciated everybody's contributions tonight.
Certainly we've appreciated Sean for putting it together. Please come play attention to the calendar or announcements or whatever and we invite all of you to come back.
Probably in, well, let's see. Stan, you told us I believe a new user log is, next week I think.
You know, but, so yeah, either see you at the, new users, log meeting let next week or I see you.
Next year. So come back and thank you all for coming tonight. And one more thing I forgot about.
We've had a couple people. Offered to watch your help assist or whatever on the server upgrade.
I've, think everybody's on system and list at this point. If you're interested in helping out and you're not on a system in list, let one of us know so we could add you.
Because I'm thinking that next week might be a holiday for a lot of people. We might wanna you know, try to get some work done.
Alright, last call, anybody else?
Just say thank you again to the presenter tonight.
Definitely. Alright.
And I'm going