Jump to content
  • entries
    0
  • comments
    0
  • views
    324

On-Site And Off-Site Backups


xxmikexx

400 views

In the Digicam thread skylab and I got off onto the tangent of data backups. I said I would open a new thread for this stuff. This is that thread ...

 

xxxxxxxxxxxxxxxxxxxx

 

skylab,

 

I'm like you -- paranoid about the possiblility of data loss, so I have everything backed up locally multiple times. However, for me this kind of approach is no longer sufficient to make me comfortable. I'm going to write about this subject again in the PC Software Tech forum, but for completeness you really should consider some kind of off-site backup storage. I was wiped out by an office fire once even though I was fully backed up so I'm sensitive to this issue. It can happen to you.

 

Until a few months ago I used to create a set of fire storage CDs in duplicate every 3-4 months and give them to my son for safekeeping. At the time this was about 12 GB of data. Even in compressed form it took a dozen CDs to hold everything. Today it's about 20GB of data, and I really don't want to monkey with the 20 CDs (plus 20 duplicates) that would be required.

 

xxxxxxxxxxxxxxxxxxxx

 

So these days I'm going to be doing offsite backup differently. Having tried the Carbonite service and having found it to be inadequate for my wife and I (watch for an upcoming blog), I decided to implement an electronic offsite backup service of my own. My FS Open Components web site has a large disk storage quota, mostly unused, and I'm now uploading my offsite backups to it.

 

However, I'm gong to tackle this problem in stages. This is because the compressed version of the 20GB of data takes up roughly 10GB. I can upload only about 100 MB per hour to the fsOC ftp site, so even the 10GB version would take about 100 hours to upload. This is simply not practical, even if done only every three months. So I'm tackling the problem in stages ...

 

xxxxxxxxxxxxxxxxxxxx

 

The first stage is currently underway. In particular, for the time being I'm reducing the offsite stuff to the essentials -- data that simply could not be reasonably be reconstructed from any source other than the offsite backups themselves. This means the photo images (irreplaceable), the latest AirBoss source code (irreplaceable), the course materials I'm developing for the FS Flight Training aspect of the joint venture with FlightSim.com (irreplaceable), and all of my downloaded payware (very costly to replace).

 

The second stage will be to upload, in compressed form, the less essential data. Most of this data is also irreplaceable, but little of it is vitally important so it can come after the irreplaceable data. The interesting thing about this data is that it doesn't change -- it is an archive. Things get added to the archive, but once in the archive they don't change and they don't get deleted.

 

Now ... Couldn't I simply put all this stuff on a third removable hard drive and give that to my son? Yes. In fact, I will be doing this because it will be the best way to hedge against our webhosting service, GoDaddy.com, going out of business or suffering some kind of unrecoverable catastrophe. But that approach will require a pair of fire storage external drives, one that will be currently in my son's possession, and another located at our condo, to be loaded with data and then exchanged for the one my son will be holding.

 

Isn't all this expensive? Yes. In fact, by the time I have the final system in place I will have as large an investment in removable drives as I would in another computer. But it's worth it to me -- computers are easily replaced but 20GB collected and organized of data is not.

Edited by xxmikexx

15 Comments


Recommended Comments

I consider putting my irreplaceable data onto CDs my "offsite" back-up. Anything that would have to be uploaded to another site is NOT a good idea in my opinion. Too many chances for loss. And there's always a chance that someone else could use the info that I might consider private, like passwords, etc. The same with external drives; something could cause them to fail just as the drive they came from. CDs seem to me to be a near fool-proof way of storing data for later retrieval. Are they in a fire-proof box? No, but I guess that's not a bad idea. All your ideas are workable, but I kinda like to keep things simple.
Link to comment

Very well said, Mike. I work in a server farm as a systems engineer, and one of my primary responsibilities is backup and recovery. Our backup solution is just an industrial strength version of what you're doing. Keep a copy on-site for X number of days for an archival copy, send a copy off-site for Y number of months for a backup, and if the system/data is important enough, we'll send it off indefinitely. It all boils down to this - if it's not sent off-site, it's just an archive.

 

While I don't advocate using a complex backup procedure for home use, I do think it's becoming more and more important to have some form of backup procedure in place for the average user. As people move away from buying music, software, and many other services in a brick and mortar store, they're not thinking about the possibility (some might say inevitability) of the PC going belly up. Many of these sources will not allow you to re-download the song/software from their online store at no cost if you lose your data.

 

Personally, I keep my home backup strategy simple. I run a weekly full backup of my system and copy that to a hard drive and leave it at the office on Monday morning. I have two drives, and rotate them out...one at home to copy to and one at the office for my backup. Critical, and irriplaceable data gets backed up to DVD quarterly and a copy left at the office and a friend's house.

 

Mike, I've seen in several of your posts that you're a former Digital guy. I have to say, they built machines very different back then. In March, my office participated in a disaster recovery drill for several systems in our data center that we support. Before the drill began, we got a tour of the DR center. They have pretty much any mainframe you can imagine from the 70's on, including many, many DEC machines. They have several that are well over 25 years old that look (and more importantly RUN) like they just came off the line. Very different than the service life of 3 - 5 years for servers today. Needless to say, I was like a kid in a candy store in there. :)

 

Cheers,

 

J Flanagan

Link to comment

skylab,

 

I'm not going to argue with you -- this whole subject is much like "How much life insurance should I carry?" I can only observe two things ...

 

First, I have had backup CDs go bad. This has never hurt me because when I back up to CD at all I do it in duplicate. (And I then remake the bad CD from its duplicate.)

 

Second, if a fire were to happen at my home office and I survived but my local data did not, once I have everything uploaded to GoDaddy there will be only an astronomically small chance that I will not be able to retrieve my data from GoDaddy.

 

xxxxxxxxxxxxxxxxxxxx

 

My goal is to make sure that, to a first approximation, only a nuclear attack on the continental USA could wipe me out. Once I get everything up on GoDaddy, and after that on quadriplicate removable hard drive, I will be bulletproof just as I was when I was sending CDs offsite.

 

My office is a room in the condo apartment in which my wife and I live. If a fire were to happen at night while we were asleep it is quite possible that we would survive, in which case the loss of data would not matter. (It's of no interest to anybody but us.) But if we did survive, I would want my data back.

Link to comment

j flanagan,

 

As I keep saying to people, it's all a question of how much time (and money) you're willing to lose -- the probability of loss interacting with the effort of guarding against that loss.

 

As for me, as a software developer I'm not willing to lose more than 10-15 minutes worth of work. Most data loss is attributable to operator error so I snapshot my work a few times per hour -- and I do it in duplicate, once to a different part of my development system and a second time to my flight computer. So if I screw up my work I have a fallback that is essentially current, it being easy to remember what I did between the snapshot and the screwup.

 

Furthermore, even in the event of loss of the C drive on my development computer, the first snapshot I made will be available on its D drive. (These are physically separate drives though they do share the drive bus and the power supply.)

 

You're a professional, I don't have to detail all the things that might go wrong. I can only say that my approach is intended to make me indifferent to the details of what might happen. The switch for me from using CDs to using removable hard drives means that I can do total backups more frequently.

 

My current weakness is that it will be weeks till I have all the archived data uploaded to GoDaddy. However, I'm managing that risk by uploading the most important current and archived data first as earlier described.

 

xxxxxxxxxxxxxxxxxxxx

 

Now let's talk about DEC computers. In fact, let's talk about them in a new thread so we can keep this one open for more discussions of backup strategy costs and benefits ...

Edited by xxmikexx
Link to comment

Mike said: "I'm not going to argue with you..."

 

And that's good because I'M the CAPTAIN!

 

Anyway, so far I haven't had a disc that I couldn't read. Always a first time I suppose, but I check them as I write to them.

Link to comment

Absolutely, you have to pair your backup solution to your business needs. Personally, I can stand to roll back my data to where it was a week ago. At work...this wouldn't fly at all. For most systems, we generally do a monthly full backup of the server followed by nightly differential backups (what's changed since the last full job) until the next full job. There are some cases where even this is unacceptable, and there is a need for real-time replication to off-site storage in a fail over data center so we can provide data and services uninterrupted even if we lose our main site.

 

Again, it's great to see someone running it up the flag pole and alerting people to the importance of backups.

Link to comment

Mike,

 

What didn't you like about the Carbonite backup system? Having got a fairly solid backup system in place at home, I'm looking at options for offsite and the Mozy and Carbonite options look decent for my use. I've found some pretty good reviews of the Mozy online backup as well.

Link to comment

Skylab,

 

I don’t know what you mean by your checking them as you write them. However, I’ll tell you what I do. Having written a CD, I read it back and then unzip it. I then make sure that the byte count of the resulting uncompressed stuff matches the byte count of what had been compressed originally. As for reading it back, I either reboot the machine on which the CD was written before re-reading the CD, or I read it on a different machine.

 

The reason for the reboot is that if you simply write a CD and then read it back, you may not be actually reading the physical media but instead looking at a buffer containing the CD image. But usually I will zip up all the data in advance in CD image sets, and then write all the CDs back to back after which I read and unzip them back to back. This makes the reboot unncessary because the previous image will be wiped out by the next one that comes along.

 

. . . And yes, you’re the captain. Are we there yet?

Link to comment

Flargan,

 

I forgot to mention that I do incrementals as I go along. The incrementals contain the data I snapshotted (gr?) as I was working, including all the successive versions of the work. A particular incremental gets swept to removable HDD every couple of days.

 

So in the event of a rollback I have all of my previous data with about fifteen minutes resolution. I have a very stylized and systematic approach to these things so that on those occasions when I do need to reach back into my archives I have no trouble finding stuff.

 

I even encountered a situation recently where I needed to go back in time to see when a particular problem with AirBoss set in. The official new base level source code versions are kept forever, but the every fifteen minutes stuff is kept only for about six months. Because my problem arose during the period when the high resolution snapshot series was still available, I was able to tease the guilty change out of an obscuring mass of version-to-version changes.

 

xxxxxxxxxxxxxxxxxxxx

 

This subject can and will be discussed over and over again, just like camera magazines cover the same matters over and over again on roughly a two-year cycle. Yes, it's healthy to stimulate thinking on the part of those folks who currently are not doing backups, or who think that what they are doing is adequate when in fact it may not be so.

Link to comment

loki,

 

I'll start that Carbonite thread during my night. Briefly, I absolutely love the Carbonite concept. However, in my particular situation it did not work for me. More information later.

Link to comment

I just meant that after I write to the CDs I eject them and then read them on the other computer to make sure everything is there. Bite for bite. Then, every so often; say months or years, I check them to see if the stuff is still there. Don't know where it'd go, but I have a look-see.

 

And.....that's Captain with a CAPITAL 'C' please!! And.....we'll get there when we get there and not before.

Link to comment

skylab,

 

You're in good shape for everything but a fire. Perhaps as Captain you will be able to make the fire go away by glaring at it or otherwise pulling rank on it. (Actually, I'll bet you were a great guy to fly with.)

 

Anyway, while I didn't write about it yesterday afternoon, by a strange coincidence we had a kind of fire then. Some lightning started a fire in the shortgrass that covers a local geographic feature known as Green Mountain. The grass is tinder because we haven't had meaningful precipitation since early May. Combine that with yesterday's high temperature, 102 F or thereabouts and it was no trouble at all for the one active thunderhead we've seen in a long time to cause ... well ... a wildfire.

 

xxxxxxxxxxxxxxxxxxxx

 

When the fire was announced first announced on radio my wife said "That sounds like Jerry and Cynthia's area. I'm going to go over there. Will you come?"

 

"No" I said, "I'm busy." And I was -- busy writing blog posts. I wasn't concerned because Green Mountain is about 500 feet high, and fire burns up, not down, right? (Wrong. Completely wrong. As I know now, a wildfire will burn in all directions.) Anyway, Evalyn returned an hour later. Or rather, my oldest grandson Alec did, startling me when I went out to the living room to see who had entered the condo without saying anything.

 

Alec was followed by my Evalyn, who was followed by our youngest grandson Zane, who was followed by Cynthia. My wife had been right to go over there. The fire was already threatening the neighborhood by the time she got there, and she was able to help Cynthia load the important stuff -- people, pets and photos -- into a car that Alec would drive to our place (and Alec does not have a solo driving license yet). After getting the cats in their cages and into the car, and the photos, they went to work on other stuff, like the jewelry ... and the computers. And then they convoyed over here with Alec driving bandito in the middle vehicle, the one with the loot and the cats.

 

That's right, folks. Sitting in our living room are that family's computers, the mission of offsite backup finally having been accomplished at last albeit the hard way, under the pressure of an advancing fast moving fire. As shown by the TV coverage, the flames sometimes were moving really fast, at one point causing some firefighters to sprint for their foothills-capable vehicle and bug out. You get much fiercer fires there in Florida, skylab, but this one wasn't fueled by underbrush, it was simply dry shortgrass that was burning.

 

(As a Native New Yorker I had only heard of wildfires -- fast moving wildfires -- and in all of thirty years out here in Colorado I had never seen one before. I did not believe that open shortgrass prairie could burn like that, even downhill. Now I know, and now I believe all the hard-to-believe old stories about tallgrass fires.)

 

xxxxxxxxxxxxxxxxxxxx

 

To make a long story short, all of Green Mountain burned along with several of the adjacent lower hills. This are is unincorporated Jefferson County, not built out, so there was nothing to stop the fire, which burned all the way down to the backs of the houses across the street from my daughter's house. My daughter's family was just three hours away from departing for a scheduled vacation in Alaska. I said to Cynthia "No, don't call us. Just enjoy your vacation. If the house burns you won't be losing anything important, and there won't be anything you can do about it, so why risk spoiling your vacation?" She was having none of it, even though when she and her brother were little ones we went on long driving vacations during which period we were completely out of touch with everybody else -- deliberately so -- in order that our vacations not be spoiled by the real world intruding on our fun.

 

 

So, Captain (and everybody else reading this thread), fire can happen to you. Whether you want to go to the effort to manage the risk is a separate question.

Edited by xxmikexx
Link to comment

As to:

"You're in good shape for everything but a fire. Perhaps as Captain you will be able to make the fire go away by glaring at it or otherwise pulling rank on it."

 

I'll have to try that if ever approached by a fire. I certainly hope I never am. I just can't think of anything much worse than FIRE. I was very fortunate to not have any uncontrolled fire aboard any flights I was on. Lots of controlled fires though. That's what kept us in the air! Except for flying over the Pacific, I always had in mind a place I would head for in the event of one on board. And, even though you're surrounded by water on a boat, I don't want one there either.

 

 

As to:

"(Actually, I'll bet you were a great guy to fly with.)"

 

Well, I'd like to think so, but I'm sure there were those who had other "favorites" to fly with. I had enough years as Co-Pilot to think I knew what I would NOT do as Captain. Doesn't always work out that way. I think most guys (and a few gals I might add) enjoyed flying with me knowing they were never gonna run out of fuel! That's right, I had three female Co-Pilots while flying Captain on the 737. Sharp gals; good pilots.

 

 

As to your Green Mountain fire:

 

Glad to hear everything turned out alright. Frightening having it that close. One of our kids lives in California. I think that whole State has been on fire at one time or another it seems! Anyway, they've been pretty much surrounded by fire so much so that all the bears have been driven from the woods into town. Now they've got bears to worry about rather than forest fires. So far I guess they're all getting along.

 

 

As to:

"You get much fiercer fires there in Florida, skylab..."

 

We don't live there any more. Moved back to Michigan a few years ago. Too hot and too many hurricanes. Had all three hurricanes go right over the house that one year. No damage, but not much fun. But we did have a wild fire right near us when we lived there. Almost died from the smoke! Don't miss it at all!

 

 

I guess we covered a few bases here, A?

Link to comment

skylab,

 

Yes, we've covered a lot of ground in this thread. That's what I like about blogging. Within reasonable limits we can talk about whatever we want.

 

I hope you didn't know her but as I recall, the first female FO ever flew for United and made Captain but was flying as FO on UAL 535 that crashed in the Springs because of the rudder hardover problem.

 

Fires in California ... New thread time. :)

Link to comment

Folks,

 

The contents of this thread have been posted to the PC Software Tech forum. Please view the thread here in my blog as closed, and make any further posts over in the forum.

Link to comment
×
×
  • Create New...