I was reading The Fading Memory of the state, and article that talks about the task of being the National Archive and Records Administration, and being faced with 347+ petabytes of data in the next 15 years or so...

The irony is that while reading it I checked the mail, well, during a break from reading the article, and there was a package from my dad with a letter and a DVD of my Grandmother's funeral. That happened in Aug 2003, and I received the DVD on 6/26/2005, and of course it is just as relevant today as it was then.

So how does one manage it all? The basic answer is that you can't really...The other day I ran across an amusing article on wired: Keep Your Privates private. It talks about a product called Couple Box which claims to be a secure application for your personal home made 'intimate' files/pictures/video.

But clearly, in the decades to come, archives such as NARA will need to be staffed by a new kind of professional, an expert with the historian's eye of an Allen Weinstein but a computer scientist's understanding of storage technologies and a librarian's fluency with metadata. "We will have to create a new profession of 'data curator'--a combination of scientist (or other data specialist), statistician, and information expert," says MacKenzie Smith of the MIT Libraries.

How do you deal with all this data? Some needs to be private, some just needs to be preserved. There is an awesome slashdot post on the subject.

The slashdot comment:

this is actually a BIG question (Score:5, Insightful) by Ralph Spoilsport (673134) Alter Relationship on Sunday June 26, @12:13AM (#12912210) (Last Journal: Monday July 12, @10:38PM) And one that I have railed about for many years.

I have been in the same position the Author discussed, and I have come to ONLY negative conclusions. In a few words, and I hate to say this, but buddy:

WE'RE FUCKED.

Digital is a loser's proposition. backing up to analogue or even digital data on analogic substrates (such as DV tape) fail. Simply nad purely.

The *only* thing that comes close is some kind of RAID, and those, even with the plummeting price of storage, are still too expensive given the needs.

Also, a RAID assumes a continuity of several things that are not likely to be continuous:

With Video:

Framerate, number of lines, colour depth, aspect ratio, file format, compression format, Operating system compatibility, etc etc etc. All of these things are variables.

With Audio:

sample rate, compression format, bit depth, file format, etc.

Basically all of it points to very bad places.

I am fairly well convinced that our age will simply disappear. They will find our garbage, the few books not pressed on acidic paper, our paintings (fat lot of good the abstract stuff will mean to them) and drawings, that's about it. the rest will just be shiny little bits of crap in the landfill.

Since we will have used up all the dense energy forms, they will be appalled at the energy requirements just to get the few remaining museum piece devices to work. Archiving the 21st century will be impossible. To the 25th century, the 21st century will be seen as a dark age - not only for the holocaust of the die caused by the failure of the petroleum based economy, but from the simple fact that very little of the information formats we are totally geared into will survive, including this note on /.

His problem of saving personal video is just the tip ofthe iceberg. His problem is the problem of our very civilisation, writ small.

That's why I am abandoning video, and going back to painting. In 500 years, my painting CAN survive. the video simply won't.

RS

--

Shoes for Industry. Shoes for the Dead.