Thought for the Day on the IRS
If you were getting investigated by the IRS, and you gave the IRS the answers that they have been giving the public over Lois Lerner's and others' lost emails, do you think that the IRS would accept your answer?
By the way, a system crash that makes a hard drive totally unreadable is just vanishingly rare nowadays. It is possible to corrupt certain system files in the root that will make it impossible to log on to the computer or access the hard drive files normally, but they are still there. Something with the hard drive's motor or read heads could fail, but the data is still there. Even if you highlighted every file in your hard drive and hit the delete key, they are still there. When you hit delete they are taken out of the file directory and may get overwritten if you add new data to the computer, but without special software, it is actually hard to totally delete files (this is why you have to be careful when you donate or dispose of computers). It actually can take the better part of an hour to really remove all files from a hard drive so that they are unrecoverable.
Given all this, I think the odds are that 6 or 7 computers of a group of senior leaders in the same office all crashed at roughly the same time in a way that wiped out all the data from their hard drives such that all data would be unrecoverable is simply beyond credulity.
Mole1:
It isn't inconceivable that, at the time, they just decided it wasn't worth the effort so they reformatted the drives, reinstalled the operating system and software, recovered what they could from backup, and put them back to use. So, by now, there would be enough use that the data desired is actually gone.
June 23, 2014, 2:05 pmArrian:
It's also not inconceivable that a tech, on seeing a machine that crashed due to a corrupted file system, pulled the drive and replaced it with a new one: If there's something physically wrong with the drive it's more likely than not that it will keep corrupting files and just be an ongoing hassle. Also, it's not inconceivable that a batch of drives went bad at roughly the same time, some models were notorious for this back when I was a PC tech. And any drive pulled really should be destroyed in a way that completely destroys all data on it. That's cheaper and easier to do than backing everything up for years, after all.
Not inconceivable, but I don't know how likely it actually is and without an idea how common it is in the organization, it's still hella fishy.
June 23, 2014, 2:45 pmLt.Col. Petch:
That sound you hear is Richard Nixon laughing.
June 23, 2014, 3:39 pmmesaeconoguy:
What this event (coverup) is leading to is a direct “good-for-thee-but-not-for-me” moment for arguably the most powerful government institution in the country.
The IRS represents our now imperial government in microcosm – it is now operating independently of any oversight, and far, far outside the will
June 23, 2014, 3:47 pmof the people, which makes it illegitimate.
marque2:
Well yes that works at my company where all valuable data is kept on central servers - but if the important data is local they need to attempt to recover it.
June 23, 2014, 3:56 pmrandian:
Credulity was never the point. Demonstrating contempt for Congress and the American people was the point.
June 23, 2014, 4:03 pmS:
Speaking as someone who works as a government contractor, the lost emails are incredibly plausible. I've seen a number of laptops die and take all of the person's email / current work with them; for email we're limited to 200 MB on the server, and it's automatically deleted after 60 days on the server. If your laptop hard drive goes, all the email you've archived there is toast. Any critical work is supposed to be on one of the central file servers, but there are constant emails asking everyone to clear space out on those (so no storing 5+ GB of email there).
One thing which I've heard is that the hard drive encryption software we use makes it much harder to recover a broken drive; it's good if someone loses / steals / improperly disposes of a laptop, but makes it much uglier to recover data when part of a drive goes bad.
Please note that I'm not defending this as a smart business practice; at least once a month I'm digging up 1-6 year old emails which save us hours of work or contain important information. The most important one I can recall was our customer signing off on an updated piece of software once we made two minor changes; I was the only person left with our group who had worked on the project, and the customer was trying to claim we hadn't met their requirements and needed to rework the whole thing for $300K+
June 23, 2014, 4:48 pmBrad Warbiany:
As someone who works for a HDD company, I can say that you're right. The chance that the data on the drive was actually corrupted to the point of being unreadable to someone who is truly interested in data recovery is unlikely. The data is most likely still there. The difficulty in actually deleting data is one of the reasons that many datacenters will crush HDDs rather than ever let them out of the building. If the IRS was properly disposing of the dead drive, they probably crushed or shredded it. (Not that I expect proper disposal to be something our government actually cares about -- after all it's only OUR private information they're holding.)
That said, if it was a head failure or a dead spindle, the process of performing data recovery is VERY difficult. It requires disassembly of the drive, and either replacing the head stack assembly (difficult) or completely removing the platters and putting them into another drive (more difficult especially as these things are tuned with weights to ensure that they don't vibrate and flutter when spinning). These operations can be done, but they cost a HELL of a lot of money.
In the case of a simple electrical failure, it's usually easier. Replace the PCB with a known good one, and you can read the data.
However, BOTH cases assume a lack of encryption. While I doubt the drives in question were encrypted, it is a possibility.
Either way, the cost and time sink of data recovery is such that almost nobody ever does it. In fact, if our government was routinely spending money to do this I'd generally consider it a waste. While I think this indicates poor backup procedures, I would not expect any entity to go through the effort of data recovery unless they had reason to believe that there was data of absolute critical importance on any given drive.
June 23, 2014, 5:02 pmjoshv:
Oh come on, the signal was given, and in the dark of night somebody came in, removed the hard drive, and subject it to massive magnetic fields. It was inserted back into the computer, and the next day Lois Lerner hit this power button and then called IT telling them that her computer had had crashed. The local IT idiots could of course recover nothing, and if it was done well, I doubt anyone can recover anything (if the drive hasn't already been destroyed).
BTW, if there is nothing to hide here, the White House should be able to provide copies of all correspondence between Lerner and White House staff. Why haven't they offered to do so?
June 23, 2014, 6:19 pmMatthew Slyfield:
You just gave me a great idea for a new source for a vast amount of carbon free electricity. Exhume the graves of several of this country's founders, transfer their remains to new high strength steel coffins and connect the coffins to the axle of an electric turbine. Free electricity for everyone! :-)
June 23, 2014, 7:10 pmmlhouse:
In a court system or even with the IRS, such a failure to provide this type of evidence would give the decision maker (a jury, judge, auditor, or administrator) the right to come to the conclusion that is least favorable to the party that was required to produce the records. In other words, if the IRS asked you for records regarding a tax return and you cannot produce them, they will make adverse decision against you. And there is nothing you can do about it.
June 23, 2014, 8:01 pmMax Lybbert:
The claim that a hard disk failure meant that the emails are completely unrecoverable is clearly aimed at the general public. Unlike Congress, the general public may not know that due to the Freedom of Information Act and similar laws, federal agencies -- especially federal agencies that enforce the law -- set up their email servers to safeguard emails I the case of any likely disaster, such as a server crash, a building fire, or even a crashed laptop.
June 24, 2014, 4:42 ammarque2:
It is pretty fishy that all the important drives had this crash. Also IRS has data management requirements. This should have all been backed up centrally and if not a procedure should be in place to attempt to recover data from crashed drives of important people. The IT tech should know to send the drive to the "data recovery department"
There are very few, if any, hard drives that fail under normal circumstances that can't have the data recovered. The hard drive wasn't dropped in the Mariana trench after all.
It seems more like, at best, they purposely have poor procedures to cover up things like this, at worst the techs were told to replace the drives because they "failed" by some higher up - trying to cover up.
June 24, 2014, 7:51 amirandom419:
Hard drives are intelligent nowadays with the Smart technology and will replace bad sectors from a pool of spares.
http://en.wikipedia.org/wiki/S.M.A.R.T.
http://www.passmark.com/products/diskcheckup.htm
June 24, 2014, 10:59 amAir Force Bureaucrat:
I work for the government and have often wondered if the stupid (did I mention it is exceedingly stupid!) limitation on our e-mail box to 90 MBytes was due to exactly this kind of situation. If somebody comes in asking for data, they can say that all e-mails more than a few weeks old just can't be recovered. There is no technical reason they can't dramatically increase our e-mail box space (a 1 TByte drive, for a whopping $100, would more than double everyone's space in the whole organization), so there has to be some bureaucratic reasoning, and this is the only one I can think of.
June 24, 2014, 1:40 pmBrad Warbiany:
There are *many* drive corruption and/or failure modes that could cause lost data. As I stated below, very few of them make it *impossible* to get the data off the platter, but depending on the cause of failure it could be very, very difficult.
While I agree that hard drives are very intelligent devices, and fully support relocation of sectors in the instance of physical defects on the media, there are still many things that could have happened to this data.
June 24, 2014, 4:50 pmBrad Warbiany:
No technical reason, yes. However, cost is not as trivial as you'd imagine. The drives used in storage servers are more than $100. Many of these servers will use RAID, sacrificing some capacity for redundancy. As a result you need a full physical server, backplane (possibly w/ active expander), RAID controller, motherboard with typically high-performance CPU and a bunch of RAM, redundant power, etc. The costs add up, and then even beyond the cost of the disks, you have the power bill to run it 24/7.
Spread out over a large organization, this is a major line item in any IT budget. One can make the claim that emails in the IRS are important enough that they *should* be kept longer and it's worth the cost, but one shouldn't minimize the cost associated with doing so.
I will state, though, that my experience with IT administrators is that if you're bumping up against the email inbox size, and you ask them to, they just increase your capacity. I suspect that the arbitrarily small limits are there to ensure they don't have to provision, say, 10GB per user up front when they know that many users will take a year to reach your 90MB initial allocation. Depending on how their storage infrastructure is allocated, doing it this way could minimize the amount of wasted and unused HDD space by not forcing them to allocate space that will never be filled.
June 24, 2014, 4:57 pmNehemiah:
The administration doesn’t care how incredulous it sounds, nor does the main stream media. So take
June 25, 2014, 8:37 amthat and lump it, what are you going to do about it? They've decided that taking the heat for this lie is better than exposing the truth in those "missing" emails.
DaveK:
The apparent "unrecoverability" of data from an individual's hard drive is entirely credible. This is especially true a laptop computer was involved and the drive in that laptop was volume-encrypted. If the recovery keys for the drive were sloppily managed or lost, even failure of a single sector of the drive could result in the loss of all data in the entire drive. That is a reason that volume-encryption is usually discouraged unless you are really, really concerned about losing control of information on the drive through something like theft of a laptop computer (and the IRS is one of those organizations that would probably require volume-encryption on laptops).
Having said that, the near-simultaneous loss of data from the personal computers of key figures in the investigation does stretch credibility to the breaking point. The additional failure to find e-mails in the servers, backups, and archives simply shatters credibility.
They're just brazenly lyin' to our faces and don't care if we know it. Holder will never appoint a special prosecutor, and Congress can't.
June 25, 2014, 9:19 amfraizer:
Richard Nixon is rolling in his grave saying "Oh what an amateur I was".
June 25, 2014, 10:12 amDaublin:
I would look more to the email software than to the hard drive issues. Email is usually stored on a server and only cached on a local machine, so the local machine crashing is not going to cause the email to go away.
June 25, 2014, 1:38 pmArrian:
A data retention policy of "Keep it on your computer" isn't a data retention policy. I've never heard of any organization that, by policy, had users store important documents on their desktops. Desktop PCs are not reliable enough to store important information on: They break and it's too inefficient to back them up when you could have the user store their important files on a network drive that backs up a thousand users in the same amount of time, space and resources desktop level backups would take for 1 user.
I can also see techs replacing a potentially bad hard drive and reimaging the PC with impunity, or even as policy: It doesn't take many people in the office losing years of work to a hard drive crash to motivate the entire office to store their important data on the server instead of the local machine. That's not even counting all the techs who look at BOFH or the guys from The IT Crowd as heroes and will reimage a PC for the sheer enjoyment of seeing the despair on a user's face.
I also wouldn't be surprised in the least if there were a policy from the email/document storage group that the user was responsible for saving anything not explicitly under a legal records hold themselves while the IT support organization's policy is that anything stored on the local machine is considered expendable. It's not uncommon at all for those two organizations to not coordinate or even be under completely separate management structures short of a level or two below the CIO/CFO.
Even in the IRS, it's not safe to assume malice where ineptitude can also explain things. And I can believe a situation like I laid out happening because I've seen it happen in the past. That doesn't mean their data retention policies aren't frighteningly hypocritical and bad, nor does it rule out malice. But if the Inspector General or a special prosecutor digs into the situation and finds that this is due to bad policy and ineptitude, I'll believe them.
June 25, 2014, 3:25 pmGattsuru:
Possible exception : SSD failures are a nightmare. Large data recovery firms can get unencrypted records off after firmware corruption, but that's well outside of the bounds of the end-user or small IT shops. Damage to the memory controller or individual memory chips can turn the data recovery process into the world's least pleasant puzzle, if there's even anything left.
That said and as you said, it's a backup problem.
June 26, 2014, 12:43 pmBrad Warbiany:
Agreed, and great point. (I actually spent 6 years of career in SSD prior to HDD). Much of the SSD failure modes are data corruption -- often induced by firmware bugs -- and if that data corruption occurs accidentally in the firmware or mapping table areas, you make it nearly impossible to reconstruct that data. Your analogy to the world's least pleasant puzzle is spot-on. Again, the data will still be on the chips, but it may be in pieces and completely out of order, and thus for all *practical* purposes is unrecoverable.
June 26, 2014, 2:39 pmCraig L:
Odds of 7 IRS hard drives crashing irretrievably: 1 in 4,018,240,425,000,000, says this woman. http://www.drakedirect.blogspot.com/2014/06/a-statisticians-view-on-irs_1432.html
June 28, 2014, 5:00 pmobloodyhell:
}}}} all data would be unrecoverable is simply beyond credulity.
18 minutes of silence, that's all it is...
But with a Democrat involved, hey, t'ain't no big thang...
Hey, Look! Chris Christie!! Bridgegate!!
July 5, 2014, 12:35 amobloodyhell:
This is partly due to Microsoft's incompetently devised, exceedingly brittle directory structure. Back in the real "old days", with Apple ][s, the structure of a file was such that you could generally recover most of the files on a floppy disk even if the directory itself was completely unreadable... because it didn't use Microsoft's beloved "big ball o'goo" data structure technique, and didn't store EVERYTHING in one central location. For a given file, the info on where the file was was kept with the file, not in the directory -- all the directory kept was a pointer to the first of these indexes.
These indexes had a usually fairly recognizable structure, so you could search for them on a disk even if you did not know where they were. You'd spot old ones, and even false ones, but it was pretty unusual to not be able to find one for any given file. And, finding them, it was then just a matter of figuring out which was which, and which ones were invalid due to being legitimately deleted files or "previous" versions of files.
M$ programmers have always sucked, which is why their shit is so random. They love big balls of goo (monolithic C drives that hold everything** -- data, OS, programs... or the "registry") and that, plus the poor design skills lead to very brittle system designs that are easy to break but not very easy to fix.
** At the VERY LEAST, you should always break up the hard drive into not less than two pieces, ideally, three or more. One for the OS, one for all programs installed, and one for all your data. Given the nature of Windows, it is hard to keep a true distinction between OS and programs, this splitting into defined roles at least allows you to replace/restore the OS to a base level and not have to wipe your data at the same time.
July 5, 2014, 1:02 amobloodyhell:
Naw. Unlike The One, he didn't have the Mainstream Media shilling for him.
The only unprofessionals involved are and have always been the media.
July 5, 2014, 1:03 amobloodyhell:
The reply: "But it could happen, right?"
July 5, 2014, 1:04 amobloodyhell:
}}}} say, 10GB per user up front when they know that many users will take a year to reach your 90MB initial allocation.
Because, you know, Google finds this expense to be so significant that they give you WAAAAY over 10gb FOR FREE ACCOUNTS.
Not that I'm overly disputing your analysis, the simple fact is that keeping every e-mail a person has ever received or sent is likely to be, in most cases, less than 10-20GB for an entire lifetime of activity. Most e-mails are text, and even if they contain reports, we're talking less than 1 or 2mb per report. Yes, there will be exceptions, people who routinely get 5mb reports will add up quickly, but they are few and far between.
The main point is, I concur that the real reason for this is so they can claim they don't have things backed up and available two years down the line when there's an investigation.
July 5, 2014, 1:10 amobloodyhell:
Indeed, I've made an adverse decision about them... and yes, there's nothing I can do about it... :-S
July 5, 2014, 1:12 amobloodyhell:
The part that strains credulity, Arrian, is the sheer *convenience* of it in obscuring what was done and by whom.
THAT is not so easily handwaved off. Or shouldn't be. When government officials lost important, potentially damning documents so "conveniently", the worst case should be the default presumption, and many of those associated should simply be told to get a Real Job.
The fact that this is the base standard of the IRS on pretty much any dealings with them only increases the irony involved.
July 5, 2014, 1:31 am