Recent Posts5151
Living Room / Re: Secure deletion: a single overwrite will do it
« Last post by IainB on June 07, 2012, 03:23 PM »I have been doing some research on security, and just thought I'd revive this thread with what I unearthed, because it seemed relevant to what seems could be a still-current myth. I inadvertently stumbled upon the link 40Hz gives in the opening post before actually seeing the post.
What I unearthed was some further, corroborating material to the OP.
In the OP made by 40Hz it says:
For posterity, the full post, dated 2009, from Heise Online (now The H Security) is in the spoiler below, minus any non-explicit embedded links:
The material I unearthed is from a 2003 post that has been revised at intervals and was last corrected in 2011. It seems quite a thorough coverage.
Caveat: I say "seems", because it is from the US NBER (National Bureau of Economic Research), which is an independent non-profit that generally researches whatever it is funded to work on. A lot of this funding could/would presumably come from the State, so I am unsure of the motivation/funding for this particular article, nor for it being kept so assiduously up-to-date.
Skepticism may be advisable.
The article is in the spoiler below, minus any non-explicit embedded links:
Some other relevant references, in the DC Forum:
What I unearthed was some further, corroborating material to the OP.
In the OP made by 40Hz it says:
Interesting article over at Heise Online
Link: http://www.heise-onl...-will-do-it--/112432Secure deletion: a single overwrite will do it
The myth that to delete data really securely from a hard disk you have to overwrite it many times, using different patterns, has persisted for decades, despite the fact that even firms specialising in data recovery, openly admit that if a hard disk is overwritten with zeros just once, all of its data is irretrievably lost.
Craig Wright, a forensics expert, claims to have put this legend finally to rest.-40hz (January 18, 2009, 01:47 PM)
For posterity, the full post, dated 2009, from Heise Online (now The H Security) is in the spoiler below, minus any non-explicit embedded links:
Spoiler
Source - http://www.h-online....ll-do-it-739699.html
17 January 2009, 11:29
Secure deletion: a single overwrite will do it
The myth that to delete data really securely from a hard disk you have to overwrite it many times, using different patterns, has persisted for decades, despite the fact that even firms specialising in data recovery, openly admit that if a hard disk is overwritten with zeros just once, all of its data is irretrievably lost.
Craig Wright, a forensics expert, claims to have put this legend finally to rest. He and his colleagues ran a scientific study to take a close look at hard disks of various makes and different ages, overwriting their data under controlled conditions and then examining the magnetic surfaces with a magnetic-force microscope. They presented their paper at ICISS 2008 and it has been published by Springer AG in its Lecture Notes in Computer Science series (Craig Wright, Dave Kleiman, Shyaam Sundhar R. S.: Overwriting Hard Drive Data: The Great Wiping Controversy).
They concluded that, after a single overwrite of the data on a drive, whether it be an old 1-gigabyte disk or a current model (at the time of the study), the likelihood of still being able to reconstruct anything is practically zero. Well, OK, not quite: a single bit whose precise location is known can in fact be correctly reconstructed with 56 per cent probability (in one of the quoted examples). To recover a byte, however, correct head positioning would have to be precisely repeated eight times, and the probability of that is only 0.97 per cent. Recovering anything beyond a single byte is even less likely.
Nevertheless, that doesn't stop the vendors of data-wiping programs offering software that overwrites data up to 35 times, based on decades-old security standards that were developed for diskettes. Although this may give a data wiper the psychological satisfaction of having done a thorough job, it's a pure waste of time.
Something much more important, from a security point of view, is actually to overwrite all copies of the data that are to be deleted. If a sensitive document has been edited on a PC, overwriting the file is far from sufficient because, during editing, the data have been saved countless times to temporary files, back-ups, shadow copies, swap files ... and who knows where else? Really, to ensure that nothing more can be recovered from a hard disk, it has to be overwritten completely, sector by sector. Although this takes time, it costs nothing: the dd command in any Linux distribution will do the job perfectly.
The material I unearthed is from a 2003 post that has been revised at intervals and was last corrected in 2011. It seems quite a thorough coverage.
Caveat: I say "seems", because it is from the US NBER (National Bureau of Economic Research), which is an independent non-profit that generally researches whatever it is funded to work on. A lot of this funding could/would presumably come from the State, so I am unsure of the motivation/funding for this particular article, nor for it being kept so assiduously up-to-date.
Skepticism may be advisable.
The article is in the spoiler below, minus any non-explicit embedded links:
Spoiler
Source: http://www.nber.org/...en-data-guttman.html
Can Intelligence Agencies Read Overwritten Data?
Claims that intelligence agencies can read overwritten data on disk drives have been commonplace for many years now. The most commonly cited source of evidence for this supposed fact is a paper (Secure Deletion of Data from Magnetic and Solid-State Memory) by Peter Gutmann presented at a 1996 Usenix conference. I found this an extraordinary claim, and therefore deserving of extraordinary proof. Thanks to an afternoon at the Harvard School of Applied Science library I have had a chance to examine the paper ( http://www.usenix.or...s/gutmann/index.html ) and many of the references contained therein.
Of course, modern operating systems can leave copies of " deleted" files scattered in unallocated sectors, temporary directories, swap files,remapped bad blocks, etc, but Gutmann believes that an overwritten sector can be recovered under examination by a sophisticated microscope and this claim has been accepted uncritically by numerous observers. I don't think these observers have followed up on the references in Gutmann's paper, however.
Gutmann explains that when a 1 bit is written over a zero bit, the "actual effect is closer to obtaining a .95 when a zero is overwritten with a one, and a 1.05 when a one is overwritten with a one". Given that, and a read head 20 times as sensitive as the one in a production disk drive, and also given the pattern of overwrite bits, one could recover the under-data.
The references Gutmann provides suggest that his piece is much overwrought. None of the references lead to examples of sensitive information being disclosed. Rather, they refer to experiments where STM microscopy was used to examine individual bits, and some evidence of previously written bits was found.
There is a large literature on the use of Magnetic Force Scanning Tunneling Microscopy (MFM or STM) to image bits recorded on magnetic media. The apparent point of this literature is not to retrieve overwritten data, but to test and improve the design of drive read/write heads. Two of the references (Rugar et al, Gomez et al) had pictures of overwritten bits, showing parts of the original data clearly visible in the micro-photograph. These were considered by the authors as examples of sub-optimal head design. The total number of bits seen was 6 in one photo and 8 in the other. Neither photo-micrograph was a total success, because in one case only transitions from one to zero were visible, and in the other case one of the transitions was ambiguous. Nevertheless, I accept that overwritten bits might be observable under certain circumstances.
So I can say that Gutmann doesn't cite anyone who claims to be reading the under-data in overwritten sectors, nor does he cite any articles suggesting that ordinary wipe-disk programs wouldn't be completely effective.
I should qualify that last paragraph a "bit". I was unable to locate a copy of the masters thesis with the tantalizing title "Detection of Digital Information from Erased Magnetic Disks" by Venugopal Veeravalli. However a brief visit to his web page shows that this was never published, he has never published on this or a related topic (his field is security of mobile communications) and his other work does not suggest familiarity with STM microscopes. So I am fairly sure he didn't design a machine to read under-data with an "unwrite" system call. In an email message to me Dr. Veeravalli said that his work was theoretical, and studied the possibility of using DC erase heads. [Since writing this paragraph the paper has been posted. It is indeed theoretical but has quantitative predictions about the possibility of recovering data with varying degrees of erasure. There isn't any suggestion that ordinary erase procedures would be inadequate].
Gutmann claims that "Intelligence organisations have a lot of expertise in recovering these palimpsestuous images." but there is no reference for that statement. There are 18 references in the paper, but none of the ones I was able to locate even referred to that possibility. Subsequent articles by diverse authors do make that claim, but only cite Gutmann, so they do not constitute additional evidence for his claim.
Gutmann mentions that after a simple setup of the MFM device, that bits start flowing within minutes. This may be true, but the bits he refers to are not from from disk files, but pixels in the pictures of the disk surface. Charles Sobey has posted an informative paper "Recovering Unrecoverable Data" with some quantitative information on this point. He suggests that it would take more than a year to scan a single platter with recent MFM technology, and tens of terabytes of image data would have to be processed.
In one section of the paper Gutmann suggests overwriting with 4 passes of random data. That is apparently because he anticipates using pseudo-random data that would be known to the investigator. A single write is sufficient if the overwrite is truly random, even given an STM microscope with far greater powers than those in the references. In fact, data written to the disk prior to the data whose recovery is sought will interfere with recovery just as must as data written after - the STM microscope can't tell the order in which which magnetic moments are created. It isn't like ink, where later applications are physically on top of earlier markings.
After posting this information to a local mailing list, I received a reply suggesting that the recovery of overwritten data was an industry, and that a search on Google for "recover overwritten data" would turn up a number of firms offering this service commercially. Indeed it does turn up many firms, but all but one are quite explicit that they can recover "overwritten files", which is quite a different matter. An overwritten file is one whose name has been overwritten, not its sectors. Likewise, partitioning, formatting, and "Ghosting" typically affect only a small portion of the physical disk, leaving plenty of potential for sector reads to reveal otherwise hidden data. There is no implication in the marketing material that these firms can read physically overwritten sectors. The one exception I found (Dataclinic in the UK) did not respond to an email enquiry, and they do not mention any STM facility on their web site.
A letter from an Australian homicide investigator confirms my view that even police agencies have no access to the technology Gutmann describes.
Of course it has been several years since Gutmann published. Perhaps microscopes have gotten better? Yes, but data densities have gotten higher too. A hour on the web this month looking at STM sites failed to come up with a single laboratory claiming it had an ability to read overwritten data.
Recently I was sent a fascinating piece by Wright, Kleiman and Sundhar (2008) who show actual data on the accuracy of recovered image data. While the images include some information about underlying bits, the error rate is so high that it is difficult to imagine any use for the result. While the occasional word might be recovered out of thousands, the vast majority of apparently recovered words would be spurious.
Another fact to ponder is the failure of anyone to read the "18 minute gap" Rosemary Woods created on the tape of Nixon discussing the Watergate breakin. In spite of the fact that the data density on an analog recorder of in the 1960s was approximately one million times less than current drive technology, and that audio recovery would not require a high degree of accuracy, not one phoneme has been recovered.
The requirements of military forces and intelligence agencies that disk drives with confidential information be destroyed rather than erased is sometimes offered as evidence that these agencies can read overwritten data. I expect the real explanation is far more prosaic. The technician tasked with discarding a hard drive may or may not have enough computer knowledge to know if running the command "urandom >/dev/sda2c1" has covered an entire disk with random data, or only one partition, nor is it easy to confirm that it was done. How would you confirm that the overwrite was not pseudo-random? Smashing the drive with a sledgehammer is easy to do, easy to confirm, and very hard to get wrong. The GPL'ed package DBAN is an apparent attempt to address this uncertainty without destroying hardware. Hardware appliances with similar aims include the Drive Erazer" and the Digital Shredder.
Surveying all the references, I conclude that Gutmann's claim belongs in the category of urban legend.
Or it may be in the category of marketing hype. I note that it is being used to sell a software package called "The Annililator".
Since writing the above, I have noticed a comment attributed to Gutmann conceeding that overwritten sectors on "modern" (post 2003?) drives can not be read by the techniques outlined in the 1996 paper, but he does not withdraw the overwrought claims of the paper with respect to older drives.
An updated copy of this memo will be kept at http://www.nber.org/...en-data-gutmann.html. Additional information may be sent to feenberg at nber dot org.
Daniel Feenberg
National Bureau of Economic Research
Cambridge MA
USA
21 July 2003
24 March 2004 (revised)
22 April 2004 (revised)
14 May 2004 (revised)
1 Oct 2011 (correction)
"Magnetic force microscopy: General principles and application to longitudinal recording media", D.Rugar, H.Mamin, P.Guenther, S.Lambert, J.Stern, I.McFadyen, and T.Yogi, Journal of Applied Physics, Vol.68, No.3 (August 1990), p.1169.
"Magnetic Force Scanning Tunnelling Microscope Imaging of Overwritten Data", Romel Gomez, Amr Adly, Isaak Mayergoyz, Edward Burke, IEEE Trans.on Magnetics, Vol.28, No.5 (September 1992), p.3141.
Wright, C.; Kleiman, D, & Sundhar S. R. S.: (2008) "Overwriting Hard Drive Data: The Great Wiping Controversy". ICISS 2008: 243-257 http://portal.acm.or...ation.cfm?id=1496285 . See also a summary at http://sansforensics...ing-hard-drive-data/
Some other relevant references, in the DC Forum:
