Main Area and Open Discussion > Living Room

Peer Review and the Scientific Process

<< < (13/47) > >>

barney:
there are many abuses in practice
-xtabber (October 22, 2013, 12:58 PM)
--- End quote ---

And that was my attempted point.  While I'm aware of the [stricter] meaning of the term, I seldom [when actually published] see it in action.  The petty jealousies of academia far surpass anything to be found in the commercial [real?] world, and ofttimes colour committee judgements.  At the end of the day, that formal process devolved to the personal judgements of the adjudicators, not all of whom are impartial and non-judgemental.  Same thing applies to the corporate world, although the motivations there [usually] tend to be a bit more clear.  Been, unfortunately, involved in both worlds, but when I discovered that my compatriots did not try as hard [as me] to clear their thoughts of possible prejudices and biases, and since I had neither the resource(s) nor the power to enforce such, I relegated the process to my mental scrapheap, where it has remained for the last decade or so.  Still read 'em, when available, but no longer place any credence in 'em.

Renegade:
Most scientific research is bunk? Apparently so...

http://reason.com/blog/2013/10/28/are-most-scientific-results-bunk

http://www.latimes.com/business/la-fi-hiltzik-20131027,0,1228881.column

http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0020124

Ahem... People were extolling the virtues of "peer review"? Hahahahaaha~! ;D 8)

IainB:
Interesting perspective of peer-reviewed literature from a science journalist - John Horgan - who kicks himself for a lack of investigation when writing an article in 1983 about Jerrold S. Petrofsky, a biomedical engineer at Wright State University who had been trying to help paralyzed patients walk by electrically stimulating their muscles with a computer-controlled device.
In 1985 he finally completed some investigation and wrote a report which corrected the matter, but nearly got him into some trouble - until his report was subsequently vindicated.
(see the link to the PLOS Medicine paper.)
(Otherwise copied below sans embedded hyperlinks/images.)
Cross-Check: Critical views of science in the news
By John Horgan | November 2, 2013

I’m moving soon, and so I’m riffling through the files I’ve accumulated in my decades as a science writer and chucking those I’ll never (I hope) need. Carrying out this archaeological dig into the strata of my career, I’m struck once again by all the “breakthroughs” and “revolutions” that have failed to live up to their hype: string theory and other supposed “theories of everything,” self-organized criticality and other theories of complexity, anti-angiogenesis drugs and other potential “cures” for cancer, drugs that can make depressed patients “better than well,” “genes for” alcoholism, homosexuality, high IQ and schizophrenia.

Nanette Davis stands at the podium during her graduation from Wright State University in 1983, an event that raised hopes that electrical stimulation of muscles would soon help paralyzed people regain control of their limbs. Davis was helped to the podium by engineer Jerrold Petrofsky (right), the designer of her muscle-stimulation system.

I graduated from journalism school in 1983 hoping to celebrate scientific advances, but from the start reality thwarted my intentions. I got a job as a staff writer for the Institute of Electrical and Electronics Engineers, a trade association. One of my first assignments was profiling Jerrold S. Petrofsky, a biomedical engineer at Wright State University trying to help paralyzed patients walk by electrically stimulating their muscles with a computer-controlled device.

Petrofsky was a lavishly honored star of the IEEE, whose research had reportedly enabled Nanette Davis, a paralyzed student at Wright State, to walk on stage and receive her diploma during her June, 1983, graduation ceremony. His work was lauded by major media, including the BBC, TIME, Newsweek, Nova and 60 Minutes. In 1985 CBS produced a television movie, First Steps, starring Judd Hirsch as Petrofsky.

I wrote a puff piece about Petrofsky–based primarily on interviews with him and materials supplied by him and Wright State–published in the November 1983 issue of The Institute, the monthly newspaper of the IEEE. It never occurred to me to question Petrosky’s claims. Who was I, a mere rookie, to second-guess him, Wright State and media like 60 Minutes?

Then other biomedical engineers wrote letters to me complaining that coverage of Petrofsky’s work was raising false hopes among paralyzed patients. At first, I thought these critics were just envious of Petrofsky’s fame, but when I investigated their complaints, they seemed to have substance.

I ended up writing an article, published in The Institute in May 1985, presenting evidence that Petrofsky’s methods for helping paralyzed subjects were less effective than he claimed. My original November 1983 article, which Petrosfsky had approved before publication, stated that Davis, while accompanied by Petrofsky during her graduation ceremony, controlled the stimulation of her own muscles and did not need his assistance.

Actually, Petrofsky held the device that stimulated Davis’s muscles, and he and another professor had to prop Davis up during the ceremony because the device malfunctioned. Davis also told me that before she met Petrofsky, she had trained herself to stand in leg braces for hours. In other words, her graduation feat was less impressive than it appeared. The muscle-stimulation method was also not risk free; Davis broke an ankle during a training session in 1984.

In my 1985 article, I argued that Petrofsky’s work raised questions that went beyond his case: “Has Petrofsky gone too far in seeking publicity for his work, as some of his peers suggest? Or should he be praised for being an effective communicator? In addressing these questions—which are echoed in other fields of research as well—perhaps some answers may be provided to a broader and more important question: What can engineers and scientists do to inform the public about their work, while ensuring that it is not misrepresented?”

This episode also taught me some lessons about science journalism that my subsequent experiences reinforced. First, researchers, when accused of hype, love to blame it on the media. But media hype can usually be traced back to the researchers themselves.

I also learned that critical journalism is much harder, more time-consuming and riskier than celebratory journalism. My 1985 investigation of Petrofsky, which I toiled over for months, made my editor so nervous that he wanted to bury it in the back pages of The Institute; I had to go over his head to persuade the publisher that my article deserved front-page treatment. After the article came out, the IEEE formed a panel to investigate not Petrofsky but me. The panel confirmed the accuracy of my reporting.

Since then, I keep struggling to find the right balance between celebrating and challenging alleged advances in science. After all, I became a science writer because I love science, and so I have tried not to become too cynical and suspicious of researchers. I worry sometimes that I’m becoming a knee-jerk critic. But the lesson I keep learning over and over again is that I am, if anything, not critical enough.

Arguably the biggest meta-story in science over the last few years—and one that caught me by surprise–is that much of the peer-reviewed scientific literature is rotten. A pioneer in exposing this vast problem is the Stanford statistician John Ioannidis, whose blockbuster 2005 paper in PLOS Medicine presented evidence that “most current published research findings are false.”

Discussing his findings in Scientific American two years ago, Ioannidis writes: “False positives and exaggerated results in peer-reviewed scientific studies have reached epidemic proportions in recent years. The problem is rampant in economics, the social sciences and even the natural sciences, but it is particularly egregious in biomedicine.”

In his recent defense of scientism (which I criticized on this blog), Steven Pinker lauds science’s capacity for overcoming bias and other human failings and correcting mistakes. But the work of Ioannidis and others shows that this capacity is greatly overrated.

“Academic scientists readily acknowledge that they often get things wrong,” The Economist states in its recent cover story “How Science Goes Wrong.” “But they also hold fast to the idea that these errors get corrected over time as other scientists try to take the work further. Evidence that many more dodgy results are published than are subsequently corrected or withdrawn calls that much-vaunted capacity for self-correction into question. There are errors in a lot more of the scientific papers being published, written about and acted on than anyone would normally suppose, or like to think.”

So whatever happened to Petrofsky? He reportedly left Wright State in 1987 and ended up at Loma Linda University in California. The only article I could find online that mentions criticism of his work at Wright State is a 1985 New York Times report on the angry reaction of biomedical researchers to the film “First Steps.” As for Nanette Davis, after her famous 1983 graduation “walk” she “returned to her wheelchair,” according to a 2010 report in the Dayton Daily News. She is now a mother and teacher.

Photo credit: National Center for Rehabilitation Engineering, Wright State University, http://www.wright.edu/~aja.ash/publicity.html.

--- End quote ---

IainB:
Most scientific research is bunk? Apparently so...

* http://reason.com/blog/2013/10/28/are-most-scientific-results-bunk
* http://www.latimes.com/business/la-fi-hiltzik-20131027,0,1228881.column
* http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0020124Ahem... People were extolling the virtues of "peer review"? Hahahahaaha~! ;D 8)
______________________
-Renegade (October 29, 2013, 04:49 AM)
--- End quote ---
Was just reviewing the posts in this thread, and re-read the links you gave there. They are not opinion pieces, seem to be quite factual, and one is an abstract from a research paper.

Some useful conclusions:

* 1. Reason: Are Most Scientific Results Bunk?
The conclusion:
Yes (in summary).
As a remedy, it is worth considering Nosek's Open Science Framework project as suggested by the quoted Economist article.
_______________________


* 2. LA Times: Science has lost its way, at a big cost to humanity
The conclusion:
...NIH and the rest of the scientific community are just now waking up to the realization that science has lost its way, and it may take years to get back on the right path. ...
_______________________

--- End quote ---


* 3. PLOS Medicine: Why Most Published Research Findings Are False
The conclusion:
Most Research Findings Are False for Most Research Designs and for Most Fields
_______________________

--- End quote ---

IainB:
Interesting post in ArsTechnica - doesn't seem to carry their usual editorial bias from John Timmer, so could be worth a read in the context of "peer review":
(Copied below sans embedded hyperlinks/images.)
Anti-GMO crop paper to be forcibly retracted
Journal editor recognizes extensive flaws, says the paper shouldn't have run.
by John Timmer - Dec 1, 2013 6:00 pm UTC

Last year, a French researcher made waves by announcing a study that suggested genetically modified corn could lead to an increased incidence of tumors in lab animals. But the way the finding was announced seemed designed to generate publicity while avoiding any scientific evaluation of the results. Since then, the scientific criticisms have rolled in, and they have been scathing. Now, the editor of the journal that published it has decided to pull the paper despite the objections of its primary author.

The initial publication focused on corn that had been genetically engineered to carry a gene that allowed it to break down a herbicide. French researchers, led by Gilles-Eric Séralini, fed the corn, with and without herbicide, to rats. Control populations were given the herbicide alone or unmodified corn. The authors concluded that the genetically modified corn led to an elevated incidence of tumors and early death.

But even a cursory glance at the results suggested there were some severe problems with this conclusion. To begin with, there were similar effects caused by both the genetically engineered crop and by the herbicide it was designed to degrade. None of the treatments showed a dose effect; in some cases, the lowest doses had the most dramatic effect. And, if the treatment populations were combined, in some cases they were healthier than the controls. Tests of whether the results were statistically significant were completely lacking.

And, since then, the scientific response has been withering. The German and EU food regulators looked the results over, but found them inadequate. The paper itself has accumulated a host of Letters to the Editor attached to it. And a different journal published an entire paper devoted to outlining its deficiencies.

All of these criticisms of the study could have been incorporated into the original press coverage, except for the fact that the people behind the study manipulated journalists to ensure that they were unable to obtain any outside evaluations of the paper. Nevertheless, even as the criticisms rolled in, the researchers remained defiant, and anti-GMO activists continued to promote the paper as evidence of the dangers posed by genetically modified crops.

Now, the editor of Food and Chemical Toxicology, the journal in which this study was published, has decided its flaws are so severe that including multiple Letters to the Editor with the study just won't cut it. In response to the initial complaints, he had set up a panel that looked over the paper and the additional data provided by Séralini. According to one letter from the editor, obtained by an anti-GMO activist group, "The panel had many concerns about the quality of the data and ultimately recommended that the article should be withdrawn." The editor has agreed with this recommendation and has already written a statement that will replace it.

Séralini has been asked to get in touch to discuss the details of the paper's withdrawal, but he has announced that he stands by his conclusions. This will ultimately force the editor to withdraw it over Séralini's objections.

This sort of retraction is a bit unusual. In one heavily publicized past example, a research group described bacteria that could supposedly replace phosphate with arsenate. Despite a large number of problems with that paper (including a failure to reproduce the original results), Science still hasn't pulled it. In contrast, a paper linking Chronic Fatigue Syndrome to a virus has been pulled, perhaps because there were more serious questions about the scientific procedures used to generate its results.

For the GMO paper, the situation is complex. According to the publisher, Elsevier, there is no evidence of any fraud or data manipulation. However, the number of animals used was insufficient to support any conclusions, and the paper certainly drew some. This goes against the journal's guidelines, and thus they seem to be admitting the paper should never have been published in the first place. That would seem to be a failure of editorial process and peer review, yet Elsevier states, "The peer review process is not perfect, but it does work."

(An additional problem that could justify retraction, noted by one of the papers linked above, is that animal welfare rules call for animals that develop tumors to be euthanized, while Séralini let the tumors grow to horrific sizes.)

If the precise grounds for retraction aren't entirely clear, the response of the groups campaigning against genetically modified foods is. A statement released by the group GMWatch basically says that the paper was fine, the editor is being unethical, and Monsanto might be behind it all. So, it seems that Séralini's paper will continue to be brought up long after it's removed from the formal scientific literature.

--- End quote ---

Navigation

[0] Message Index

[#] Next page

[*] Previous page