Main Area and Open Discussion > Living Room
Peer Review and the Scientific Process
Renegade:
Tangentially related and funny:
IainB:
Well this was a surprise! (NOT)
(There are some quite amusing bits in here.)
Psychologists Call Out the Study That Called Out the Field of Psychology
By Rachel E. Gross
458637324-amy-cuddy-speaks-onstage-during-cosmopolitan-magazines
Independent researchers have had trouble replicating the famous findings of Harvard psychologist Amy Cuddy (pictured).
Craig Barritt/Getty Images for Cosmopolitan magazine and WME Live
Remember that study that found that most psychology studies were wrong? Yeah, that study was wrong. That’s the conclusion of four researchers who recently interrogated the methods of that study, which itself interrogated the methods of 100 psychology studies to find that very few could be replicated. (Whoa.) Their damning commentary will be published Friday in the journal Science. (The scientific body that publishes the journal sent Slate an early copy.)
Rachel E. Gross Rachel E. Gross
Rachel E. Gross is a Slate editorial assistant.
In case you missed the hullabaloo: A key feature of the scientific method is that scientific results should be reproducible—that is, if you run an experiment again, you should get the same results. If you don’t, you’ve got a problem. And a problem is exactly what 270 scientists found last August, when they decided to try to reproduce 100 peer-reviewed journal studies in the field of social psychology. Only around 39 percent of the reproduced studies, they found, came up with similar results to the originals.
That meta-analysis, published in Science by a group called the Open Science Collaboration, led to mass hand-wringing over the “replicability crisis” in psychology. (It wasn’t the first time that the field has faced such criticism, as Michelle N. Meyer and Christopher Chabris have reported in Slate, but this particular study was a doozy.)
Now this new commentary, from Harvard’s Gary King and Daniel Gilbert and the University of Virginia’s Timothy Wilson, finds that the OSC study was bogus—for a dazzling array of reasons. I know you’re busy, so let’s examine just two.
The first—which is what tipped researchers off to the study being not-quite-right in the first place—was statistical. The whole scandal, after all, was over the fact that such a low number of the original 100 studies turned out to be reproducible. But when King, a social scientist and statistician, saw the study, he didn’t think the number looked that low. Yeah, I know, 39 percent sounds really low—but it’s about what social scientists should expect, given the fact that errors could occur either in the original studies or the replicas, says King.
His colleagues agreed, telling him, according to King, “This study is completely unfair—and even irresponsible.”
Upon investigating the study further, the researchers identified a second and more crucial problem. Basically, the OSC researchers did a terrible job replicating those 100 studies in the first place. As King put it: “You’d think that a test about replications would actually reproduce the original studies.” But no! Some of the methods used for the reproduced studies were utterly confounding—for instance, OSC researchers tried to reproduce an American study that dealt with Stanford University students’ attitudes toward affirmative action policies by using Dutch students at the University of Amsterdam. Others simply didn’t use enough subjects to be reliable.
The new analysis “completely repudiates” the idea that the OSC study provides evidence for a crisis in psychology, says King. Of course, that doesn’t mean we shouldn’t be concerned with reproducibility in science. “We should be obsessed with these questions,” says King. “They are incredibly important. But it isn’t true that all social psychologists are making stuff up.”
After all, King points out, the OSC researchers used admirable, transparent methods to come to their own—ultimately wrong—conclusions. Specifically, those authors made all their data easily accessible and clearly explained their methods—making it all the easier for King and his co-authors to tear it apart. The OSC researchers also read early drafts of the new commentary, helpfully adding notes and clarifications where needed. “Without that, we wouldn’t have been able to write our article,” says King. Now that’s collaboration!
“We look forward to the next article that tries to conclude that we’re wrong,” he adds.
--- End quote ---
IainB:
@Renegade: By the way, I just watched Alternate Viewpoint-Cancelling Headphones - We the Internet Sketch 9
I've met quite a few people who might like to buy these headphones, so, thanks for the link. ;)
IainB:
I had always been interested in "shaken baby syndrome" since watching a UK TV documentary about it as a child. It was the first time that I learned that parents could snap and lose control and actually seriously harm their babies out of a sort of mental state of pent-up frustrated anger, without actually intending to harm them.
Whilst I had realised that diagnosis was based on a hypotheses rather than established facts, I had not realised that in the UK one is apparently not allowed to talk about it as being a hypothesis, but only as an established fact.
This shaken baby syndrome case is a dark day for science – and for justice
Clive Stafford Smith
A leading doctor faces being struck off for challenging the theory about the infant condition. It’s like Galileo all over again
Image: Father holds newborn baby on shoulder
‘Shaken baby syndrome is almost unique among medical diagnoses in that it is not focused on treating the child.’ Photograph: Moodboard/Alamy
Monday 14 March 2016 09.30 GMT
Last modified on Tuesday 15 March 2016 11.42 GMT
On Friday, I witnessed something akin to a reenactment of the trial of Galileo, precisely four centuries after the original. Dr Waney Squier faces being struck off by the General Medical Council (GMC) for having the temerity to challenge the mainstream theory on shaken baby syndrome (SBS).
For years, the medical profession has boldly asserted that a particular “triad” of neurological observations is essentially diagnostic of SBS. Since the Nuremberg Code properly prevents human experimentation, this is an unproved hypothesis, and there has been rising doubt as to its validity.
Doctor who doubted shaken baby syndrome misled courts, panel rules
I am convinced that Squier is correct, but one does not have to agree with me to see the ugly side to the GMC prosecution: the moment that we are denied the right to question a scientific theory that is held by the majority, we are not far away from Galileo’s predicament in 1615, as he appeared before the papal inquisition. He dared to suggest that the Bible was an authority on faith and morals, rather than on science, and that 1 Chronicles 16:30 – “the world is firmly established, it cannot be moved” – did not mean that the Earth was rigidly lodged at the epicentre of the universe. It was not until 1982 that Pope John Paul II issued a formal admission that the church had got it wrong.
Shaken baby syndrome is almost unique among medical diagnoses in that it is not focused on treating the child. If an infant has bleeding on the brain (a subdural hematoma), the doctor wants to relieve the pressure – it is of little relevance how the infant came about the injury. SBS is, then, a “diagnosis” of a crime rather than an illness, and when a brain surgeon comes into the courtroom and “diagnoses” guilt, the defendant, mostly a parent, is likely to go to prison – or worse.
I have defended a number of emotionally charged capital cases where doctors have opined that a child had to have been shaken by an angry parent because it was “impossible” for the triad of neurological sequelae to result from an accident – it “had” to be caused by shaking. Many American doctors adhere to a bizarre notion that an infant cannot suffer a fatal head injury from a fall of less than three storeys. While we cannot drop a series of infants on their heads to test this, it would appear to be plain folly. The velocity of a five-foot fall means a child’s head can hit the ground at roughly 15mph, which is faster than most people – short of Usain Bolt - can sprint. I invited a series of neurosurgeons to run headlong into a hardwood wall in one courtroom, so we could see what happened to them. They politely declined, and stuck to their silly theory.
What other doctor will be prepared to question the prosecution theory if it means the end of a career?
Squier has now been branded a “liar” by the panel, and found “guilty” of paying insufficient respect to her peers. Dr Michael Powers, perhaps the eminent QC in the area of medico-legal practice in the UK, believes that the GMC tribunal – made up of a retired wing commander, a retired policeman and a retired geriatric psychiatrist – was not qualified to understand the complex pathology of the developing brain. “It is therefore sad, but not surprising, that they have reached the wrong conclusion,” he said. “The proper forum for debating these issues is the international neuroscience community.”
Powers has a point: Michele Codd, the chair of the panel, was a general duties officer in the RAF for 32 years. One might doubt whether Stephen Marr, a retired Merseyside police officer, would hold up a constable’s hand to a prosecution theory that has sent so many people to prison.
Nisreen Booya was the sole person with any meaningful medical qualifications on the panel, but in a rather different area: she is a retired psychiatrist specialising in geriatric issues such as Alzheimer’s, an illness that, like infant head trauma, is “poorly understood”. She is quoted as saying that she “made a career of trying to provide innovative services” in her field – and yet she condemns Squier for thinking outside her own rigid box. All three are doubtless honourable people, but they are simply wrong to hold SBS up as the fifth gospel.
At the risk of being diagnosed with “I told you so” syndrome, I wrote an article 20 years ago questioning whether forensic hair analysis was really science. I was pleased therefore when, in 2015, the FBI admitted that they had got it wrong for decades – but this came after thousands of men, women and children had been convicted on the basis of latter-day snake oil, and scores had been sent to death row.
Those deemed to be blasphemers often suffer a gruesome fate. Although Squier may be struck off, at least she will not be burned at the stake. But the impact on medical science will be immense, because what other doctor will be prepared to question the prosecution theory if it means the end of a career? This is a very dark day for science, as it is for justice.
______________
--- End quote ---
IainB:
Eisenhower foresaw this kind of thing...
UK Government Forbids Publicly-Funded Scientists And Academics From Giving Advice It Disagrees With
Navigation
[0] Message Index
[#] Next page
[*] Previous page
Go to full version