Main Area and Open Discussion > Living Room

Peer Review and the Scientific Process

(1/47) > >>

dspelley:
Over the last couple of weeks I've been watching a lot of YouTube videos on science and mathematics-related subjects. Most of these have been made by Australian-born video journalist and film-maker Brady Haran - many in conjunction with scientists and mathematicians at the University of Nottingham.

Peer Review and the Scientific Process Brady Haran

This morning I watched one of the videos on his Peer Review and the Scientific Process Test Tube web site.

The video is about a breakthrough in the field of "atomic switches" by physicist Phil Moriarity and his team, but what I found interesting and important was Professor Moriarity's discussion about the absolute requirement for peer review in the scientific process.  This discussion starts at about the 8:43 point in the video.

I work in an industry where opposition groups bring forth so-called "research" or "evidence" that has not undergone this peer review process, and don't seem to understand why our whole scientific process depends on it.

eleman:
Well, peer review has its load of problems. At the moment we don't have a better alternative though.

IainB:
...but what I found interesting and important was Professor Moriarity's discussion about the absolute requirement for peer review in the scientific process.  This discussion starts at about the 8:43 point in the video.
I work in an industry where opposition groups bring forth so-called "research" or "evidence" that has not undergone this peer review process, and don't seem to understand why our whole scientific process depends on it.
-dspelley (April 14, 2013, 03:11 PM)
--- End quote ---

Psychologists do tell us that we seem to be an irrational species by nature, and critical thinking therefore requires learning and practice - i.e., because it doesn't come naturally to us. (It certainly didn't come naturally to me either - I had to work at it. Critical thinking is no friend to the ego.)
Thus, you may find that a lot of people might accept that some absurd piece of reasoning, or loudness or strength of opinion, was sufficient to prove something.
So the scientific method, whilst being something that is recommended for use in science, is not necessarily always used, whether by "laymen" or "scientists". Such people - assuming that they know about the scientific method in the first place - seem to sometimes choose to (say) consider it as being optional or de rigeur only, especially where, if they did use the method, then their pet beliefs/theories could be at risk of being debunked.

Peer Review and the Scientific Process   Peer Review and the Scientific Process   Peer Review and the Scientific Process

Of course, critical peer review is a very useful thing, but if the reviewers are unable to review something critically, or are not skeptical, or lack rigour in their critical/scientific thinking, or are of one mind with the author of the thing being reviewed - or some combination of these things - then you are as likely as not going to just get GIGO (Garbage In, Garbage Out). The only thing that could generally and probably forestall this is the proper and rigorous application of the scientific method.

Peer review of itself has been shown to be unreliable for the elimination of any bad or fraudulent science.
For example, there is an informative report in Forbes.com (2013-01-09) with various links to investigations of published bad/fraudulent science (these links are all well worth a read), which would all presumably have had to get through some "gates" in a peer review process, before being published. The examples given in these links are quite egregious:
A Barrage Of Legal Threats Shuts Down Whistleblower Site, Science Fraud

The thing is that we generally seem to irrationally expect/assume/believe scientists to be good people and good scientists, incorruptible and upholding the highest standards of scientific integrity and following the scientific method at all times.
The vexing reality is that there have been many cases where so-called "scientists" have fallen far short of this  expectation, and have been seen to be sadly deficient, the facts showing them to be variously outright frauds or con merchants, or just severely unscientific, incompetent/misguided - regardless of their qualifications. The modern degree-mill universities would seem to have a lot to answer for, regarding the inferior and mediocre intellectual/academic outputs.

There has been quite a bit of discussion on some aspects of this in the DC Forum - e.g. here, and in my neck of the woods (Australasia) there has been a collection of very recently discovered examples of this in the case of the Australian CSIR (see post from The Age, dated 2013-04-12, copied below), and in 2010, NIWA (the New Zealand government's National Institute of Water and Atmospheric research) suffered a legal defeat over the important matter of apparently falsifying some of its climate data ("tainted climate reconstruction") implicated in temperature data fraud - e.g., see here.

Here is The Age's report re the CSIR:
(Copied below sans embedded hyperlinks/images.)
Call for inquiry as CSIRO comes under the microscope
April 12, 2013
Nicky Phillips and Linton Besser

EXCLUSIVE
Demanding answers: Science Minister Don Farrell.
Call for answers: Science Minister Don Farrell. Photo: Supplied

Confidential reviews of the CSIRO by some of the world's most accomplished scientists show that the once great institution is now unable to act in the best interests of advancing research.

They found the Commonwealth Scientific and Industrial Research Organisation was being strangled by a bureaucratic labyrinth stifling innovation and persuading science leaders to abandon the 87-year-old institution, the reviews say.

One of Australia's most renowned scientists, who wished to remain anonymous, said the nation's peak research body had lost its way and should ''remove the S from its name''.
Critical: Former division chief Max Whitten believes CSIRO has lost worldwide credibility.

Critical: Former division chief Max Whitten believes CSIRO has lost worldwide credibility. Photo: Supplied

On Thursday night Science Minister Don Farrell demanded answers from the CSIRO after Fairfax Media reported that officials and others involved in a spin-off joint venture knowingly passed off cheap Chinese chemicals as their trade-secret formula.

In a deal believed to be worth $2.5 million, the venture sold the technology to the Swiss drug company Novartis, one of the biggest pharmaceutical makers.

It was part of its high-security anti-counterfeit technology to protect hundreds of millions of injectable Voltaren ampoules distributed overseas. Voltaren is an anti-inflammatory.
Former CSIRO CEO Dr Geoff Garrett: Introduced the controversial 'matrix' management system.

Former CSIRO Chief Executive Dr Geoff Garrett. Photo: Jessica Shapiro

Novartis has confirmed it has begun an investigation into the affair and the federal opposition has called for an independent inquiry into the entire organisation. A dozen previously unreleased assessments reveal the organisation had become bogged down in bureaucracy, doubling the number of managers and putting excessive emphasis on basic paid consulting work at the cost of time and resources for real science.

Its focus on short-term projects was ''paralysing the ability of the groups to act creatively and strategically in the best interests of advancing the science''.

Former CSIRO staff, including division chief Max Whitten, said it was no longer recognised as a world-leading scientific institution, an accusation it vigorously disputes, citing a separate review by a former chief scientist in 2006.

One previously unpublished review, of the earth science and resource engineering division, reported consistently negative responses from all research groups it interviewed about the management model.

''The panel considers that this is … seriously undermining the quality of the research,'' the review says. ''In our opinion, the costs significantly outweigh the putative advantages.'' The sentiments were echoed in many other reviews, including the nutrition group which found its ''once world-leading laboratories have lost that position, and with a number of exceptions, are now followers of the best front-line centres''.

The reviews commend some areas for world-class research but repeatedly criticise the management structure, which it has dubbed the ''matrix''.

This matrix was incrementally introduced from 2003 by former chief executive Geoff Garrett, aimed at conducting more science targeted to specific problems facing industry, government and the community. Dr Garrett dismantled many of the 22 divisions. In their place he introduced entities called ''flagships'', which are more focused on generating revenue.

Critics say that while the goals of many flagships were worthy, it was inappropriate for the research of the country's leading scientific organisation to be tied to financial benchmarks because it stifled scientific discovery.

Under the present structure, the 12 divisions host the organisation's scientific capacity - its staff, infrastructure and expertise. But these resources are mainly used to service projects run not by the divisions but the flagships.

In the past, the CSIRO's reputation for producing highly valuable and independent science was based on its divisions, led by internationally respected scientists. ''Now CSIRO doesn't enjoy a good reputation in many areas,'' said Dr Whitten.

The reviewers found the matrix fragmented researchers among multiple projects and answerable to several managers. Reviewers of the land and water division found the needs and priorities of the flagship dominated decisions about what science to undertake.

Despite the criticism of the inner workings, staff scientists have achieved successes in the past few years, including developing a hendra vaccine and securing Australia as a co-location for the world's biggest radio telescope. The review's complaints also contrasted sharply with a review of the flagship program conducted by the former Australian chief scientist Robin Batterham in 2006, which praised the matrix structure. The deputy chief executive, science strategy and people, Craig Roy, rejected suggestions the matrix had increased management, saying the organisation had reduced its 27 divisions and flagships in 2003 to 23 entities now.

''In 2002 the organisation wasn't structured to focus on the big issues of low emissions energy, water, oceans, health, food. Those are the places where, in many cases, we're leading the national R&D agenda today,'' he said.

The organisation was also addressing criticism its divisional research was fragmented and researchers were too stretched. ''In the last six months we've been working … to address … [the issue] of fragmentation [to] make life easier for scientists so they can focus more on their science,'' he said.

The general manager of science excellence and standing, Jack Steele, said only a ''sliver'' of the CSIRO's work was contract testing for industry. ''Almost all of our activity has a component of discovery associated with it.''

In 2012 the organisation made $410 million, almost 30 per cent of its total revenue, providing services to the private sector, government and other research groups.

Do you know more? [email protected]
___________________________________

--- End quote ---

Slashdot had a post referring to the above, on 2013-04-14, which adds even more unsavoury stuff to the pile:
Corruption Allegations Rock Australia's CSIRO
Posted by samzenpus on Sunday April 14, @12:38PM
from the say-it-aint-so dept.

An anonymous reader writes "Australia's premiere government research organization the CSIRO has been rocked by allegations of corruption including: dishonesty with 60 top-class scientists bullied or fired, fraud against drug giant Novartis, and illegally using intellectual property, faking documents and unreliable testimony to judicial officers. CSIRO Boss Megan Clark has refused to discipline the staff responsible and the federal police don't want to get involved. Victims are unimpressed and former CSIRO scientists are calling for an inquiry."
___________________________________

--- End quote ---

kyrathaba:


As the human species becomes more enlightened (assuming we don't annihilate ourselves first) I think there will have to come a time when there will be an admission that there are things we cannot know/learn by reflection, analysis or discovery. Though I'm sure the opinion I'm putting forth is in opposition to that of many site members, I believe there are absolute limits to human ability, and that there are phenomena in the universe that are not susceptible to scientific investigation, however advanced our tools become.

app103:
(see attachment in previous post)
As the human species becomes more enlightened (assuming we don't annihilate ourselves first) I think there will have to come a time when there will be an admission that there are things we cannot know/learn by reflection, analysis or discovery. Though I'm sure the opinion I'm putting forth is in opposition to that of many site members, I believe there are absolute limits to human ability, and that there are phenomena in the universe that are not susceptible to scientific investigation, however advanced our tools become.
-kyrathaba (April 19, 2013, 12:46 PM)
--- End quote ---


We all have faith...maybe not religious faith, but faith nonetheless.

A priest once explained it to me like this:

Faith is believing or trusting in something when you have no proof or when common sense tells you not to.

Then he went on to state that we all buy cans of soup on faith, that we trust that the label is truthful and purchase it without any proof beforehand that what is in the can is what it says on the label. You don't really know for sure what you are going to get. The label could be wrong. It could be a can of corn and not soup...or it could be a different kind of soup than what it says on the label. But you will continue to believe that it is soup in that can and trust in that label until you open it, get your proof and know for sure.

We have faith in the people we love, even when they do something wrong and common sense tells you not to. You are willing to forgive the mistakes of your children and trust again, even when there is no proof that they will not make a mistake again. We trust people that have never hurt us, even though we have been hurt by others. There is no proof that this new person in our lives will not hurt us. There is never any proof they won't...even after knowing them for 50 years and them never hurting us. They could still hurt us tomorrow. But we have faith that they won't. This is part of what makes a marriage work...faith in each other.

Religious people are like that. They have their beliefs that they accept on faith. They trust that they are the truth. It won't be until they get to open their can after their death that they will get their proof of whether there is a god inside, something else, or nothing at all. One way or another they will have their proof that what they have believed all their lives is true or not. And if it's not true, they will never know. Either way, it doesn't matter to them, just like it wouldn't matter to you if someone came along and told you that the soup you bought isn't soup, without any proof that it isn't soup. You'll be content to keep believing the label until you open it and see for yourself. And you'll go on having faith in the people you love.

I am not saying that I agree with the religious about what they believe, but I am willing to admit that I understand how faith works and why it is hard for them to believe otherwise. Nobody has given them a can opener yet, and until then, they don't have their proof and anyone that tells them otherwise without proof themselves is just as looney as you think they are for having the faith they have. And if you hand them anything other than the can opener they need, they will reject it as being the wrong tool for the job.

And if you can't understand what I am trying to say, then try applying the scientific method, with or without peer review, the next time you are shopping for canned goods, before you make your purchase. (I hope you don't starve)

Navigation

[0] Message Index

[#] Next page