topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • Saturday December 14, 2024, 4:15 am
  • Proudly celebrating 15+ years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Last post Author Topic: Peer Review and the Scientific Process  (Read 157623 times)

dspelley

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 90
  • But it's a dry heat.
    • View Profile
    • Read more about this member.
    • Donate to Member
Peer Review and the Scientific Process
« on: April 14, 2013, 03:11 PM »
Over the last couple of weeks I've been watching a lot of YouTube videos on science and mathematics-related subjects. Most of these have been made by Australian-born video journalist and film-maker Brady Haran - many in conjunction with scientists and mathematicians at the University of Nottingham.

Brady_Haran.jpgPeer Review and the Scientific Process Brady Haran

This morning I watched one of the videos on his Test_tube.jpgPeer Review and the Scientific Process Test Tube web site.

The video is about a breakthrough in the field of "atomic switches" by physicist Phil Moriarity and his team, but what I found interesting and important was Professor Moriarity's discussion about the absolute requirement for peer review in the scientific process.  This discussion starts at about the 8:43 point in the video.

I work in an industry where opposition groups bring forth so-called "research" or "evidence" that has not undergone this peer review process, and don't seem to understand why our whole scientific process depends on it.
We are at the very beginning of time for the human race. It is not unreasonable that we grapple with problems. But there are tens of thousands of years in the future. Our responsibility is to do what we can, learn what we can, improve the solutions, and pass them on.
--- Richard Feynman (1918-1988)

eleman

  • Spam Killer
  • Supporting Member
  • Joined in 2009
  • **
  • default avatar
  • Posts: 413
    • View Profile
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #1 on: April 15, 2013, 03:33 AM »
Well, peer review has its load of problems. At the moment we don't have a better alternative though.

IainB

  • Supporting Member
  • Joined in 2008
  • **
  • Posts: 7,544
  • @Slartibartfarst
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #2 on: April 15, 2013, 06:22 AM »
...but what I found interesting and important was Professor Moriarity's discussion about the absolute requirement for peer review in the scientific process.  This discussion starts at about the 8:43 point in the video.
I work in an industry where opposition groups bring forth so-called "research" or "evidence" that has not undergone this peer review process, and don't seem to understand why our whole scientific process depends on it.

Psychologists do tell us that we seem to be an irrational species by nature, and critical thinking therefore requires learning and practice - i.e., because it doesn't come naturally to us. (It certainly didn't come naturally to me either - I had to work at it. Critical thinking is no friend to the ego.)
Thus, you may find that a lot of people might accept that some absurd piece of reasoning, or loudness or strength of opinion, was sufficient to prove something.
So the scientific method, whilst being something that is recommended for use in science, is not necessarily always used, whether by "laymen" or "scientists". Such people - assuming that they know about the scientific method in the first place - seem to sometimes choose to (say) consider it as being optional or de rigeur only, especially where, if they did use the method, then their pet beliefs/theories could be at risk of being debunked.

Scientific Method - diagram 01.jpgPeer Review and the Scientific Process   Scientific Method - diagram 02.jpgPeer Review and the Scientific Process   Science vs Faith 02.jpgPeer Review and the Scientific Process

Of course, critical peer review is a very useful thing, but if the reviewers are unable to review something critically, or are not skeptical, or lack rigour in their critical/scientific thinking, or are of one mind with the author of the thing being reviewed - or some combination of these things - then you are as likely as not going to just get GIGO (Garbage In, Garbage Out). The only thing that could generally and probably forestall this is the proper and rigorous application of the scientific method.

Peer review of itself has been shown to be unreliable for the elimination of any bad or fraudulent science.
For example, there is an informative report in Forbes.com (2013-01-09) with various links to investigations of published bad/fraudulent science (these links are all well worth a read), which would all presumably have had to get through some "gates" in a peer review process, before being published. The examples given in these links are quite egregious:
A Barrage Of Legal Threats Shuts Down Whistleblower Site, Science Fraud

The thing is that we generally seem to irrationally expect/assume/believe scientists to be good people and good scientists, incorruptible and upholding the highest standards of scientific integrity and following the scientific method at all times.
The vexing reality is that there have been many cases where so-called "scientists" have fallen far short of this  expectation, and have been seen to be sadly deficient, the facts showing them to be variously outright frauds or con merchants, or just severely unscientific, incompetent/misguided - regardless of their qualifications. The modern degree-mill universities would seem to have a lot to answer for, regarding the inferior and mediocre intellectual/academic outputs.

There has been quite a bit of discussion on some aspects of this in the DC Forum - e.g. here, and in my neck of the woods (Australasia) there has been a collection of very recently discovered examples of this in the case of the Australian CSIR (see post from The Age, dated 2013-04-12, copied below), and in 2010, NIWA (the New Zealand government's National Institute of Water and Atmospheric research) suffered a legal defeat over the important matter of apparently falsifying some of its climate data ("tainted climate reconstruction") implicated in temperature data fraud - e.g., see here.

Here is The Age's report re the CSIR:
(Copied below sans embedded hyperlinks/images.)
Call for inquiry as CSIRO comes under the microscope
April 12, 2013
Nicky Phillips and Linton Besser

EXCLUSIVE
Demanding answers: Science Minister Don Farrell.
Call for answers: Science Minister Don Farrell. Photo: Supplied

Confidential reviews of the CSIRO by some of the world's most accomplished scientists show that the once great institution is now unable to act in the best interests of advancing research.

They found the Commonwealth Scientific and Industrial Research Organisation was being strangled by a bureaucratic labyrinth stifling innovation and persuading science leaders to abandon the 87-year-old institution, the reviews say.

One of Australia's most renowned scientists, who wished to remain anonymous, said the nation's peak research body had lost its way and should ''remove the S from its name''.
Critical: Former division chief Max Whitten believes CSIRO has lost worldwide credibility.

Critical: Former division chief Max Whitten believes CSIRO has lost worldwide credibility. Photo: Supplied

On Thursday night Science Minister Don Farrell demanded answers from the CSIRO after Fairfax Media reported that officials and others involved in a spin-off joint venture knowingly passed off cheap Chinese chemicals as their trade-secret formula.

In a deal believed to be worth $2.5 million, the venture sold the technology to the Swiss drug company Novartis, one of the biggest pharmaceutical makers.

It was part of its high-security anti-counterfeit technology to protect hundreds of millions of injectable Voltaren ampoules distributed overseas. Voltaren is an anti-inflammatory.
Former CSIRO CEO Dr Geoff Garrett: Introduced the controversial 'matrix' management system.

Former CSIRO Chief Executive Dr Geoff Garrett. Photo: Jessica Shapiro

Novartis has confirmed it has begun an investigation into the affair and the federal opposition has called for an independent inquiry into the entire organisation. A dozen previously unreleased assessments reveal the organisation had become bogged down in bureaucracy, doubling the number of managers and putting excessive emphasis on basic paid consulting work at the cost of time and resources for real science.

Its focus on short-term projects was ''paralysing the ability of the groups to act creatively and strategically in the best interests of advancing the science''.

Former CSIRO staff, including division chief Max Whitten, said it was no longer recognised as a world-leading scientific institution, an accusation it vigorously disputes, citing a separate review by a former chief scientist in 2006.

One previously unpublished review, of the earth science and resource engineering division, reported consistently negative responses from all research groups it interviewed about the management model.

''The panel considers that this is … seriously undermining the quality of the research,'' the review says. ''In our opinion, the costs significantly outweigh the putative advantages.'' The sentiments were echoed in many other reviews, including the nutrition group which found its ''once world-leading laboratories have lost that position, and with a number of exceptions, are now followers of the best front-line centres''.

The reviews commend some areas for world-class research but repeatedly criticise the management structure, which it has dubbed the ''matrix''.

This matrix was incrementally introduced from 2003 by former chief executive Geoff Garrett, aimed at conducting more science targeted to specific problems facing industry, government and the community. Dr Garrett dismantled many of the 22 divisions. In their place he introduced entities called ''flagships'', which are more focused on generating revenue.

Critics say that while the goals of many flagships were worthy, it was inappropriate for the research of the country's leading scientific organisation to be tied to financial benchmarks because it stifled scientific discovery.

Under the present structure, the 12 divisions host the organisation's scientific capacity - its staff, infrastructure and expertise. But these resources are mainly used to service projects run not by the divisions but the flagships.

In the past, the CSIRO's reputation for producing highly valuable and independent science was based on its divisions, led by internationally respected scientists. ''Now CSIRO doesn't enjoy a good reputation in many areas,'' said Dr Whitten.

The reviewers found the matrix fragmented researchers among multiple projects and answerable to several managers. Reviewers of the land and water division found the needs and priorities of the flagship dominated decisions about what science to undertake.

Despite the criticism of the inner workings, staff scientists have achieved successes in the past few years, including developing a hendra vaccine and securing Australia as a co-location for the world's biggest radio telescope. The review's complaints also contrasted sharply with a review of the flagship program conducted by the former Australian chief scientist Robin Batterham in 2006, which praised the matrix structure. The deputy chief executive, science strategy and people, Craig Roy, rejected suggestions the matrix had increased management, saying the organisation had reduced its 27 divisions and flagships in 2003 to 23 entities now.

''In 2002 the organisation wasn't structured to focus on the big issues of low emissions energy, water, oceans, health, food. Those are the places where, in many cases, we're leading the national R&D agenda today,'' he said.

The organisation was also addressing criticism its divisional research was fragmented and researchers were too stretched. ''In the last six months we've been working … to address … [the issue] of fragmentation [to] make life easier for scientists so they can focus more on their science,'' he said.

The general manager of science excellence and standing, Jack Steele, said only a ''sliver'' of the CSIRO's work was contract testing for industry. ''Almost all of our activity has a component of discovery associated with it.''

In 2012 the organisation made $410 million, almost 30 per cent of its total revenue, providing services to the private sector, government and other research groups.

Do you know more? [email protected]
___________________________________

Slashdot had a post referring to the above, on 2013-04-14, which adds even more unsavoury stuff to the pile:
Corruption Allegations Rock Australia's CSIRO
Posted by samzenpus on Sunday April 14, @12:38PM
from the say-it-aint-so dept.

An anonymous reader writes "Australia's premiere government research organization the CSIRO has been rocked by allegations of corruption including: dishonesty with 60 top-class scientists bullied or fired, fraud against drug giant Novartis, and illegally using intellectual property, faking documents and unreliable testimony to judicial officers. CSIRO Boss Megan Clark has refused to discipline the staff responsible and the federal police don't want to get involved. Victims are unimpressed and former CSIRO scientists are calling for an inquiry."
___________________________________
« Last Edit: April 15, 2013, 06:49 AM by IainB, Reason: Minor corrections. »

kyrathaba

  • N.A.N.Y. Organizer
  • Honorary Member
  • Joined in 2006
  • **
  • Posts: 3,200
    • View Profile
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #3 on: April 19, 2013, 12:46 PM »
Science vs Faith 02.jpg

As the human species becomes more enlightened (assuming we don't annihilate ourselves first) I think there will have to come a time when there will be an admission that there are things we cannot know/learn by reflection, analysis or discovery. Though I'm sure the opinion I'm putting forth is in opposition to that of many site members, I believe there are absolute limits to human ability, and that there are phenomena in the universe that are not susceptible to scientific investigation, however advanced our tools become.

app103

  • That scary taskbar girl
  • Global Moderator
  • Joined in 2006
  • *****
  • Posts: 5,885
    • View Profile
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #4 on: April 19, 2013, 06:28 PM »
(see attachment in previous post)
As the human species becomes more enlightened (assuming we don't annihilate ourselves first) I think there will have to come a time when there will be an admission that there are things we cannot know/learn by reflection, analysis or discovery. Though I'm sure the opinion I'm putting forth is in opposition to that of many site members, I believe there are absolute limits to human ability, and that there are phenomena in the universe that are not susceptible to scientific investigation, however advanced our tools become.


We all have faith...maybe not religious faith, but faith nonetheless.

A priest once explained it to me like this:

Faith is believing or trusting in something when you have no proof or when common sense tells you not to.

Then he went on to state that we all buy cans of soup on faith, that we trust that the label is truthful and purchase it without any proof beforehand that what is in the can is what it says on the label. You don't really know for sure what you are going to get. The label could be wrong. It could be a can of corn and not soup...or it could be a different kind of soup than what it says on the label. But you will continue to believe that it is soup in that can and trust in that label until you open it, get your proof and know for sure.

We have faith in the people we love, even when they do something wrong and common sense tells you not to. You are willing to forgive the mistakes of your children and trust again, even when there is no proof that they will not make a mistake again. We trust people that have never hurt us, even though we have been hurt by others. There is no proof that this new person in our lives will not hurt us. There is never any proof they won't...even after knowing them for 50 years and them never hurting us. They could still hurt us tomorrow. But we have faith that they won't. This is part of what makes a marriage work...faith in each other.

Religious people are like that. They have their beliefs that they accept on faith. They trust that they are the truth. It won't be until they get to open their can after their death that they will get their proof of whether there is a god inside, something else, or nothing at all. One way or another they will have their proof that what they have believed all their lives is true or not. And if it's not true, they will never know. Either way, it doesn't matter to them, just like it wouldn't matter to you if someone came along and told you that the soup you bought isn't soup, without any proof that it isn't soup. You'll be content to keep believing the label until you open it and see for yourself. And you'll go on having faith in the people you love.

I am not saying that I agree with the religious about what they believe, but I am willing to admit that I understand how faith works and why it is hard for them to believe otherwise. Nobody has given them a can opener yet, and until then, they don't have their proof and anyone that tells them otherwise without proof themselves is just as looney as you think they are for having the faith they have. And if you hand them anything other than the can opener they need, they will reject it as being the wrong tool for the job.

And if you can't understand what I am trying to say, then try applying the scientific method, with or without peer review, the next time you are shopping for canned goods, before you make your purchase. (I hope you don't starve)

IainB

  • Supporting Member
  • Joined in 2008
  • **
  • Posts: 7,544
  • @Slartibartfarst
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #5 on: April 20, 2013, 03:51 AM »
...I believe there are absolute limits to human ability, and that there are phenomena in the universe that are not susceptible to scientific investigation, however advanced our tools become.
Yes. The great imponderables, and I have had direct experience of a few of them, such that I know (from that direct experience) certain things to be true, without knowing why/how they can be, or are so. I do not "believe" them.
However, though I would not care to predict whether our tools would be able to advance to the point where they could enable us to understand these previously inexplicable phenomena, given our progress to date in philosophy, metaphysics, science and engineering, it wouldn't surprise me if they did.

Such imponderables may be always susceptible/open to scientific investigation to some limited extent, but currently seem to remain defiantly opaque to our further enquiries, and so the best we will be able to do is create often contradictory theories and/or beliefs about them - e.g., including the Big Bang theory v. constant creation; life v. death; Higgs boson v. non-Higgs models.
Many of these theories might be half-baked, cobbled-together and even simply "made-up" theories explaining away the inexplicable, founded on an ego-centric rationale that we absolutely must be able to explain everything away that we cannot understand, rather than simply admitting that "We just don't know" - e.g., "near death experiences".
In this way, we can have the security blanket of a belief or a theory for everything, and skitter away from the terrifying abyss of our ignorance and the inexplicable - the Unknown - comforted in our assumed knowledge, which is in reality but knowledge of very little indeed.

So we have (say) a theory of Evolution (Darwinism), which, whilst it rather seems to knock the theory/belief of Creationism into a cocked hat, arguably is not necessarily of itself true (it remains an unproven theory) and does not of itself necessarily defeat the theoretical concept of God - though some might prefer to believe or perceive that it does, of course.    ;)
Whilst thinking about this, it may be useful to reflect that Darwin was himself a devout Christian and though his research and his theory perturbed him, it apparently did not cause him to lose his faith in God.

However, as I have argued above, getting research through the gates of a peer review process apparently does not of itself prove anything about anything, especially where the scientific method has been abused in the fist place (QED). Thus, if we had some research that seemed to show that Creationism was true after all, and, after several peer reviews it was published in Nature or something, I would have to recommend one read such a publication with a high degree of scepticism.

@app103: I have to say that I feel that your priest's can of soup analogy is an absurd analogy for Faith. Have faith in the love of God, by all means, but not in a can of soup, for goodness' sake. A can is not God. Opening a can of soup will reveal that it is made of pieces of rolled and crimped iron sheet, sometimes galvanised on the outside, and lined with a plating of tin (a silvery metal) or a film of plastic on the inside.
Generally, the soup contained within will be found to be edible, and seems to keep without perishing whilst it is in the unopened can. You don't have to trust the can, but you do need to place some reliance on the proper manufacture of the can and the canning process at the cannery.
You can inspect a can and determine whether the food inside is likely to be safe to eat. Treat with circumspection any cans where there is evidence of damage, corrosion/leakage of the can - the contents may have perished and could be fatally poisonous as air may have entered the can and microbes will have bred in that environment. If the flat ends of the can are bulging outwards, then that is a sure sign of microbial gas production and the contents will almost certainly be fatally poisonous - destroy the can and its contents to avoid the risk of poisoning others.
If otherwise the can looks OK, and if the contents smell and taste OK, then they're likely to be safe to eat. Trust or faith doesn't seem to come into it. Theory and observation do.
"Action which is not based on sound theory or "best"/good practice is irrational by definition." (WE Deming)

I could be wrong, of course, but the sort of Faith you seem to be talking about is religious faith in the eternal, in (say) a God. A Christian believer might recite the dogma of the Nicene Creed: "I believe...in the Father, The Son and the Holy Ghost...etc.", perhaps not knowing that history tells us that the creed was apparently invented in about 325AD by the leaders of the RC Church as a compromise to avoid a faction forming amongst the religion's orders. An Islamist would be quick to point that out to you and say furthermore that the Creed is a double-whammy blasphemy in Islam, because:
  • (a) it divides the indivisible one God into three bits (this is also potentially idolatrous), and
  • (b) one of the three bits is a man (Jesus Christ) who, though he is believed by Muslims to be a true prophet of Allah, was but a man nevertheless, just like Mohammed (pbuh), who is believed by Muslims to be the second and last true prophet of Allah.

That's why:
(i) a Christian cannot convert to being a Muslim without first renouncing the Nicene Creed (the belief in the Holy Trinity), which blasphemy is a sin and otherwise blocks his spiritual eligibility to enter Islam.
(ii) a Christian who expresses his faith in the Holy Trinity is apparently expressing faith in a deliberate and artificial (i.e., made up) fiction.

Of course there's no doubt lots of sophistry to say this is not really how it is, or "look how many angels you can fit on the head of a pin!", or "our Book of God/Allah is more true than yours!", but it is all absurd  - arguments over fantasy/myth - and likely to prove of no productive use of your cognitive surplus and may even be potentially life-threatening. So, before we start cutting off each other's head's or blowing ourselves up over whose fantasy/belief is thickest, we might be better off discussing something more useful and enjoyable - e.g. science fiction/fantasy.    ;)

app103

  • That scary taskbar girl
  • Global Moderator
  • Joined in 2006
  • *****
  • Posts: 5,885
    • View Profile
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #6 on: April 20, 2013, 05:10 AM »
@app103:[/b] I have to say that I feel that your priest's can of soup analogy is an absurd analogy for Faith. Have faith in the love of God, by all means, but not in a can of soup, for goodness' sake. A can is not God. Opening a can of soup will reveal that it is made of pieces of rolled and crimped iron sheet, sometimes galvanised on the outside, and lined with a plating of tin (a silvery metal) or a film of plastic on the inside.
Generally, the soup contained within will be found to be edible, and seems to keep without perishing whilst it is in the unopened can. You don't have to trust the can, but you do need to place some reliance on the proper manufacture of the can and the canning process at the cannery.
You can inspect a can and determine whether the food inside is likely to be safe to eat. Treat with circumspection any cans where there is evidence of damage, corrosion/leakage of the can - the contents may have perished and could be fatally poisonous as air may have entered the can and microbes will have bred in that environment. If the flat ends of the can are bulging outwards, then that is a sure sign of microbial gas production and the contents will almost certainly be fatally poisonous - destroy the can and its contents to avoid the risk of poisoning others.
If otherwise the can looks OK, and if the contents smell and taste OK, then they're likely to be safe to eat. Trust or faith doesn't seem to come into it. Theory and observation do.

Read what I wrote again. Where is the proof before you buy it that there is actually soup in the can and not corn? Where is the proof that it has tomato soup and not cream of mushroom? The label could be wrong. THAT is where faith comes in. You don't truly know what is in the can till you buy it, take it home, and open it. You buy it on faith.

This isn't about anything you can observe, not about whether the contents is safe to eat or not...it's about the actual contents itself, something you can't see through the metal, something you have to believe without any proof at the time of purchase. Is a label proof? No. So, it requires faith. Not religious faith, just common ordinary faith, but faith, nonetheless.

IainB

  • Supporting Member
  • Joined in 2008
  • **
  • Posts: 7,544
  • @Slartibartfarst
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #7 on: April 20, 2013, 06:39 AM »
Where is the proof before you buy it that there is actually soup in the can and not corn? Where is the proof that it has tomato soup and not cream of mushroom? The label could be wrong. THAT is where faith comes in. You don't truly know what is in the can till you buy it, take it home, and open it. You buy it on faith.
That is one of the most absurd pieces of reasoning I have come across in a while. Not the worst by any means, but it's up there amongst the worst. I hadn't realised that your priest was making such a fundamental and naively elementary error.

The premise is that you don't know what is in the can, so it is bought on blind faith/trust.
It is a false premise, and this is why:

Soup is a food item. The can of soup is bought under a simple contract.
When you buy the can of soup, it is as a result of accepting an "invitation to treat" - an offer to sell the can at a certain price - from the vendor. Three things need to occur to complete the contract:
  • 1. Offer.
  • 2. Acceptance.
  • 3. Consideration (Payment).

The prevailing local/international Trade Descriptions Act and labelling standards ensure that the can is labelled in such a way as to specify its contents in precise detail. There is no uncertainty. The product must (as in Mandatory) be properly described and labelled and be fit for human consumption. It had better be too, because otherwise the producer will face hefty fines/penalties and may even lose his licence to manufacture the product, and this could put him out of business.
There are a string of cases in Contract Case law which set the precedent and the penalties for any breach of contract by the supplier/producer, and the penalties/damages awarded by the courts are usually hefty and inevitably in favour of the consumer.
Not only that, but local and international Food and Drug Administration Authorities come down like the proverbial ton of bricks on producers who do not meet the requisite product standards - hence more hefty fines/penalties in addition to any judgements of contractual breach.

Unsurprisingly, the contents of cans of soup are precisely as labelled - always allowing for the statistical chance of human error and the placing of the wrong labels on the product, which is also an offence and a breach of contract - so it doesn't happen very often. In fact, years ago when I was working as a consultant in the UK on a contract to audit and review/improve a cannery's production systems in Liverpool (UK), I couldn't identify any real potential areas of improvement in their production processes, because they were already squeaky-tight.
Things have only got better since then, because factory automation has almost eliminated human error in large batch production bottling and canning factories.

app103

  • That scary taskbar girl
  • Global Moderator
  • Joined in 2006
  • *****
  • Posts: 5,885
    • View Profile
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #8 on: April 20, 2013, 09:03 AM »
IainB, You seem to be very eager to pick apart one example, zooming the entire focus in on that one example. So, much so that you are failing to see the big picture, completely missing the point. Do you even know what my point was? It wasn't about soup, or canned goods, or even religion.

And I feel as though you are making some assumptions about me, believing something without any proof.

Why do you keep referring to him as my priest, and not that priest? Are you aware of what that choice of words implies? Do you think I am a religious person, a Catholic? Because that's the only way he could be my priest. Is that why you are attacking and picking apart my words with such focused energy, the kind of nit-picking energy I normally see reserved for when atheists attack the religious?

I never said he was my priest. I said he was a priest. In fact he isn't my priest at all, he's my sister-in-law's priest, and I don't share his or her faith. At least not their religious faith. I am not a religious person. I hold no belief in deities that can not be proven both real and perfect.

However, I am willing to keep an open mind about some things (not religious things, though). One of the things I am willing to keep an open mind about is the idea that it is perfectly normal and natural for human beings to believe in things for which they have no proof or when common sense tells them not to. Maybe it's a part of human nature to do so. And that the occurrences of this kind of thinking is a lot more common than those that claim they have no faith, or that they don't believe faith is normal, are willing to admit. That even people that claim not to have any faith, do in fact have some faith and that faith plays an important role in some of the things they do and the decisions they make, and that without it, they would be stuck, unable to move forward. Perhaps the very survival of the human species depends on faith, common every day faith.

But are you so willing to be closed minded to that possibility and completely dismiss the idea that faith is a normal part of being human, asserting that it is not natural for the human mind to believe in things for which there is no proof or when common sense tells them not to?

And are you willing to state that nothing in normal daily life requires faith? Nothing in daily life requires a belief in something for which you have no proof? Nothing in daily life requires you to have a belief that acts against your common sense? Nothing at all? Think, observe, watch your own thoughts and actions for awhile before you answer that with a "no". Don't answer until you are sure.

Because that was my point. We are creatures of faith, even those that claim they have no faith. The faith they refer to is religious faith. But that's not the only kind that exists. Normal people are creatures of faith. And sometimes that faith gets in the way of things, leads people to believe things that are pretty far fetched, things that can't be proven false to their satisfaction because they are not the ones making the first hand discoveries that prove their beliefs false. They are stubborn skeptics, in their own way.

And that ties into what kyrathaba said, and why I quoted him in the first place:

As the human species becomes more enlightened (assuming we don't annihilate ourselves first) I think there will have to come a time when there will be an admission that there are things we cannot know/learn by reflection, analysis or discovery. Though I'm sure the opinion I'm putting forth is in opposition to that of many site members, I believe there are absolute limits to human ability, and that there are phenomena in the universe that are not susceptible to scientific investigation, however advanced our tools become.

and I believe that common everyday faith may be a sign that those limitations may be very real. Sometimes we are unwilling or unable to see or believe the truth, and that's not due to the limitations of our tools.

kyrathaba

  • N.A.N.Y. Organizer
  • Honorary Member
  • Joined in 2006
  • **
  • Posts: 3,200
    • View Profile
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #9 on: April 20, 2013, 12:17 PM »
Religious people are like that. They have their beliefs that they accept on faith. They trust that they are the truth. It won't be until they get to open their can after their death that they will get their proof of whether there is a god inside, something else, or nothing at all.

You are correct in this. People who place faith in Christ as a supernatural being and who belief they will be resurrected will either have that faith someday rewarded; or, they'll never no they were wrong because they're forever dead.

However, not all proofs are scientific proofs. Or, perhaps more correctly I should say, 'Scientifc proofs are the be-all-end-all' in terms of understanding what is real. Many very logical, intelligent people, including some prominent scientists (some of the physicists) and medical doctors have come to accept preponderances of anecdotal proofs that their religious faith is valid. Is each and every such case an example of self-delusion?

I recommend, for those skeptical of the claims of Christianity, that they read, with intellectual rigor, Josh McDowell's "The New Evidence That Demands A Verdict." Mr. McDowell was a true-blue atheist and top-notch investigative reporter who took 2-3 years to finally critically collect evidence, analyze his preconceptions, etc.

I believe that religious faith is part of our nature as human belings, but that many have hardened their hearts against it because the world's religions proscribe certain behaviors, and it's also in our nature not to like to be constrained in anyway. We don't want anyone passing judgment on our behaviors or lifestyles, and religions do that, in that they generally contain thou shalts and thou shalt nots.

I also highly recommend, for anyone willing to be intellectually honest and open-minded, that they read Lee Strobel's "The Case for Christ / The Case for Faith" (2 books in one).

There's an old story that says that when scientists finally reach the pinnacle of knowledge/revelation (speaking metaphorically), they will find the people of religious of faith sitting there waiting for them.

I don't want to derail the topic by inserting my opinion that most self-proclaimed atheists and agnostics have not been attitudinally/intellectually willing to investigate faith and revelation as alternative tools. But I wanted to challenge skeptics to read the two books I've named above. It'd be an interesting experiment to see how many of them, if any, had a different take on religious faith afterwards.
« Last Edit: April 20, 2013, 12:24 PM by kyrathaba »

cmpm

  • Charter Member
  • Joined in 2006
  • ***
  • default avatar
  • Posts: 2,026
    • View Profile
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #10 on: April 20, 2013, 12:30 PM »
let me find my ten foot pole, and maybe i could address some things...

kyrathaba

  • N.A.N.Y. Organizer
  • Honorary Member
  • Joined in 2006
  • **
  • Posts: 3,200
    • View Profile
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #11 on: April 20, 2013, 01:16 PM »
LOL. I'm gonna end here, so you won't need your ten-foot-pole. I respect everyone's nature as a free moral/intellectual agent, capable of clinging to whatever methodologies, preconceptions, biases, etc., that they choose. I include myself among them. I can't very well argue in the Basement against self-censorship in an effort to avoid stirring up the masses with a post, yet continue to press a minority view that (both now and historically) has lead to heated/scornful reactions.

Since religious faith seems to have come into the thread, should we spiral this off into the basement and let the original thread stick narrowly to its subject? Better yet I welcome PMs on the topic of religious faith.

app103

  • That scary taskbar girl
  • Global Moderator
  • Joined in 2006
  • *****
  • Posts: 5,885
    • View Profile
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #12 on: April 20, 2013, 01:27 PM »
Since religious faith seems to have come into the thread, should we spiral this off into the basement and let the original thread stick narrowly to its subject?

Yeah, maybe from my post onward, since that seems to have sparked it....my claim that common everyday faith, that has nothing to do with religious faith, actually exists.  :-[

kyrathaba

  • N.A.N.Y. Organizer
  • Honorary Member
  • Joined in 2006
  • **
  • Posts: 3,200
    • View Profile
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #13 on: April 20, 2013, 02:50 PM »
I agree completely with your point, BTW: everyday faith (that chair will hold me up when I sit on it) is different than religious faith.

IainB

  • Supporting Member
  • Joined in 2008
  • **
  • Posts: 7,544
  • @Slartibartfarst
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #14 on: April 20, 2013, 07:13 PM »
Just attempting to pull this back on topic, after @kyrathaba's interesting diversion with the Science v, Faith diagram/discussion:
Where we seemed to have got to: It looks like I had established pretty solidly the case for the argument that the peer-review process was not in practice actually as much use to science audit as one might have intuitively expected (QED), and that the validity of the results of the scientific method seemed to be down to the rigour and integrity of the application of the scientific method.

Though discussions about the strength/rightness of one's personally held fantasies/beliefs can be interesting, could we please split that issue off into a separate thread?

IainB

  • Supporting Member
  • Joined in 2008
  • **
  • Posts: 7,544
  • @Slartibartfarst
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #15 on: April 21, 2013, 09:11 AM »
Very interesting - if not scary - report via motherjones, about the trial of AstraZeneca's Seroquel drug, relevant to this thread:
The Deadly Corruption of Clinical Trials
This is just an excerpt (my emphasis):
...Yet the more I examined the medical and court records, the more I became convinced that the problem was worse than the Pioneer Press had reported. The danger lies not just in the particular circumstances that led to Dan's death, but in a system of clinical research that has been thoroughly co-opted by market forces, so that many studies have become little more than covert instruments for promoting drugs. The study in which Dan died starkly illustrates the hazards of market-driven research and the inadequacy of our current oversight system to detect them...

IainB

  • Supporting Member
  • Joined in 2008
  • **
  • Posts: 7,544
  • @Slartibartfarst
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #16 on: April 21, 2013, 09:59 AM »
Just provisionally copying the "Faith" discussion posts across to another (new) thread...under the name of Faith under the microscope. - though the name may change, or the thread might get moved to the Basement or expunged.
« Last Edit: April 21, 2013, 10:42 AM by IainB »

IainB

  • Supporting Member
  • Joined in 2008
  • **
  • Posts: 7,544
  • @Slartibartfarst
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #17 on: April 22, 2013, 05:41 AM »
Very relevant analysis here: Moriarty on peer review
There is compelling evidence that, across the disciplines, peer review often fails to root out science fraud. Yet even basic errors in the literature can now be extremely difficult to correct on any reasonable timescale. –Philip Moriarty, Times Higher Education, 18 April 2013
« Last Edit: April 24, 2013, 12:01 PM by IainB, Reason: Added quote. »

IainB

  • Supporting Member
  • Joined in 2008
  • **
  • Posts: 7,544
  • @Slartibartfarst
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #18 on: April 30, 2013, 01:05 AM »
Informative WUWT post on How a scientist becomes a con man.
Excerpt (my emphasis):
...finding that Stapel’s fraud went undetected for so long because of “a general culture of careless, selective and uncritical handling of research and data.” If Stapel was solely to blame for making stuff up, the report stated, his peers, journal editors and reviewers of the field’s top journals were to blame for letting him get away with it. The committees identified several practices as “sloppy science” — misuse of statistics, ignoring of data that do not conform to a desired hypothesis and the pursuit of a compelling story no matter how scientifically unsupported it may be.

Well worth a read, though likely to be a bit depressing if you (like me) used to place a high degree of trust in "science". Mind you, Stapel was apparently a "Sociological scientist", or something, and thus arguably not a scientist per se, as one of the commenters points out:
Paraphrasing Robert Heinlein, “Any discipline with the word ‘science’ in the name, such as ‘social science’, isn’t one.”

IainB

  • Supporting Member
  • Joined in 2008
  • **
  • Posts: 7,544
  • @Slartibartfarst
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #19 on: June 18, 2013, 06:03 PM »
Over at hunch.net the Machine Learning (Theory) blog has a cogent and useful post on the subject of peer reviews:
Representative Reviewing
6/16/2013
Tags: Conferences, Reviewing , Workshop — jl@ 10:09 am

When thinking about how best to review papers, it seems helpful to have some conception of what good reviewing is. As far as I can tell, this is almost always only discussed in the specific context of a paper (i.e. your rejected paper), or at most an area (i.e. what a “good paper” looks like for that area) rather than general principles. Neither individual papers or areas are sufficiently general for a large conference—every paper differs in the details, and what if you want to build a new area and/or cross areas?

An unavoidable reason for reviewing is that the community of research is too large. In particular, it is not possible for a researcher to read every paper which someone thinks might be of interest. This reason for reviewing exists independent of constraints on rooms or scheduling formats of individual conferences. Indeed, history suggests that physical constraints are relatively meaningless over the long term — growing conferences simply use more rooms and/or change formats to accommodate the growth.

This suggests that a generic test for paper acceptance should be “Are there a significant number of people who will be interested?” This question could theoretically be answered by sending the paper to every person who might be interested and simply asking them. In practice, this would be an intractable use of people’s time: We must query far fewer people and achieve an approximate answer to this question. Our goal then should be minimizing the approximation error for some fixed amount of reviewing work.

Viewed from this perspective, the first way that things can go wrong is by misassignment of reviewers to papers, for which there are two easy failure modes available.
  • 1. When reviewer/paper assignment is automated based on an affinity graph, the affinity graph may be low quality or the constraint on the maximum number of papers per reviewer can easily leave some papers with low affinity to all reviewers orphaned.
  • 2. When reviewer/paper assignments are done by one person, that person may choose reviewers who are all like-minded, simply because this is the crowd that they know. I’ve seen this happen at the beginning of the reviewing process, but the more insidious case is when it happens at the end, where people are pressed for time and low quality judgements can become common.

An interesting approach for addressing the constraint objective would be optimizing a different objective, such as the product of affinities rather than the sum. I’ve seen no experimentation of this sort.

For ICML, there are about 3 levels of “reviewer”: the program chair who is responsible for all papers, the area chair who is responsible for organizing reviewing on a subset of papers, and the program committee member/reviewer who has primary responsibility for reviewing. In 2012 tried to avoid these failure modes in a least-system effort way using a blended approach. We used bidding to get a higher quality affinity matrix. We used a constraint system to assign the first reviewer to each paper and two area chairs to each paper. Then, we asked each area chair to find one reviewer for each paper. This obviously dealt with the one-area-chair failure mode. It also helps substantially with low quality assignments from the constrained system since (a) the first reviewer chosen is typically higher quality than the last due to it being the least constrained (b) misassignments to area chairs are diagnosed at the beginning of the process by ACs trying to find reviewers (c) ACs can reach outside of the initial program committee to find reviewers, which existing automated systems can not do.

The next way that reviewing can go wrong is via biased reviewing.
  • 1. Author name bias is a famous one. In my experience it is real: well known authors automatically have their paper taken seriously, which particularly matters when time is short. Furthermore, I’ve seen instances where well-known authors can slide by with proof sketches that no one fully understands.
  • 2. Review anchoring is a very significant problem if it occurs. This does not happen in the standard review process, because the reviews of others are not visible to other reviewers until they are complete.
  • 3. A more subtle form of bias is when one reviewer is simply much louder or charismatic than others. Reviewing without an in-person meeting is actually helpful here, as it reduces this problem substantially.

Reviewing can also be low quality. A primary issue here is time: most reviewers will submit a review within a time constraint, but it may not be high quality due to limits on time. Minimizing average reviewer load is quite important here. Staggered deadlines for reviews are almost certainly also helpful. A more subtle thing is discouraging low quality submissions. My favored approach here is to publish all submissions nonanonymously after some initial period of time.

Another significant issue in reviewer quality is motivation. Making reviewers not anonymous to each other helps with motivation as poor reviews will at least be known to some. Author feedback also helps with motivation, as reviewers know that authors will be able to point out poor reviewing. It is easy to imagine that further improvements in reviewer motivation would be helpful.

A third form of low quality review is based on miscommunication. Maybe there is silly typo in a paper? Maybe something was confusing? Being able to communicate with the author can greatly reduce ambiguities.

The last problem is dictatorship at decision time for which I’ve seen several variants. Sometimes this comes in the form of giving each area chair a budget of papers to “champion”. Sometimes this comes in the form of an area chair deciding to override all reviews and either accept or more likely reject a paper. Sometimes this comes in the form of a program chair doing this as well. The power of dictatorship is often available, but it should not be used: the wiser course is keeping things representative.

At ICML 2012, we tried to deal with this via a defined power approach. When reviewers agreed on the accept/reject decision, that was the decision. If the reviewers disgreed, we asked the two area chairs to make decisions and if they agreed, that was the decision. It was only when the ACs disagreed that the program chairs would become involved in the decision.

The above provides an understanding of how to create a good reviewing process for a large conference. With this in mind, we can consider various proposals at the peer review workshop and elsewhere.
  • 1. Double Blind Review. This reduces bias, at the cost of decreasing reviewer motivation. Overall, I think it’s a significant long term positive for a conference as “insiders” naturally become more concerned with review quality and “outsiders” are more prone to submit.
  • 2. Better paper/reviewer matching. A pure win, with the only caveat that you should be familiar with failure modes and watch out for them.
  • 3. Author feedback. This improves review quality by placing a check on unfair reviews and reducing miscommunication at some cost in time.
  • 4. Allowing an appendix or ancillary materials. This allows authors to better communicate complex ideas, at the potential cost of reviewer time. A standard compromise is to make reading an appendix optional for reviewers.
  • 5. Open reviews. Open reviews means that people can learn from other reviews, and that authors can respond more naturally than in single round author feedback.

It’s important to note that none of the above are inherently contradictory. This is not necessarily obvious as proponents of open review and double blind review have found themselves in opposition at times. These approaches can be accommodated by simply hiding authors names for a fixed period of 2 months while the initial review process is ongoing.

Representative reviewing seems like the real difficult goal. If a paper is rejected in a representative reviewing process, then perhaps it is just not of sufficient interest. Similarly, if a paper is accepted, then perhaps it is of real and meaningful interest. And if the reviewing process is not representative, then perhaps we should fix the failure modes.

Edit: Crossposted on CACM.

This is coincidentally the same website as @mouser referred to in another DC Forum discussion thread in 2006: Nice blog essays on Fixing Peer Reviews and Collaborative Research « on: 2006-09-19, 00:54:05 » - where he quoted from the hunch.net post What is missing for online collaborative research?:

I've been reading http://hunch.net/ for their take on machine learning articles but they've been posting some nice essays recently on the underlying frameworks for reviewing papers, etc.
Reviewing is a fairly formal process which is integral to the way academia is run. Given this integral nature, the quality of reviewing is often frustrating. I’ve seen plenty of examples of false statements, misbeliefs, reading what isn’t written, etc…, and I’m sure many other people have as well.

Recently, mechanisms like double blind review and author feedback have been introduced to try to make the process more fair and accurate in many machine learning (and related) conferences. My personal experience is that these mechanisms help, especially the author feedback. Nevertheless, some problems remain.

The game theory take on reviewing is that the incentive for truthful reviewing isn’t there. Since reviewers are also authors, there are sometimes perverse incentives created and acted upon. (Incidentially, these incentives can be both positive and negative.)

Setting up a truthful reviewing system is tricky because their is no final reference truth available in any acceptable (say: subyear) timespan. There are several ways we could try to get around this.
...

IainB

  • Supporting Member
  • Joined in 2008
  • **
  • Posts: 7,544
  • @Slartibartfarst
    • View Profile
    • Read more about this member.
    • Donate to Member
Restoring the good name and integrity of UK academic research work.
« Reply #20 on: June 18, 2013, 06:29 PM »
I wondered when the UK would take some action to stop the evident decline in academic research standards achieved over the last few years, as displayed in some sub-standard output from some UK universities and some other UK research bodies (e.g., in the domains of climate science and medical research). Looks like they have started to confront/address the issues by focussing on aligning funding allocation with research integrity. (As the saying goes, "Follow the money"?)
Better late than never, I suppose. Yay for Britain!    :up:
Let's hope it has some good effect. We shall see.
Britain’s Bad Science Scandal: UK Research Position Threatened By Fact-Fabricators
    Date: 18/06/13
    John Lawless, The Independent

Britain’s leading science institutions will be told on Monday that they will be stripped of many millions of pounds in research grants if they employ rogue researchers who fake the results of experiments, The Independent has learnt.

The clampdown comes as retractions of scientific claims by medical journals are on course to top 500 for the first time in 2013 – having been just 20 a year in the late 1990s, when Andrew Wakefield notoriously claimed that the MMR vaccine caused autism in children. In April, the UK’s first researcher was jailed for falsifying data over a prolonged period.

The Government is concerned that Britain’s prized second place in global research behind the US will be at threatened if more fact-fabricators are exposed. It knows that hundreds of thousands of jobs could easily go to foreign rivals if British laboratories do not keep coming up with new product ideas, to be made by major multinational companies in UK factories.

All of the country’s 133 universites and colleges of higher education are being forced to sign a new Concordat for Research Integrity – having been warned by major fund providers that those who do not will be refused access to more than £10 billion in research grants funded each year by British taxpayers – and as much again from the private sector.

A spokesman for Universities UK, which chaired negotations with the grant providers, said: “From next year, universities in the UK will have to prove compliance with the research integrity concordat in order to receive research grant. They are doing this to help demonstrate to government, business, international partners and the wider public that they can continue to have confidence in the research.”

Full article in The Independent: The bad science scandal: how fact-fabrication is damaging UK's global name for research.

TaoPhoenix

  • Supporting Member
  • Joined in 2011
  • **
  • Posts: 4,642
    • View Profile
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #21 on: June 18, 2013, 06:49 PM »

Separately from the money side, I have pondered a bit on different types of "weighted rating" systems for papers, then the reviewers. So very basically, Researcher A produces a Good Paper, and gets a "Good Researcher" score. Reviewer A reviews it correctly, and gets a Good Reviewer score. So in a simple world, if you get a paper from those two, you can generally "trust it".

But if one or the other goes rogue, then they will begin to collect "bad scores", and eventually like crying wolf game theory, they won't be believed even if later they start to turn around.

I know, everything can be gamed, but at the top of the basic theory, it becomes a shorthand for quality.

barney

  • Charter Member
  • Joined in 2006
  • ***
  • Posts: 1,294
    • View Profile
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #22 on: June 19, 2013, 02:42 AM »
Separately from the money side, I have pondered a bit on different types of "weighted rating" systems for papers, then the reviewers. So very basically, Researcher A produces a Good Paper, and gets a "Good Researcher" score. Reviewer A reviews it correctly, and gets a Good Reviewer score. So in a simple world, if you get a paper from those two, you can generally "trust it".

But if one or the other goes rogue, then they will begin to collect "bad scores", and eventually like crying wolf game theory, they won't be believed even if later they start to turn around.

I know, everything can be gamed, but at the top of the basic theory, it becomes a shorthand for quality.

Personalities weigh in as well, sometimes quite heavily.  For instance, say I dislike TaoPhoenix.  Any paper submitted with that name on it - even if it was research done by someone else! - is likely to get a bad review from me ... assuming I bother to read it!  Peer review has a place, as does any other form of review,  but consideration has to be afforded the attitudes of the reviewers, regardless their position(s).

mahesh2k

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 1,426
    • View Profile
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #23 on: June 19, 2013, 06:36 AM »
I believe there are absolute limits to human ability, and that there are phenomena in the universe that are not susceptible to scientific investigation, however advanced our tools become.

If we set such goalpost at first while making any theory and then try to base our assumptions thereafter, then it doesn't become critical thinking theory but more of theory based on beliefs and the goal post always changes when any proof threatens the goalpost placed earlier. Such reviews then become biased rather than critical. If the root of any theory isn't challenged it becomes biased and more personal.

barney

  • Charter Member
  • Joined in 2006
  • ***
  • Posts: 1,294
    • View Profile
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #24 on: June 19, 2013, 07:26 PM »
Y'know, it strikes me that DC is a source of much peer review.  Someone requests a coding snack, for instance.  One (1) of you takes up the challenge, offers a product.  Then, usually, several folk in addition to the requester critique the product.  Then the creator revises it, republishes it - often with code that purveyors can examine - and it goes through review again, often, multiple times.

Yeah, I realize that's not the greatest analogy, but the process is not dissimilar.  We have out naysayers - a few - and we have our innovators and we have our reviewers (wallflowers like me don't count  :P.).  There's not a significant difference, save for the range of recognition, methinks.