topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • Saturday December 14, 2024, 1:03 am
  • Proudly celebrating 15+ years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Last post Author Topic: Peer Review and the Scientific Process  (Read 157617 times)

IainB

  • Supporting Member
  • Joined in 2008
  • **
  • Posts: 7,544
  • @Slartibartfarst
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #225 on: December 23, 2016, 05:22 AM »
@Renegade: Yes, as you say, of course it's garbage science, but some people might quite rightly ask: "What's wrong with publishing it?" The alternative might be, for example, to (say) call it "fake news" or something, and then censor and censure the authors and put them in prison for nothing more than being nutjobs or simply exceedingly barmy.

Fortunately we have stopped the inhumane process of locking all barmy people and nutjobs up. It's not their fault that they seem to occupy a widely different reality to the rest of us.
As a child, I was fascinated by the concept of monasteries and convents. I asked my mother why these men and women would lock themselves up like that with groups of people who all believed the same sorts of things. She said that they had anomalous beliefs and thus had difficulty fitting in with a society that did not share their beliefs, and found that they were happier in the sort of "tribal lockups" (echo chambers) afforded by the convent/monastery. There they were happy and their collective energies could often be directed to doing useful/good things for others - enabling them to live potentially productive/useful lives -  rather than them being inhumanely locked up and whiling their lives away in the quiet Hell of a lunatic asylum.

The problem comes though, when people with strongly-held anomalous beliefs and views take to a public stage and adopt a charismatic messianic stance, and people start to fall under their spell. So an Austrian nutjob like Hitler, for example, could lobotomise a whole nation of apparently intelligent German people, who would then blindly follow him to perdition. He could do no wrong, and therefore, by following him - and by extension - neither could they (QED) - as the Nuremberg trials revealed. (Ring any bells?)

Such leaders do this by creating a hateful false and artificial dichotomy between "people who think correctly and believe correctly as we do" and those who don't - the "Others"). The "Other" will be variously regarded as being stupid, evil, deplorable unbelievers, or similar, and thus they are not real people deserving of a life of freedom and tolerance, and so must be variously converted, rehabilitated, re-educated, imprisoned, beaten up, or (at worst) tortured and beheaded/killed and have their property expropriated. Since the "Other" is not a real person like we are, then there is no ethical wrong in treating them like the scum/deplorables/unbelievers that they are - it is entirely legitimate to do so. This is simple fascism, and it seems to have invaded and permeated religion, Science and politics alike - including (say) from recent scary examples in the US, the EU, and of course the Middle East.

IainB

  • Supporting Member
  • Joined in 2008
  • **
  • Posts: 7,544
  • @Slartibartfarst
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: Peer Review and the Scientific Process - bias, blindness and fraud
« Reply #226 on: January 15, 2017, 10:34 AM »
Fascinating paper here with more on the “replication crisis”. Helps to explain a lot as to how we can erroneously take something as "fact" when it is not proven.
Research: Publication bias and the canonization of false facts
Silas Boye Nissen Tali Magidson Kevin Gross  Carl T Bergstrom
University of Copenhagen, Denmark; University of Washington, United States; North Carolina State University, United States
DOI: http://dx.doi.org/10.7554/eLife.21451
Published December 20, 2016
Cite as eLife 2016;5:e21451

Abstract
Science is facing a “replication crisis” in which many experimental findings cannot be replicated and are likely to be false. Does this imply that many scientific facts are false as well? To find out, we explore the process by which a claim becomes fact. We model the community’s confidence in a claim as a Markov process with successive published results shifting the degree of belief. Publication bias in favor of positive findings influences the distribution of published results. We find that unless a sufficient fraction of negative results are published, false claims frequently can become canonized as fact. Data-dredging, p-hacking, and similar behaviors exacerbate the problem. Should negative results become easier to publish as a claim approaches acceptance as a fact, however, true and false claims would be more readily distinguished. To the degree that the model reflects the real world, there may be serious concerns about the validity of purported facts in some disciplines.

DOI: http://dx.doi.org/10...7554/eLife.21451.001
(...read more at the link)

I think this could also be relevant to the condition Scott Adams refers to with a new term -  "cognitive blindness":
"It probably does have a name. It’s a mix of cognitive dissonance and confirmation bias at the least, but a special case in my opinion."
...
"It isn’t that you disagree with the strong form of the argument on the other side so much as you don’t know it exists no matter how many times it is put right in front of you."

Such blindness may extend to self-serving fraud. In this example report, scientists/management investigated at the US DOE had apparently planned a scheme that deliberately withheld relevant information from Congress, so that members of Congress could not see the strong form of an argument that risked disproving the DOE's management's preferred "truth" and politicised line of research funding: Dept of Energy gov’t scientist fired for answering Congressional questions contrary to DOE management views.

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,291
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #227 on: March 04, 2017, 09:30 PM »
Such blindness may extend to self-serving fraud. In this example report, scientists/management investigated at the US DOE had apparently planned a scheme that deliberately withheld relevant information from Congress, so that members of Congress could not see the strong form of an argument that risked disproving the DOE's management's preferred "truth" and politicised line of research funding: Dept of Energy gov’t scientist fired for answering Congressional questions contrary to DOE management views.

I think you're right about outright fraud, but what might be even worse is simple error in "settled science".

I'm quite a fan of the Thunderbolts Project.

http://www.thunderbolts.info/

http://www.holoscience.com/

https://www.youtube....qXK_Hz79tjqRosK4tWYA

If you go through their materials, it's flat out terrifying that we've made so many fundamental errors in science.

The most recent video walks through some issues:

https://youtu.be/Bl4fVY2d5ok

In particular, here:

https://youtu.be/Bl4fVY2d5ok?t=400

The 3 points he makes at 7:30 can't be ignored.

We should re-examine basic science as it appears that much of what we think we know is simply wrong.

Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

IainB

  • Supporting Member
  • Joined in 2008
  • **
  • Posts: 7,544
  • @Slartibartfarst
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #228 on: March 05, 2017, 12:34 AM »
@Renegade:
The Thunderbolts Project looked interesting. Thanks. Good questioning approach as well - so, useful for educational purposes and for stimulating the enquiring mind (like my 15 y/o daughter's).

If the scientific error seems to be present as:
(a) deliberate error in cases of outright fraud - apparently in some areas of commercially-driven science and at least that one US government department (QED).
(b) errors in the application of the scientific method - e.g., in cases of gross error in other supposedly scientific institutions, "settled science", etc.

 - then, I find it not so much "terrifying" as simply appalling that, when one goes go through their materials, one can see so many egregious fundamental errors in scientific conclusions.

The quote they give is rather pertinent:
"l know that most men, including those at ease with problems of the greatest
complexity, can seldom accept even the simplest and most obvious truth, if
it be such as would oblige them to admit the falsity of conclusions which
they have delighted in explaining to colleagues, which they have proudly
taught to others, and which they have woven, thread by thread,
into the fabric of their lives."
— Leo Tolstoy

We seem to invest so much of ourselves into our beliefs/paradigms that we inhibit our ability to think freely/critically, where nothing must be allowed to contradict those beliefs - which have (to us) become seemingly sacred truths. Edward De Bono called this "intellectual deadlock", and saw it as the single biggest obstacle to one being able to improve one's thinking skills (ref. "Teaching Thinking", by Edward De Bono). He theorised that, at bottom, it was the ego protecting itself.

As far as De Bono could see, this self-crippling phenomenon had a tendency to occur in the minds of most people, but from experience was more likely in intelligent people  - regardless of profession (i.e., whether they were business people, politicians, scientists or peer reviewers. etc.).

EDIT: 2017-08-03 - Major correction to wording in the last para re "fallacy".
And this is before one considers the self-evident fallacy that peer review proves the truth of some research, when in fact it proves nothing except that a review has taken place (QED). The material reviewed could still be false/flawed.
« Last Edit: August 02, 2017, 08:37 PM by IainB, Reason: Major correction to wording in the last para re "fallacy". »

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,291
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #229 on: March 05, 2017, 11:23 PM »
Ego is definitely an issue. And a difficult one. Admitting that you're wrong is hard. The first point in being honest, is being honest with oneself.
Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

IainB

  • Supporting Member
  • Joined in 2008
  • **
  • Posts: 7,544
  • @Slartibartfarst
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #230 on: August 02, 2017, 08:47 PM »
There is an interesting bit of psychological research that has been done on the attention-distracting effect of things like smartphones, as reported on here: Your iPhone Reduces Your Brainpower, Even If It's Just Sitting There.

Rather than the research itself, or its conclusions, I find this comment honest and very wise - in light of what we now know about the falsifiability of a lot of psychological research: (my emphasis)
...“We can yell our opinions at each other, and people are going to agree or disagree with them, and set up luddites-versus-technolovers debates. But I wanted to get data,” he told me.

It’s worth noting that the type of psychological research Ward conducts—trials on willing, Western undergrads, often participating in studies to fulfill course credit—has suffered a crisis of confidence in recent years. Psychologists have had difficulty replicating some of the most famous experiments in their field, leading some to argue that all psychology experiments should be replicated before they are published. Ward’s study has not yet been replicated. ...

« Last Edit: August 03, 2017, 03:34 PM by IainB, Reason: Typo correction. »

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,291
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #231 on: August 22, 2017, 12:02 AM »
@IainB - I suppose I shouldn't find any of that surprising. Hey... let's do a study! And pay people! Surely people willing to do our study for f**k all aren't any different than people that have productive and meaningful lives and that couldn't give a s**t less about us!!!

The idiocy is deafening.

For the most part, science is dead.
Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

IainB

  • Supporting Member
  • Joined in 2008
  • **
  • Posts: 7,544
  • @Slartibartfarst
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #232 on: August 30, 2017, 05:01 AM »
...For the most part, science is dead.
___________________________________
Well, maybe. Here's a scientist who persists in calling out the weaknesses in peer review. He's about to be fired for blowing the whistle, it being a crime - "uncollegial", or something. Looks like a witch-hunt: Concerned About Prof Peter Ridd - Jennifer Marohasy

Here he is in a recent radio interview. Worth a listen: University Professor censored

He has consistently been saying the same thing for ages: Peer reviews are a poor form of quality control, and bad science may be falling through the cracks because of peer review.
It's all about money/status, apparently.

IainB

  • Supporting Member
  • Joined in 2008
  • **
  • Posts: 7,544
  • @Slartibartfarst
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: Peer Review and the Scientific Process
« Reply #233 on: October 16, 2017, 12:52 AM »
Examining the scientific method, this post (link follows) looks at how traditional peer review seems to be failing. It seems to be correct, as well. A very interesting synopsis: Misuse of the scientific method has led to peer review failures with significant implications

I still hold out hope that reason will prevail.