Main Area and Open Discussion > Living Room

Peer Review and the Scientific Process

<< < (10/47) > >>

IainB:
Thanks! I shall try that later. I use FlasgGot + Getright. I was download ing 230 or so items the other night and to my great surprise GetRight repeatedly crashed. I wonder how it will cope with 600 or so?

xtabber:
I then just now finally got around to watching the vidcast of Prof./PhD Don Easterbrook testifying before a hearing for a senate commission in Washington on 2013-03-26. He is a geologist. Start watching at 10 minutes and 30 seconds. Basically, using just raw, unadulterated data, Easterbrook explains to the senators all about "climate change" and why the theories, models and manipulated data (GISS, NASA, CRU) used by IPCC/CAGW alarmists are bunk. It's like watching a curious and highly rational child knock down a row of standing dominoes, each one onto the next.
Scam exposed. Time spent: approx. 1:20hrs, including Q&A.

Towards the end of it, even though he has kept mentioning that this or that point has been substantiated/verified by other scientists with whom he works, Easterbrook is asked if his work has been peer reviewed, and he says "Everything I have spoken about today, all this work, has been peer reviewed by other scientists, astronomers, physicists" (OWTTE).
He is also asked if he can explain how the IPCC with its peer-reviewed material can come to such different conclusions, and he politely says he can't explain it.
-IainB (October 04, 2013, 07:03 AM)
--- End quote ---
I'm not a geologist or a climatologist, so I won't address specifics, but watching the Easterbrook testimony reminded me of many people I have come across who have built what they believe to be an irrefutable case for some crackpot theory or another.  Unless you have a really deep understanding of the issues involved, it can be hard to challenge them because they know enough to sound as if they really do know what they are talking about.

As it happens, the geology department of Western Washington University, from which Dr. Easterbrook retired some time ago, issued a statement dissociating themselves from his testimony, which they describe as "filled with misrepresentations, misuse of data and repeated mixing of local vs. global records."  I'd suggest reading that before accepting his ideas as valid.

Renegade:
I then just now finally got around to watching the vidcast of Prof./PhD Don Easterbrook testifying before a hearing for a senate commission in Washington on 2013-03-26. He is a geologist. Start watching at 10 minutes and 30 seconds. Basically, using just raw, unadulterated data, Easterbrook explains to the senators all about "climate change" and why the theories, models and manipulated data (GISS, NASA, CRU) used by IPCC/CAGW alarmists are bunk. It's like watching a curious and highly rational child knock down a row of standing dominoes, each one onto the next.
Scam exposed. Time spent: approx. 1:20hrs, including Q&A.
-IainB (October 04, 2013, 07:03 AM)
--- End quote ---

Darn thing doesn't want to play for me. Either stalls or just won't play. I'll have to check back later.

IainB:
I'm not a geologist or a climatologist, so I won't address specifics, but watching the Easterbrook testimony reminded me of many people I have come across who have built what they believe to be an irrefutable case for some crackpot theory or another.  Unless you have a really deep understanding of the issues involved, it can be hard to challenge them because they know enough to sound as if they really do know what they are talking about.
As it happens, the geology department of Western Washington University, from which Dr. Easterbrook retired some time ago, issued a statement dissociating themselves from his testimony, which they describe as "filled with misrepresentations, misuse of data and repeated mixing of local vs. global records."  I'd suggest reading that before accepting his ideas as valid.
______________________________
-xtabber (October 04, 2013, 05:13 PM)
--- End quote ---

@xtabber: Thankyou for that comment. Being something of an information/data junkie by training and inclination, I followed up the link and other, related references on the issue that I could find. I found your comments and those at the link rather illuminating.
Your comment was in response to mine (above) where I refer to three instances of apparently peer-reviewed research seemingly being abused by publication bias (for whatever reason) or being otherwise abused, buried or even avoided so as to (deliberately or otherwise) obfuscate or pervert the valid conclusions of science and/or avoid testing and the risk of falsifiability:

* (a) in the fields of space-related science and climate science.
* (b) in the field of scientific medical research.
* (c) in the field of climate science.
Points that should probably be made here:

* 1. The subject of this thread: is categorically about Peer Review and the Scientific Process.
Thus, the issue is not "whether what Easterbrook says is true" but more like "whether this adds to our knowledge and understanding of the use of peer-reviews in the scientific process".
Whilst you profess ignorance of the facts or specific issues involved, your comments above would nevertheless seem to be arguing against or pointing to other people's "arguments" against the truth of what Easterbrook says in his presentation.
This would seem to be irrelevant to the subject - ignoratio elenchi (a "red herring" or genetic fallacy).
Furthermore you seem to have introduced:

* - argumentum ad hominem (argument against the person).
* - argumentum ad ignorantiam (forwarding a proposition without any certain proof).
* - argumentum ad populum (appeal to the people/consensus, popular sentiment - appeal to the majority; appeal to loyalty).
* - argumentum ad verecundiam (appeal to authority; conventional propriety).
* - non sequitur ("it does not follow"; or irrelevant conclusion: diverts attention away from a fact in discussion rather than addressing it directly.
______________________
Examples of outcomes of fallacy in "scientific" or "rational" thought:
Spoiler  "Man will never reach the moon regardless of all future scientific advances."
 -- Dr. Lee DeForest,  "Father of Radio & Grandfather of  Television."
 
"The bomb will never go off. I speak as an expert in  explosives."
 -- Admiral William  Leahy , US Atomic Bomb Project

"There is no likelihood man can ever tap the power of the  atom."
 -- Robert Millikan,  Nobel Prize in Physics, 1923
 
"Computers in the future may weigh no more than 1.5  tons."
 -- Popular  Mechanics, forecasting the relentless march of science,  1949
 
"I think there is a world market for maybe five computers."
 -- Thomas Watson,  chairman of IBM, 1943
 
"I have traveled the length and breadth of this country and talked with the best people, and I can assure you that data processing is a fad that won't last out the year."
 --The editor in charge  of business books for Prentice Hall,  1957
 
"But what is it good for?"
 -- Engineer at the  Advanced Computing Systems Division of IBM, 1968, commenting  on the microchip.

"640K ought to be enough for anybody."
 -- Bill Gates, 1981

"This 'telephone'has too many shortcomings to be seriously  considered as a means of communication. The device is  inherently of no value to us,"
 -- Western Union  internal memo, 1876.
 
"The wireless music box has no imaginable commercial value. Who would pay for a message sent to nobody in  particular?"
 -- David Sarnoff's  associates in response to his urgings for investment in the  radio in the 1920s.
 
"The concept is interesting and well-formed, but in order to earn better than a 'C,' the idea must be feasible,"
 -- A Yale  University management professor in response to Fred Smith's  paper proposing reliable overnight delivery service. (Smith  went on to found Federal Express  Corp.)
 
"I'm just glad it'll be Clark Gable who's falling on his face and not Gary Cooper,"
 --Gary Cooper on his  decision not to take the leading role in  "Gone With The Wind."

"A cookie store is a bad idea. Besides, the market research reports say America likes crispy cookies, not soft and chewy cookies like you make,"
 -- Response to Debbi  Fields' idea of starting Mrs. Fields'  Cookies.
 
"We don't like their sound, and guitar music is on the way  out,"
 -- Decca Recording  Co. rejecting the Beatles,  1962.

"Heavier-than-air flying machines are impossible,"
 -- Lord Kelvin,  president, Royal Society,  1895.

"If I had thought about it, I wouldn't have done the  experiment.  The literature was full of examples that said you can't do this,"
 -- Spencer Silver on  the work that led to the unique adhesives for 3-M "Post-It"  Notepads.
 
"Drill for oil?  You mean drill into the ground to try and find oil? You're crazy,"
 -- Drillers who  Edwin L. Drake tried to enlist to his project to drill for  oil in 1859.
 
"Stocks have reached what looks like a permanently high  plateau."
 -- Irving Fisher,  Professor of Economics, Yale University ,  1929.
 
"Airplanes are interesting toys but of no military value,"
 -- Marechal Ferdinand  Foch, Professor of Strategy, Ecole Superieure de Guerre ,  France .
 
"Everything that can be invented has been invented,"
 -- Charles H. Duell,  Commissioner, US Office of Patents,  1899.

"The super computer is  technologically impossible. It would take all of the water that flows over Niagara Falls to cool the heat generated by the number of vacuum tubes required."
 -- Professor of Electrical  Engineering, New York University

"I don't know what use any one could find for a machine that would make copies of documents. It certainly couldn't be a feasible business by itself."
 -- the head of IBM,  refusing to back the idea, forcing the inventor to found  Xerox.

"Louis Pasteur's theory of germs is ridiculous fiction."
 -- Pierre Pachet,  Professor of Physiology at Toulouse ,  1872

"The abdomen, the chest, and the brain will forever be shut from the intrusion of the wise and humane surgeon,"
 -- Sir John Eric  Ericksen, British surgeon, appointed Surgeon-Extraordinary  to Queen Victoria 1873.

"There is no reason anyone would want a computer in their  home."
-- Ken Olson,  president, chairman and founder of Digital Equipment Corp.,  1977.

--- End quote ---

I would therefore recommend the approach that I try to take, which is that one takes responsibility for making the effort to do one's own thinking, using the available data, rather than deferring to or allowing the thinking of others to be a substitute for one's own thinking.

In other words, one follows the Royal Society's motto: "Nullius in verba/verbo." Literally, "Take nobody's word for it; see for yourself".
Spoiler"Nullius in verba/verbo." Motto of the Royal Society, London. Literally, "Take nobody's word for it; see for yourself".
This motto indicates that currently, legitimate science seems to be based on the rejection of trust.
Thus, saying something purely on the basis of trust does not resemble genuine knowledge.
This is a new paradigm from the old, where scientific method can be seen to have developed from the 16th century perspective (Montaigne - no harm in the fact that "almost all the opinions we have are taken on authority and credit".) to the 17th century perspective (Gilbert, Bacon, Descartes and Boyle) where the approach is to take nothing on trust/authority.
So today we seek natural knowledge founded in evidence in nature - using individual reason - NOT in authority of tradition.
Thus real knowledge is NOT based on trust but on direct experience. - because reliance on the views of others produces errors.
The best scientist is thus incapable of functioning as a member of society.
The puzzle is that objective truth may exist, but human nature may preclude us from being able to experience it.

--- End quote ---

More on broken trust in peer review and how to fix it:
SpoilerShoring Up the Mantra of Science: Take Nobody's Word for It

Shoring Up the Mantra of Science: Take Nobody's Word for It
Ronald Bailey|Aug. 17, 2012 12:34 pm

Broken trust in peer review
The cited mantra is a general translation of "Nullius in verba," the motto of the British Royal Society, one of the world's first scientific organizations. Real science does not credit arguments from authority, but accepts the results from experiment and demonstration. The idea is that other researchers would check each others results to see if they could be reproduced. In the modern world, there's a lot less experimental replication and the result is lots of unreproduced experimental results are strewn throughout the scientific literature.

Earlier this year, two cancer researchers reported that that nine out of 10 preclinical peer-reviewed cancer research studies cannot be reproduced. As I explained in my column, "Can Most Cancer Research Be Trusted?":

    The academic system encourages the publication of a lot of junk research, and former vice president for oncology research at the pharmaceutical company Amgen Glenn Begley and M.D. Anderson Cancer Center researcher Lee Ellis agree. “To obtain funding, a job, promotion or tenure, researchers need a strong publication record, often including a first-authored high-impact publication,” they note. And journal editors and grant reviewers make it worse by pushing researchers to produce “a scientific finding that is simple, clear and complete—a ‘perfect’ story.” This pressure induces some researchers massage data to fit an underlying hypothesis or even suppress negative data that contradicts the favored hypothesis. In addition, peer review is broken. If an article is rejected by one journal, very often researchers will ignore the comments of reviewers, slap on another cover letter and submit to another journal. The publication process becomes a lottery; not a way to filter out misinformation.

The company Science Exchange has proposed its "Reproducibility Initiative" as an innovative way to fix this problem at the heart of experimental science. As Science Daily reports:

    Scientists who want to validate their findings will be able to apply to the initiative, which will choose a lab to redo the study and determine whether the results match.

    The project sprang from the growing realization that the scientific literature - from social psychology to basic cancer biology - is riddled with false findings and erroneous conclusions, raising questions about whether such studies can be trusted. Not only are erroneous studies a waste of money, often taxpayers', but they also can cause companies to misspend time and resources as they try to invent drugs based on false discoveries.

    "‘Published' and ‘true' are not synonyms," said Brian Nosek, a psychology professor at the University of Virginia in Charlottesville and a member of the initiative's advisory board....

    The initiative's 10-member board of prominent scientists will match investigators with a lab qualified to test their results, said Elizabeth Iorns, Science Exchange's co-founder and chief executive officer. The original lab would pay the second for its work. How much depends on the experiment's complexity and the cost of study materials, but should not exceed 20 percent of the original research study's costs. Iorns hopes government and private funding agencies will eventually fund replication to improve the integrity of scientific literature.

    The two labs would jointly write a paper, to be published in the journal PLoS One, describing the outcome. Science Exchange will issue a certificate if the original result is confirmed.

Here's hoping that lots of researchers will take advantage of this new initiative. For more background check out epidemiologist John Ioannides' 2005 classic article, "Why Most Published Research Findings are False" at PLoS Medicine.

--- End quote ---

I would also suggest that, in some contradiction to what you suggest, the evidence would seem to demonstrate that one categorically does not necessarily need to be a geologist or a climatologist - or, for that matter a member of any secular priesthood - to be able to contribute rational discussion on the subject of peer reviews or outcomes of climate science research, or other science for that matter. For example, many of the IPCC report authors/contributors over the years are or have been similarly and unashamedly not necessarily qualified and/or nor from what you might consider to be relevant scientific disciplines, but that apparently does not preclude their contribution - mistaken or otherwise (QED) - though some of them apparently do seem to consider themselves to be select members of a priesthood (QED per Climategate emails).


* 2. The need for critical thinking, reason/rationality when discussing the subject of peer-review in science:
Given the above, having even (say) 2 or so classical logical fallacies in a row would seem to be bad enough, but 5 or more would probably generally be regarded as going a tad too far and a pretty poor showing for the principles of rational thought.
Furthermore, as supposed substantiation of what you say or refer to, material from the link you provide would seem  to employ the use of some of these fallacies - apparently by scientists/academics.
Just to recap on the importance of this: Critical thinking helps us to look for the presence of a fallacy in a rational argument, which would indicate an invalid point in its logical structure. If a single point in a logical structure is invalid, then the whole structure is deemed invalid (not true). If groups of people employ or use the same logical fallacy/fallacies, then that does not substantiate or reduce the fallacy nor validate what they say - e.g., including groups advocating witch-burning, Lysenkoism, McCarthyism, Climate Catastrophism, Heaven's Gate, and Phrenology - though it might appeal to our confirmational bias or inherent irrationality (beliefs).


* 3. The implications of this for peer review in the scientific process:
The link you pointed to - statement - was illuminating, not so much for what refutation or truth it might have contained but for the light it shed on the use of:

* - argumentum ad hominem (argument against the person).
* - argumentum ad ignorantiam (forwarding a proposition without any certain proof).
* - argumentum ad populum (appeal to the people/consensus, popular sentiment - appeal to the majority; appeal to loyalty).
* - argumentum ad verecundiam (appeal to authority; conventional propriety).
* - non sequitur ("it does not follow"; or irrelevant conclusion: diverts attention away from a fact in discussion rather than addressing it directly.
* - peer review consensus ( argumentum ad populum and  argumentum ad verecundiam)- as mechanisms for making something out to be absolutely and unquestionably true because unfortunately they were for some reason unable to provide proof to do that (though the proof might have been available nonetheless).

Some conclusions we could arrive at here would include:

* A. Truth: You can't make something true out of a collection of logical fallacies. That would be an assault upon reason. Once you accept one invalid premise, you can accept infinitely more.
However, the depressing reality seems too often to be that many people are so unable to think rationally for themselves that they seem gullible to this kind of barrage of logical fallacy. One's head would be full of a confusing and probably conflicting mass of invalid premises, with ergo no real knowledge or understanding of truth.


* B. Peer review per se is not crucial as it cannot and does not certainly establish truth: We have already seen, in this discussion thread and others - e.g., including, the thread on CAGW, Thermageddon? Postponed! - that there is plenty of evidence to demonstrate pretty conclusively that peer review is an unreliable instrument for determining truth, as it can be and has been, and probably will continue to be used/abused to rationalise whatever careless or unethical/misguided scientists might want, because they cannot otherwise scientifically prove a pet theory or preferred/biased conclusion.
This is also well-documented in the literature - e.g., including as referred to in one of the spoilers above("…on broken trust in peer review and how to fix it").


* C. Falsifiability is crucial:
Falsifiability or refutability is the property of a statement, hypothesis, or theory whereby it could be shown to be false if some conceivable observation were true. In this sense, falsify is synonymous with nullify, meaning not "to commit fraud" but "show to be false". Science must be falsifiable. - Wikipedia.

--- End quote ---

As to a discussion of whether the IPCC theory/research and conclusions of CAGW are provable, or if the contrary facts/conclusions that Easterbrook was providing that it was falsifiable stood up to scrutiny - i.e., whether what the IPCC or Easterbrook are saying can be discredited/refuted on a factual and rational footing - I think that an evidence-based discussion of that would be very interesting, but is probably best be left to another, more relevant discussion thread, and so I shall continue with it in the discussion Thermageddon? Postponed!
(I hope that's OK.)

IainB:
If that part of the scientific process called "peer review" is per se not crucial, as it cannot and does not certainly establish truth (QED), then of what use is it?
From personal experience of having worked in a consulting environment where all one's technical reports had to undergo collegial peer review and standards review prior to going out to the client, my view is that peer review can be an extremely helpful process for giving one's output a final, rigorous sanity check. One is provided by a brief review report from each reviewer, which highlights any errors or omissions of fact or approach, which enables you to correct the report. We all make mistakes, so this was a useful checkpoint.
It was also just good risk-avoidance. As consultants, we were being billed out at a rate of (typically) $1,000 to $1,500 per day. A client might often make significant expenditure based on the recommendations/advice in our reports at the end of an assignment. If we gave duff advice, then we could have been liable for the consequences, so everything had to be rational, factual, substantiated and the potential errors/risks had to be stated. We had insurance to cover us for professional negligence or mistake, but if we ever had to claim against it, then the subsequent premiums would have skyrocketed and could have driven us out of business.

In science, the (non-crucial) peer review part of the process would seem to have not only a potential (desirable) governing effect, but also could have an enabling effect on the (undesirable) propensity for inherent bias/error/fraud - and the latter might not become apparent unless you checked the science for falsifiability - which is crucial (QED).
This necessarily forces you back to the Royal Society's motto: "Nullius in verba/verbo." Literally, "Take nobody's word for it; see for yourself".
The signs of bad/bogus science are only likely to be revealed if you follow the Royal Society's motto - and remember, the rule is: science must be falsifiable.

Interestingly, this discussion thread mentions some evidence (as "bad science") to show that for a great number of years, parts of the scientific community may have been only too well aware of the potential of and propensity for inherent bias/error/fraud in peer review (above), and of how to use it to promote lobbying for preferred religio-political or other biased conclusions in their "science".

Checking the science for falsifiability would seem to be about the only real test that we have at our disposal, and where scientists seem determined to avoid the risk of research being exposed to that test (e.g., refusing to reveal methods, or refusing/obfuscating FOI requests), one can presume that something professionally unethical is probably going on.
_____________________________
"The rule of thumb is that, if a business process can not stand the hard light of scrutiny, then there is probably something unethical about it". - Sir Adrian Cadbury (Chairman of the then Quaker family-owned Cadbury's) in his prize-winning article on Business Ethics for Harvard Business Review circa 1984.

--- End quote ---

Navigation

[0] Message Index

[#] Next page

[*] Previous page