Good article. Cochrane has its place but also has its issues. I like the phrase mentioned in the article “methodological fetishism” to describe their style.
The bigger issue for me is the anti-science stance so many are pushing/following now, but whenever one study comes out reinforcing what they already wanted to believe they pounce on it. That’s not how the scientific consensus works and that “methodological fetishism” fed right into it. We still need experts’ help to understand what’s significant.
“methodological fetishism"
On a lighter note: Hey, no kink shaming, Naomi 😸
Oh I agree. I’m coming at this from the medicine side though and there’s some specific concerns and criticisms for Cochrane reviews and the meta analysis method in general, though on the whole I generally agree with what they’re going for. A really simplistic example but I think illustrates the point well would be this:
https://www.bmj.com/content/327/7429/1459
A meta analysis of the effectiveness of parachutes concludes there is no evidence to suggest their effectiveness, due to lack of high quality double blind randomized controlled trials.
I’m a huge proponent of evidence based medicine, but sometimes people are quick to point to a Cochrane review or other systematic review as biblical scientific dogma that cannot be defied, and don’t consider the nuances of the situation. Like what went into the meta analysis and why? What data might be missing from the meta review and why? Is only including double blind randomized controlled trials the best research method to answer the specific question being addressed? Was everything included together actually comparable? Was the specific question the review asked the right question to ask? How broad can this finding actually be applied?
Part of the problem in this case though was the Cochrane reviews findings being somewhat misrepresented in the popular media by those not familiar with the format or the nuances. No doctor is gonna read that review and rip off their masks when treating covid, flu, or other airborne or droplet diseases. Masks were used in the hospital long before covid and will continue to do so, for good reason.
what went into the meta analysis and why? What data might be missing from the meta review and why? Is only including double blind randomized controlled trials the best research method to answer the specific question being addressed? Was everything included together actually comparable?
The critiques / rebuttals to the masking review typcially ask these questions. This review almost falls apart after considering the answers.
If there’s not enough information to form a solid conclusion, maybe they have no business analyzing it until there is. At the very least, if they’re going to include partially unrelated studies, then reflect that in the title / opening statement. Don’t say the analysis is of apples, then analyze fruit in general.
