May 19, 2022
A site that tells you if the studies you find are bullshit or not
In short, this idea is to provide a web service where someone can put in a title or link to a study, and then get a report that shows the flaws (low confidence, small sample size, dubious methodology, etc.) as well as any positives (popularly referenced, independently verified, etc.).
That way, if you're discussing a topic and get a study referenced to you, you don't need to do an in-depth literature review before continuing.
1.8 million articles are published in scholarly journals every single year. Even if you are a professional researcher in a certain field, it's unlikely you'll be able to consume all of the literature published in a year. Not to mention all of the literature from history.
Furthermore, it's unreasonable to assume that someone who is just discussing a topic will have the necessary academic background to break down a report and determine its veracity.
Instead, one of two things will likely happen:
Either way, our ability to use science to advance our collective knowledge hangs in the balance.
This is a service that would help rectify this problem.
Let anyone put in the link to a study, and give them a report. Assign a score for reliability, show the weak points, show the strengths.
It would have to be a thoroughly unbiased site in order for it to be effective. But, if you can achieve this feat in the same way as Snopes, you can change argumentative discourse for the better.
Here's how I think you could do it. Bipartisanship is super important. If you get branded as a liberal media outlet or a right wing conspiracy forum, your days are numbered.
First, you should find three sympathetic politicians -- one from each of the major parties. They will be the founding members and your advisory committee.
This will have the first-order effect of getting you some publicity. Any bipartisanship is newsworthy.
The second-order effect will be that, assuming the members of that party like the politician, your site will immediately gain credibility from all sides. They may not love that you also work with the other party, but they will respect it.
So now you have a committee. But how do you actually make and promote the product?
These are certainly the more challenging dimensions. You'd obviously need to hire researchers to write the report. But how do you choose the articles? Easy. Just scrape the news for the most often referenced studies of that day / week. These are the ones people will be searching for.
How do you promote it? This is way harder. My suspicion is that the best way would be to hire a ton of workers to post the link to your review anytime one of these studies is mentioned. They'd need to be a bit inflammatory to get anyone to click ("Hey that study you posted is total bullshit! _____ only gives it a 42 for reliability. Even papers on *insert something very wrong here* got a 60! link")