Sabine
patreon
Do your own research. But do it right.
🕑 Added 2023-08-26 12:00:04 +0000 UTC[This is a transcript with links to references.]
I’ve noticed that a lot of scientists on social media make jokes about people who do their own research. This has been going on ever since COVID and it’s only been getting worse. We’re now at the point where they’re claiming you’re only allowed to talk about a topic if you’re an expert yourself. Don’t you dare commenting on anything you don’t have a PhD in!
I think this nonsense has been going on for long enough. So I’m here to say: It’s of course completely okay if you do your own research, provided you do it right. But how do you do that? That’s what we’ll talk about today.
This video is not a guide for writing a PhD thesis, setting up a placebo-controlled drug trial, or building a particle collider. I’m not talking about how to do a scientific research project. I’m talking about how to research resources that you can for the most part find online. It’s this context in which the phrase ‘doing your own research’ has become common.
As with most problems on social media, the idea that you shouldn’t talk about topics you don’t have a degree in started from a good place and then went horribly wrong.
It began in the early days of the COVID pandemic when few people knew what was going on, but they were difficult to hear amidst the noise of non-experts who generously handed out uninformed opinions. People who “did their own research” became a running joke for good reasons, and quite a few of them ended up on emergency wards.
Unfortunately, the joke continued after the reasons were gone. And so here we are today, where you’re not allowed to quote from governmental reports on, say, gender affirming care, because that counts as “doing your own research”.
And yes, I am talking about this because it personally affects me. I frequently get people complaining, both here and on other social media, that I’m talking about topics I’m not an expert in. Marijuana, climate change, obesity, bees, ai, neurodiversity. I don’t know much about any of those, or at least I didn’t until I had to figure out how to pronounce tetrahydrocannabinol, yet that hasn’t stopped me. Some people really really don’t like that. You’ve probably seen them in the comments here saying I should “stick to physics”.
Indeed, why doesn’t Sabine stick to physics? I talk about other topics not because I like to annoy people. Though, now that I think about it, that’s definitely a bonus. But mostly it’s because I like to be well-informed about what’s going on in science, engineering, and technology, broadly speaking, not just my own niche in physics. And I’ve found it difficult to find good information, that is (a) comprehensible (b) doesn’t dumb things down, is (c) politically neutral and (d) brief. So I decided to do it myself.
Unlikely as that might have seemed three years ago, I’m now the owner of a science news business. When I talk on this channel, I don’t talk as a physicist, I talk as a science communicator with 20 years’ experience backed up by a team of script writers and fact checkers. Of course, I talk about topics I’m not an expert in myself. Do you think the guy in the evening news is an expert on child soldiers in Congo?
But I’ve found it difficult to find people to work for me who are good at doing their own research. And so I was thinking, I’ll do a video and explain how to do it.
Before we talk about doing your own research, let’s talk about when *not to do it. That’s when there’s nothing to research, when there’s no information online, other than that coming from people who know more than you do.
The early days of the COVID pandemic were an example of that. If you didn’t know your way around with virology and epidemiology you couldn’t make sense of the little data that were available. Another case may be when information is classified. Nothing to research, so all you find online is just noise. In such cases, the best thing you can do is shut up listen to the experts. This is why I don’t talk about UFOs.
This brings up the question of how to recognize an expert. That’s a good question but it would lead us somewhat astray. If you’re interested in it, let me know and I’ll do another video on it.
A big part of doing your own research begins before you research anything. Before you start, spend a moment to think about what you can reasonably expect. It takes about 10 years to do a PhD. It takes another 5 to 10 years to get a good overview on a research area. You’ll not reach the same level of understanding by spending a few hours on Google. No, you won’t.
You’ll have to draw a line somewhere, usually that’s at a point where it just doesn’t matter for your purposes. For example, when I was talking about whether to legalize marijuana, I didn’t really need to know the biochemical pathways of THC. When I was talking about climate models, I didn’t really need to know how to numerically integrate the Navier Stokes equation.
I find that it helps to make a list of concrete questions I want answers to before I even start. That prevents me from going down rabbit holes. Sometimes.
Besides having reasonable expectations, I also have to constantly remind myself to be honest with myself and acknowledge what I don’t understand. I know from past experience that if I start glossing over incomprehensible sentences, that much increases the risk that I make mistakes.
A good example is the blunder in my video on the greenhouse effect, where I said that the gravitational force decreases with !!altitude and that’s why air density decreases.
Now it’s right that the gravitational forces decrease with altitude and it’s right that the air density also decreases with altitude, but the former isn’t the reason for the latter. Obviously not. If you think about it for more than a millisecond that doesn’t make any sense, the density would decrease even if the force didn’t depend on altitude.
And why did I get this wrong? Because the combination of words sounded kind of plausible and I didn’t think about it. I left the video up the way it is because it doesn’t matter for anything else I said there, but I’m still embarrassed about it.
Another example is the story about how the core of Earth supposedly started spinning in the other direction, remember that? Someone from my team put this into the script that I was supposed to read. I couldn’t see how that was possible. Where did all that angular momentum go? How could the core suddenly start going the other way when it’s coupled to the mantel through a liquid layer in between. Just like that?
Wouldn’t you at least expect a very good explanation for how this can possibly happen? I didn’t understand what my script writer had written, so I had a close look at the paper. And sure enough, it turned out that the paper wasn’t about the absolute rotation of the core, but the rotation relative to that of the mantel. The core of earth used to spin a little bit faster than the mantel, now it seems to spin a little slower. And yes, that actually does have a tiny effect on the length of the day. Suddenly it all made sense. Now I understood it.
Why did I not just read what my script writer had written and make the same mistake as many others? Because I didn’t understand it and I didn’t want to say something I didn’t understand.
So when you do your own research, keep track of what you do and don’t understand.
Something else to get out of the way before you start is acknowledging your own biases. If you’re doing research on a topic that you have previously put forward strong opinions about, you might be reluctant to admit that you were wrong. The best way I find to deal with this is to remind myself that admitting to being wrong is the hallmark of a good scientist. Yeah, I still don’t like it. But it does help a little. I find it difficult especially with topics that I have worked on myself, such as specific dark matter models.
Another factor that can bring in bias is if you or someone you know is personally affected. Suppose you had an expensive and painful procedure that a doctor said would help you, but new research finds that this intervention doesn’t have any clear benefits. You’re likely to insist that it worked for you anyway, because that makes a good psychological protection. And that’s fine if it’s just your personal way of dealing with it. Not so fine if you tell others the study is wrong because *you don’t like the result.
So we have reasonable expectations, honesty with yourself, acknowledging biases. And that puts us in a good position to start with the actual research.
I suggest you start your research by looking for peer-reviewed review articles, governmental reports, lecture notes, and textbooks. Then. You read them.
Yes, this is going to take time. No one said it would be easy! But you don’t have to read all of this in great detail. Those sources usually begin with an overview, and most of the details that follow later won’t be so important for you. The important thing to take away is what the key questions are that you need to pay attention to.
A review article is basically an overview on the status of research, and some scientific journals specialize in them. The nature group publishes a lot of them, and if you can find one on the topic you’re interested in, that’s usually pure gold.
You find governmental reports either on the governmental websites or that of the ministry of science and education or whatever it’s called in your country. Other good resources are specialized national institutions such as the National Academy of Sciences in the USA or the Fraunhofer Institute in Germany.
After you know what you need to pay attention to, you can move on to meta-reviews, if they exist in the field you’re looking at. A meta-review is a summary of previous research studies. Meta reviews can be great and save you a lot of work, but enjoy them with some caution. A summary of crappy studies does not make a good study. You should only trust meta-reviews if they assess the quality of the studies they summarise, and not just lump together different data.
And after that you can move on to the most recent studies. Then you should have a pretty good impression of what is going on in the field and be well prepared to put new stuff into context.
Okay, now let’s talk a bit about how to search for scientific literature and how to understand what you’re looking at.
First of all, if you’re searching for scientific literature, I suggest you don’t use Google’s main interface, but you use Google Scholar. Indeed, in many cases Google will suggest that you switch to scholar. Google scholar shows you quickly whether a paper has been published in a scientific journal, and it has an extremely useful feature which is that it lets you search for articles that have referred to an earlier article.
This is great if you have a fairly old reference, and you want to know what’s happened recently. All you have to do is search for all papers that have cited the old one, and sort them by date. I did this a lot when I was working on the video on collective intelligence. That’s because only thing I knew about the topic were some of the classical studies in the field, and looking for recent papers that cite the classics is a good way to find reviews.
Next thing you need to do is figure out whether an article has been peer reviewed. At present, almost all peer review is done at submission to a scientific journal. If the paper passes peer review, it will be accepted for publication. So you have to check whether the paper has a journal reference, and if it’s a decent journal. It can be difficult to figure out what’s with the journal, but if you don’t know the journal, at the very least check if it’s on the list of predatory journals because such journals will basically publish anything for money.
Also make sure that what you’re looking at is not a pre-print server. Pre-print servers should really be called pre-peer review servers. It’s where scientists upload papers before they submit them to a journal. You should stay away from pre-prints unless you can consult an expert. The most widely used pre-print servers are the arXiv in physics and related disciplines, and the bioRxiv in the life science, though there are a few others.
Also be very careful with reports or surveys from non-profit organizations or corporations because these are rarely peer reviewed. It’s not that they’re necessarily wrong, but that they often don’t disclose how they arrived at their results, and there’s a risk they have omitted relevant information.
Conference proceedings can be difficult to understand because their purpose strongly depends on the field. In some areas, like particle physics, conference proceedings are where you dump preliminary results or summaries of your own work that wouldn’t get published in a journal anyway, not because they’re wrong, but because they’re not new or not interesting enough.
In other fields, like engineering or computer science, conference proceedings are the final product. They do their research, present it at a conference, write it up and publish it in the proceedings. And that’s that. They’re usually peer reviewed, or at least they’re supposed to be, so don’t dismiss conference proceedings. In some fields they’re vital.
Let’s then talk about some things you should not do.
The number one mistake that I see among people who “do their own research” is that they just pick one or two papers that kind of say what they want to say and leave it at that. This is not how research works.
The second most important “don’t” is to never trust a second-hand reference. Don’t trust websites referring to other websites, press releases referring to papers, newspapers referring to reports, or magazines quoting scientists. Don’t trust books referring to other books, don’t trust Wikipedia, and don’t trust people talking on YouTube. You have to track every piece of information down to its source, and then check how reliable that source is.
That is not to say that second-hand references are useless. They are pretty good for pointing you into the right direction. Wikipedia in particular is a good starting point to launch into the scientific literature. Just don’t make it the end point.
My final advice for today is that if there’s data, look at the data, don’t trust the text. I see this all the time that data in a paper don’t support what the abstract or the conclusion says. For this you’ll have to know what p-values and R-coefficients are and stuff like that, and that would make for another video entirely. So for today let me just say that while outright scientific fraud is rare, it happens all the time that written text is sloppy or doesn’t properly represent the data.
Ok, in summary. Before you set out to do your own research, clarify for yourself which questions you are trying to answer and if you have a reasonable chance of answering them in the first place. Acknowledge the limits of your understanding and possible biases. Once you’ve done that, start with the basics and try to get an overview by help of reviews and reports. Then work your way up to more recent publications. Don’t cherry pick studies, don’t trust text if there’s data, don’t trust second hand sources, and don’t trust me either. Did I forget anything important? Let me know in the comments.
Comments
I agree and I think that it's very sad. The whole substack world of news and reporting has turned out to be highly limited because people like Matt have lost whatever credibility they once had from working with legitimate outlets and so are working 'remotely', siloed with little to no contact with real sources but conspiracists.
I used to really like Matt Taibbi's reporting once upon a time, but he really fell down a conspiracy nutjob hole these last few years.
At the start of the Covid-19 pandemic, I knew of Vincent Racaniello, a virologist, from his coursera.org virology course and so I searched for him and found his microbe.tv web site and got my pandemic information there. I found that the media was not competent to discuss most of the news that they were and so provided them links to his site so that they could a) get better information to educate themselves and b) contact him directly. I doubt any of them did and know for an absolute fact that those in the 'alternate' news universe, such as Matt Taibbi, rejected that outright, for whatever reason, maybe because their substack universe is built on conspiracy and not on providing true, as in objective reality, information.
good to know, whom you trust, above all, if it´s about health. BigPharma does a lot research (Forschung) for the health of all, and got to be payed, but also made a lot of mistakes and lies, that led to mistrust in the population. It reminds me at the errors of the nuclear industry in my country.
I've been told by anti-pharma quatsch types to 'do my research', my 'research' involves specialists who want to save my life and GPs who know more than these pflanze-kopf fools ever will. It's annoying, frustrating and kind of depressing.
English is a poor language sometimes: in german there are two different words for "research": "Recherche" (which is a french loanword) and "Forschung", two quite different topics. That leads to some confusion in the social media comment sections, as I could notice.
There is an area of possible physical research that is greatly underestimated. These are logical errors in current physics which have an amazingly long life because physicists do not dare to reconsider them when they are so firmly accepted by the community. Here is a well-known historical case: In 1932, John von Neumann presented a proof that the notion of a deterministic world is incompatible with quantum mechanics. This proof attracted great interest from all the major proponents of QM such as Heisenberg and all his colleagues in the field, and it was widely discussed. It took a full 33 years until John Bell showed that there was a serious and also easily detectable error in this proof. This demonstrates how definite errors in physical theories are never questioned again after general acceptance. And there are other cases, e.g., in Einstein and de Broglie, which are still undiscovered. And even when well-known researchers (professors serving in universities) have raised these cases, they are reliably ignored by the mainstream. So it's worth investigating these cases. Even though it is to be expected that working on these cases may take a lot of effort, it is necessary and therefore valuable work.
Excellent point. But Sabine H., to all appearences, is human. So it applies to her at least the beginning of the saying: "Errare humanum est, perseverare autem diabolicum." As she does not persevere on being wrong, as far as I've noticed, she is not exactly diabolical. Also the first part, in English, often has this, added by Alexander Pope: "To forgive is divine." I suspect she is not divine, either. A good explanation and history of the saying: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2842843/
Tanj
There comes a point where you need to be familiar with how the subject you are looking at works, to understand the publications. For example, Sabine's recent essay on transgender swallowed without filter some biased works based on talking to parents selected from biased pools, and which never talked to the transgender people themselves, and those papers were then given unwarranted weight in Sabine's discussion. To understand the flaws in those papers you need to be alert to methodology pitfalls for the subject (though, lack of peer review should have been a flag, too). If you do choose to research unfamiliar subjects you will need to put in the work to gain some ability to ask questions about the methods, not be in a rush. It can help to prefer papers which are old enough to have been referenced, so you can then read what else was said about the paper, whether it builds up or was disputed. This will limit your use of the most cutting edge - but then, that would be where being an expert is most needed, and as this blog was about doing research where you are not an expert.
We have to live with such people and even may need them for something or other, for example a good car mechanic, or in the case of an old friend or a close relative who is, overall, a fine person. With them, when they mention something as fact that I think is not, or is clearly a conspiracy theory or a superstition, I either do not comment, or if necessary, merely state what I believe or know, so there are no misunderstandings. But do not try, or encourage the other to engage further on the subject. So telling it as I see it, but no preaching or teaching. For all I know, this might even work for the better.
It helps though, to be able to have the tools to try to debunk stuff that hasn't been accepted to a peer reviewed publication, but that has still been published (or aired as a podcast or video) on the internet and affected millions of people. There are many people who are skeptical of mainstream science, but totally gullible to any scientist who dissents from the mainstream consensus. Why not be an equal opportunity skeptic, instead of a biased towards dissention and against mainstream skeptic? They have no tools to debate and debunk such claims, and want to believe them because of all kinds of biases having to do with their political orientation. I've seen this with both covid and climate. On the other hand, they are not swayed if a new analysis contradicts their favorite dissenter. So maybe it's a waste of time.
As Sabine mentioned, a basic understanding of statistics and statistical variables such as p-values and R-coefficients can be very important when evaluating scientific articles. A good book that explains how statistics are used (or misused to intentionally mislead) is Risk Savvy: How to Make Good Decisions by Gerd Gigerenzer.
