{"id":286,"date":"2016-06-03T00:48:30","date_gmt":"2016-06-03T00:48:30","guid":{"rendered":"http:\/\/www.authorfreeman.com\/blog\/?page_id=286"},"modified":"2022-01-07T19:19:42","modified_gmt":"2022-01-07T19:19:42","slug":"how-science-works","status":"publish","type":"page","link":"https:\/\/www.authorfreeman.com\/blog\/how-science-works\/","title":{"rendered":"How to study studies"},"content":{"rendered":"<p><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" class=\"aligncenter\" src=\"https:\/\/i0.wp.com\/www.authorfreeman.com\/blog\/wp-content\/uploads\/2016\/06\/science.jpg?resize=600%2C225&#038;ssl=1\" alt=\"science\" width=\"600\" height=\"225\" \/><\/p>\n<p>There\u2019s a lot of cynicism about science these days, fed by every headline that declares one day that some food is good for you and the next that it\u2019s bad for you, every report about some researcher faking his experimental results, and every \u201cstudy\u201d trotted out by some con artist claiming that his new supplement will cure all ills. (Note: throughout this post, I\u2019m going to use \u201cstudy\u201d and \u201cexperiment\u201d interchangeably, though they don\u2019t quite mean the same things, and I\u2019ll be thinking mostly about biological research, though what I say will apply to greater or lesser degrees to the other branches of science.) It can seem to some like science is not working very well, or worse, that it\u2019s fundamentally flawed. And indeed, anyone trying to devise an experiment to discover a truth about the world faces many potential difficulties, including:<\/p>\n<ul>\n<li>Flawed design. It\u2019s really, really hard to design an experiment that proves anything complex enough to make a real difference in the world.<\/li>\n<li>Human error. Even well trained, well intentioned scientists can make mistakes in the conducting of experiments.<\/li>\n<\/ul>\n<ul>\n<li>Randomness. Especially if you\u2019re experimenting on living things, you\u2019re going to get some degree of randomness. Life is complex, and one frog might not react exactly the same as another frog.<\/li>\n<li>Unintentional bias. Until our robot overloads take over, scientific experiments are going to be designed, conducted, and interpreted by human beings with conscious or unconscious biases.<\/li>\n<li>Intentional fraud. Yes, it does happen that scientists will fake results, whether for purely monetary reasons (e.g. to advance their own careers or enable some scam) or ideological ones. (To support some preexisting view they hold.)<\/li>\n<\/ul>\n<p>Faced with these obstacles, we might doubt that science can prove anything, but what we\u2019d be forgetting is the power of human intelligence. We\u2019re a problem solving species, and over the centuries, we\u2019ve developed a process for deriving truth from the messiness of experimental science.<\/p>\n<ol>\n<li>We design our experiments in accordance with a set of best practices that we\u2019ve continue to refine over the years. Two good examples of these practices would be the requirements that studies involving people be \u201c<a href=\"https:\/\/en.wikipedia.org\/wiki\/Blind_experiment#Double-blind_trials\" target=\"_blank\" rel=\"noopener\">double blinded<\/a>\u201d and \u201c<a href=\"https:\/\/en.wikipedia.org\/wiki\/Randomized_controlled_trial\" target=\"_blank\" rel=\"noopener\">randomized<\/a>.\u201d A more recent example would be the requirement that researchers state their objectives ahead of time to avoid the practice of \u201c<a href=\"http:\/\/www.authorfreeman.com\/blog\/2016\/06\/06\/science-working\/#p-hacking\" target=\"_blank\" rel=\"noopener\">p-hacking<\/a>.\u201d As we discover new ways that studies can be flawed, we develop new guidelines to help us avoid those pitfalls.<\/li>\n<li>We require scientific papers to be peer reviewed before they can be published. This allows experts in the same field to confirm that the design of the study or experiment was sound and that the researchers properly processed and interpreted the data.<\/li>\n<li>We talk non-judgmentally about the strengths of studies. A study that followed all the best practices and involved a large and sufficiently random sample size would be considered strong, while a study that involved a smaller sample size or that had some design flaws might be thought of as weaker. However, \u201cweak\u201d is not necessarily a pejorative. Many studies are intentionally weak, because it\u2019s more expensive and time consuming to conduct a strong study, and in a world of limited resources, we can use weaker studies to probe for new areas that might be worth more rigorous testing later.<\/li>\n<li>We attempt to reproduce experiments. This is such an important point that it bears repeating. We attempt to reproduce experiments! The biggest misunderstanding that non-scientists have about science is that it relies on single studies to prove anything. After a study is published that seems to prove some new thing, other scientists try to repeat the experiment, following its steps precisely and seeing if they get the same results. This helps to address almost all the problems that studies can have: human error, randomness, unconscious bias, and intentional fraud. Every time a study is reproduced successfully, it becomes less likely that the same errors are being made, or that the randomness is breaking the same way every time, or that the new experimenters share the biases or evil intentions of the original team.<\/li>\n<li>Only when a sufficient strength of findings clusters convincingly enough around a given point do we consider that the point has been proven. And the key word here is \u201ccluster.\u201d It\u2019s expected that whatever the truth of a question is, the experiments trying to prove it will lead to results that have some randomness in them. So, for example, if a given vitamin can, in truth, extend the average human life span by 1%, then we should expect that some weaker studies might even come up with answers below zero (i.e. that this vitamin is harmful) while others might show that it has no benefit or a much bigger benefit. But the critical question is: after we adjust how much weight we put on a given study based on its strength, do all the studies generally cluster around the same point? If so, then that point is probably the right one.<\/li>\n<\/ol>\n<p>Most of the \u201cproblems\u201d we see in science are based on a misunderstanding about how it\u2019s supposed to work. We think that experiments are supposed to be self sufficient proofs unto themselves, when they are really just pieces of a very big and complicated puzzle. In fact, the most serious legitimate problem with science today is that not enough researchers are doing the unglamorous but necessary work of replicating past studies. In an ideal world, science would have the funding, and scientists would have the incentives, to complete the process outlined above for everything we want to know \u2013 and then, ironically, there\u2019d be even more cases of apparently conflicting studies that the news media could sensationalize and that na\u00efve readers could take as signs that science was broken. But then, in that ideal world, everyone would also understand that these studies weren\u2019t conflicting at all, but merely science homing in on truth.<\/p>\n<p>Meanwhile, the next time you hear about some study that seems to disprove everything that science was saying about the subject just last year, run it through the following gauntlet:<\/p>\n<ol>\n<li>How strong was this study in comparison to the studies that had been done before? If the studies that had been done before were not very strong in the first place, then maybe the question is simply still up in the air, and it\u2019s natural to see results going back and forth for a while. In these cases, it\u2019s important not to oversell any given result, as if some established law of nature was being overthrown. Real scientists rarely sensationalize their findings like this, but the PR departments of their labs might, and news media headline writers do it all the time.<\/li>\n<li>If the new study is very weak in comparison to the studies that had been done before, then you shouldn\u2019t give it much weight. Weak studies have the power to suggest new avenues for more rigorous research, or to bolster existing knowledge, but they can\u2019t overthrow strong past findings, unless they reveal some big design flaw in those past studies. (In which case, stronger follow up studies are still needed before any conclusions can be reached.)<\/li>\n<li>If the design of the contradictory study was strong, has it been reproduced yet? Until it can be reproduced by independent third parties, there\u2019s always the possibility of human error, bias, or outright deception.<\/li>\n<li>If a study passes all these tests, then how big was the difference that it found in the first place? Remember randomness and clustering. It\u2019s almost guaranteed that whatever the truth is, your results will cluster around that point rather than sit right on top of it exactly. So, for example, if the truth is that, say, there\u2019s no correlation between cracking your knuckles and getting arthritis later in life, it\u2019s almost certain that if enough studies are done on the subject, you\u2019ll get some that seem to show that cracking your knuckles does increase your chances of arthritis, as well as others that show it reduces your odds! Taken individually, these studies can lead to confusion and science cynicism, but the key is to take them together. Collectively, they cluster around the true answer: zero correlation.<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"<p>There\u2019s a lot of cynicism about science these days, fed by every headline that declares one day that some food is good for you and the next that it\u2019s bad for you, every report about some researcher faking his experimental results, and every \u201cstudy\u201d trotted out by some con artist&#8230; <a href=\"https:\/\/www.authorfreeman.com\/blog\/how-science-works\/\">Read more &raquo;<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"open","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-286","page","type-page","status-publish","hentry"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.authorfreeman.com\/blog\/wp-json\/wp\/v2\/pages\/286","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.authorfreeman.com\/blog\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.authorfreeman.com\/blog\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.authorfreeman.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.authorfreeman.com\/blog\/wp-json\/wp\/v2\/comments?post=286"}],"version-history":[{"count":9,"href":"https:\/\/www.authorfreeman.com\/blog\/wp-json\/wp\/v2\/pages\/286\/revisions"}],"predecessor-version":[{"id":1229,"href":"https:\/\/www.authorfreeman.com\/blog\/wp-json\/wp\/v2\/pages\/286\/revisions\/1229"}],"wp:attachment":[{"href":"https:\/\/www.authorfreeman.com\/blog\/wp-json\/wp\/v2\/media?parent=286"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}