One of my new favorite games is the lyrically-named Potato Flowers in Full Bloom. Let me tell you why I love this game so much and why I hope you will also play it. Potato Flowers in Full Bloom is a first-person dungeon-crawling game in the tradition of Wizardry or Might and Magic. However, it does a lot of exciting new things within this framework that I hope future games will imitate. The…
A few weeks ago, I started a new job as a data scientist at the logistics division of a pharmaceutical company. This was primarily motivated by family reasons. My wife works at an art museum in Chicago and could not find satisfactory work in my small college town. We'd been living apart while I struggled to find a new professorship that might put us in a city she'd like. The academic job market is…
This is the story of how I found what I believe to be scientific misconduct and what happened when I reported it. Science is supposed to be self-correcting. To test whether science is indeed self-correcting, I tried reporting this misconduct via several mechanisms of scientific self-correction. The results have shown me that psychological science is largely defenseless against unreliable data. I…
Killing time in the UChicago stacks in the summer of 2019, I found a book from 1995 called Fraud and Erroneous Judgment in the Social Sciences . It's been an interesting read, because despite having been written nearly 25 years ago, much of it reads like it was written today. Specifically, there is very little substance about actually preventing, detecting, or prosecuting fraud, presumably because…
Nick Brown asks: When faked data are uncovered, they are very often crude and "obvious" in their construction. Are we not catching smarter fakers because most fakers are not smart, or because the data they fake is too realistic for us to spot them? — Nick Brown (@sTeamTraen) January 30, 2020 My answer is that we are not spotting the competent frauds. This becomes obvious when we think about all…
It's been a rich week of readings for wondering just what the hell we're doing. Loyka et al. (2019) present a framework for considering external validity , and this framework reminds us just how poorly we are doing at considering actual real-world human behavior. Tal Yarkoni has a preprint up that describes how implausible it is that the situations and stimuli we study will generalize to other…
Recent research by Chang & Bushman (2019) reports how video games may cause children to be more likely to play with a real handgun. In this experiment, children participate in the study in pairs. They play one of three versions of Minecraft for 20 minutes. One version has no violence (control), another has monsters that they fight with swords (sword violence), and another has monsters that they…
I’m approaching the end of my first semester teaching Intro to Social Psychology. As someone who came of age during the peak of the replication crisis (Bem, Stapel, Reproducibility Project), studies publication bias, and has had a hard time finding statistically significant results, I generally have a dim view of big chunks of the literature. I was worried that we would have very little to talk…
The prediction market is a way to try to assign probabilities to events. Bettors buy YES bets on things they think are likely to happen (relative to the market price) and NO bets on things they think are unlikely to happen (relative to the market price). Market dynamics lead the market price to settle on what is, across the bettors, the best subjective probability of the event. This is useful if…
I'm a new assistant professor trying to set up my research laboratory. I thought I'd try making the jump to PsychoPy as a way to make my materials more shareable, since not everybody will have a $750+ E-Prime or DirectRT license or whatever. (I'm also a tightwad.) My department has a shared research suite of cubicles. Those cubicles are equipped with Dell Optiplex 960s running Windows 7. I'm…
At long last, our article " Overstated Evidence for Short-Term Effects of Violent Games on Affect and Behavior: A Reanalysis of Anderson et al. (2010) " is released from its embargo at Psychological Bulletin. (Paywalled version here .) In this paper, Chris Engelhardt, Jeff Rouder, and I re-analyze the famous Anderson et al. (2010) meta-analysis on violent video game effects. At the time, this…
The last couple years have seen an exciting explosion in new techniques for publication bias. If you're on the cutting edge of meta-analysis, you now can choose between p-curve, p-uniform, PET, PEESE, PET-PEESE, Top-10, and selection-weight models. If you're not on the cutting edge, you're probably just running trim-and-fill and calling it a day. Looking at all these methods, my colleagues and I…
The reliability of scientific knowledge can be threatened by a number of bad behaviors. The problems of p-hacking and publication bias are now well understood, but there is a third problem that has received relatively little attention. This third problem currently cannot be detected through any statistical test, and its effects on theory may be stronger than that of p-hacking. I call this problem…
In DataColada [58] , Simonsohn argues that funnel plots are not useful. The argument is, for true effect size δ and sample size n : Funnel plots are based on the assumption that r (δ, n ) = 0. Under some potentially common circumstances, r (δ, n ) != 0. When r (δ, n ) != 0, there is the risk of mistaking benign funnel plot asymmetry (small-study effects) for publication bias. I do not think that…
It is a common goal of meta-analysis to provide not only an overall average effect size, but also to test for moderators that cause the effect size to become larger or smaller. For example, researchers who study the effects of violent media would like to know who is most at risk for adverse effects. Researchers who study psychotherapy would like to recommend a particular therapy as being most…
A few months ago, I had the opportunity to attend a symposium on research integrity. The timing was interesting because, on the same day, Retraction Watch ran a story on two retractions in my research area , the effects of violent media. Although one of these retractions had been quite swift, the other retraction had been three years in coming, which was a major source of heartache and frustration…
Growing up, I played a lot of role-playing games for the Super Nintendo. One trope of late-game design for role-playing games are rare drops -- highly desirable items that have a low probability of appearing after a battle. These items are generally included as a way to let players kill an awful lot of time as they roll the dice again and again trying to get the desired item. For example, in Final…
Some months ago, a paper argued for the validity of an unusual measurement of aggression. According to this paper, the number of pins a participant sticks into a paper voodoo doll representing their child seems to be a valid proxy for aggressive parenting. Normally, I might be suspicious of such a paper because the measurement sounds kind of farfetched. Some of my friends in aggression research…
Yesterday, Perspectives on Psychological Science published a 17-laboratory Registered Replication Report, totaling nearly 1900 subjects. In this RRR, researchers replicated an influential study of the Facial Feedback Effect, showing that being surreptitiously made to smile or to pout could influence emotional reactions. The results were null, indicating that there may not be much to this effect.…
Fail-Safe N is a statistic suggested as a way to address publication bias in meta-analysis. Fail-Safe N describes the robustness of a significant result by calculating how many studies with effect size zero could be added to the meta-analysis before the result lost statistical significance. The original formulation is provided by Rosenthal (1979), with modifications proposed by Orwin (1983) and…
Inspired by a recent excellent lecture by Nick Brown , I decided to finally sit down and read Diederik Stapel's confessional autobiography, Ontsporing . Brown translated it from Dutch into English; it is available for free here . In this account, Stapel describes how he came to leave theater for social psychology, how he had some initial fledgling successes, and ultimately, how his weak results…
Brent Roberts suggests the replication movement solicit federal funding for the organization of federally-funded replication daisy chains. James Coyne suggests that the replication movement has already made a grave misstep by attempting to replicate findings that were always hopelessly preposterous. Who is in the right? It seems to me that both are correct, but the challenge is in knowing when to…
Everyone seems to agree with the saying "extraordinary claims require extraordinary evidence." But what exactly do we mean by it? In previous years, I'd taken this to mean that an improbable claim requires a dataset with strong probative value, e.g. a very small p-value or a very large Bayes factor. Extraordinary claims have small prior probability and need strong evidence if they are to be…
Last post , I talked about the benefits a manuscript enjoys in the process of scientific publication. To me, it seems that the main benefits are that an editor and some number of peer reviewers read it and give edits. Somehow despite this part coming from volunteer labor, it still manages to cost $1500 an article. And yet, as researchers, we can't afford to try to do without the journals. When the…
@PLOS 's financials reveal that they are merely trying to maximize their personal and corporate profit, like any company 30/40 — Andrew Kern (@pastramimachine) March 15, 2016 They are merely another Nature or Science that aims to maximize profits while cloaking itself in the white robes of OA. 32/40 — Andrew Kern (@pastramimachine) March 15, 2016 . @pastramimachine i agree - I've always want to…