In 1981 , Reagan's Secretary of Education commissioned a report that became the 1983 A Nation at Risk: the Imperative for Educational Reform . It opens with a jump scare: Our nation is at risk . Political, but not polemical, the report bypasses academic framing and goes straight for national alarm to ensure that the reader comes away with the right conclusion. Nation's diagnosis was a "rising tide…
Introduction Stephen Jay Gould promoted the idea of non-overlaping magisteria , or ways of knowing the world that can be separated into mutually exclusive domains, where each "holds the appropriate tools for meaningful discourse and resolution." The tension Gould was trying to resolve was between religion and science: Science tries to document the factual character of the natural world, and to…
The student/faculty ratio, which represents on average how many students there are for each faculty member, is a common metric of educational quality. The ratio shows up in the Common Data Set (CDS) and college guides, presumably so prospective students can compare colleges. The standard way to count students and faculty for the CDS calculation is to equate the total number of students into a…
I'm scheduled to give a talk on grade statistics on Monday 10/26, reviewing the work in the lead article of JAIE's edition on grades . The editors were supportive in not just accepting my dive into the statistics of course grades and their relationship to other measures of learning, but they recruited others in the assessment community to respond. I got to have the final word in a synthesis. That…
"How much data do you have?" is an inevitable question for program-level data analysis. For example, assessment reports that attempt understand student learning within an academic major program typically depend on final exams, papers, performance adjudication, or other information drawn from the seniors before they graduate: a reasonable point in time to assess the qualities of the students before…
In 2015 I bought a house on the slope of Paris Mountain in Greenville, South Carolina. A flat spot for the house was bulldozed, creating a steep hill behind the house, leaving a slope from the floor of the garage to the street, so the driveway in between is slanted at about 15 degrees. That turns out to be a lot of degrees. I first noticed that this could be a problem when I drove up from Florida…
The annual NACUBO report on tuition discounts was covered in Inside Higher Ed back in April, including a figure showing historical rates. (More recently this was covered in the Economist .) The discount rate is the fraction of posted tuition that is "given back" by the institution in financial aid awards. Usually these are unfunded awards, meaning there was never any money involved. For a primer…
This post is the first of a series on student achievement. The National Center for Education Statistics (NCES) summarizes graduation rates for first-time full-time undergraduates here . Overall rates have increased from 34% to 47% between start years of 1996 and 2014-an average increase of .7% per year. Why is this? Educated Guesses "Why" questions are a trap, like a free trial subscription to the…
Sign, sign Everywhere a sign Blockin' out the scenery Breakin' my mind Do this, don't do that Can't you read the sign? -"Signs" by Les Emmerson (Five Man Electrical Band) A February 2023 Ezra Klein podcast pokes at his audience with the title "How Liberals — Yes, Liberals — Are Hobbling Government." He interviews Nick Bagley, a law professor at the University of Michigan, who has fascinating ideas…
Sorry for the clickbait headline, but it's accurate: the way we assign grade points (A = 4, B = 3, etc.) is arbitrary, which makes me wonder if it's sub-optimal. That statement--GPA being sub-optimal--is a little mind-bending. What does that even mean? We certainly treat GPA like it means something. The statistic is used as a measure of Satisfactory Academic Progress (SAP), which has the weight of…
Introduction I recently received a copy of a new book on assessment, one that was highlighted at this year's big assessment conference: Fulcher, K. H., & Prendergast, C. (2021). Improving Student Learning at Scale: A How-to Guide for Higher Education . Stylus Publishing, LLC. This is not a review of that book; I just want to highlight an important idea I came across, from pages 60-63 of the…
How long does it take to complete a bachelor's degree? Furman's four-year graduation rate runs around 75% and the six-year rate is about 81%, so take a guess what the average time to graduation is. The answer is an average of 3.8 years from start to graduation, a rate that has been steady for years. Surprised? I was, because the math doesn't seem to add up. Wouldn't it have to be more than four…
My academia.edu account emails me links to articles they think I'll like, and the recommendations are usually pretty good. A couple of weeks ago I came across a paper on the reliability of rubric ratings for critical thinking that way: Saxton, E., Belanger, S., & Becker, W. (2012). The Critical Thinking Analytic Rubric (CTAR): Investigating intra-rater and inter-rater reliability of a scoring…
Last time I showed how we can think of matrix multiplication as (1) a matrix of inner products, or (2) a sum of outer products, and used the latter to unpack the "positive definite" part of the product \(X^tX\). At the end I mentioned that the outer-product version of matrix multiplication makes it easy to show that trace \(AB\) = trace \( BA \). In the late 1980s I took a final exam that asked…
In much of my work, a data summary looks like a spreadsheet, with column headers like StudentID, Cohort, Year, GPA, Retained, and other potentially interesting data points. Hadley Wickham calls this arrangement tidy data, where the cases are in rows and the attributes of each case are columns. When these are all scalars (e.g. an ID number is not a scalar, because it isn't meaningful to take half…
2021 AAHLE The Case for Grades 2021 New England Assessment Conference: The Future of Assessment (skips the seven minutes of chatting before the presentation). 2021 App State Analytics Day: Grades and Catalogs as Data with Amanda Whalen.
Introduction A common difficulty in regression modeling is to figure out which input variables should be included. One tool at our disposal is the added-variable plot. Given an existing model, the added-variable plot lets us visualize the improvement we would get by adding one more variable of our choosing. This idea is described nicely in A Modern Approach to Regression with R by Simon J.…
Introduction We all develop an intuitive sense of cause and effect in order to navigate the world. It turns out that formalizing exactly what "cause" means is complicated, and I think it's fair to say that there is not complete consensus. A few years ago, I played around with the simplest possible version of cause/effect to produce "Causal Interfaces," a project I'm still working on when I have…
Introduction The use of the ordinal responses as scalar values was on my list of problems with rubric data. I showed how to estimate an underlying "latent" scale for ordinal data here and elaborated on the methods here . These methods cast the response values into a normal (or logistic, etc.) distribution, from which we can get the latent mean. However, by itself this isn't very interesting,…
If you haven't read it yet, I recommend that you pick up a copy of Paul Tough's book The Years that Matter Most: How College Makes or Breaks Us . In particular, the section starting on page 182 of the hardcover is a revealing picture of how college finances affect recruiting. You can find the same (or very similar) content in this New York Times article. In short, most private colleges need to…
Introduction The last article showed how to take ordinal scale data, like Likert-type responses on a survey, and map them to a "true" scale hypothesized to exist behind the numbers. If the assumptions make sense for the data, this can lead to better estimates of averages, for example. In this article, I'll compare some different ways to calculate the latent scale. As with most math exercises, the…
Introduction A few days ago , I listed problems with using rubric scores as data to understand learning. One of these problems is how to interpret an ordinal scale for purposes of doing statistics. For example, if we have a rating system that resembles "poor" to "excellent" on a 5-point scale, it's a usual practice to just equate "poor" = 1, ..., "excellent" = 5, and compute an average based on…
I have a guest post that appears on John Warner's blog in Inside Higher Ed this morning. You can find it here . I spent much of the weekend working on latent trait scales derived from ordinal responses--the kind of data we often get from surveys and rubric raters. More on that soon.
Introduction Within the world of educational assessment, rubrics play a large role in the attempt to turn student learning into numbers. Usually this starts with some work students have done, like research papers for an assignment (often called "artifacts" for some reason). These are then scored (i.e. graded) using a grid-like description of performance levels. Here's an outline I found at…
As announced in InsideHigherEd this morning, the College Scorecard is now reporting financial information for graduates by institution and program , when there are enough samples. I used the opportunity to play what-if. Back in ancient times, when I was an undergraduate studying math, I decided I'd need another major in order to find a job (don't ask me why I thought that), so I picked up…