I'm (Still) Building A Crime Data Assistant

I created a crime data chat bot a while ago and I told you all about it two-ish months ago. Here’s the post I wrote back then in case you missed it or want to re-read:

The assistant is at NIBRSAI.com and it is designed to answer your crime data questions while highlighting the limitations of the available data and avoiding analytic AI slop. The goal remains, when ready, to add it to our newly released Crime Index website over at CrimeIndex.org. It wasn’t perfect before, and it isn’t perfect now. There are undoubtedly bugs and responses it won’t generate perfectly, but I think it’s getting better and want to figure out by how much!

This project is quite ambitious in my humble opinion.

There are more than a billion rows of NIBRS data, millions of rows of current and historical crime, clearance and staffing data, hundreds of thousands of homicide reports, and tens of thousands of rows of hate crime data (oh, I added hate crimes data). Historical SRS data captures crime back to the 1930s and clearances and staffing back to the late 1950s. Supplementary Homicide Report and current crime data are completely different sources covering different aspects of crime.

I want to include all of it and have the bot produce accurate answers in just a few seconds.

There have been challenges.

I built the assistant using my knowledge and expertise, but I wasn’t initially very good at querying the assistant from the perspective of someone who doesn’t have a strong working knowledge of the data sources and subject matter. Getting hundreds of real-world queries was an enormous boon that showed what it did well (a lot!) and what it didn’t do well (a lot!). In the end, all I want is the bot to query a database and return an answer, but the beauty of the chat bot is also its main challenge — there are a lot of ways to ask questions.

The main problem I found was twofold: first, the assistant wasn’t always very good at understanding what a user was asking and translating that into a usable SQL query to find the correct data. And second, users weren’t always very good at picking sources and phrasing questions in a way that maximized the assistant’s potential for generating an appropriate SQL query.

Basically, both the users and model had too much freedom and it was leading to fewer correct answers.

The problem started on the original main page:

Users were asked to choose a data source before asking a question. This was a great system for me because I usually know if I want to query NIBRS data or SHR data or current crime data. These are all distinct data sources in my mind with clearly distinct outputs.

This was not a great system for users who do not know the difference between NIBRS and SHR, which is going to be many if not most users. It’s unfairly limiting to expect users to know the data schema as a prerequisite to using the assistant.

Now there is a new home page which grounds your query in the question you’re asking rather than the source you’re asking it from. There are model queries below that hopefully help guide a user into both using appropriate language and highlighting the different kinds of questions that can be asked thereby subtly introducing the available data sources. The purpose of the home page is to help users craft a better query, not make them crime data sourcing experts.

Another big change is that the assistant is now far more deliberate in determining exactly what you’re asking for before asking the model to weigh in. Some crimes and geographies are ambiguous or not covered in the available data, so the site now works with the user to solve those ambiguities on the front end to craft a better query that avoids confusing the model. As a result, the AI model guesses far less often and uses more concrete rules to match the user’s question to the available data.

Let’s say you want to know how many carjackings there have been each month in Memphis since 2021. The assistant confirms the correct query is being run by figuring out the data source (carjackings are determined through NIBRS), the place (Memphis, TN which has just the police department), the metric being requested (carjacking count), the timeframe (since 2021), and the unit (count of incidents).

This process takes all the guesswork out of crafting the proper query for both the user and model. The result is this lovely chart.

There are a ton of other changes to make responses faster and honest to what’s available in the FBI’s (still flawed) datasets. For example, the FBI does not track shootings, but if you ask for shooting data it’ll help you pick something we do have data on.

The assistant is also much better at picking out geographic ambiguity and forcing the user to decide before it gets to building the query.

And it still gives you the option to choose data sources when multiple sources could be used to answer the same question:

The assistant still runs on Claude’s Haiku though I’ve used both Claude and Codex (OpenAI’s answer to Claude Code) as a sort of tag team engineering team. I asked Codex to summarize the changes we’ve made over the last two months and here’s how it summarized the improvements:

“In practical terms, the site is better at three things: answering common questions quickly, refusing questions it cannot answer honestly, and making its remaining uncertainty visible instead of hiding it behind a polished but unsupported result.

The main limitation is unchanged: the underlying FBI data still has uneven agency participation and coverage. What has improved is that the site is much less likely to confuse those data limitations with actual crime trends.”

In other words, the site should work much better now than before and you should give it another whirl if you’re so inclined. It should be better constrained from giving bad answers and it should be better at helping users to give it questions it can properly answer.

The assistant is getting closer to being a truly helpful crime data assistant. Please feel free to use it and provide whatever feedback you’d like. Hopefully the next update post about this project will be me telling you it is nearly perfect and ready for adding to the Crime Index!

This week’s guest is Insha Rahman, the president of the Vera Institute of Justice, the oldest and largest justice reform organization in the country. Insha has spent the past ten years building out everything from bail reform research to a 501c4 focused squarely on the politics of public safety and brings a fascinating perspective to the conversation around making perceptions of crime and safety more responsive to what the data says is actually happening.

We talk about what criminal justice reform looks like today in 2026 and how it might look into the future. We get into why the data behind reforms like bail reform rarely matches the political story told about them, what cities like Camden, NJ got right during their first homicide-free summer in fifty years, and why Insha thinks the gap between crime perception and crime reality might finally be starting to close.

Apple

Spotify

You can also catch it on the Jeff-alytics YouTube page where I’ll be posting episodes and video clips, so be sure to like and subscribe there if you’re so inclined!

And while you’re here, be sure to check out these other recent great episodes:

Director, NORC Center on Public Safety & Justice John Roman

Criminologist Andy Wheeler

Filmmaker Lynn Novick

Georgetown Professor Christy Lopez

Leave a comment