Building research tools with Claude

Part of the process of thinking about large scale conjoint has been using Claude Code AI as a sounding board in addition to actual creating testable code. For me, the ability to have a good conversation about technical ideas and issues has been a revelation. Combining this with the ability to create code, and for Claude to create tests and validation checks along the way makes it much much easier to think creative about quantitative research.

For the most part, AI in market research has been based on relatively simple tasks like transcription and coding, or on attempts to simulate respondents, or to act as a helper for some of the classic research output steps like building a PowerPoint deck, or for sense checking or first-drafting a questionnaire.

Claude Code’s focus on programming might not get a look in because many researcher never code and probably think it is not for them. Scripting is what the DP department does, and advanced analysis is what your in-house statistician does.

However, Claude is potential a huge addition to the researcher’s armory when used appropriately. Unlike ChatGPT Claude Code can sit on your computer (in a folder you limit it to) and you can share files with it, and Claude will generate code and scripts.

The actual AI bit still happens on Anthropic’s servers so care has to be taken with actual real-world data which might be confidential. But what Claude allows you to do is build tools that can be run locally, so that the data itself never leaves your system.

Unfortunately I would say most researchers have a low-level of coding skills. Researchers design questionnaires, manage fieldwork and read and interpret results, but the basics of scripting a questionnaire is passed to a dedicated DP department who handle the coding. Some researchers may use SPSS or other off-the-shelf analysis tools for data output, but only those at the top end of the analytics tree are using R (or Python).

However there are plenty of opportunities where researchers need tools to handle the data formats and inputs and outputs.

One simple example is middleware - the sort of code that needs to be written to translate data output from one software system so it can become input for another piece of analysis software.

A common example is converting a questionnaire and data in to an SPSS .sav file type. The precise set up will depend on which survey system you use and what data is being exported. With Claude, that becomes an instruction to write a converter (not do the conversion, which would involve Claude accessing the real data). You then have a dedicated converter tool that can be used every time you need to make that transfer. Write Once Use Often, without needing AI tokens downstream.

A second common use-case would be as a helper for R. R is the open-source statistics system which can do every type of statistics you can think of and is more commonly used by statistics professionals than some of the older statistics packaged.

However, handling and converting your data into the right format for the right R script, particularly if you need to use dummy coding of any type, can be extremely frustrating.

A typical market research example would be using R to run cluster analysis. However, it is easy to become daunted by R’s syntax and set up and the proliferation of options around which type of cluster analysis to use or how to set the R commands up correctly.

Claude makes this so much easier. It can either create a tool, or write you the script that will run your data through R. If you let Claude see your data (remembering that it will be shared with Anthropic - but it can be anonymised) it will define the script and run the R code and can then be asked to provide guidance and advice about the results.

So more than just taking you through to create an R script, Claude has the potential to become a tools and techniques advisor. You can ask it what cluster analysis options are applicable and which ones to test and run.

If you allow Claude to look at your data and questionnaires, you can ask for advice on what analyses to do (assuming you are OK with data going to Anthropic’s servers)

We can go one stage further (and then one stage beyond that). Claude will actually build complete tools for you - which might be copies of existing tools that you do not have.

But because Claude can build the tool, you can also ask for extensions and work with Claude to create completely new applications and approaches. For me, large scale conjoint is one such idea, but I also have a ‘salesman-conjoint’ design in my head, where I can see the design, but the build would be experimental and long. Too long to merit investigation if I was writing it myself.

However, before that, I started with Claude because someone on LinkedIn said they had written a version of Hierarchical Bayes (HB) analysis that uses a GPU (graphics processor) to accelerate the calculations - so instead of taking minutes or longer to run, HB would run in seconds. However, HB with a GPU for acceleration requires a serious, non-trivial, piece of code with a deep statistical underpinning.

This is a great example of using Claude for building a difficult, real-world toolset.

HB is a tool used for analysing conjoint and maxdiff to estimate individual utility scores. Often it is done with off-the-shelf tools such as Sawtooth’s CBCHB (which has a price tag), but it can also be done in R which is open source, but a bit fiddly to get the data in the right format.

The first thing to say, is that if you are using Claude for tool building, do not think of using it aiming for a single-shot prompt like you would for ChatGPT or image creation. Using Claude is like having a technical discussion with an expert in the field who will generate code once you have had a decent discussion about how to approach the problem and to layout what you need and how it will work. Discuss the specification first so Claude understands the domain and what you are looking for.

Once we had had that discussion, around conjoint analysis initially and then about conjoint analysis, the first place I started was to ask for a non-accelerated HB (with a connector to read directly from the Cxoice survey system). This means Claude needs to look up everything about HB, and to understand the Cxoice data structures and to make suggestions about an appropriate approach..

Once the approach was agreed, Claude would start to generate code. Once the code is written it then tests that code for basic errors like missing functions or syntax issues. And then it generates test data and runs the final code to test the run works and outputs correctly.

For anything which requires an external command to be run, or a new file to be written, Claude asks permission first.

With any code output, the testing and validation stages are hugely important, but just because the code runs, does not mean the output is correct. I also needed a comparison to see if the output was any good - does it actually work statistically?

So for validation, I asked Claude to generate a second script for R that would take the same test data and run it through R, using an appropriate library (Claude chose BayesM) and then compare outputs.

Looking good so far. But then I checked the test data file and realized it was missing a ‘None’ term. In conjoint analysis often a product can be chosen, or ‘None of them’ selected, so the HB code must handle a None parameter.

I told Claude the problem, and it generated new test data, and fixed all the code. Everything looking good.

Nope. The output was actually wrong. I tested the HB code on an old conjoint dataset and the None parameter was flattening all the actual utilities for the attributes. Something was going wrong.

This is where the real magic starts to happen.

Finding and fixing deep calculation issues is tough and tedious for real programming. With a difficult script, and only summary-level aggregate data, identifying where calculations are going wrong is a difficult thing to trace.

But here, I could just tell Claude that there were errors in the output and explain those errors, and the Claude did all the heavy lifting with detailed diagnostics of the code it had generated to identify why there was a discrepancy, and how to fix it.

Initially it thought the error was in the priors - so the statistical setup was wrong, and we went around a couple of loops solidifying the statistical elements.

However, when this still led to a difference between the code with None and the code without None, it eventually discovered it had a mis-coding problem. The input design was 1-based (ie 1,2,3), but the code - and R validation script - was expecting the design to be 0-based (ie 0,1,2).

These are deep and nasty bugs for programmers to find. But here, Claude just needed to be told that it was wrong, and it eventually found and corrected the problem. All I needed to do was look over its shoulder to make sure the output met the pre-existing standards.

If you are experienced with AI text or image generation, it can feel like it is impossible to bring the AI system back on track with subsequent requests if the initial prompt fails to deliver the quality you want. Here, Claude is much more thorough and stays much more on track.

With an unaccelerated HB working, testing and verified against R it was time to ask for the GPU accelerated version.

With the systems and algorithms already bedded down, it was much easer for Claude to create the accelerated version using the Nvidia graphics card and CUDA for acceleration (the speed up is due to parallel processing and dedicated matrix and tensor calculations on the graphics card)

Obviously (by now), Claude tested it against the other versions (with none and without none) and ran timing tests for. Larger designs that would take R several minutes to run, were now taking seconds.

What’s more, having the code and tooling itself, meant I could also ask Claude to add extensions, or change the input and output formats it would accept - for example for MaxDiff data.

Consequently, I now have a HB that mirrors what is in the market today but is super fast. And Claude would even tell me how to host it on AWS GPU if I wanted it as an online analytics module without burdening the survey servers.

The story might feel like it has finished. Claude build a tool. Whoopee do. That’s what vibe coding is all about. It might be new in the research world, but it is now business as usual in software development.

The part that is missing, is that now I can also ask Claude to adapt the tool exactly to what I need. I just ask for a new version with the extra functionality and Claude will generate it and automatically suggest doing all the checks and validations against what works already.

Better still, I can ask ‘What if’ questions to explore new ideas (eg around large scale conjoint) and get what sound like solid suggestions and explanations and working code to experiment with.

Remember, this isn’t trivial code and it did need guidance and checking to get it right. Consequently, the ‘guide’ person needs to know what they are looking for and have points of comparison to check and validate the output.

But if you wanted your own tool for say cluster analysis that allows overlaps, or key driver analysis that has multi-collinearity that needs to be accounted for, or data interpolation tools for fusing data, or creating digital twins statistically then Claude will generate them, and let you experiment.

It will also probably write the converter you need to get data from your tabs package into Powerpoint, or to write a tabs package that will actually read your questionnaire and data directly. And you would be able to use it to create a validator that checks contact lists are set up and de-duped correctly, and to look for consistency checks and outliers in your survey data. Or use it to convert charts into interactive ‘Observable’ charts and animated data graphics showing customer flows.

So if you have an idea, or want to explore a new method or technique, you do not have to wait for someone else to write it. You can build it with Claude yourself.

And the best bit. Most of the time it just takes a couple of hours to get the tool that you are looking for. If you can imagine it, it can be built.