AI Builders week reflections and opportunities

I recently had the pleasure of running an event challenging product managers to experiment with AI to enhance their processes.

The genesis of this event was discussion with Adam, CEO at Colab, an organisation that focuses on development of product management skills. My affiliation with them is simply as a happy customer and member of their community.

The insight Adam had was that whilst AI is the current “big thing”, uptake and penetration is actually quite low, and many members of the community haven’t really done much with it, and felt like they were at risk of being left behind.

So aligning with Colab’s mission to build the skills of product people, we devised an “AI Builder’s Week”.

Some data about our participants

Having put out the call to the Colab community and beyond, we received registrations of interest from 109 people.

We asked that people registering their interest for builders week completed a brief survey to help build our understanding of the group.

We had people from the UK, the Netherlands, Spain, Egypt, Kenya, South Africa, Saudi Arabia, India, Indonesia, Vietnam, Australia and New Zealand. (Side-bar - isn’t the internet amazing?)

People registered from all sorts of organisations: start-ups, scale-ups, enterprise, government departments and NGOs. We had people from early in their career through to Heads of department and one CxO.

Along with demography, we also gathered a small amount of data about the current level of AI use.

The responses were skewed towards people already using AI at work, although a quarter of respondents rarely or never use AI work. We didn’t go into exploring use cases or tools. That’s a job for a more in-depth survey with a broader population.

We also asked about which areas of product management people wanted to explore using AI.

Top line summary - pretty much everything is interesting to people to explore. My university statistics modules were sufficient for me to understand that this is not especially scientific, but the themes with the most interest were:

• Research & Insights (competitor & customer research)

• Product Definition & Design (PRDs, prototyping, user stories)

• Planning & Strategy (roadmaps, priorities, GTM)

There was also strong interest in exploring AI as a research partner, drafting assistant, and workflow automaton.

So yeah, pretty much everything.

How we structured the week

The central premise of the week was about getting people to explore and build their confidence with use AI without this being a formal training exercise. It was also designed to be something that people could do with a small investment of time - pitched at 30 minutes a day. It was also designed to be asynchronous, which, given the spread of locations was pretty much vital.

The week followed a representative product journey:

• Competitor Research

• Customer Research

• PRD writing

• Prioritisation

• User Story writing

• Prototyping

• Story-telling

As I said at the kick-off call, this may not be how you do product, but was chosen to cover a range of common activities, and also have some flow through the week.

We also gave people the option of participating in a more free-form “choose your own adventure” deep dive into an area of interest.

We had a slack channel that everyone who registered was able to see the daily challenges, access resources and share their learning.

As an example, the challenge for day one was to:

produce a competitive brief with insights on opportunities for your product to move ahead of the competition.

Suggested tasks: Pick a competitor, use AI to analyse their product, pricing, messaging, and recent updates

All the challenges had a similar style, perform activities that form part of typical product management responsibilities, using AI.

The week culminated in people submitting their pitch for the rest of the community to vote on. I even vibe-coded a product hunt like site to facilitate this last bit. It’s still there, with the submissions for people to enjoy.

Subscribe now

So what did I learn?

The post-event survey was immensely valuable, and there was a load of really useful feedback that would make the week even more valuable, plus lots of ideas for follow-on activities.

And most importantly the week was immensely successful in terms of helping people get moving with AI tools, and try things out. Per one verbatim piece of feedback:

Alongside the feedback and opportunities to improve how people learn together and benefit from a cohort of peers exploring the same topic, there was also quite a lot to reflect on in terms of the state of AI and the challenges and opportunities that accompany the technology.

AI can help create a good first draft

When a blank page is an obstacle to progress, AI tools, particularly the foundational models in “chat” mode can help you quickly move to step 2. Even just asking a meta-question about how to approach a task can be a great accelerator.

Choose the right tool for the job

This was one of the great aspects of having a large cohort of people trying out different tools for the same exercise and sharing their findings. Given the probabilistic nature of AI, you get different outputs from the same input, sometimes from the same tool.

Evaluating the quality of outputs is something that needs serious thought if you are going to trust your tool.

Across the week people discussed 15 different AI tools they used. Most frequently used were the big names - ChatGPT, Claude, Gemini, Co-Pilot.

One of the most interesting challenges was when we wrote PRDs. Alongside the big names, a lot of people tried ChatPRD, which is a specialist application focused on writing product management documentation. Participants reported that they found the outputs from ChatPRD to be of a higher quality and more relevant than the general tools.

This was also reflected in the last challenge, where several people tried, unsuccessfully, to convince the general tools to make slides, whilst the winning pitch was put together in Gamma, which specialises in creating slides.

One swallow does not a summer make, but will the next iteration of AI tools tend towards being smaller and more use-case focused?

Context and Data are critical

That ChatPRD is superior to the foundational models probably reflects the importance of context. Prompting and the inputs provided are hugely influential to how good the outputs are.

Participants repeatedly demonstrated that a well-crafted prompt yielded a better output than a brief prompt.

ChatPRD is reliant on a foundational model itself, so it’s working with the same base data and algorithms as if you would by inputting the prompt directly. But you wouldn’t have the same context and prompting as Claire Vo (the maker of ChatPRD) has spent the last couple of years refining and tuning.

Coming back to the question of the next iteration of AI tools tending towards being smaller and more use-case focused, what about the underlying data sets?

Whilst the rate of improvement in these models over the last three years has been breath-taking, there is evidence of slowing marginal improvements. Specialised, domain-trained “Small Language Models” are able to outperform the large models on benchmarking tasks. They also have the added benefit of being far less resource hungry.

Vibe coding is both a threat and an opportunity

The energy through challenge week wasn’t constant, it was highest at the start, and then spiked with the last couple of challenges, particularly the prototyping challenge. This is where people got to take all the research, thinking and writing they had done over the previous few days and turn it into something visual and interactive. We saw people try Lovable, bolt, v0, as well as the likes of ChatGPT and Gemini.

The fact that you can go from a simple prompt to a clickable, functional prototype or application in a matter of minutes is simply incredible. As many product people will attest, the power of something visual and interactive as a communication tool is absolutely incredible.

But these tools also represent a risk for product people working with others. You can easily get pulled into “vibe coding”, asking the tool to build more and more without necessarily checking on the code.

We saw people fighting the tools, spending time and credits to get the tools to produce exactly what they wanted, and sometimes found themselves trapped in a bug-bashing death spiral.

The message here is to be really clear on what you are trying to do with these tools. Whilst some of these applications are positioning themselves as the future of no-code, and claim to allow you to build fully functional, production-grade applications, the majority of people shouldn’t be trying to do that with these tools.

Keep a human in the loop

Ironically, one of the best pieces of advice through the week came from Gemini:

This was apparent time and again through the week. I posed questions for reflection along with each challenge. One topic I frequently encouraged people to reflect on was to what extent they would trust the output to share with colleagues or customers as-is.

The answer was not at all. Back to that rough first draft finding.

The level of trust did vary through different types of activity, and also varied due to both concern with accuracy and matters of taste. The ability to pose a good question or prompt well were leading indicators of getting a good first draft, but the ability to judge the output is absolutely critical.

AI has the potential to be truly transformational

During the challenge week we scratched the surface of how AI could be applied to product management activities. We did however demonstrate that there is opportunity everywhere.

One of the highest leverage uses for AI at the moment for organisations is to use it as a massive lever for process re-engineering.

During our wrap up, one participant bemoaned the fact that he had wasted all his time doing the early writing challenges and could have gone straight to the prototyping challenge and got to the same point. I challenged that - how could you give the right input to the tool if you haven’t done that research and thinking up front?

However, it does seriously challenge the need for all the steps an organisation goes through in order to put something in the hands of a customer in order to generate feedback.

Using AI as an opportunity (or a tool?) to ask the question of why you are doing all the things you do could be hugely powerful. Why are you writing documents? Is it to clarify thinking and share assumptions? Is it for alignment? Is it because that’s the way things are done around here?

I expect successful organisations will be doing things radically differently in the next few years.

Where to from here?

I am personally going to continue trying out different AI tools for different use cases, and encouraging others to do the same.

There was a recent MIT study that found that the vast majority of organisations are getting zero return from their AI investment.

Let’s just remember that whilst cloud computing is now used by almost all organisations, that has taken probably twenty years; perhaps longer depending on your definition. For AI, we’re still very early in the cycle, so largely “investment” at the moment is experimental. And given the level of complexity that most organisations operate in, if all your experiments are successful, then you’re probably not taking enough risks.

I am also going to look at how to help people explore and learn with this new technology, probably in a group-based format, building on the success and learning of the first AI builders week.

Subscribe now