The New Economics of Experimentation

AI has made building dramatically faster. It hasn't made knowing what deserves to be built any easier.

It's a distinction that came up recently on The Cognitive Revolution podcast. In a conversation about AI-powered software development, OutSystems CEO Woodson Martin observed that deciding what to do remains the hardest question. As the cost and effort of building fall, knowing where to focus becomes even more important. 

“What should I build to actually solve problems in my business? …What to do is still the hardest question… Where can I get the biggest bang for my buck? …What business process does it make sense to transform into something Agentic?”

At Co-Created, we've run thousands of experiments with more than 90 companies over 15 years. Today, most are built with the help of AI agents and workflows we’ve created. A prototype that once took months to build now takes an afternoon. We can explore more ideas, get working solutions in front of people sooner, and learn more along the way.

But just because we can build something doesn't mean we should. With tighter budgets and growing pressure to show results from AI, companies need to be more thoughtful about where they invest.

That's where experimentation comes in. Not just as a way to test ideas, but as a way to make better decisions. For us, AI makes it possible to run more experiments, often in parallel, and get working solutions in front of people faster. That means faster learning, stronger evidence, and better decisions about where to invest.

But it still takes time, attention and resources to manage and decide what’s worth doubling down on.

Two applications, one discipline.

We use experimentation in two ways:

1. Growth experiments help us understand whether customers want something new, whether they'll pay for it, and whether there's a viable business behind it.
2. Transformation experiments help us understand whether a new way of working is actually better than what's in place today, whether that's measured by quality, speed, cost, or reliability.

The questions are different, but the approach is much the same. Start small, gather evidence, and use what you learn to decide what comes next.

There are four things we try to get right in every experiment:

• Know what you're testing. Start with a clear hypothesis and be willing to be proven wrong.
• Make sure it matters. Focus on a real customer problem or business outcome, not just something interesting to build.
• Decide what success looks like. Establish a baseline, what you'll measure, and what would make the idea worth pursuing.
• Keep real people in the loop Test in realistic conditions, with the right controls and oversight in place. Synthetic data and AI-generated feedback can be valuable, but we still love hearing from real humans. There's no substitute for seeing how people actually respond: what they say, what they do, where they hesitate, and the reactions we didn't anticipate.

For new growth opportunities, our updated Venture Fundamentals framework remains as relevant as ever. It helps us assess venture maturity across the key dimensions of a business, identify the most critical assumptions at each stage, and focus experiments where they can reduce uncertainty and de-risk further investment.

For AI transformation, we start by understanding the underlying business needs and pain points (this is often where we spend the most time), then experiment with solutions to find what works best. We look at how a proposed solution performs against the existing way of working. Is it more accurate? More reliable? Faster? What does it actually cost, including the human effort involved?

In either case, we're trying to answer the same question: Do we have enough conviction to justify our next bet?

Better evidence, earlier.

One of the things we love about AI is how much more we can learn, and how quickly.

Not long ago, we were testing new app experiences with clickable Figma prototypes, walking users through screens over Zoom and watching where they clicked, hesitated, or got confused. It worked, but creating the prototypes, coordinating sessions, and interpreting feedback took (a lot of) time.

Today, with tools like our own hotTake platform, we send a link to a working application, design, or prototype and let people explore it on their own time. No sessions to schedule, no moderator in the room, and no one being polite to our faces. Within a day we have honest reactions from the people we’re building for and a clear read on what to fix, keep, or kill.

The difference isn't just speed. We're seeing how people interact with something functional, rather than asking them to imagine how they might use it. And we can learn from more people, across more iterations, without all the manual effort.

Of course, we've been testing real behavior long before AI came along (painted door tests, anyone?). What's changed is how quickly we can move from an idea to something people can actually use, and from their reactions to the next decision.

Finding the right problem before building the solution.

A blood donation organization came to us concerned about losing younger donors. It seemed like a reasonable place to focus and a common industry concern, but - as always - we wanted to dig in and validate the need to understand what was really happening before jumping into solutions.

Through customer research, we found that the real problem was different. Getting one-time donors back a second time was the real unlock. Donors of all ages were tuning out, overwhelmed by too many generic communications and a lack of personal attention and appreciation throughout the process.

That changed where we focused. Rather than investing in a solution aimed at younger donors, we developed a prototype to address the broader engagement problem. We then worked with three trial centers to test it with real donors.

The results gave us evidence that the approach worked (i.e. significant measured lift in real donor response) and generated interest from additional centers, creating a path to expand. The important part wasn't how quickly we built the prototype. It was that we took the time to make sure we were solving the right problem.

Turning experiments into paying customers.

With a large consumer electronics manufacturer, we ran more than 50 experiments to turn an existing consumer print platform into a recurring-revenue service.

We used our live experimentation sandbox to put new offers in front of real customers, testing everything from pricing and acquisition channels to the checkout experience. That gave us room to try things, see what resonated, and make changes at pace that incrementally created more value without putting the broader brand at risk.

Within four months of kickoff, we had paying customers and real evidence to guide the next phase of growth. As the client later described it, “What they delivered wasn't a pilot. It was an experimentation engine.” Each experiment helped inform the next decision, ultimately laying the foundation for a successful, growing business at a fraction of the cost of a conventional internal build.

When building is cheap, judgment is premium.

We've all seen impressive demos that don't translate into business value. Or pilots that keep running because no one has decided what success should look like. When building takes less time and money, it's tempting to give an idea another week, another feature, another round of testing.

A few things become even more important:

• Compare it with what happens today. Run an AI solution alongside the existing process. See whether it actually improves quality, speed, or cost—including the human effort involved.
• Watch what people do. Enthusiasm for a demo is encouraging. Whether people use it, pay for it, or come back tells you more.
• Decide what comes next. Scale it, change it, or stop. An experiment that tells you not to invest can be every bit as valuable as one that leads to a launch.

Founders learn this early: with limited time and money, you can't afford to pursue everything. That's the experience our Entrepreneurs in Residence bring to client teams. They've built businesses, made difficult trade-offs, and learned when to back an idea and when to walk away.

AI gives us more room to explore. We still have to choose where to spend our time—and act on what we learn.

Build faster. Learn smarter. Invest in what works.

Recognition reflects a decade-long track record of helping organizations turn ambitious ideas into real-world impact through deep, enduring partnerships.

A decade ago, we set out to build a different kind of firm. One that brings both strategic thinking with hands-on execution, working closely with corporate leaders and entrepreneurs to develop, launch and scale new businesses, products and capabilities.

We have always believed that the best outcomes come from true partnership. From understanding the challenges our clients face, sharing in the complexity of the work and staying committed for the duration. Today, we're incredibly proud to share that Co-Created has been named a 2026 Inc. Power Partner, recognizing companies that help businesses grow and succeed.

For us, this recognition is especially meaningful because it reflects the very principles on which Co-Created was founded.

The need for partnership has never been greater

The business environment has changed dramatically over the past decade, and the pace of change continues to accelerate. Technology is advancing faster than ever. AI is reshaping what's possible across industries. And expectations for corporate teams have never been higher. Leaders are being asked to innovate faster, deliver measurable results and make the right decisions in an environment where the opportunities and risks are constantly evolving.

But while technology can accelerate what's possible, turning that potential into meaningful business outcomes remains hard. It requires more than a compelling idea or a well-designed strategy. It takes the ability to navigate organizational complexity, align stakeholders, build the right capabilities, execute effectively and adapt along the way.

And increasingly, it requires partners who are willing to do the work alongside you.

Built for the journey, not just the moment

At Co-Created, we've always believed that the most valuable partnerships extend well beyond any single project, milestone or deliverable. We work alongside our clients from the earliest stages of exploring what's possible through the hard work of building, implementing and scaling solutions that deliver real value. 

That means being there when the path forward isn't obvious. Bringing the right expertise to the table. Challenging assumptions, solving problems and adapting as circumstances change. It also means sharing a commitment to the outcome, not simply the process. This approach has shaped how we've built Co-Created, the relationships we've developed and the work we're privileged to do every day.

A recognition we share

Being named an Inc. Power Partner is a meaningful milestone for our firm, but it's also a reflection of the people and partnerships that have made this possible. We're proud of what we've accomplished together over the past decade.

Here's to the next decade of building, creating and scaling what comes next.

‍

A system that looks intelligent and a system that is reliable are not the same thing — especially when you cannot tell them apart.

This difference rarely matters in casual use. It matters enormously when the output informs real business decisions.

***

Exec Summary. The rapid improvement of LLMs has created a widespread belief that better models alone will drive enterprise transformation. We believe that view is incomplete. What organizations experience as AI is not just a model, but a composite system: orchestration logic, data gathering, validation, constraint enforcement, and presentation layers working alongside probabilistic generation. In enterprise environments, real impact does not come from unconstrained autonomy or clever prompting. It comes from disciplined structure. Durable agentic systems require deliberate data collection, deterministic analytical playbooks, traceable reasoning paths, dynamic replanning, and consistent presentation. Intelligence at scale (that’s meant to inform real business decisions) emerges from the interaction between agency and constraint; encoding methodology into systems that humans can understand, validate, and act upon.

***

Consider a simple example. A corporate innovation team runs a competitive analysis on a relatively new company they are considering partnering with. They delegate the task to an AI-powered workflow. Within minutes, it returns a polished report: messaging themes, positioning analysis, market perception, strategic recommendations. The output is fluent, well-structured, and confident.

It is also wrong from the very first step.

The company name had been used by a different organization about a decade earlier. That older company had been acquired but left behind a large digital footprint: press coverage, archived product pages, conference talks, and commentary. The newer company, by contrast, had a sparse website and minimal public presence. Faced with this imbalance, the system gravitated toward the richer dataset. There was simply more to find. From the system's perspective, this looked like success: results were plentiful, synthesis could proceed, nothing explicitly failed.

But every subsequent stage – data collection, messaging analysis, competitive positioning, conclusions – was now applied to a company that no longer existed in the relevant form. The result was coherent and professional-looking. It was also completely wrong. Because the error occurred at the level of identity selection, no amount of downstream quality could recover correctness.

A human analyst encountering the same situation would notice the mismatch in timelines, question why an apparently new company had a decade of historical content, and look for confirming signals: incorporation dates, current products, leadership continuity. They would recognize that abundance of data is not evidence of correctness. If uncertainty remained, they would pause and seek clarification before proceeding.

The AI system did none of this, not because the underlying model lacked capability, but because no one had designed the surrounding process to ask those questions. And this is just one of many silent failure modes where AI completes the task it was given, produces a professional-looking deliverable, and still misses the mark, because the methodology governing how it approached the work was flawed from the start.

The distinction that matters

The failure above is not a model failure. It is a system failure. Most of what makes AI useful — or unreliable — in professional contexts does not live in the model. It lives in the system around it.

When people reference tools like ChatGPT or Claude, they tend to attribute the observed intelligence to the model itself. A common assumption follows: as models improve, today’s shortcomings will simply disappear. This framing treats model capability as the binding constraint on every problem, when in practice most of the limitations organizations encounter have little to do with the model at all. In reality, what users interact with is a composite system: an orchestration layer responsible for planning, tool invocation, data retrieval, validation, formatting, and constraint enforcement, wrapped around a model that primarily performs language understanding and generation.

This distinction is not academic. It determines whether the output can be trusted.

When a system retrieves financial data, constructs tables, cites sources, or decides to branch into a follow-up search, these are not spontaneous properties of the model. They are the result of explicit design decisions — decisions that are invisible to the end user. Every chat-based AI tool makes choices about how to search, what to retrieve, how much context to retain, and how to structure its response. These choices shape the output as much as the model's own reasoning does, and the user typically has no visibility into any of them. The companies behind these tools have made their own design and process decisions – how to search, what to prioritize, and when to stop. Those decisions may or may not align with how your organization would approach the same task.

You are not just using a model.
You are deferring to someone else's methodology, whether you realize it or not.

Consider a basic constraint most users never think about. When a chat conversation gets long enough, the system quietly drops earlier context to stay within its processing limits. It re-reads the entire conversation history every time you send a message, and at some point, begins discarding information it decides is no longer a priority. Your analyst doesn't forget the first half of the brief because it got too long. This is one example of an invisible system constraint that shapes output quality independent of model capability.

Models are genuinely improving. That is not the issue. What this paper is arguing is that the visible successes mask invisible assumptions that matter enormously in professional contexts. When fluent outputs and successful demos are treated as evidence that the hard problems are solved, attention shifts away from system design and methodology. Complexity does not disappear; it simply moves out of view.

Where things quietly break

To see why this matters in practice, consider what happens when an organization uses AI for competitive analysis — not as a one-off question, but as part of a structured process that informs strategy, budgets, and priorities.

The wrong-target problem. The entity confusion described above is not an edge case. Any company with an ambiguous name, a recent rebrand, or a limited digital footprint is vulnerable. The system's default behavior is to follow the richest data trail, which often means the most visible entity rather than the correct one. The output reads like a thorough analysis of the right question — about the wrong company. This failure mode is especially dangerous precisely because it is invisible in the final deliverable.

Shallow collection disguised as research. Even when the correct entity is identified, speed bias shows up in what happens next. A system runs a search, retrieves a press release, a generic "About Us" page, and a two-year-old blog post. It declares data collection complete and moves to synthesis. The output reads well, but anyone with domain experience would recognize that the signal is weak: the material is shallow, dated, and reveals almost nothing about how the company actually positions its products or frames its strategy. A human analyst would not stop there. They would explore the company's website in depth, use targeted searches to surface relevant pages, actively judge relevance, discard superficial content, and continue until coverage felt sufficient, not until a default result limit was exhausted.

Representative sampling matters. A chatbot might collect a handful of Reddit posts or social media comments and summarize them as if they reflect the broader market. The narrative may sound authoritative, but the evidence is fragile. Twelve posts cannot represent the range of sentiment in a market, and a different twelve posts selected on a different day might tell a different story.

The gap is not about effort or volume; increasing sample size is easy. The issue is whether the system understands that data gathering is an evaluative act requiring judgment about sufficiency, not simply executing a retrieval step.

What counts as “sufficient” is specific to your organization — your standards, your stakeholders’ expectations, and the decisions the analysis is meant to inform. A threshold suitable for an internal briefing is not the same as one required for a board-level recommendation. No general-purpose tool can know that standard unless someone encodes it into the process.

Non-determinism across runs. Given the same input, an LLM-driven workflow may produce outputs that vary significantly from one execution to the next. One run might emphasize messaging and tone; another might focus on product features; a third might omit entire dimensions of analysis. For casual exploration, this variability feels like creativity. For an organization paying for a consistent methodology — a way of answering, not just an answer — it is a liability. A report that changes structure, emphasis, or analytical framing from run to run undermines trust and makes downstream consumption harder, not easier.

Asymmetric comparisons. Now imagine the system is comparing four competitors. It collects website content, social media data, and analyst commentary for three of them. The fourth has no social media presence and minimal coverage. A simplistic workflow proceeds regardless: it generates whatever output it can and presents the result alongside the others. Three competitors analyzed in depth, one barely sketched — but the system does not flag this asymmetry or adjust. A human analyst would stop and ask: does this company still belong in the comparison set? Can the missing data be sourced elsewhere? Should a replacement be identified? If so, does the analysis need to be rerun from the start to preserve like-for-like comparability? This is not a minor procedural point. It is the moment where analytical integrity is either maintained or quietly abandoned.

These failure modes share a common trait: the output still looks good. The prose is fluent, the formatting is clean, the confidence is high. The errors are not in the language. They are in the invisible decisions that preceded it — what was collected, what was compared, what was assumed, and what was silently omitted.

Why structure is the answer

If the problem lives in the system, the solution does too. That solution is structure – and this is where most AI conversations go wrong.

The instinct many organizations have at this point is to improve their prompts: give the AI more detailed instructions, more context, a better brief. And this helps. More sophisticated prompting genuinely produces more consistent, higher-quality outputs. It is a meaningful step beyond naive usage.

But it does not solve the underlying problem. However well-crafted the prompt, the model is still probabilistic. The orchestration layer is still opaque. And the system still lacks the persistent state, evaluative judgment, and methodological discipline that professional analytical work requires.

Some teams go further and build their own automated workflows, chaining AI calls together in tools designed for that purpose. This represents real investment and real progress. But it also introduces a challenge that is easy to underestimate: building a workflow that works in a demo is not the same as building a system that works reliably in production. Edge cases, error handling, state management, and graceful degradation when data is missing are all software engineering problems. They require software engineering discipline. The fact that the building blocks are AI calls rather than traditional code does not make the surrounding system design any less critical. If anything, the non-determinism of the components makes it more so.

Organizations that extract durable value from AI-powered analysis impose structure at every layer. In practice, this means codifying the analytical methodologies that already exist implicitly in experienced teams. Over years of work, corporate teams, just like agencies and consultancies, develop standard approaches to competitive intelligence, market analysis, and strategic research. These approaches define not just what data to examine, but how to examine it: which questions to ask, how to structure outputs, how to communicate findings, and how to handle gaps and ambiguity.

Encoding these methodologies into software involves a mix of conventional programming and LLM-based components. Some steps are purely procedural – data collection, filtering, deduplication, categorization. Others rely on the model's ability to reason, summarize, or interpret. Crucially, the system constrains the model's role. It does not decide whether a competitive analysis includes pricing, positioning, or tone. Those decisions are already embedded in the playbook. The model operates within a defined scope, producing outputs in specific formats, using consistent language, addressing specific analytical dimensions of the problem.

This is not a reduction of intelligence. It reflects how human expertise actually works. Experienced analysts do not reinvent their methodology from scratch for every project. Their skill lies in selecting the right approach for a given context and executing it consistently — not in arbitrarily varying their process each time. Structured AI systems operate the same way: agency at the level of planning and routing, discipline at the level of execution.

And structure does not eliminate the model's capacity for novel insight. It creates conditions where such insights can be reliably captured, evaluated, and acted upon — rather than appearing and disappearing randomly across runs.

What good looks like

The difference between a system that generates text and one that supports real decisions comes down to three capabilities.

Traceability. When an end recipient challenges a claim (e.g.  "competitor messaging focuses on cost reduction") a well-designed system can show the evidence trail: how many data points were collected, how they were categorized, what representative examples look like within each category, and how the conclusion was derived. A weak system can only restate the claim in different words. This is the difference between output and decision support. The value is not just accountability; it is that humans can interrogate conclusions, challenge them, and build on them with confidence.

Regeneration from source, not surface editing. Professional contexts often require the same analysis to be reframed through a different lens, e.g. shifting from competitive positioning to technology evaluation, from strategic overview to investor briefing. A naive approach takes the final text and rewrites it, swapping vocabulary and adjusting emphasis. But conclusions that made sense in a positioning context may not hold in a technical one. A more rigorous approach propagates the reframing through the entire analytical pipeline, regenerating intermediate analyses where necessary and ensuring that the questions asked of the data change when the lens changes. The rewrite should be a genuine reanalysis, not a veneer over a misaligned foundation.

Consistent presentation as part of the analytical contract. Even when AI systems produce well-formatted outputs, consistency across reports and across time still matters. A monthly competitive update should feel like a continuation, not a reinvention. End recipients should not have to re-learn the structure of a deliverable each time they receive one.

In professional settings, presentation is part of the analytical contract. When structure, framing, and layout change from run to run, it creates friction for the people consuming the work. It also makes it difficult to make comparisons and track patterns over time. Treating presentation as a first-class component of the system — not something left to generative variability — reinforces the broader point: reliability emerges from disciplined structure, not from unbounded generation.

What this means for your organization

Most organizations are already using AI for research, analysis, or strategic input. The question is no longer whether AI is capable enough. The question is whether you understand the system shaping your outputs.

Every AI tool your team uses makes invisible decisions about how to search, what to retrieve, how much context to retain, and how to structure its response. These decisions constitute a methodology — one you did not choose, cannot inspect, and have no guarantee is aligned with how your organization thinks about the problem.

Most organizations build structured methodologies for research, innovation, and competitive analysis precisely to counteract bias, gut feel, and strategic inertia. These frameworks exist to enforce discipline in how evidence is gathered and interpreted. Introducing AI-generated outputs into these processes without holding them to the same standards creates a new vulnerability. An answer may sound well-reasoned yet be based on insufficient sampling, misidentified entities, or hidden methodological assumptions. When such outputs enter reports or decision-making forums unchallenged, they can quietly distort direction. The danger is not obvious error, rather it is a gradual misalignment.

For internal strategy teams, this plays out in planning conversations. When an AI-generated analysis is presented and looks thorough and well-sourced, no one in the room has reason to question it. Budgets shift. Priorities adjust. The cost rarely appears as dramatic failure. It appears as wasted effort, flawed prioritization, and opportunity cost that only becomes visible later.

For consultancies and agencies, the implications are sharper still. Your clients are paying for judgment — contextualized, structured, and defensible. That judgment is embedded in your process. If analysis is delegated to a tool that does not reflect your methodology, you are substituting your intellectual framework with a commodity layer. Over time, this erodes differentiation. If the underlying method is indistinguishable from what anyone else can access, the insight becomes indistinguishable too.

The organizations that extract durable value from AI will not be those chasing the most fluent outputs. They are those that design systems where structure, validation, and transparency are first-class concerns, where intelligence is not just generated, but disciplined.

And that leads to a deeper risk.

If you do not understand the method shaping an output, you are outsourcing judgment.

And in business, outsourced judgment always carries consequences.

*********

Co-Created is a venture studio that helps organizations design and build structured AI systems for research, competitive intelligence, and strategic analysis. We work with companies that have outgrown chatbot-level AI usage and need systems that reflect their own methodology, not someone else's.