What Building AI Taught Me About Leading AI Transformation

Why understanding how AI works can change the way we approach business transformation.

Over the past couple of years, I’ve spent quite some time reading about Artificial Intelligence, following new developments, and thinking about how these technologies could change the way companies operate. Like many people working in digital transformation, I’ve also been exploring different AI tools, testing their capabilities, and trying to understand where they could bring real value to a business.

But at some point, I realized that using AI and understanding how it actually works are two very different things. I could have a conversation about AI strategies, identify potential use cases, or discuss the opportunities with technical teams. However, I wanted to understand what happens behind the scenes. What does it really take to build an AI application? How complicated is it to connect an AI model to company information? And, perhaps more importantly, what separates an interesting experiment from something a business can actually use?

So I decided to get my hands dirty and spend some time building AI applications myself.

Not because I’m planning to become an AI developer. That’s not my objective, and I certainly don’t pretend that a few months of learning can replace years of technical experience. I did it because I believe that, when you’re responsible for leading technology and business transformation initiatives, understanding how things work helps you make better decisions.

And I have to say, the experience has changed the way I look at some aspects of AI transformation.

Building Something Is Different from Talking About It

One of the things I enjoyed most was moving from theory into practice. During my learning journey, I worked with language models, explored how they process information, experimented with different ways of improving their responses, and built applications that connected AI capabilities with other tools and sources of information.

Some use cases were relatively simple. Others required more work than I expected. There were moments when something worked almost immediately, which was quite exciting, and others when I spent more time trying to understand why an application wasn’t behaving as expected.

One particularly interesting use case involved building a question-and-answer application that could work with information from external documents. The idea sounds simple: instead of relying only on what a language model already knows, you connect it to a specific collection of information that it can use to answer questions.

This approach is commonly known as Retrieval-Augmented Generation, or RAG. In practice, the application needs to process the documents, organize the information into smaller pieces, and make it searchable. When someone asks a question, the system looks for the most relevant information and provides it to the language model, which uses that context to prepare its answer.

But here’s something I found particularly interesting: giving an AI model access to the right information doesn’t automatically mean it will provide the right answer. The system might retrieve information that is not relevant enough, miss an important detail, or combine different pieces of information incorrectly. And, as we’ve probably all experienced when using AI tools, models can sometimes produce answers that sound perfectly reasonable but are simply wrong. That’s what we call hallucinations.

So part of the challenge is not just building an application that answers questions, but making sure those answers can be trusted. You need to think about how the information is selected, how the model is instructed to use it, and what should happen when the available information isn’t sufficient to answer a question. For example, you can instruct the model to base its answers only on the retrieved documents, provide references to the sources, and clearly acknowledge when it doesn’t have enough information rather than inventing an answer.

Of course, these measures don’t eliminate every possible error. You still need to test the application with different questions, check whether the answers are accurate, and understand where it might fail. Depending on the business use case, you may also need people to review certain responses before they are used to make decisions.

What I found encouraging is that these challenges are not impossible to manage. There are already approaches and tools that help improve reliability, evaluate results, and monitor how AI applications behave. The important thing is to understand that this doesn’t happen automatically just because we’ve connected a powerful language model to some documents.

While working with this technology, I started thinking less about the application itself and more about what it could mean for a business.

And, coming from a sales and digital transformation background, my mind naturally went to the sales force.

Imagine What This Could Mean for Your Sales Team

Think about a company selling complex products, with hundreds or thousands of product references, technical specifications, application guides, compatibility rules, and other documents. The sales team needs to understand not only what the company sells, but also how its products can solve specific customer problems.

Even experienced salespeople cannot remember everything. Information may be available in product databases, technical documents, internal portals, presentations, or even emails. But finding the right answer often takes time, especially when the customer asks a question that goes beyond the standard product description.

Now imagine that a salesperson could simply ask an internal AI assistant: “Which of our products would be suitable for this application?” or “What are the main differences between these two models?” The assistant could search the relevant documentation, provide an answer, and indicate where that information comes from.

The potential is quite interesting. Salespeople could spend less time searching for information and more time talking to customers. New team members could become productive faster, and companies could make better use of the product knowledge they already have.

We could even take the idea further. What if this assistant could also connect to the CRM, understand the customer context, and suggest relevant products or opportunities? Suddenly, we’re not just talking about searching documents. We’re starting to explore how AI could support the sales process itself.

Of course, this is where we need to be careful. A working demonstration is one thing. Giving a sales team an assistant they can trust when talking to customers is something very different. Product recommendations must be accurate, information must be current, and the system needs to know when it doesn’t have enough information to provide a reliable answer.

And that brings me to one of the most valuable lessons from the experience.

The Prototype Is Often the Easy Part

When you start building AI applications, it’s surprisingly easy to become excited about what you can achieve with relatively little effort. There are powerful models available, plenty of frameworks, and tools that make it possible to connect different components without having to develop everything from scratch.

But making something work in a controlled environment is not the same as making it work reliably in a company.

Let’s go back to the sales assistant example. Before deploying something like this, we would need to answer several questions. Where does the product information come from? Who is responsible for keeping it updated? What happens when two documents contain conflicting information? Can different sales teams access the same information? And how do we make sure the assistant doesn’t recommend a product that is not available in a particular market?

These are not simply technical questions. They involve product management, sales, IT, data governance, and potentially legal or compliance teams. They also require decisions about how the solution will be maintained, how its performance will be measured, and who will take responsibility when something goes wrong.

There is also the question of integration. An AI assistant that works independently might be useful, but its value can increase significantly when it becomes part of the systems and processes people already use. Connecting it with a CRM, product information management system, or other business applications introduces another level of complexity.

None of this makes the opportunity less attractive. Quite the opposite. It helps us understand what is required to turn a promising idea into something that can deliver real value.

In fact, this is something I’ve seen many times in digital transformation projects, long before AI became such a popular topic. Technology can open the door to new opportunities, but delivering the expected business results requires much more than implementing the technology itself.

AI is no exception.

Who Should Own AI in the Organization?

Another thought that became clearer to me during this experience is that AI cannot be treated exclusively as an IT initiative.

Of course, technical expertise is essential. We need people who understand the models, architecture, integrations, security, and many other aspects involved in developing and operating AI solutions. Without them, even a good idea can quickly become a problem.

However, the business needs to play a central role, especially when it comes to defining the problems we want to solve and the outcomes we expect.

If we’re building an AI assistant for the sales organization, who should define what success looks like? Is it the IT department because they manage the technology? Or should the sales organization be responsible because the objective is to improve sales productivity, customer service, or conversion?

For me, the answer is relatively clear. Technology teams should own the technical delivery and operation, while business teams need to take responsibility for the business value. But both sides must work closely together, from the initial idea through implementation and continuous improvement.

The same applies to governance. We need clear rules around data, security, responsible use, and quality. At the same time, we need to avoid creating so many procedures that experimenting with AI becomes unnecessarily difficult.

Finding the right balance between innovation and control will be one of the challenges for companies developing their AI capabilities.

And there’s another question worth considering: how much should a company centralize its AI expertise, and how much freedom should individual business units have to explore their own opportunities?

I don’t think there’s a single answer that works for every organization. Company size, existing capabilities, business complexity, and the level of AI adoption will all influence the right approach. But I do believe that leaving every department to experiment independently, without shared standards or a clear strategy, can easily lead to duplicated efforts and solutions that are difficult to maintain.

What Business Leaders Should Take from This

I don’t believe every executive needs to learn Python or understand the technical details of how a language model is trained. There are specialists for that, and their expertise should be respected.

But I do believe that leaders responsible for AI initiatives need enough understanding to ask the right questions, challenge assumptions, and recognize the difference between an impressive demonstration and a realistic business opportunity.

For example, when someone presents an AI solution, we should be able to discuss more than its functionality. Where will the information come from? How reliable are the results? What level of human supervision is required? How will it fit into existing processes? What will it cost to operate and maintain? And, most importantly, what business problem are we actually solving?

We should also be willing to challenge whether AI is the right solution in the first place. Not every business problem requires a language model, an AI agent, or a complex new application. Sometimes improving an existing process, organizing the data properly, or using a simpler technology might deliver better results.

These are strategic decisions as much as technical ones.

One thing my experience reinforced is the importance of learning through experimentation. Reading about AI is useful, but trying to build something gives you a different perspective. You begin to understand both the possibilities and the limitations, and that makes conversations with technical experts much more productive.

It doesn’t mean that everyone needs to follow the same path I did. But companies should think about how to help their business leaders and teams develop a practical understanding of AI, rather than limiting their knowledge to what they see in presentations or product demonstrations.

From Understanding AI to Building AI Capabilities

Looking back, I’m glad I decided to spend time exploring how AI applications are built. I learned a lot, had the satisfaction of seeing some of my experiments work, and gained a better understanding of the challenges involved.

Perhaps the most valuable result, though, is that it reinforced something I’ve believed for a long time about digital transformation: understanding technology matters, but understanding how to make it work for the business matters even more.

AI creates exciting opportunities, and we’re still discovering many of its potential applications. But having access to powerful models and tools doesn’t automatically mean that a company is ready to take advantage of them.

Organizations will need the right skills, processes, data, governance, and business ownership to move beyond isolated experiments and turn AI into something that supports their daily operations.

That raises a bigger question, and one I’d like to explore in a future article: how should companies organize themselves to build these AI capabilities, and where should they start?

Because building an AI application is one challenge. Building an organization that knows how to use AI effectively is another.

Jaime Porta Avatar

Posted by

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.