Separating Signal From Noise: An Evidence-Based Framework for AI’s Impact

• While a majority of organizations are actively using AI, most report little to no benefits, or inefficiencies that undermine its effectiveness. 
• Determining what type of LLM (large language model) and level of complexity for tasks can help organizations invest in adequate resources to optimize and maximize AI’s potential.
• To best serve an organization, an AI model must be designed to scale, adapt, and learn across all silos of your business.   

As discussions of AI continue to increase across industries due to its promise of increased productivity and unlocked new revenue opportunities, the potential applications of AI appear limitless. Yet, despite the widespread buzz, a significant disconnect remains between AI investment and its real‑world impact.  

While many organizations see the opportunity, far fewer succeed in deploying AI in a way that actually delivers sustained value. This gap highlights the need for a more disciplined, evidence‑based approach to deciding when and how to use AI.  

In a discussion hosted by the Detroit Regional Chamber, Thomas Hancock, a cognitive scientist, AI innovator, and Senior Manager of AI LLM Technology Architecture at Accenture, spoke on how organizations can move beyond hype and toward real impact. 

Watch the full event recording below.

Despite widespread enthusiasm, Hancock pointed to a significant disconnect between AI investment and execution. While the opportunity is widely recognized, far fewer organizations are seeing measurable returns.  

He cited recent research showing that roughly 80% of organizations are investigating AI and see the opportunity, yet only 40% move to pilot programs, and just 5% implement and deploy AI effectively. The gap, Hancock explained, is rarely about a lack of ambition, but rather stems from treating AI as a standalone tool rather than as part of a broader organizational ecosystem.  

“When we’re thinking about actually deploying AI to solve a specific use case or multiple use cases, it’s not an individual tool,” he said, “… but it’s actually trying to understand what type of system you need it to design.”

Hancock cautioned against focusing solely on a single AI model or product. While individual tools may work for specific, well‑defined tasks, they can put a business at a disadvantage. As tasks become more involved, additional sources of information and data are needed to provide the more complex knowledge expected. Because of this, AI models must be designed as an interconnected architecture that can scale, adapt, and learn across all silos of your business.  

“One of the key things I always want you to think about when we’re discussing or thinking about AI and its use is, ‘Are we creating an opportunity where the AI can learn?’” Hancock said. 

Hancock also stressed the importance of understanding how large language models differ from one another, and the aspects that directly influence their performance:  

• Training determines what data a model has learned from. This includes text, audio, visuals, or a combination of both. Without the right training foundation, models can lack relevance or accuracy.  

• Reinforcement governs how well a model learns from feedback. If reinforcement is not aligned to a specific business use case, even well‑trained models may fail to deliver useful outputs.  

• Inference determines how efficiently a model understands context and generates responses. Poor inference can quickly increase costs and reduce reliability.  

AI’s value depends heavily on the type of work its user is expecting. In routine work, such as scheduling meetings, generating reports, or processing large volumes of structured information, AI can deliver immediate efficiency gains by reducing manual effort and freeing up time for higher‑value tasks. In more advanced work, including reviewing medical records, legal documents, or conducting financial risk analysis, AI can play a meaningful supporting role, highlighting patterns or generating quick summaries. However, Hancock stressed that human oversight is still necessary and that AI is most effective with human involvement.  

By defining meaningful metrics, evaluating performance honestly, and designing systems that can learn and scale responsibly, organizations can move beyond experimentation and toward real impact. AI’s greatest value can be found through deliberate, evidence‑based deployment aligned with real organizational needs.