Customer interactions with colleagues represent critical moments of truth that directly shape customer outcomes and brand perception. Outcome audit testing plays a vital role in maintaining Lloyds quality standards and banking regulatory compliance. However, we knew auditing covered less than 1% of total colleague-to-customer interactions. It really was a drop in the ocean. How could quality assurance be reimagined?
Context was key for the project. This was a highly specialised role in the bank with significant responsibility. We needed to map out the current as-is process and observe how colleagues worked day to day to understand existing pain points and opportunities. This was a highly pressured role: they needed to ensure fellow colleagues met banking regulatory compliance alongside delivering great results to customers with every interaction. No small task.
Working closely with the product owners and business analysts, we quickly mapped the journeys for typical audit task completion, with a focus on the mortgages department, our first beta area. We did similar work in other business areas to understand the broader business challenges. This was a critical step, as a wider holistic LBG vision was going to amplify the transformation potential of this project. My role was to elevate ambition and reach, reimagining interaction auditing, not micro improvements for a singular department which couldn’t be replicated elsewhere. Transformation was the target.
As part of the broader vision, we began looking beyond auditing to explore how we could support colleagues before and after the audit itself. A consistent theme emerged. We could improve the auditing process, but without effective coaching alongside it, we risked setting colleagues up to fail.
This was a pivotal moment. Our journey mapping revealed that for greater impact, we needed to look upstream, focusing on what happens before an audit occurs.
For complex interactions, reviews could take upwards of 2 hours per case, limiting capacity and forcing skilled colleagues to spend time listening to call playbacks rather than actively coaching colleagues and directly improving customer interactions. The role had become a checkbox activity, focused on how many interactions had been reviewed. How could we bring the focus back towards customer outcomes?
Reviews often occurred weeks or months after the interaction took place, significantly reducing the opportunity to influence outcomes, reinforce best practices, or correct issues in real time. Auditing colleagues rarely saw their impact linked back to improving customer outcomes, whilst colleagues taking calls felt they were being caught out.
The bank generally struggled to deliver consistent, high-quality interactions and appropriate interaction oversight, auditing and coaching. Auditing covered less than 1% of total colleague-to-customer interactions per quarter.
I started by reimagining the North Star journey. The intention was to bring the vision to life quickly and establish an early feedback loop. This also created an opportunity for the team to get their ideas, thoughts and assumptions out of their heads and onto paper, making previously implicit knowledge visible and accessible in a shared space.
This was a significant step towards democratising information. By making our thinking tangible, we gave everyone the opportunity to engage with, challenge and objectively review the emerging artefacts. It created a shared understanding and provided a common foundation for moving forward together.
Orion Test
Orion Test is transforming outcome testing from reactive sampling to proactive, AI-powered oversight. The in-house GenAI product transcribes, analyses, grades and summarises customer interactions, using sentiment analysis to identify critical moments of truth and customer vulnerability at scale. By automating interaction reviews and providing actionable insights, Orion enables comprehensive quality oversight, supporting colleague development, regulatory oversight, and directly improves customer outcomes by helping to deliver quality, consistent LBG colleague-to-customer interactions.
Interactions audit reviewed in minutes, not hours
Behavioural change
Orion Test is enabling a fundamental shift in how audit teams operate, moving from manual review and administration towards insight-led coaching that delivers better outcomes for customers and colleagues.
Caring
With over 5 million transcribed customer interactions and counting, Orion Test can identify patterns in sentiment, behaviour and vulnerability at a scale previously impossible, empowering teams to uncover insights and act on risks that might otherwise go unnoticed.
Consistency
Orion brings consistency to customer interactions at scale with bias-free auditing for any interaction type and less error-prone reviews. Helping raise quality standards across LBG.
Orion Test
Orion Test is transforming outcome testing from reactive sampling to proactive, AI-powered oversight. The in-house GenAI product transcribes, analyses, grades and summarises customer interactions, using sentiment analysis to identify critical moments of truth and customer vulnerability at scale. By automating interaction reviews and providing actionable insights, Orion enables comprehensive quality oversight, supporting colleague development, regulatory oversight, and directly improves customer outcomes by helping to deliver quality, consistent LBG colleague-to-customer interactions.
150mins
For complex case reviews
Complex audit reviews that previously took over two hours can now be completed in as little as five minutes using Orion Test. By automating transcription, analysis and summarisation, the platform significantly reduces manual effort, enabling audit teams to review interactions more efficiently and at greater scale.
Reduction in coach review preparation time
Historically, audit handlers reviewed customer interactions by replaying call recordings and making notes throughout the conversation. This was time-consuming, highly manual and less accessible for colleagues who preferred alternatives to audio playback. Orion Test has transformed this process. Colleagues now have access to over 5 million fully transcribed customer conversations, enabling them to search, analyse and connect interactions across entire customer journeys, supported by AI-generated summaries and insights. Reviews are significantly faster, shifting time towards colleague coaching and customer outcomes. Following a successful beta phase, Orion Test has been rolled out across the Mortgage audit teams, with expansion underway into Transport, Complaints and Consumer Growth.
Audit interaction coverage
Audit teams do an incredible job, but often lack the scale and capacity needed to meet the growing volume of digital customer interactions. Orion Test has demonstrated the potential to transform this business model. Moving from random spot-checking to auditing 100% of customer interactions provides greater oversight and consistency. Automated test-point generation reduces reliance on localised departmental expertise, enabling audit teams to support one another more effectively and improve operational resilience.
Orion Coach has supercharged coaching, extending its reach, improving operational efficiency and driving better customer outcomes. It has shortened the coaching feedback loop, shifting colleagues from reactive support towards proactive self-service and insight-led, manager-driven coaching. Feedback and insights are now surfaced at the point of need, rather than months after an interaction.
Importantly, Orion doesn’t replace the human element. Our research confirms that colleagues value human-led coaching. Instead, Orion simplifies the preparation, enabling more targeted, data-informed and meaningful conversations. Orion is the perfect wingman, by your side when you need them most.
Orion Coach
Calls transcribed
Managers have limited time to review colleague interactions, particularly full end-to-end journeys, often ranging between 20 minutes and 2 hours per case. This makes regular coaching difficult to deliver at the frequency they desire. Orion Coach is estimated to reduce coaching preparation time by 20% per review, freeing managers to focus on meaningful coaching conversations rather than manual auditing beforehand. This enables more frequent, timely and impactful colleague development.
Customer recovery opportunities identified vs manual review (est)
Even the best colleagues can make mistakes. Today, issues are often identified too late, or not at all, limiting the ability to proactively remediate and deliver the best for customers. Orion Coach acts as an intelligent support partner, continuously monitoring interactions and highlighting potential areas for review. It can link multiple interactions to build a complete understanding of the customer journey and align to LBG policies. This empowers colleagues to take timely corrective action, turning potential missteps into opportunities to strengthen customer relationships.
Interaction coverage before vs after
Manually reviewing colleague interactions was highly time-consuming for managers, resulting in coaching coverage of less than 5% of monthly interactions. Orion Coach has transformed this process, increasing coaching coverage by 90% within the beta group. As interaction volumes continue to grow, Orion Coach has significant potential to scale, providing consistent quality assurance and actionable coaching insights for every digital interaction.
Key highlights
Orion Test
Delivered the first in-house fully operational colleague transcription capability within LBG, transforming how customer interactions are captured and audited. The solution significantly reduces administrative burden by eliminating lengthy call listening and manual note-taking, enabling colleagues to focus on higher-value activities. The product is estimated to save over 100 working hours per colleague annually, improving efficiency, audit readiness and colleague experience at scale.
Transformed a labour-intensive quality assurance process into a scalable AI-enabled solution, significantly improving productivity while increasing oversight and consistency of customer interactions.
Orion Coach
Influenced strategic decisions around AI-enabled colleague experiences, helping position Orion Coach as a scalable capability with the potential to improve coaching effectiveness, operational efficiency and customer outcomes. Led the creative vision for Coach, turning ideas and aspirations into a live working product.
Led key stakeholder conversations around integrating Orion with the bank’s internal knowledge source, Athena, providing a strong use case for improving colleague coaching and policy and process adherence.
Roles and teams reinvented
Orion has enabled teams to fundamentally rethink their structure and roles. As AI takes on repetitive tasks, colleagues are transitioning from auditors to coaches, focusing on higher-value activities rooted in improving customer outcomes.
Previously, audit teams were small, stretched and highly specialised by department, leaving limited resilience when unexpected absences or changes in demand arose. By democratising knowledge and reducing reliance on departmental expertise, Orion has enabled teams to support one another across departmental boundaries, creating a more flexible, collaborative and resilient banking operation without compromising on quality.
The feedback
Orion has had a far-reaching impact, growing from humble beginnings as a beta into a solution now scaling across new departments and reaching thousands of new colleagues.
“One of the standout moments and without doubt a highlight for many in the room was the introduction of Orion… absolutely blew us away with what’s coming”
Assistant manager, Operations
We knew Orion was an ambitious and revolutionary GenAI product, achieving several UK banking firsts. But this quote captures something more: the genuine excitement and recognition that Orion wasn’t just another tool, but something with the potential to deliver real value for our colleagues. Here are a few more Orion mentions from across the bank.
1. A culture focused on delivering something, anything
As the UK’s largest digital bank, driving change can be challenging, particularly in a culture where success is often measured against short-term delivery goals. We all love an ambitious target, but is it moving us towards a destination we actually want, or simply saying we’ve shipped? One of my biggest challenges was influencing beyond these short-term goals. I believed the greatest risk was building a solution too narrowly focused on an individual department and missing the bigger opportunity to transform auditing and coaching more broadly.
I learnt the importance of building relationships, visualising the vision and constructively challenging thinking. Defining and measuring impact through meaningful metrics was equally important, helping galvanise people around a shared goal.
2. AI has changed the rules, everybody can be a designer
AI has transformed how we work, and design is no exception. Design has become democratised. For the first time, colleagues were presenting prototypes and concepts for me to develop while I was still sketching initial ideas on paper. What might have taken me days to produce was now taking typical non-design roles such as business analysts or product owners just a few hours. With these lower barriers to entry, they all wanted to be setting the design direction and turning their ideas into reality.
I quickly embraced this new approach and realised my traditional design process was too slow. AI could get colleagues 80% of the way when problems were well understood, but the final 20% mattered. Concepts might look good, but did they solve the right problem in the right way? Were they accessible, intuitive and considered? Did they create moments of delight?
I learnt that AI could accelerate creation, but good design judgement was still needed. I embraced a more collaborative approach, co-designing with non-design colleagues and spending less time pushing pixels and more time guiding others, challenging and shaping the direction. Whilst stepping in when necessary, particularly to ensure Lloyds brand and experience standards are applied. Colleagues appreciated the design input and recognised the quality difference when I refined and pitched against earlier iterations.
Tools
Figma
Maze
Teams
FigJam
Expertise
Research
Prototyping
Usability testing
Accessibility
Visual design
Platform
Internal colleague platform