The Five Failure Categories

The survey asked each respondent to rate 100 indicators of the engagement, 20 in each of five failure categories. Financial institution readiness covers the financial institution’s data, systems, processes, staff, management of the engagement and business case. Co-management covers the structures through which the two firms governed the engagement together. Vendor capability covers the vendor’s leadership, commercial model, methods, knowledge and talent. AI technology covers the reliability, control and observability of the agentic AI tools. The third-party ecosystem covers the outside providers, interfaces and data feeds on which the agents depended.

Two findings govern this Part. The first is that the indicators describing the two firms were rated worse than the AI technology and the outside systems it depended on. Financial institution readiness had the lowest mean of the five categories, 3.55 on the 10-point scale, followed by co-management at 3.65 and vendor capability at 3.67 (n = 150 engagements). AI technology and the third-party ecosystem were rated higher, at 4.60 and 4.64, and remained under the midpoint of the scale (n = 150 engagements). Each figure is the mean of the category’s 20 indicators across the 150 engagements.

The second finding is uniformity. Each of the 100 indicators was associated with all four core study measures, and each of the 22 robustness checks applied to each indicator held, 2,200 checks in total. Against cancellation, the 100 correlations lay between r = −0.69 and r = −0.56, each computed on 150 engagements. An engagement rated low on one indicator was, as a rule, rated low on all 100.

Net of the engagement indicator mean, no indicator retained an association with outcomes (Part 1). The indicators are therefore not ranked against one another by the strength of their associations. They are distinguished by their written evidence, the account the delivery teams gave of what happened when each part of the engagement failed. In each category, respondents described the delivery team absorbing the failure by hand.

Every indicator is reported in its own category, including the governance indicators of financial institution readiness and co-management. The examination of controls draws on those indicators without reporting them again (Part 3).

The five failure categories divide into two groups. Financial institution readiness, co-management and vendor capability, the three categories that describe the two firms and the way they worked together, were rated between 3.55 and 3.67 (n = 150 engagements). AI technology and the third-party ecosystem, the two categories that describe the tools and the outside systems the tools depended on, were rated at 4.60 and 4.64 (n = 150 engagements). The gap between the groups is about one point on the 10-point scale, and no category reached the midpoint.

In this study set, the delivery teams rated AI technology and the third-party ecosystem as the stronger parts of the engagement. The parts of the engagement they rated lowest were the financial institution’s data and processes, the joint management of the engagement and the vendor’s own readiness to deliver agentic work. Some respondents who commented on the tools placed the difficulty in those conditions.

“The tool was the least of it, in truth. My people spent their evenings re-keying metadata on deal documents the agent had already processed once, and kindly note that is what wore them down, not anything the model got wrong.”Engagement Manager, BPO, Corporate and back-office functions

“The tool is not what fails here. It reads a long spec paper and finds the right section, that part is fine … I would ask you to understand nobody gave it clean test data either.”Individual Contributor, ITO, Trade, settlement and reconciliation

“To be fair the agent itself is steady, it gives me the same answer most times and the facts come out right. What moves underneath it is the manager Dashboard, they push a change and the flow breaks, then everyone says the AI is unreliable. It is not the AI, it is the platform below it”Individual Contributor, CXM, Corporate and back-office functions

“The tool gets blamed for things that are not its fault. Half the flags it raises wrong are because the exposure file arrives before the month close is done.”Individual Contributor, KPO, Credit, underwriting and suitability

“the security approval queue is the real blocker here, not the model”Individual Contributor, ITO, Technology engineering and operations

The ranking has a firm limit. The categories were rated together, and an engagement rated low in one was rated low in the others. Every one of the 100 indicators was associated with all four study measures, and none was associated with outcomes once the engagement indicator mean was taken into account (Part 1). The ranking therefore describes where the delivery teams placed the weakest parts of the engagement. It does not identify one category as the source of poor outcomes, and within a category the indicators cannot be ranked against one another by their associations.

The written evidence gives the categories their distinct content. Across all five, respondents described the same response to a failing part of the engagement. The delivery team corrected the financial institution’s data, rebuilt its processes, reviewed agent output line by line and performed by hand the tasks that third-party systems did not perform. The categories are reported in order of their mean rating, from financial institution readiness at 3.55 to the third-party ecosystem at 4.64 (n = 150 engagements).

1.1 Concentration of Distress in Three Categories

The distressed share draws a firmer line between the categories than the mean. In financial institution readiness, 8,219 of the 15,966 answers (51.48%) were distressed. Co-management followed at 7,866 of 15,968 answers (49.26%) and vendor capability at 7,687 of 15,943 answers (48.22%). In the two technology categories the share was about a third, 5,113 of 15,911 answers (32.14%) for AI technology and 5,062 of 15,947 (31.74%) for the third-party ecosystem. About one rating in two placed an indicator of the two firms in the distressed range, against about one in three for AI technology and the third-party ecosystem.

The same order held in every part of the study set. Respondents on canceled engagements rated every category far lower than respondents on active engagements. Individual contributors, the role that performs the engagement’s daily tasks, rated every category lower than engagement managers and portfolio directors. Table 7 gives, for each group, the mean across a category’s 20 indicators of the indicator’s mean rating among the group’s respondents.

Two features of the table matter. Respondents on canceled engagements rated co-management the lowest of the five categories (Table 7). Among active engagements, co-management also showed the largest difference between stages, at 4.67 in Build-Out against 4.07 in Production, and the joint management of the engagement was rated lower on engagements in live operation. Portfolio directors rated every category more than one point higher than the individual contributors who worked within those circumstances (Table 7).

1.2 Uniform Association of Every Indicator With the Study Measures

Each of the 100 rated indicators was associated with all four core study measures. Engagements that rated an indicator higher had a lower cancellation rate, recorded higher customer satisfaction among active engagements and had a higher engagement indicator mean, computed without the indicator. Their respondents reported lower intent to leave. For every indicator, each of its 22 robustness checks held, 2,200 checks in total.

The associations also lay in a narrow band. Against cancellation, the 100 correlations ranged from r = −0.69 to r = −0.56 (n = 150 engagements each). Against customer satisfaction among active engagements, they ranged from r = 0.59 to r = 0.75 (n = 94 engagements each). Against the engagement indicator mean they ranged from r = 0.76 to r = 0.88 (n = 150 engagements each). Against intent to leave, they ranged from r = −0.69 to r = −0.56.

No category stood outside the band. Each indicator of the two technology categories, rated highest, was associated with cancellation at r = −0.63 or stronger (n = 150 engagements). The three categories rated lowest ranged from r = −0.65 to r = −0.56 (n = 150 engagements each).

The uniformity is itself a finding about the engagements. Conditions did not fail one at a time. An engagement whose data was poor was, as a rule, also an engagement whose joint management was weak, whose vendor was unprepared and whose tools were judged unreliable. The delivery teams rated the conditions of their engagement as a whole, and the four study measures were associated with that whole (Part 1).

The finding governs how each indicator in this Part is reported. Each indicator section states the indicator’s association with the four measures, and each association is part of the shared pattern. What distinguishes one indicator from another is its distribution, the segments that rated it lowest and, most of all, its written evidence.

Financial institution readiness was the lowest-rated of the five failure categories. Its 20 indicators had a mean of 3.55 on the 10-point scale across the 150 engagements, and 8,219 of the 15,966 answers (51.48%) were distressed. The indicators cover what the financial institution brought to the engagement. They cover its data, its systems, its processes, its staff, its management of the engagement, its governance of AI and its business case. Each describes a part of the engagement the financial institution controls or could have prepared before the engagement began.

The 20 indicators were rated within a narrow range of one another, from a mean of 3.47 to 3.61 (n = 789 to 811 respondents). Every one had about half of its answers in the distressed range. No single part of the financial institution’s readiness stood out as the weak point. The delivery teams rated the financial institution’s preparation as uniformly poor.

Mean ratings in Table 8 are computed across respondents. The category figure of 3.55 is computed across engagements.

The written evidence describes what the low ratings meant for the delivery team. When the financial institution’s preparation failed, its tasks moved to the vendor. The delivery team corrected the financial institution’s data by hand before the agents could use it and wrote down processes that the financial institution had never documented. It made decisions for which the financial institution had named no decision-maker. Respondents described doing this every cycle and, in many accounts, without telling the financial institution. The survey closed the section with an open invitation to comment on the financial institution, and 557 of the 850 respondents (65.53%) wrote a comment.

“Every file come in different. One branch send payslips as pdf, other one send a photo of the payslip taken on a phone, and the tool cannot read a photo so that file come to me. I do maybe thirty of these a day … Nobody on their side seem to know this is happening.”Individual Contributor, BPO, Credit, underwriting and suitability

“The agent is fine. It is being fed rubbish and asked to be careful with it…”Individual Contributor, ITO, Technology engineering and operations

Many respondents who rated the financial institution low took care to separate its people from its preparation. They described staff who answered quickly and knew their own business, working inside an organization whose data had no owner, whose systems gave an agent no way in and whose decisions had no named decision-maker. The failures the delivery teams recorded were failures of organization and preparation, and they are reported in that order.

The category includes three indicators of the financial institution’s governance of agentic AI, covering the ownership of AI across its functions, its capacity to oversee agent risk and its change control. The three are reported in section 2.6 with the other indicators of the category. They also inform the examination of the controls meant to govern work performed by agents (Part 3).

2.1 Financial-Institution-Provided Data

Four indicators describe the data the financial institution provided, covering its quality, its accessibility to the engagement, its normalization across source systems and the governance over it. Each had about half of its answers in the distressed range, from 387 of 795 answers (48.68%) for quality to 413 of 809 (51.05%) for normalization. All four indicators were associated with each core study measure, without exception across the 22 robustness checks.

The quality of the financial institution’s data was the subject of more written evidence than any other indicator of the category, 582 responses from 415 respondents on 133 engagements. Negative responses numbered 499 of the 582 (85.74%). Most came from the survey’s questions on altering financial-institution-provided data. Respondents described records that arrived with missing fields, duplicate entries and out-of-date reference data, corrected by hand before the agents could use them and received again with the same faults in the next delivery.

“The problem is data quality, it is worst I have seen. My last project was service desk for same type of bank, at least there the user list is clean. Here the joiner leaver file still has people who left in 2019.”Individual Contributor, ITO, Technology engineering and operations

The four failures compound one another. Data that the delivery team could not reach arrived as extracts and screenshots. Data recorded differently in each source system had to be reconciled before use. Data with no owner reached the vendor without validation, and the financial institution could not say which version was correct. In each case the correction passed to the delivery team, and in many accounts it was reported within the vendor and not to the financial institution.

“The people on their side are not the problem and I want that on record … The same vendor sits in there three times with three tax numbers, spend categories are free text in one region and coded in another, and nobody owns that record. We are asking agents to reason over a file that no human has agreed on.”Engagement Manager, BPO, Corporate and back-office functions

“in march i needed the sector holdings file from their old research store, it comes only as a weekly pdf dump. so for two days i am typing figures by hand into the model sheet. nobody on their side could say who owns that system”Individual Contributor, KPO, Corporate and back-office functions

Data failure is the context of the manual alteration of financial-institution-provided data. Respondents reported altering financial-institution-provided data in 421 of 839 answers (50.18%), on 127 of the 150 engagements. Those alterations, the reasons respondents gave for them and whether they reached the financial institution are examined with the other forms of manual alteration (Part 5).

2.1.1 Quality of Financial-Institution-Provided Data

Respondents rated the quality of the financial institution’s data across the systems the engagement depended on. The mean rating was 3.6 and the median 4.0 (n = 795 respondents), and 387 of the 795 answers (48.68%) were distressed. Respondents on canceled engagements rated the data lowest (mean 2.4, n = 305 respondents). Among the roles, individual contributors, who handle the data, rated it lowest (mean 3.4, n = 614 respondents).

The rating of the quality of the financial institution’s data was associated with all four core study measures, and each association held under the 22 robustness checks. Engagements that rated the data higher had a lower cancellation rate (r = −0.60, 95% CI −0.70 to −0.49, n = 150 engagements). They recorded higher customer satisfaction among active engagements (r = 0.68, 95% CI 0.55 to 0.77, n = 94 engagements). The rating was associated with the engagement indicator mean, computed without the indicator, at r = 0.81 (95% CI 0.74 to 0.86, n = 150 engagements). Respondents who rated the data higher reported lower intent to leave (r = −0.61, cluster bootstrap 95% CI −0.67 to −0.53, n = 748 respondents).

The written responses addressing the quality of the data numbered 582, from 415 respondents on 133 engagements, and 499 of the 582 (85.74%) were negative. In 226 of the 582 responses (38.83%), the response received the strongest negative score. Most came from the questions on altering financial-institution-provided data, where respondents described correcting the financial institution’s data by hand before it could be used, in many accounts every cycle or most days. Many of the corrections were reported within the vendor and not to the financial institution.

“CMDB is half empty so my team is filling asset details by hand before we can trust any playbook. It is not difficult work but it is night work, because we can only touch prod inside their window. Two of us are on that shift now instead of building.”Individual Contributor, ITO, Technology engineering and operations

“Their user directory is old, half the entries have no manager listed and no location… so the agent cannot even tell who is asking for what. For me this means my day is phone calls to verify identity, one by one … Two of us have left already.”Individual Contributor, CXM, Corporate and back-office functions

“Second week the agent restarted a claims application node at 3am because the asset register shows it as non production, it is very much production. Their own infrastructure team did not know that record was wrong either. Consequently we now verify every entry by hand before the playbook is allowed near it”Engagement Manager, ITO, Technology engineering and operations

“the problem is files come in already broken before we even open them, every single time, i fix it so exception queue does not build up but honestly it never stop, i clean same three field every week and next week same three field broken again”Individual Contributor, BPO, Corporate and back-office functions

“I dont raise a ticket for every duplicate, there are too many, I just fix it and move on, my lead knows but nobody tells the client, it eats an hour most days and nobody counts that hour anywhere.”Individual Contributor, KPO, Corporate and back-office functions

Respondents who wrote rated the data lower than respondents who did not, at a mean of 3.0 (n = 389 respondents) against 4.2 (n = 406 respondents). Where respondents rated the data well, they named a different failure at the financial institution, such as access for the agents, ownership of the data or the authority to decide. The written evidence places the cost of poor data on the delivery team, in hours spent correcting records that the agents were meant to process.

2.1.2 Accessibility of Financial Institution Data to the Engagement

Respondents rated how accessible the financial institution’s data was to the engagement when and where the engagement required it. The mean rating was 3.5 and the median 3.0 (n = 798 respondents), and 402 of the 798 answers (50.38%) were distressed. Respondents on canceled engagements rated access lowest (mean 2.4, n = 305 respondents), and individual contributors rated it lower than any other role with 10 or more answers (mean 3.4, n = 618 respondents).

All four core study measures were associated with the accessibility of the financial institution’s data, and none of the 22 robustness checks overturned an association. Higher ratings were associated with less cancellation (r = −0.57, 95% CI −0.67 to −0.45, n = 150 engagements). They were associated with higher customer satisfaction among active engagements (r = 0.67, 95% CI 0.54 to 0.77, n = 94 engagements). The engagement indicator mean computed without the indicator was higher where access was rated higher (r = 0.78, 95% CI 0.71 to 0.84, n = 150 engagements). Intent to leave was lower (r = −0.60, cluster bootstrap 95% CI −0.67 to −0.52, n = 749 respondents).

The written evidence locates the problem in reaching the data, more than in the data itself. The written responses addressing access numbered 190, from 176 respondents on 101 engagements, and 163 of the 190 (85.79%) were negative. Respondents described waiting days or weeks for logins, service accounts and firewall rules, and working meanwhile from extracts sent by email, screenshots and files delivered once a day. In the questions on altering financial-institution-provided data, the same gap appears as data pulled by hand from exports or copied from the displays of the financial institution’s systems. The connection to the source systems had never been approved.

“What they did was give us read only rights on the case system for two weeks, we could see the alert but not close it, so my queue kept growing. i asked twice, no answer. Then one day the access is there, nobody tells us.”Individual Contributor, ITO, Financial crime and client onboarding

“Last month I wait nine days for a service account on their claims warehouse… On my previous project, a health insurer also, the access came in two days and there is a named person for it. Here nobody owns that.”Individual Contributor, ITO, Technology engineering and operations

“In March we wait eleven days for a firewall rule so the agents could reach their case system, and the queue builds the whole time. Their monitoring platform is from 2009 and everything we take out of it comes through a nightly file, there is no interface.”Engagement Manager, BPO, Financial crime and client onboarding

“i put in a request for read access on their HR feed in january and still we do not have it, so every joiner list comes to us as a file by email once a day. And by the time we get it the person already started.”Individual Contributor, ITO, Technology engineering and operations

“Last week we needed three months of prior notes and their system only gives us export twice a week. So we work from whatever came in the last export and flag the gap in the summary”Individual Contributor, KPO, Credit, underwriting and suitability

Respondents who wrote rated access lower than respondents who did not, at a mean of 3.1 (n = 171 respondents) against 3.6 (n = 627 respondents). An agent cannot work on data it cannot reach. Where access was withheld or delayed, the delivery team brought the data to the agent by hand.

2.1.3 Normalization of Financial Institution Data Across Source Systems

Respondents rated how well the financial institution’s data was normalized, with consistent formats and definitions across its source systems. The mean rating was 3.5 and the median 3.0 (n = 809 respondents), and 413 of the 809 answers (51.05%) were distressed. Respondents on canceled engagements rated normalization lowest (mean 2.3, n = 311 respondents), and individual contributors rated it lowest of the roles (mean 3.4, n = 627 respondents).

Every association of the normalization of the financial institution’s data with the four core study measures held, in each of the 22 robustness checks. Engagements that rated normalization higher had a lower cancellation rate (r = −0.60, 95% CI −0.69 to −0.49, n = 150 engagements). Their customer satisfaction among active engagements was higher (r = 0.68, 95% CI 0.56 to 0.78, n = 94 engagements). So was the engagement indicator mean computed without the indicator (r = 0.81, 95% CI 0.75 to 0.86, n = 150 engagements). Their respondents reported lower intent to leave (r = −0.63, cluster bootstrap 95% CI −0.69 to −0.56, n = 758 respondents).

The written responses addressing normalization numbered 199, from 184 respondents on 106 engagements, and 180 of the 199 (90.45%) were negative. They describe the same account holder, vendor, employee or account recorded under different codes, names and formats in different systems of the financial institution, and file layouts that changed from one delivery to the next. On several engagements the differences arose because business units, regions or merged companies had kept their own codes, and the financial institution had named no owner of a single definition. The delivery team reconciled the records by hand before the agents could use them.

“Last month we tried to build a single joiner record for one branch. Same person appeared three times across their directory, their HR feed and the core entitlement store, each with different identifier and different spelling of name. Nobody owns reconciliation.”Engagement Manager, ITO, Technology engineering and operations

“Three systems hold job codes and none of them agree, and we spent the first six weeks just proving which one was authoritative. Thus the agent inherited a mess nobody on either side had owned before.”Engagement Manager, BPO, Corporate and back-office functions

“Here every business unit sends employee data in its own format. One uses employee id, one uses email, one sends a pdf. Nobody on their side thinks that’s a problem.”Individual Contributor, CXM, Corporate and back-office functions

“What i did is keep a small mapping file of how each system writes the trade reference, because settlement side and confirmations side do not write it the same way… one puts the counterparty code in front, the other one puts it at the end. So before the agent picks up the exception queue i run my mapping over the file and make the references match. It take maybe twenty minutes each morning.”Individual Contributor, BPO, Corporate and back-office functions

“I feel angry writing this again, if I am honest. Same column means one thing in file today and different thing in file next week, client dont check before sending it to us, so I fix number myself before we can use it in system”Individual Contributor, CXM, Corporate and back-office functions

Many respondents who rated normalization low said that the financial institution’s people were responsive and that the failure lay in the data. Respondents who wrote rated the indicator lower than respondents who did not, at a mean of 2.8 (n = 178 respondents) against 3.7 (n = 631 respondents). An agent asked to act on one account holder recorded under three identifiers depends on the delivery team to decide which of the three is correct.

2.1.4 Absence of Ownership and Validation in Financial Institution Data Governance

Respondents rated the maturity of the financial institution’s governance over the data the engagement used. The mean rating was 3.6 and the median 3.0 (n = 797 respondents), and 401 of the 797 answers (50.31%) were distressed. Ratings were highest among active engagements in Build-Out (mean 4.4, n = 160 respondents) and lowest among canceled engagements (mean 2.5, n = 305 respondents). Individual contributors gave the lowest ratings of the roles (mean 3.5, n = 614 respondents) and portfolio directors the highest (mean 4.8, n = 31 respondents).

The maturity of the financial institution’s data governance showed an association with each of the four core study measures that survived all 22 robustness checks. Engagements with higher ratings had a lower cancellation rate (r = −0.56, 95% CI −0.66 to −0.44, n = 150 engagements). Customer satisfaction among active engagements was higher where governance was rated higher (r = 0.67, 95% CI 0.54 to 0.77, n = 94 engagements). Higher ratings were associated with a higher engagement indicator mean computed without the indicator (r = 0.76, 95% CI 0.68 to 0.82, n = 150 engagements). They were associated with lower intent to leave (r = −0.60, cluster bootstrap 95% CI −0.67 to −0.53, n = 750 respondents).

The written responses addressing data governance numbered 172, from 155 respondents on 99 engagements, and 152 of the 172 (88.37%) were negative. Two failures recur in them. The financial institution had named no owner of the data the engagement used, and the data reached the vendor without validation. Respondents described asking who decided what a field meant and receiving several answers or none, and the delivery team validating and correcting the data by hand before processing.

“The thing a number cannot show is that their data is not so bad, it is the ownership of it which is missing … when we ask who decides what a field is supposed to mean, there is nobody. Two teams give two answers and both are confident.”Engagement Manager, BPO, Trade, settlement and reconciliation

“on their side the account classification has no owner. every month different person tell us which portfolios sit in which composite, next month it change again, and no one can say who approved it.”Individual Contributor, KPO, Trade, settlement and reconciliation

“In week three we asked for the data dictionary for the client positions warehouse. What came back was a spreadsheet from 2021 with half of tables missing. Since then we have been mapping fields by reading the source tables ourselves.”Engagement Manager, ITO, Technology engineering and operations

“In May we found dormant-account flags on core system were three days behind the servicing platform, so agents were writing to customers whose accounts had already been reactivated. Took client eleven days to confirm which system was master.”Engagement Manager, BPO, Payments, cards and disputes

“But nobody upstream owns the fix, so i keep cleaning the same file every month and it keeps arriving dirty, i if you would flag it each cycle and my lead knows, nothing changes on their side though.”Individual Contributor, BPO, Trade, settlement and reconciliation

Respondents who wrote rated governance lower than respondents who did not, at a mean of 3.0 (n = 145 respondents) against 3.7 (n = 652 respondents). The absence of an owner links this indicator to the manual alteration of financial-institution-provided data (Part 5). Where the financial institution could not say which value was correct, respondents described the delivery team choosing one.

2.2 Legacy Systems and Technical Infrastructure

The second barrier at the financial institution was technical. Two indicators describe it, the support the financial institution’s legacy systems gave to the integration the engagement required and the readiness of its infrastructure for agentic AI. Both were rated in the distressed range in just over half of the answers, 416 of 801 answers (51.94%) for legacy systems and 419 of 803 (52.18%) for infrastructure. Each of the two was associated with every core study measure, and the associations held under all 22 robustness checks.

The two indicators drew markedly different volumes of written evidence. Legacy systems were addressed in 231 written responses, and 127 of the respondents who wrote named the financial institution’s old systems among the three challenges that most reduced their job satisfaction. Infrastructure was addressed in 23. Together they describe systems that gave an agent no way in and environments that did not reflect production. Where no interface existed, the delivery team typed or copied the data by hand. Where the test environment no longer matched production, the team discovered the difference on release.

“Legacy HR systems still fight the agent at every handoff”Engagement Manager, BPO, Corporate and back-office functions

2.2.1 Legacy Systems Without an Interface for Agents

Respondents rated how well the financial institution’s legacy systems supported the integration the engagement required. The mean rating was 3.5 and the median 3.0 (n = 801 respondents), and 416 of the 801 answers (51.94%) were distressed. Respondents on canceled engagements rated the systems far lower than respondents on active engagements in any stage (mean 2.3, n = 308 respondents).

The support the legacy systems gave to integration was associated with cancellation, customer satisfaction, the engagement indicator mean and intent to leave, and each association held under the 22 robustness checks. Higher ratings were associated with less cancellation (r = −0.58, 95% CI −0.67 to −0.46, n = 150 engagements). Customer satisfaction among active engagements was higher where the systems were rated higher (r = 0.72, 95% CI 0.60 to 0.80, n = 94 engagements). The engagement indicator mean computed without the indicator was also higher (r = 0.82, 95% CI 0.76 to 0.87, n = 150 engagements). Respondents who rated the systems higher reported lower intent to leave (r = −0.60, cluster bootstrap 95% CI −0.67 to −0.53, n = 755 respondents).

The written responses addressing legacy systems numbered 231, from 212 respondents on 116 engagements, and 216 of the 231 (93.51%) were negative. Asked to name the three challenges that most reduced their job satisfaction, 127 respondents named legacy systems or legacy integration. Elsewhere, respondents described systems with no interface for an agent, data delivered as nightly files or copied by hand from a system’s display, and integrations that broke with each release.

“the systems it sits in are very old and there is no way for us to reach them directly, everything comes through a person, and that person is on leave or in another meeting.”Individual Contributor, BPO, Corporate and back-office functions

“the platform the expenses sit in is very old and there is no clean way in or out, so what reaches us is an extract, and the extract is only as good as whoever set it up years back. And nobody there today knows who set it up or why the columns are in that order.”Individual Contributor, BPO, Trade, settlement and reconciliation

“Last month we ask for the case API and their SIEM version does not expose it, so agent enrichment runs off a nightly export instead of live.”Engagement Manager, ITO, Technology engineering and operations

“Legacy platform integration broke constantly and nobody owned the fix”Individual Contributor, BPO, Trade, settlement and reconciliation

Several respondents did not blame the financial institution’s staff, who had inherited the systems. Respondents who wrote more negatively about the legacy systems reported higher intent to leave, and engagements with more negative written sentiment had a higher cancellation rate. An agent placed in front of a system with no interface depends on a person to bring the data to it, and in the written evidence that person was a member of the delivery team.

2.2.2 Readiness of Financial Institution Infrastructure for Agentic AI

Respondents rated the readiness of the financial institution’s technical infrastructure to support the agentic AI the engagement deployed. The mean rating was 3.5 and the median 3.0 (n = 803 respondents), and 419 of the 803 answers (52.18%) were distressed. Respondents on canceled engagements rated the infrastructure far lower than respondents on active engagements in any stage (mean 2.3, n = 297 respondents).

Every association of infrastructure readiness for agentic AI with the four core study measures held, in each of the 22 robustness checks. Higher infrastructure ratings were associated with less cancellation (r = −0.60, 95% CI −0.69 to −0.49, n = 150 engagements). Their customer satisfaction among active engagements was higher (r = 0.74, 95% CI 0.63 to 0.82, n = 94 engagements). Their engagement indicator mean computed without the indicator was higher (r = 0.84, 95% CI 0.79 to 0.88, n = 150 engagements). Their respondents reported lower intent to leave (r = −0.61, cluster bootstrap 95% CI −0.67 to −0.53, n = 755 respondents).

Few written responses addressed the infrastructure, 23 from 23 respondents on 22 engagements, and 18 of the 23 were negative. The evidence is thin and supports no general statement on its own. The negative responses named test environments that no longer matched production, servers that could not handle the volume, systems that timed out and environments not ready for cloud services. Each of the five positive responses described the infrastructure as ready and named a different problem.

“The test environment is the thing nobody asks about. It is a copy of prod from about two years ago, and half the deal types we support now do not exist in it. So we build against something that isn’t real and find out on release night.”Individual Contributor, ITO, Technology engineering and operations

“It comes down to one thing, the log sources feeding our triage agent are only about half mapped, so severity keeps coming back wrong on anything touching the acquiring platform.”Engagement Manager, ITO, Technology engineering and operations

“We were handed a genuinely solid environment to work in, access and identities were provisioned quickly and that is not always our experience. What would help now is one named owner for the custodian data definitions.”Engagement Manager, BPO, Trade, settlement and reconciliation

How widely these problems occurred is measured by the ratings, and just over half of the answers placed the infrastructure in the distressed range. The written responses describe what that rating meant on the engagements where respondents wrote.

2.3 Process Documentation, Maturity, and Redesign

Four indicators describe the financial institution’s processes. They cover how completely the processes were documented, how standardized they were, how consistently they were performed across business units and regions, and how willing the financial institution was to redesign them for agentic AI. Each had about half of its answers in the distressed range, from 396 of 789 answers (50.19%) for standardization to 430 of 795 (54.09%) for documentation. Documentation had the highest distressed share of any indicator in the category. The four were associated with every core study measure, and none of the 22 robustness checks overturned an association.

The four indicators describe a single difficulty. An agent follows a defined process, and on the delivery teams’ account the financial institution had none to give it. Processes were undocumented, or documented in a form that no longer matched what staff did. Where they were performed, each desk, branch or region performed them its own way. Where the vendor proposed to redesign a process around the agent, the negative written responses described a financial institution that asked for the existing process to be copied as it stood.

“they told us up front not to change any of the workflow, just make the agent do it the way the underwriters already do it. so my whole week is teaching it to copy a bad process.”Individual Contributor, ITO, Client servicing and communications

In each case the design of the process moved to the delivery team. Respondents described watching the financial institution’s staff at work and writing down the steps themselves, choosing one regional variant among several and teaching the agent to reproduce checks that had been built for people. They described doing so, in many accounts, without a ruling from the financial institution. Respondents reported altering the financial institution’s processes in 490 of 841 answers (58.26%), the most reported of the three methods counted by a direct question. Respondents linked those alterations to this gap (Part 5).

2.3.1 Undocumented Processes Rebuilt by Observation

Respondents rated how completely the financial institution’s processes were mapped and documented to the depth agentic AI requires. The mean rating was 3.5 and the median 3.0 (n = 795 respondents). Distressed ratings made up 430 of the 795 answers (54.09%), the highest distressed share among the 20 indicators of financial institution readiness. Respondents on canceled engagements rated documentation far lower than respondents on active engagements in any stage (mean 2.4, n = 303 respondents).

The documentation of the financial institution’s processes showed an association with each of the four core study measures that survived all 22 robustness checks. Higher ratings were associated with less cancellation (r = −0.56, 95% CI −0.66 to −0.44, n = 150 engagements). They were associated with higher customer satisfaction among active engagements (r = 0.70, 95% CI 0.57 to 0.79, n = 94 engagements). Documentation was associated with a higher engagement indicator mean computed without the indicator (r = 0.81, 95% CI 0.74 to 0.86, n = 150 engagements). Respondents who rated it higher reported lower intent to leave (r = −0.57, cluster bootstrap 95% CI −0.65 to −0.49, n = 748 respondents).

The written responses addressing documentation numbered 270, from 238 respondents on 121 engagements, and 229 of the 270 (84.81%) were negative. The largest group came from the question on why the respondent had altered the financial institution’s processes. Respondents described processes that had never been written down, documents that did not match what the financial institution’s staff actually did, and workflows the delivery team rebuilt by watching those staff at work.

“the process doc they handed us was from 2019. it didnt match anything. the desk does it one way, the doc says another, and the two floors dont even do it the same. so we sat with one of their people for a week and wrote down what actually happens. that isnt our job but nobody else was going to.”Individual Contributor, BPO, Payments, cards and disputes

“In fairness nobody there was hiding, they simply dont have the process written anywhere, so each person answers from memory.”Individual Contributor, CXM, Corporate and back-office functions

“The signed off process is not what the finance leads actually do day to day… i adjust my workflow to match how they really work, otherwise the model calibration breaks every month.”Individual Contributor, KPO, Corporate and back-office functions

“Here the reconciliation steps are mostly in peoples heads, we ask and somebody explains it on a call.”Individual Contributor, BPO, Trade, settlement and reconciliation

“what the numbers don’t show is nobody has written down how they actually handle a disputed charge. every senior agent on their side does it slightly diffrent.”Individual Contributor, CXM, Payments, cards and disputes

Respondents who wrote rated documentation lower than respondents who did not, at a mean of 3.0 (n = 225 respondents) against 3.7 (n = 570 respondents). Questions put to the financial institution about its processes went unanswered for days or weeks. The delivery team followed what worked in place of the document, in many accounts without a ruling from the financial institution. Respondents linked the manual alteration of the financial institution’s processes they reported to this gap (Part 5).

2.3.2 Unstandardized Processes Without a Single Version to Automate

Respondents rated the maturity of the financial institution’s processes in scope, meaning how standardized and repeatable they were. The mean rating was 3.5 and the median 3.0 (n = 789 respondents), and 396 of the 789 answers (50.19%) were distressed. Respondents on canceled engagements rated the processes far lower than respondents on active engagements in any stage (mean 2.4, n = 295 respondents).

All four core study measures were associated with the maturity of the financial institution’s processes, and the 22 robustness checks left each association intact. Engagements that rated the processes higher had a lower cancellation rate (r = −0.57, 95% CI −0.67 to −0.45, n = 150 engagements). They recorded higher customer satisfaction among active engagements (r = 0.72, 95% CI 0.61 to 0.81, n = 94 engagements). Process maturity was associated with a higher engagement indicator mean computed without the indicator (r = 0.80, 95% CI 0.74 to 0.85, n = 150 engagements). It was associated with lower intent to leave (r = −0.56, cluster bootstrap 95% CI −0.64 to −0.49, n = 742 respondents).

The written responses addressing the maturity of the processes numbered 85, from 79 respondents on 59 engagements, and 74 of the 85 (87.06%) were negative. Respondents described processes that each person, desk or region at the financial institution performed its own way. The delivery team could not train an agent on a single version, and when the financial institution did not rule, the team chose a version itself. Each of the four comparisons of written sentiment with the study measures reached significance.

“Every desk contact on their side confirms trades a different way. Some use the Portal. Some just email a PDF. Some call and read numbers off a screen. And there is no single process document that says which one is correct for which counterparty, so we made our own version months ago and just follow that now.”Individual Contributor, BPO, Trade, settlement and reconciliation

“each fund still runs its attribution its own way, so what we built for one desk did not carry across to the next. They were never going to redesign how attribution is done — only who does it.”Portfolio Director, KPO, Trade, settlement and reconciliation

“The problem is every exception gets handled a different way depending which person from Client team responds that day, so I make the call on Process myself and note it internally, revert back if anyone ask.”Individual Contributor, BPO, Corporate and back-office functions

“On my last account, a card client, the process was written down and every team followed it. Here every desk route their tickets their own way.”Individual Contributor, CXM, Technology engineering and operations

An agent trained on one version of a process will apply that version everywhere. Where the financial institution had several and named none, the version the agent applied was the one the delivery team had chosen.

2.3.3 Inconsistency of Processes Across Business Units and Regions

Respondents rated how consistently the financial institution performed the same processes across its business units and regions. The mean rating was 3.6 and the median 3.0 (n = 801 respondents), and 408 of the 801 answers (50.94%) were distressed. Respondents on canceled engagements rated consistency far lower than respondents on active engagements in any stage (mean 2.4, n = 300 respondents).

The consistency of the financial institution’s processes across business units and regions was associated with cancellation, customer satisfaction, the engagement indicator mean and intent to leave, and each association held under the 22 robustness checks. More consistent processes were associated with less cancellation (r = −0.58, 95% CI −0.68 to −0.47, n = 150 engagements). They were associated with higher customer satisfaction among active engagements (r = 0.74, 95% CI 0.63 to 0.82, n = 94 engagements). The engagement indicator mean computed without the indicator was higher where consistency was rated higher (r = 0.83, 95% CI 0.78 to 0.88, n = 150 engagements). Intent to leave was lower (r = −0.61, cluster bootstrap 95% CI −0.67 to −0.53, n = 755 respondents).

The written responses addressing consistency numbered 44, from 44 respondents on 35 engagements, all in comment on the financial institution, and 35 of the 44 (79.55%) were negative. They describe the same process performed differently by each region, desk, branch or business unit. On one engagement the financial institution had been formed from four companies that never merged their rules.

“the client is four companies that never merged their HR rules. Leave entitlement, asset return, even the shift codes are different in each one, also nobody there could tell us which version was correct. We were asked to automate a process that has four versions and no owner.”Engagement Manager, CXM, Corporate and back-office functions

“Each region books the same account closure with a different reason code. What I do is keep a small list of which code belongs to which branch group, and I map them before the file goes in, otherwise the agent will raise exception on every one of them.”Individual Contributor, BPO, Payments, cards and disputes

“New business in the west region orders requirements one way, the east does it different, and the agent has to be told twice. Same product, two workflows.”Individual Contributor, BPO, Credit, underwriting and suitability

“What has not changed is every region does invoce approval a different way, so rules we set for one business unit do not fit the next one, and nobody wrote any of it down.”Individual Contributor, BPO, Corporate and back-office functions

Written sentiment, on a base of 44 responses, was not associated with the study measures, and the associations rest on the ratings. The written responses describe why inconsistency mattered. A process run several ways gave the agent no single set of rules, and the delivery team mapped the variants by hand or chose one, in some cases keeping private lists to do so.

2.3.4 Automation of Processes as They Stand Rather Than Redesign

Respondents rated the financial institution’s willingness to redesign its processes for agentic AI, rather than automate them as they stood. The mean rating was 3.5 and the median 3.0 (n = 804 respondents), and 429 of the 804 answers (53.36%) were distressed. Respondents on canceled engagements rated the financial institution’s willingness far lower than respondents on active engagements in any stage (mean 2.4, n = 307 respondents).

The financial institution’s willingness to redesign its processes for agentic AI showed an association with each of the four core study measures that survived all 22 robustness checks. Engagements where the willingness was rated higher had a lower cancellation rate (r = −0.56, 95% CI −0.66 to −0.44, n = 150 engagements). Their customer satisfaction among active engagements was higher (r = 0.70, 95% CI 0.58 to 0.79, n = 94 engagements). Willingness to redesign was associated with a higher engagement indicator mean computed without the indicator (r = 0.82, 95% CI 0.76 to 0.87, n = 150 engagements). Respondents who rated it higher reported lower intent to leave (r = −0.59, cluster bootstrap 95% CI −0.66 to −0.52, n = 755 respondents).

The written responses addressing redesign numbered 43, from 43 respondents on 38 engagements, and 34 of the 43 were negative. They describe the request respondents met, to automate the process as it stood or to run it the same way, only faster. The agent was taught to reproduce steps designed around human checks, and some processes stopped working once those checks were removed.

“the client wants the agent to produce the summary in exactly the same shape their underwriters have always done it, box for box, including the sections nobody reads. We ask twice about redesigning the template for a machine and each time we are told to keep it as is. And that one decision costs us more rework than anything else in month one.”Engagement Manager, KPO, Credit, underwriting and suitability

“They want same close process as before, only faster, no appetite to change any step. Their data itself is fine, we get clean trial balance every month, accordingly I went higher on those. Its the willingness which is missing, not the informations.”Individual Contributor, BPO, Trade, settlement and reconciliation

“Nobody re-engineered the process before handing it to the agent”Individual Contributor, CXM, Technology engineering and operations

“A specific case last month, the change sat for two weeks because nobody would take the decision, so we applied it ourselves. Nobody ever updated the process for how an agent actually works, so it falls on us to catch what should be flagged”Individual Contributor, KPO, Trade, settlement and reconciliation

On a base of 43 responses, written sentiment on redesign showed no association with the study measures, and the associations rest on the ratings. A process designed for people contains checks and handoffs that exist because people make particular types of mistake. Copied unchanged, it asked the agent to perform those steps and the delivery team to perform the checks.

2.4 Organizational Readiness and Staff Capability

Three indicators describe the financial institution’s people. They cover the organization’s readiness to absorb the changes agentic AI makes to daily work, the agentic AI knowledge and skills of its staff on the engagement, and the availability of its subject matter experts. The knowledge of the staff was distressed for 418 of 793 respondents (52.71%) and the availability of experts for 427 of 793 (53.85%). Readiness to absorb change was distressed in 396 of 794 answers (49.87%). The associations of all three with the four core study measures held under each of the 22 robustness checks.

The written evidence separates two qualities the ratings combine. Respondents credited the financial institution’s people for their willingness and responsiveness. They described staff who answered the same day, knew their own business and stayed late on calls. The same respondents did not credit those people with experience of working alongside agents, and many described staff who had never done so. The low ratings record the second, in several cases in answers that praise the first.

The written evidence on the availability of experts describes the delivery team’s daily work. Experts at the financial institution were few, shared with other work and absent at the moments the engagement needed them, at a month-end close or while a deal was live. When the delivery team could not reach them, the engagement’s tasks did not stop. Respondents described making the financial institution’s decisions in its place, correcting its data on their own judgment and rebuilding its rules from old reports.

“It’s that there is nobody on the deal team who can answer a question. I sent a question about which entity a set of payroll files belonged to on a Tuesday, got an answer nine days later, and the answer was a guess.”Individual Contributor, KPO, Corporate and back-office functions

2.4.1 Organizational Readiness to Absorb Changes to Daily Work

Respondents rated the financial institution’s readiness to absorb the changes to daily work that agentic AI introduces. The mean rating was 3.5 and the median 4.0 (n = 794 respondents), and 396 of the 794 answers (49.87%) were distressed. Respondents on canceled engagements rated the organization’s readiness far lower than respondents on active engagements in any stage (mean 2.3, n = 305 respondents).

Every association of the organization’s readiness to absorb changes to daily work with the four core study measures held, in each of the 22 robustness checks. Engagements that rated the organization’s readiness higher had a lower cancellation rate (r = −0.59, 95% CI −0.69 to −0.48, n = 150 engagements). Customer satisfaction among their active engagements was higher (r = 0.73, 95% CI 0.61 to 0.81, n = 94 engagements). Readiness was associated with a higher engagement indicator mean computed without the indicator (r = 0.83, 95% CI 0.78 to 0.88, n = 150 engagements). It was associated with lower intent to leave (r = −0.60, cluster bootstrap 95% CI −0.66 to −0.53, n = 746 respondents).

Few written responses addressed the indicator, 26 from 25 respondents on 23 engagements, and the evidence is divided. Negative responses numbered 16 and positive responses 10. The positive responses described staff who were open to the change and placed the difficulty elsewhere. The negative responses described staff who continued working as before and adoption reported as complete without being measured.

“their own staff are not ready for any of this, half of them still ask us to mail the same note across because they dont open the tool.”Individual Contributor, KPO, Corporate and back-office functions

“i felt bad putting one against their people but its not really the people, its that their night shift still works off the old runbook and wont touch the triage queue at all. day team is fine with it. thats the whole of that score.”Individual Contributor, ITO, Technology engineering and operations

“The vendor reported system uptime as the primary success metric on a tool where the actual risk was answer accuracy, so the dashboard stayed green through months of advisors quietly stopping use of the assistant.”Individual Contributor, CXM, Client servicing and communications

Written sentiment on the organization’s readiness was not associated with the study measures. On a base of 26 responses, the written evidence records that both conditions occurred, staff who took to the change and staff who did not. The frequency of each is measured by the ratings, and half of the answers placed the financial institution’s readiness in the distressed range.

2.4.2 Agentic AI Knowledge of Financial Institution Staff

Respondents rated the agentic AI knowledge and skills of the financial institution’s staff working with the engagement. The mean rating was 3.5 and the median 3.0 (n = 793 respondents), and 418 of the 793 answers (52.71%) were distressed. Respondents on canceled engagements rated the staff’s knowledge far lower than respondents on active engagements in any stage (mean 2.3, n = 307 respondents).

All four core study measures were associated with the agentic AI knowledge and skills of financial institution staff, and none of the 22 robustness checks overturned an association. Where the staff’s knowledge was rated higher, cancellation was less common (r = −0.59, 95% CI −0.69 to −0.48, n = 150 engagements). Among active engagements, customer satisfaction was higher where the staff’s knowledge was rated higher (r = 0.70, 95% CI 0.58 to 0.79, n = 94 engagements). The engagement indicator mean computed without the indicator was higher as well (r = 0.82, 95% CI 0.75 to 0.86, n = 150 engagements). Respondents who rated the staff’s knowledge higher reported lower intent to leave (r = −0.60, cluster bootstrap 95% CI −0.67 to −0.53, n = 750 respondents).

The written responses addressing the staff’s knowledge numbered 36, from 36 respondents on 30 engagements, and 24 of the 36 were negative. The negative responses described staff at the financial institution who had never worked with an agent, sometimes in the same answer that credited their responsiveness. The positive responses said the staff understood the agent and placed the difficulty in the data, in access or in ownership.

“their manager is fine, hes responsive, the analysts under him just dont have any experience with this stuff yet.”Individual Contributor, ITO, Technology engineering and operations

“What they did, please note, is answer us quickly – when we raised the failed release in June their platform team was on the call the same afternoon, which I cannot fault. The difficulty is the people sitting with us have never run a pipeline with an agent in it, so I explain the same rollback twice … ”Engagement Manager, ITO, Payments, cards and disputes

“Their process documents are actually fine, this i will say. But their people have never worked with an agent before, so every request come back with question. And access to the Config environment took five weeks.”Individual Contributor, ITO, Client servicing and communications

Respondents who wrote more negatively about the staff’s knowledge reported higher intent to leave. Their written sentiment was not associated with cancellation or customer satisfaction. The staff’s inexperience placed the task of explaining the agent, and of checking that the financial institution’s people had understood it, on the delivery team.

2.4.3 Availability of Financial Institution Subject Matter Experts

Respondents rated the availability of the financial institution’s subject matter experts when the engagement needed them. The mean rating was 3.5 and the median 3.0 (n = 793 respondents), and 427 of the 793 answers (53.85%) were distressed. Respondents on canceled engagements rated availability far lower than respondents on active engagements in any stage (mean 2.3, n = 299 respondents).

The availability of the financial institution’s subject matter experts was associated with all four core study measures, and each association held under the 22 robustness checks. Engagements that rated the experts more available had a lower cancellation rate (r = −0.59, 95% CI −0.68 to −0.47, n = 150 engagements). They recorded higher customer satisfaction among active engagements (r = 0.70, 95% CI 0.58 to 0.79, n = 94 engagements). Availability was associated with a higher engagement indicator mean computed without the indicator (r = 0.82, 95% CI 0.77 to 0.87, n = 150 engagements). Respondents who rated the experts more available reported lower intent to leave (r = −0.58, cluster bootstrap 95% CI −0.64 to −0.50, n = 748 respondents).

The written responses addressing the availability of experts numbered 220, from 202 respondents on 113 engagements, and 184 of the 220 (83.64%) were negative. They describe experts at the financial institution who were few in number and shared with other work. When the delivery team could not reach them, it decided, corrected or inferred alone. The positive responses described experts who answered the same day and placed the difficulty in the data or in access.

“There is exactly one person on client side who understands how their shift allowance and arrears rules actually work, she sits in one meeting a week with us and she is covering three other projects … We had a cycle where fourteen queries sat nine days for want of one answer, and then payroll ran anyway because payroll always runs.”Engagement Manager, BPO, Corporate and back-office functions

“The bankers are never free when a book is live, which is exactly when we need them. So we dont ask in the moment anymore. We keep our own note of how each desk has allocated in the past, deal by deal, and we work from that, then we confirm after pricing rather than before.”Engagement Manager, KPO, Trade, settlement and reconciliation

“the problem is we wait weeks for one modeler to answer a question, then everything lands at once. my team ends up burning weekends to catch up. and it repeats every quarter”Individual Contributor, KPO, Credit, underwriting and suitability

“there is one guy who knows the expense policy and he is booked out three weeks. so my team just guesses and eats it when its wrong.”Individual Contributor, BPO, Corporate and back-office functions

“the problem is a break on one Fund in March, we send the query up on Monday and no informations come back for two weeks, by then NAV is already signed off. My team lead says the SME on Bank side is put on other project.”Individual Contributor, BPO, Trade, settlement and reconciliation

Cancellation was more common on engagements whose written responses on the experts were more negative. The rating measures how readily the experts could be reached. The written responses show what their absence meant. The engagement’s tasks continued, and the decisions the experts would have made were made by the delivery team.

2.5 Management of the Financial Institution’s Side of the Engagement

Two indicators describe how the financial institution managed its own side of the engagement. They measure the effectiveness of that management and the speed with which the financial institution responded when the engagement raised an issue that needed its action. The management of the engagement was distressed in 398 of 811 answers (49.08%), and the speed of response in 404 of 789 (51.20%). Both indicators held every association with the four core study measures through the 22 robustness checks.

The written evidence places the weakness in decision rights more than in effort. Individual staff at the financial institution answered questions quickly, and many respondents said so. Decisions that required someone with authority were different. They went up within the organization and did not return in time, or returned from several people with several answers. The delivery team then made the decision itself, to keep the engagement moving, and in many accounts the financial institution was not told.

“we get an answer same day now, used to be a week. what hasnt changed is who owns it when the tool misroutes something. everybody agrees it needs fixing, nobody signs anything.”Individual Contributor, BPO, Payments, cards and disputes

On these two indicators, written sentiment was itself associated with the outcomes. Engagements whose written responses on the financial institution’s management were negative had a far higher cancellation rate than other engagements. The written sentiment on response speed tracked three of the measures, cancellation, customer satisfaction and intent to leave. The written evidence on delay at the financial institution, and on decisions taken in its place, was associated with how the engagement ended.

2.5.1 Decision Rights Unassigned in Financial Institution Management of the Engagement

Respondents rated the effectiveness of the financial institution’s management of its side of the engagement. The mean rating was 3.6 and the median 4.0 (n = 811 respondents), and 398 of the 811 answers (49.08%) were distressed. Respondents on canceled engagements rated the financial institution’s management far lower than respondents on active engagements in any stage (mean 2.4, n = 305 respondents).

The effectiveness of the financial institution’s management of its side was associated with all four core study measures, and each association held under the 22 robustness checks. Engagements whose delivery teams rated the financial institution’s management of its side higher had a lower cancellation rate (r = −0.56, 95% CI −0.66 to −0.44, n = 150 engagements). They recorded higher customer satisfaction among active engagements (r = 0.72, 95% CI 0.60 to 0.80, n = 94 engagements). Effective management by the financial institution was associated with a higher engagement indicator mean, computed without the management indicator (r = 0.82, 95% CI 0.76 to 0.87, n = 150 engagements). Respondents who rated the management higher reported lower intent to leave (r = −0.59, cluster bootstrap 95% CI −0.66 to −0.51, n = 763 respondents).

The written responses addressing the financial institution’s management numbered 196, from 173 respondents on 104 engagements, and 184 of the 196 (93.88%) were negative. In 157 the response received the strongest negative score. One account recurs in nearly all 196. The financial institution had named neither an owner of the engagement’s tasks nor a person with authority to decide, and decisions waited until the delivery team made them.

“The people we sat with were fine, they answered on chat, they showed us things when they could. The problem is above them. Nobody there was allowed to say yes to anything. Their engineer told me straight, he cant open a service account for a bot without a sign off that does not exist yet, so we wait. Four weeks we wait and then the project is closed.”Individual Contributor, ITO, Technology engineering and operations

“In the meantime the product team in London was using one definition of the peer group and the team in Singapore was using another, and both told us theirs was the approved one … Nobody on their side owned it, so we didnt get a decision, only opinions. By the time an answer came the deal review had already passed.”Engagement Manager, KPO, Corporate and back-office functions

“To be blunt it is a joke, we correct their identity data every single week and nobody above them seems to care, they defer every decision to us and we absorb it, as such I stopped raising it and just fix things myself now.”Individual Contributor, ITO, Technology engineering and operations

“nobody on their side owns this piece, that is the real problem. on my first desk, card refunds, there was one name on the wall and i could ask her. here i send question and it goes round three teams, nobody answer”Individual Contributor, BPO, Trade, settlement and reconciliation

The written evidence on the financial institution’s management was associated with the outcomes in its own right. Engagements with a negative written response on the management had a far higher cancellation rate than other engagements. Respondents who wrote more negatively about it reported higher intent to leave, and their written sentiment was not associated with customer satisfaction. The respondents located the weakness in decision rights, not in the effort or competence of the financial institution’s staff.

2.5.2 Speed of Financial Institution Response to Issues Raised by the Engagement

Respondents rated how quickly the financial institution responded when the engagement raised an issue that needed its action. The mean rating was 3.5 and the median 3.0 (n = 789 respondents), and 404 of the 789 answers (51.20%) were distressed. Respondents on canceled engagements rated the speed of response far lower than respondents on active engagements in any stage (mean 2.4, n = 297 respondents).

All four core study measures were associated with the speed of the financial institution’s response to issues raised by the engagement, and none of the 22 robustness checks overturned an association. Faster response was associated with less cancellation (r = −0.60, 95% CI −0.69 to −0.48, n = 150 engagements). Faster response by the financial institution was associated with higher customer satisfaction among the active engagements (r = 0.74, 95% CI 0.63 to 0.82, n = 94 engagements). The engagement indicator mean computed without the indicator was higher where response was rated faster (r = 0.84, 95% CI 0.78 to 0.88, n = 150 engagements). Respondents who rated the response faster reported lower intent to leave (r = −0.59, cluster bootstrap 95% CI −0.65 to −0.51, n = 740 respondents).

The written responses addressing the speed of response numbered 498, from 382 respondents on 142 engagements, and 437 of the 498 (87.75%) were negative. Most came from the questions on manual alteration. There, respondents described questions that sat for days or weeks, answers that arrived after the deadline, and a delivery team that made the decision without the financial institution. The comments on the financial institution were more divided. Many respondents said the people they dealt with answered quickly, and others described waits that stalled the engagement.

“every question we raise on the billing cycle sits for two weeks, and the business case has been rewritten three times in eight weeks.”Engagement Manager, BPO, Payments, cards and disputes

“I raise it, they say we will discuss about it in the next call. Next call is cancelled. We build the note anyway because the deadline does not move, then I rebuild the same note three times when the list changes again.”Individual Contributor, KPO, Corporate and back-office functions

“Escalation goes to my lead, my lead sends to vendor manager, vendor manager sends to their side. Answer comes back in three or four days. Agent sends fifty responses in that time. Nothing else from me on it”Individual Contributor, CXM, Client servicing and communications

“i have not had a full weekend since January. Their environment team answer once a week, Friday evening usually, so my team sit idle four days then works the whole night to make the window… two of us have asked for transfer.”Individual Contributor, ITO, Technology engineering and operations

“last week ticket needed system access but their Security team wont approve connection, again, same as every time, and i had to enter data by hand instead, like always.”Individual Contributor, ITO, Technology engineering and operations

Written sentiment on the speed of response was associated with cancellation, customer satisfaction and intent to leave. The written responses separate two types of response. Individual staff answered quickly, on many respondents’ account. Decisions that needed someone with authority went up and did not come back in time, and when they did not, the delivery team made them. In many accounts the financial institution was not told.

2.6 Governance of Agentic AI Within the Financial Institution

Three indicators describe how the financial institution governed agent work on its own side. They cover the assignment of ownership and accountability for AI, the capacity to oversee the risks of agents performing the engagement’s tasks and the capacity of change control to handle the changes the engagement required. About half of the answers on each were distressed, 417 of 802 answers (52.00%) for ownership of AI, 417 of 797 (52.32%) for oversight of agent risk and 411 of 800 (51.38%) for change control. None of the 22 robustness checks overturned an association of the three with the four core study measures. Diffuse ownership of AI had the lowest mean rating of the 20 indicators of the category, 3.47 (n = 802 respondents).

The written evidence on the three indicators is modest in volume and consistent in content. Respondents described AI that no single function at the financial institution owned, model-risk review and audit frameworks not written for agents, and change boards that met weekly or monthly against agent behavior that changed daily. They located each gap beyond the staff they worked with, at the level where accountability is assigned and controls are designed.

“Moreover their governance model assumes changes happen at release pace, not the pace an agent generates cases, so we adapted the workflow ourselves.”Individual Contributor, KPO, Financial crime and client onboarding

The three indicators record the financial institution’s own controls over work performed by agents. With the joint controls of co-management, they inform the examination of how the controls meant to govern agent work were bypassed or absent (Part 3).

2.6.1 Diffuse Ownership of AI Across Financial Institution Functions

Respondents rated how distinctly ownership and accountability for AI were assigned inside the financial institution. The mean rating was 3.47, the lowest of the 20 indicators of financial institution readiness, and the median 3.0 (n = 802 respondents). Distressed ratings made up 417 of the 802 answers (52.00%). Respondents on canceled engagements rated the assignment of ownership far lower than respondents on active engagements in any stage (mean 2.3, n = 306 respondents).

Every association of the assignment of ownership for AI inside the financial institution with the four core study measures held, in each of the 22 robustness checks. Clearer ownership of AI at the financial institution was associated with less cancellation (r = −0.57, 95% CI −0.67 to −0.45, n = 150 engagements). It was associated with higher customer satisfaction among active engagements (r = 0.70, 95% CI 0.59 to 0.79, n = 94 engagements). Engagements where ownership of AI was rated clearer had a higher engagement indicator mean, computed without the ownership indicator (r = 0.81, 95% CI 0.75 to 0.86, n = 150 engagements). Intent to leave was lower among respondents who saw clearer ownership of AI (r = −0.59, cluster bootstrap 95% CI −0.66 to −0.52, n = 755 respondents).

The written responses addressing the ownership of AI numbered 46, from 46 respondents on 40 engagements, and 41 of the 46 were negative. Nearly all said the same thing. The financial institution had named no owner of the AI. Accountability rested with several functions, and each pointed to another, and decisions on what the agent was allowed to do waited for days or weeks.

“there is nobody on their side who owns AI. Risk own the watchlist, technology own the platform, and when the agent flags an obligor wrongly, both of them say it is the other one. Their credit analysts do turn up when we ask, that is fair, but they cannot approve anything.”Engagement Manager, KPO, Credit, underwriting and suitability

“the client does not have anybody who owns what the agent is allowed to do. thats the whole problem right there … i flagged it same day. it took eleven days to get a yes or no on whether we keep repairing them or return the whole batch. eleven days of me sitting on a queue and aging it.”Individual Contributor, BPO, Payments, cards and disputes

“I came away impressed, and the softer answers here come down to one thing. Their desk leads are strong and the endpoint data is clean. But nobody on their side owns which ticket categories an agent may close unsupervised, and that call keeps moving between people”Portfolio Director, ITO, Technology engineering and operations

“My last engagement was a smaller bank and their finance team gives us one owner for the model, one person to ask. Here nobody owns it. I send a question on the entity heirarchy and it goes round three teams and comes back to me.”Individual Contributor, KPO, Corporate and back-office functions

Written sentiment on the ownership of AI was not associated with the study measures. Respondents were careful to separate the gap from the financial institution’s staff, several of whom they described as willing and capable. Where several functions each held a part of the AI, no single function could decide what the agent was allowed to do, and the decision waited or fell to the delivery team.

2.6.2 Missing Instruments of Financial Institution Agent-Risk Oversight

Respondents rated the financial institution’s capacity to govern and oversee the risks of AI agents doing the engagement’s work. The mean rating was 3.5 and the median 3.0 (n = 797 respondents), and 417 of the 797 answers (52.32%) were distressed. Respondents on canceled engagements rated the financial institution’s capacity far lower than respondents on active engagements in any stage (mean 2.3, n = 306 respondents).

The financial institution’s capacity to oversee the risks of AI agents showed an association with each of the four core study measures that survived all 22 robustness checks. Cancellation was less common on engagements that rated the oversight capacity higher (r = −0.57, 95% CI −0.67 to −0.45, n = 150 engagements). Among active engagements it was associated with higher customer satisfaction (r = 0.75, 95% CI 0.64 to 0.83, n = 94 engagements). Oversight capacity was associated with a higher engagement indicator mean, computed without the oversight indicator (r = 0.83, 95% CI 0.77 to 0.88, n = 150 engagements). Respondents who rated the oversight higher reported lower intent to leave (r = −0.61, cluster bootstrap 95% CI −0.67 to −0.54, n = 750 respondents).

The written responses addressing oversight of agent risk numbered 45, from 44 respondents on 38 engagements, and 37 of the 45 were negative. Most came from the question on the challenges that most reduced job satisfaction, where respondents listed challenges in a few words each. They named the oversight the financial institution lacked, model-risk review that arrived late or never closed, audit frameworks that did not exist for agents and vendor-risk terms not written for this form of work. A small number described an institution whose risk function was strong, or whose AI oversight forum existed before the engagement began.

“nobody on their side is really watching what agent does with it”Individual Contributor, CXM, Corporate and back-office functions

“Asking how quickly they govern AI risk when nobody has been named to govern it is not a question anyone here can answer.”Engagement Manager, CXM, Corporate and back-office functions

“Client could not audit the agent’s recon decisions and we had no answer”Individual Contributor, BPO, Trade, settlement and reconciliation

“Model-risk sign-off lags the build”Individual Contributor, ITO, Technology engineering and operations

Written sentiment on agent-risk oversight was not associated with the study measures. The written responses indicate which parts of the financial institution’s oversight were missing more than they describe how oversight failed in a given case. The rating measures the capacity, and just over half of the answers placed it in the distressed range.

2.6.3 Financial Institution Change Control Calibrated to Human-Paced Release Cycles

Respondents rated how well the financial institution’s change control process handled the changes the engagement required. The mean rating was 3.6 and the median 3.0 (n = 800 respondents), and 411 of the 800 answers (51.38%) were distressed. Respondents on canceled engagements rated the financial institution’s change control far lower than respondents on active engagements in any stage (mean 2.4, n = 304 respondents).

The financial institution’s change control, as rated by the delivery team, was associated with cancellation, customer satisfaction, the engagement indicator mean and intent to leave, and each association held under the 22 robustness checks. Engagements that rated the financial institution’s change control higher had a lower cancellation rate (r = −0.59, 95% CI −0.69 to −0.48, n = 150 engagements). Their customer satisfaction among active engagements was higher (r = 0.75, 95% CI 0.64 to 0.83, n = 94 engagements). So was their engagement indicator mean computed without the indicator (r = 0.84, 95% CI 0.79 to 0.89, n = 150 engagements). Their respondents reported lower intent to leave (r = −0.62, cluster bootstrap 95% CI −0.69 to −0.55, n = 751 respondents).

The written responses addressing change control numbered 85, from 80 respondents on 64 engagements, and 69 of the 85 were negative. The central complaint was pace. Respondents described change processes and boards built for a change a month, or meeting weekly or monthly, against work that changed several times a day, and changes made without waiting for approval as a result. The positive responses described a quick change process and placed the difficulty elsewhere.

“it is sign-off where we lose everything. A fix we write in two days sits four weeks in their change queue.”Engagement Manager, BPO, Trade, settlement and reconciliation

“What they did not have was a change process that could absorb a rule set moving weekly rather than quarterly, and I don’t think that is a failing so much as an artifact of how regulatory systems are governed everywhere. Their own model validation group has a mandate written for annual releases.”Portfolio Director, ITO, Regulatory, audit, tax and legal

“the problem is the way client set up change process only expect maybe one update a month, but we adjust ticket routing rule several time a day just to keep queue moving, so i just update it myself when it need fixing”Individual Contributor, CXM, Client servicing and communications

“My team sits late every second Thursday waiting on their approval, the window opens at 11pm and we are still online at 2. Two people already ask for shift change because of it”Individual Contributor, ITO, Technology engineering and operations

Written sentiment on the financial institution’s change control was not associated with the study measures. A control built for periodic releases met a delivery that changed daily, and respondents described closing the gap by changing first and seeking approval afterward. The change record the financial institution held and the changes that reached its systems then differed (Part 3).

2.7 Provisioning of System Access and Agent Identities

Access is a readiness failure with a consequence for control. An agent needs its own identity and permissions to act in the financial institution’s systems, and only the financial institution can issue them. Respondents rated its ability to do so in the distressed range in 410 of 801 answers (51.19%). Where access did not arrive, the engagement continued by other means, through manual entry, emailed extracts and logins never issued for the purpose. Each of these means leaves the financial institution less able to say who, or what, acted in its systems.

2.7.1 Agent Access Provisioned Through Borrowed and Shared Credentials

Respondents rated the financial institution’s ability to provision the system access and identities that AI agents require. The mean rating was 3.5 and the median 3.0 (n = 801 respondents), and 410 of the 801 answers (51.19%) were distressed. Respondents on canceled engagements rated provisioning far lower than respondents on active engagements in any stage (mean 2.4, n = 307 respondents).

The 22 robustness checks left intact each association between the provisioning of system access and agent identities and the four core study measures. Cancellation was less common where provisioning was rated higher (r = −0.58, 95% CI −0.67 to −0.46, n = 150 engagements). Among active engagements, better provisioning was associated with higher customer satisfaction (r = 0.69, 95% CI 0.57 to 0.79, n = 94 engagements). The engagement indicator mean, computed without the provisioning indicator, was higher on engagements that rated provisioning higher (r = 0.80, 95% CI 0.74 to 0.85, n = 150 engagements). Respondents who rated provisioning higher reported lower intent to leave (r = −0.59, cluster bootstrap 95% CI −0.66 to −0.51, n = 751 respondents).

The written responses addressing access and identities numbered 437, from 348 respondents on 135 engagements, and 403 of the 437 (92.22%) were negative. They describe access requests that waited weeks or months and agents with no identity of their own. Delivery teams worked from emailed files, typed data in by hand or entered systems through borrowed and shared logins.

“Their identity team treats a service account for an agent exactly like a request for a new joiner, same form, same four week SLA, and nobody has explained to them that it is a different animal. So we are building dashboards off extract files that somebody mails us on Fridays.”Engagement Manager, KPO, Corporate and back-office functions

“We waited five weeks for processor sandbox credentials and in the end we got a shared login that three teams were using, which is its own problem … Every screen wants a human login. So we sat a human in front of it and called it agentic.”Individual Contributor, ITO, Payments, cards and disputes

“i send request, it goes to a team mailbox, three approvers are named and each one thinks other two will do it. so nothing move for four five days.”Individual Contributor, ITO, Technology engineering and operations

“access provisioning never finish before we go live so the agents patch around missing permissions constantly, happens most days.”Engagement Manager, CXM, Corporate and back-office functions

“Access provisioning for the agent took weeks instead of days”Individual Contributor, ITO, Technology engineering and operations

Written sentiment on access was associated with the outcomes. Engagements with a negative written response on access had a higher cancellation rate and recorded lower customer satisfaction. Respondents who wrote more negatively about access reported higher intent to leave. The responses also show a cost the rating does not measure. An agent that runs under a person’s account, and a member of staff who reaches a system through someone else’s credentials, leave the financial institution unable to say who acted in its systems.

2.8 Stability of the Business Case

The last indicator of the financial institution’s readiness concerns the business case on which the engagement rested. Respondents rated its stability, and that of the financial institution’s definition of return, in the distressed range in 420 of 794 answers (52.90%). Where the case moved after work began, the written responses describe scope that changed with it and tasks that had to be redone.

2.8.1 Business Case and Return Definition That Moved After Work Began

Respondents rated the stability of the financial institution’s business case and its definition of return on investment for the engagement. The mean rating was 3.5 and the median 3.0 (n = 794 respondents), and 420 of the 794 answers (52.90%) were distressed. Respondents on canceled engagements rated the stability of the case far lower than respondents on active engagements in any stage (mean 2.3, n = 298 respondents).

The stability of the business case showed an association with each of the four core study measures that survived all 22 robustness checks. A more stable case and definition of return were associated with less cancellation (r = −0.57, 95% CI −0.67 to −0.46, n = 150 engagements). Customer satisfaction among active engagements was higher where the case was rated more stable (r = 0.72, 95% CI 0.60 to 0.80, n = 94 engagements). Business-case stability was associated with a higher engagement indicator mean, computed without the stability indicator (r = 0.82, 95% CI 0.76 to 0.87, n = 150 engagements). Intent to leave was lower among respondents who rated the case more stable (r = −0.59, cluster bootstrap 95% CI −0.65 to −0.51, n = 746 respondents).

The written responses addressing the business case numbered 34, from 33 respondents on 25 engagements, and 30 of the 34 were negative. The negative responses described a case and a scope that kept moving after delivery began, rewritten several times within weeks or redefined each quarter. The four positive responses described a financial institution that knew what it wanted from the first week.

“We ask, it goes to their platform team, then to their security, then it stops. The business case also changes twice in that period, so we are never sure what we are building towards.”Engagement Manager, ITO, Technology engineering and operations

“Here it is a ticket for every access and the ROI story changes each quarter, consequently i stopped asking”Individual Contributor, ITO, Technology engineering and operations

“Scope on their side moves after work is already underway. What that costs my team is not the rework itself, it is that I must ask same four people to redo something they finished on Friday, and I am running out of ways to ask it”Engagement Manager, KPO, Regulatory, audit, tax and legal

“Business case kept shifting to justify a bad bet”Individual Contributor, KPO, Corporate and back-office functions

Written sentiment on the business case was not associated with the study measures. The written evidence is thin and consistent. It records the effect of a moving case on the delivery team, scope that changed and work that was rebuilt, and it does not show why the case moved.

Co-management is the category of indicators that exist only because two firms share the engagement. Its 20 indicators describe how the financial institution and the vendor governed the engagement together, managed its risks and issues and controlled changes to the agents. They also describe how the two firms reviewed agent output and adapted their joint structures to work performed by agents. The category had a mean of 3.65 across the 150 engagements, second lowest of the five, and 7,866 of its 15,968 answers (49.26%) were distressed. Its indicators are reported in the order of the survey section from which they come.

Mean ratings in Table 9 are computed across respondents. The category figure of 3.65 is computed across engagements.

The category differs from the others in two respects. Respondents on the 56 canceled engagements rated it lower than any other category, at a mean of 2.22 across its indicators (Table 7). Among active engagements it was rated lower by respondents on the 20 engagements in Production (4.07) than on the 31 in Build-Out (4.67), the widest difference between stages of any category. Its indicators were associated with intent to leave as closely as any in the study, from r = −0.69 to r = −0.64. Their associations with customer satisfaction among active engagements lay in the lowest range of any category, from r = 0.59 to r = 0.63 (n = 94 engagements each).

The written evidence describes two firms that managed agent work through joint structures designed for staffed delivery. Decisions, risks, issues and changes were handled in two places, one in each firm, or in none. Human review was sized for the output of people and received the output of agents, and joint decisions were taken at a slower tempo than the agents acted. The survey closed the section with an invitation to comment on the joint management of the engagement, and 551 of the 850 respondents (64.82%) wrote a comment.

“The joint calls work, but I am the one holding the gap between an agent that acts in seconds and a decision that comes on Thursday. That sits with me personally, and my leads know to call me at any hour when the queue backs up.”Engagement Manager, CXM, Corporate and back-office functions

“It is that neither side see what the other changed until the allocation is already posted… for that reason we find out at month end, every month”Individual Contributor, BPO, Trade, settlement and reconciliation

Where a joint structure failed, respondents described the delivery team acting first and seeking approval afterward, or not at all. The examination of controls takes up that pattern and the controls it bypassed (Part 3).

3.1 Joint Governance and Decision Rights

Four indicators describe how the two firms managed the engagement together and settled who decides. They cover the effectiveness of joint management, the boundary of agent autonomy, the ownership of each step in a shared workflow and the accountability for problems arising from agent action. Each had about half of its answers in the distressed range, from 388 of 808 answers (48.02%) for the ownership of workflow steps to 406 of 808 (50.25%) for the boundary of autonomy. Across the four, every association with the core study measures held under the 22 robustness checks.

The written evidence describes joint structures that existed in documents and decided nothing. Forums with terms of reference met rarely or stopped meeting. No document stated what the agent was allowed to do without a person, and no name stood against each step of the shared process. When a problem arose from an agent’s action, the two firms spent days on whose fault it was before either fixed it.

“What it costs us is Fridays. Because it has never been settled which policy questions the agent may answer alone, my leads sit in a two-hour joint review every week clearing a backlog of held tickets that mostly should never have been held. The team notices, and it is the part of the week they resent.”Engagement Manager, CXM, Corporate and back-office functions

“The problem is nobody can tell me who is allowed to switch the thing off. In February the tool post same vendor exception into queue forty times, maybe more, and I ring my lead, my lead ring somebody else, and by then it is next morning already.”Individual Contributor, BPO, Corporate and back-office functions

The consequence was that decisions about agent action passed to each analyst on each case. Engagements in which the joint structures worked were also present in the written evidence. On those, the two sides met, decided and closed issues, and respondents placed the remaining weakness in the depth of review or in the means to stop an agent quickly.

3.1.1 Joint Management Structures That Existed on Paper

Respondents rated how effectively the financial institution and the vendor jointly managed the engagement now that agents performed part of its tasks. The mean rating was 3.7 and the median 4.0 (n = 789 respondents), and 385 of the 789 answers (48.80%) were distressed. Respondents on canceled engagements rated joint management far lower than respondents on active engagements in any stage (mean 2.2, n = 299 respondents).

Joint management of the engagement by the two firms was associated with every core study measure, without a single association overturned in the 22 robustness checks. Engagements that rated joint management higher had a lower cancellation rate (r = −0.61, 95% CI −0.70 to −0.50, n = 150 engagements). Customer satisfaction among active engagements was higher where joint management was rated higher (r = 0.62, 95% CI 0.48 to 0.73, n = 94 engagements). Effective joint management was associated with a higher engagement indicator mean, computed without the joint-management indicator (r = 0.84, 95% CI 0.78 to 0.88, n = 150 engagements). Intent to leave was lower among respondents who rated joint management higher (r = −0.67, cluster bootstrap 95% CI −0.72 to −0.60, n = 739 respondents).

The written responses addressing joint management numbered 209, from 199 respondents on 115 engagements. They were more evenly divided than for most indicators, with 127 of the 209 responses negative (60.77%) and 79 of the 209 responses positive (37.80%). The negative responses described joint management that did not exist, or that met without deciding anything, while the engagement proceeded through separate trackers and private records. The positive responses described joint calls that worked and placed the remaining weakness in the review step or in the means to stop an agent.

“There was a joint operating forum on paper — I have the terms of reference still, names on it, a weekly slot — and in eight months it met four times, twice with only my side in the room. So the ones are not a judgment on the people, they are a judgment on a structure that existed only in a document.”Engagement Manager, CXM, Corporate and back-office functions

“Two firms is generous, it is whoever picks up the exception queue that morning deciding what a stalled contract exception does next, and whatever we decide is what happened. Nobody above either side ever looks at it.”Individual Contributor, BPO, Corporate and back-office functions

“We had a finding on duplicate vendor payments sitting for eleven days, nobody would say if it was theirs to sign or ours. In the end my lead just approved it himself, and the review after that was one person clicking through forty items in an hour.”Individual Contributor, KPO, Regulatory, audit, tax and legal

“On my previous ITO account we have a joint bridge for every P2 and both sides sit on it. Here the client incident manager joins only when it is already a P1, so the middle cases drift for a day or two.”Engagement Manager, ITO, Technology engineering and operations

“my last account was pension admin, one weekly call. here there is a joint call every morning, my lead sits in it and sometimes an answer comes back same day. that is better than what i had before, even if none of it reaches my queue”Individual Contributor, KPO, Client servicing and communications

Written sentiment on joint management was strongly associated with the outcomes. Engagements with a negative written response on joint management had a far higher cancellation rate and recorded lower customer satisfaction. Respondents who wrote more negatively about joint management reported higher intent to leave. Where the joint structure decided nothing, the decision passed to whoever held the case.

3.1.2 Unsettled Boundary of Agent Autonomy

Respondents rated how far the two firms had settled who decides what agents may do without human approval. The mean rating was 3.6 and the median 3.0 (n = 808 respondents), and 406 of the 808 answers (50.25%) were distressed. Respondents on canceled engagements rated the settlement of autonomy far lower than respondents on active engagements in any stage (mean 2.2, n = 309 respondents).

The settlement of agent autonomy between the two firms was associated with cancellation, customer satisfaction, the engagement indicator mean and intent to leave, and no robustness check overturned an association. Cancellation was less common where the boundary of autonomy was rated settled (r = −0.62, 95% CI −0.71 to −0.51, n = 150 engagements). Among active engagements, customer satisfaction was higher where the boundary of autonomy was settled (r = 0.59, 95% CI 0.44 to 0.71, n = 94 engagements). The engagement indicator mean, computed without the autonomy indicator, was higher where the boundary was settled (r = 0.84, 95% CI 0.79 to 0.88, n = 150 engagements). Respondents who rated the boundary settled reported lower intent to leave (r = −0.67, cluster bootstrap 95% CI −0.73 to −0.60, n = 758 respondents).

The written responses addressing the boundary of autonomy numbered 48, from 48 respondents on 42 engagements, and 34 of the 48 were negative. Half said the two firms had never settled or written down what the agent could do without a person, and each analyst decided case by case. The remedy respondents proposed was a single agreed list. Where the list existed, respondents said it held.

“Give us checklist for what the agent is allowed to write without review, because right now every one of us decide by ourself.”Individual Contributor, CXM, Client servicing and communications

“We set the decision rights in the first two weeks and wrote them down, that part has held up. Anything the agent does above a sev-3 goes to a named person on the bank side and a named person on ours. Where it is thinner is review depth.”Engagement Manager, ITO, Technology engineering and operations

“In my last job on the fraud queue there is one name against every rule, if it break you know who to call. Here nobody know who says yes when the agent close an alert by itself. My Lead ask them, they ask us back, and we keep triaging by hand in the middle.”Individual Contributor, ITO, Technology engineering and operations

“At present every material disposition still comes back to a person on our side, which is safe but it does not scale, and we have not had the conversation with their compliance leadership about which alert types the agent may close on its own. That conversation has been deferred twice now.”Portfolio Director, BPO, Financial crime and client onboarding

“Decision rights are clear, we know what the agent can file without a person signing off.”Individual Contributor, BPO, Corporate and back-office functions

Written sentiment on the boundary of autonomy was not associated with the study measures. The written evidence presents the boundary of autonomy as a document that either existed or did not. Where it did not, the decision passed to each analyst on each case, and the agent saved little time. Where it did, respondents turned to the next question, whether the people reviewing the agent’s output could keep pace with it.

3.1.3 Unowned Handoffs in Shared Workflows

Respondents rated how distinctly ownership of each workflow step was assigned where agents, vendor staff and the financial institution’s staff worked in the same process. The mean rating was 3.7 and the median 4.0 (n = 808 respondents), and 388 of the 808 answers (48.02%) were distressed. The assignment of step ownership was rated far lower on canceled engagements than on active engagements in any stage (mean 2.2, n = 305 respondents).

Every association of workflow-step ownership with the four core study measures held, in each of the 22 robustness checks. Engagements that rated the ownership of each workflow step higher had a lower cancellation rate (r = −0.65, 95% CI −0.73 to −0.55, n = 150 engagements). Customer satisfaction among their active engagements was higher (r = 0.61, 95% CI 0.47 to 0.72, n = 94 engagements). Clear ownership of each step was associated with a higher engagement indicator mean, computed without the ownership indicator (r = 0.85, 95% CI 0.80 to 0.89, n = 150 engagements). Respondents who rated step ownership higher reported lower intent to leave (r = −0.69, cluster bootstrap 95% CI −0.74 to −0.62, n = 758 respondents). After Holm correction the rating also differed by process family.

The written responses addressing the ownership of workflow steps numbered 82, from 81 respondents on 58 engagements, and 70 of the 82 were negative. Among the 68 responses in the comment field on joint management, 40 described a step in the shared process that belonged to neither firm. When the agent or a person handled that step wrongly, each firm said it belonged to the other, and the case waited. The remedy respondents proposed was a name against each step.

“When the assistant sends a HR case to wrong queue and it sits four days, we have meeting about it and both sides explain why it is not their part. And then next week same thing. My lead says we cannot change the routing rules ourself because it is joint decision, but joint decision takes longer than the SLA it is breaking.”Individual Contributor, CXM, Corporate and back-office functions

“A single attribution rule change sat for eleven weeks … Word came back it was our call to make, our lead said it was theirs, and in the meantime the agent kept producing breakdowns on the old basis.”Individual Contributor, KPO, Trade, settlement and reconciliation

“Nobody owned the step where a generated test case gets accepted, we assumed their QA lead did, he assumed we did, and it sat there. Also every decision we raised took three weeks to revert back, which is not a pace anything can be built at.”Engagement Manager, ITO, Technology engineering and operations

“Last month close we get 400 exceptions in one night and only two of us to review them. But nobody decide who signs off the ones the agent already cleared. i check them all myself, hence we finish late”Individual Contributor, BPO, Trade, settlement and reconciliation

“We keep our own sheet now. Every morning my senior analyst lists which summaries the agent touched overnight and who on their side signs each one, and we walk through it on a fifteen minute call before their clinical lead starts. It is not governance. But it is the only place both firms see the same list.”Engagement Manager, KPO, Credit, underwriting and suitability

Written sentiment on step ownership was itself associated with the outcomes. Engagements with a negative written response on step ownership had a higher cancellation rate, and respondents who wrote more negatively about it reported higher intent to leave. A step that belonged to neither firm became the point at which cases waited and errors went unclaimed.

3.1.4 Accountability for Problems Arising From Agent Action Dissolved Across the Two Firms

Respondents rated how distinctly accountability landed on one firm when a problem arose from an agent’s action in the engagement. The mean rating was 3.7 and the median 3.0 (n = 799 respondents), and 402 of the 799 answers (50.31%) were distressed. Respondents on canceled engagements rated the assignment of accountability far lower than respondents on active engagements in any stage (mean 2.2, n = 299 respondents).

Accountability for problems arising from agent action showed an association with each of the four core study measures that survived all 22 robustness checks. Clearer accountability was associated with less cancellation (r = −0.64, 95% CI −0.73 to −0.54, n = 150 engagements). Among active engagements, customer satisfaction was higher where accountability was rated clearer (r = 0.61, 95% CI 0.46 to 0.72, n = 94 engagements). The engagement indicator mean, computed without the accountability indicator, was closely associated with it (r = 0.87, 95% CI 0.82 to 0.90, n = 150 engagements). Intent to leave was lower among respondents who rated accountability clearer (r = −0.68, cluster bootstrap 95% CI −0.73 to −0.61, n = 750 respondents).

The written responses addressing accountability numbered 51, from 50 respondents on 37 engagements, and 45 of the 51 were negative. When a problem arose from an agent’s action, respondents described days spent deciding which firm was at fault before either fixed it, and incident records that named a symptom in place of an owner. Several said that stopping the agent was quick and that deciding who owned the harm was not.

“both firms are perfectly able to describe what went wrong and neither is willing to write down who carries it. We have a joint issue log, on paper. In practice the log records symptoms. When a suite the agent generated missed a defect that later surfaced in UAT, the entry reads environment instability, which is true enough, and also convenient for everyone in the room, ourselves included.”Portfolio Director, ITO, Technology engineering and operations

“There was a letter that went out with the wrong deal date in it last quarter and eleven days went by before anyone would say where the fault sat, and in the end no one did.”Individual Contributor, CXM, Client servicing and communications

“escalation is the whole issue here. when the agent dispositions an alert wrong nobody owns it after. my side says compliance owns disposition, compliance says we built the rule. so it sits in queue for weeks and then ages out. which is why the backlog number they report is meaningless.”Individual Contributor, ITO, Financial crime and client onboarding

“when the assistant mis-reads an underwriting requirement and a new-business file sits for a week, there is nobody on our side who carries it. I raise it, my manager raises it, and it comes back as a question for the client.”Engagement Manager, BPO, Credit, underwriting and suitability

“nobody owns errors when the agent ships something broken”Engagement Manager, ITO, Payments, cards and disputes

Written sentiment on accountability was not associated with the study measures. The written evidence describes accountability that dissolved across the hands through which an agent’s output passed. Where neither firm had agreed in advance which party would remediate, the question was argued after each incident, and the incident record left it unanswered in the accounts respondents gave.

3.2 Shared Risk and Issue Management

Four indicators describe how the two firms managed the risks and issues that agent work produced. They ask whether the firms kept one view of agent risk and managed issues as one team. The indicators also ask whether the firms resolved issues at the speed at which agent error spreads and could stop an agent behaving badly in live operation. Each had about half of its answers in the distressed range, from 383 of 793 answers (48.30%) for the shared view of risk to 406 of 804 (50.50%) for the authority to stop an agent. All four held their associations with the core study measures under the 22 robustness checks.

The written evidence reduces shared risk and issue management to two concrete questions. The first is whether the two firms kept one list of risks and issues or two. Where they kept two, the same problem had two statuses, was discussed in two meetings and was sometimes resolved in neither. The second is whether a named person on each side could pause an agent. Where one could, respondents described agents stopped within minutes. Where none could, or the authority was reachable only through the weekly call, the agent continued at its own pace while the firms worked out who held the switch.

“The change control part is honestly fine, we see their updates before they land and they see ours… you should ask about the stopping instead. If the agent starts pushing rubbish into the pipeline on a Friday night, I do not know who I ring, my lead says it is their side, they say it is ours.”Individual Contributor, ITO, Technology engineering and operations

An error made by a person stops when the person stops. An error made by an agent continues until someone with authority pauses the agent. The joint structures described in the written evidence were built for errors made by people.

3.2.1 Separate Risk Records Maintained by the Two Firms

Respondents rated how well the two firms maintained a single shared view of the risks that agent work introduced. The mean rating was 3.6 and the median 4.0 (n = 793 respondents), and 383 of the 793 answers (48.30%) were distressed. Respondents on canceled engagements rated the shared view of risk far lower than respondents on active engagements in any stage (mean 2.2, n = 297 respondents).

The 22 robustness checks left intact each association between a shared view of agent risk and the four core study measures. Engagements where the two firms kept a shared view of risk had a lower cancellation rate (r = −0.62, 95% CI −0.71 to −0.51, n = 150 engagements). A shared view of risk was associated with higher customer satisfaction among the active engagements (r = 0.62, 95% CI 0.48 to 0.73, n = 94 engagements). The engagement indicator mean, computed without the risk-view indicator, was higher where the view was shared (r = 0.85, 95% CI 0.80 to 0.89, n = 150 engagements). Respondents who rated the shared view higher reported lower intent to leave (r = −0.67, cluster bootstrap 95% CI −0.73 to −0.61, n = 744 respondents).

Few written responses addressed the shared view of risk, 26, all in comment on joint management, and 19 of the 26 were negative. Separate risk lists, one kept by each firm and never merged, were described in 16 of the 26 responses. The same risk was raised in two meetings and resolved in neither. The positive responses described a single shared log reviewed together and placed the remaining gap elsewhere.

“What did not exist, and would have to next time, is one risk log rather than two, and some way for each side to see what the other’s agents had actually done on a ticket without having to ask … None of that is hard. It just needs doing before day one rather than at month two.”Portfolio Director, CXM, Corporate and back-office functions

“same for the shared view of risk. two firms, two lists, nobody put them side by side.”Individual Contributor, ITO, Financial crime and client onboarding

“What is missing is a common risk list. Their side keeps their own, we keep ours, and QA only catches a bad suggestion in the pricing code when somebody remembers to compare them.”Individual Contributor, ITO, Technology engineering and operations

“Both sides keep one risk log for the agents, this works well. our Lead meets their dispute team every week and issues get closed in that call.”Individual Contributor, BPO, Payments, cards and disputes

Engagements whose written responses on the shared view of risk were more negative had a higher cancellation rate. Respondents who wrote more negatively about it reported higher intent to leave. For the respondents who wrote, the shared view of risk was a single document, present or absent. Where there were two lists, a risk known to one firm remained unknown to the other until something failed.

3.2.2 Issue Management Through Separate Lists

Respondents rated how effectively the two firms managed issues arising from agent work as one team, rather than through two separate processes. The mean rating was 3.7 and the median 4.0 (n = 799 respondents), and 391 of the 799 answers (48.94%) were distressed. Its mean of 3.73 was the highest of the 20 indicators of co-management (n = 799 respondents). Respondents on canceled engagements rated joint issue management far lower than respondents on active engagements in any stage (mean 2.3, n = 302 respondents).

Joint management of issues as one team was associated with cancellation, customer satisfaction, the engagement indicator mean and intent to leave, and no robustness check overturned those associations. Cancellation was less common where issues were managed jointly (r = −0.61, 95% CI −0.70 to −0.50, n = 150 engagements). Joint issue management was associated with higher customer satisfaction among the active engagements (r = 0.61, 95% CI 0.47 to 0.72, n = 94 engagements). It was associated with a higher engagement indicator mean, computed without the issue-management indicator (r = 0.85, 95% CI 0.79 to 0.89, n = 150 engagements). Respondents who rated joint issue management higher reported lower intent to leave (r = −0.66, cluster bootstrap 95% CI −0.72 to −0.60, n = 749 respondents).

The written responses addressing issue management numbered 54, all in comment on joint management, and 39 of the 54 were negative. In 26 of the 54 responses, each firm logged issues in its own tracker and the two trackers were never joined. The same problem had two statuses, was discussed in two calls and was sometimes fixed in neither. In some engagements the vendor’s team kept a third, private list. The positive responses described a joint call or a single shared list that worked.

“we and the bank keep two separate issue logs, and neither is complete. Ours records what the agent got wrong, theirs records what reached their desk, and the overlap is maybe half. What that means in practice is this: when we sit down every fortnight we spend the first half hour agreeing what happened, before anyone can decide anything.”Engagement Manager, KPO, Credit, underwriting and suitability

“when an inquiry lands with the wrong relationship team, nobody on their side takes it, our lead takes it, thus it is our fault by default. Accordingly I stopped raising them. Two processes, and me in the middle.”Individual Contributor, CXM, Client servicing and communications

“In March an agent put hold on around forty accounts wrongly. Our side found it in an hour, but their servicing team was running its own investigation separately and we only joined up next day… same issue, two diffrent calls.”Individual Contributor, BPO, Payments, cards and disputes

“Weekly call with their service manager, Tuesday. Issues get logged both sides, one in their tool one in ours, so same ticket sits in two places.”Individual Contributor, ITO, Technology engineering and operations

“My marks in this section are high because of one thing, the joint triage call on thursday actually works, and their middle office lead reads every classification we flag. It is a person, not a process.”Engagement Manager, KPO, Corporate and back-office functions

Respondents who wrote more negatively about issue management reported higher intent to leave. Where the two firms kept one list, the written evidence moved on to other questions, the depth of review and the speed of stopping an agent. Where they kept two, the same issue was handled twice and owned by neither.

3.2.3 Resolution Speed Against the Spread of Agent Error

Respondents rated how well the speed of joint issue resolution matched the speed at which problems arising from agent action could spread. The mean rating was 3.7 and the median 4.0 (n = 809 respondents), and 400 of the 809 answers (49.44%) were distressed. Respondents on canceled engagements rated the speed of resolution far lower than respondents on active engagements in any stage (mean 2.3, n = 306 respondents).

Each of the four core study measures was associated with the speed at which the two firms resolved issues together, and all 22 robustness checks upheld the associations. Faster resolution was associated with less cancellation (r = −0.62, 95% CI −0.71 to −0.51, n = 150 engagements). Among active engagements, customer satisfaction was higher where resolution was rated faster (r = 0.60, 95% CI 0.46 to 0.72, n = 94 engagements). The engagement indicator mean, computed without the resolution-speed indicator, was higher as well (r = 0.84, 95% CI 0.79 to 0.88, n = 150 engagements). Intent to leave was lower among respondents who rated resolution faster (r = −0.64, cluster bootstrap 95% CI −0.71 to −0.57, n = 758 respondents).

Few written responses addressed the speed of resolution, 18, all in comment on joint management, and 16 of the 18 were negative. Most described an agent that kept acting while the two firms resolved the problem. One described 50 responses sent during a three-day escalation, and another 20 more new hires moved through a misfiring step before anyone saw the alert. A third described 100 more lines processed while the firms discussed a wrong supplier record.

“When an onboarding journey misfires, the agent has already moved twenty more new hires through the same step before anyone reads the alert. What we need is a standing call, not a weekly one, and one named individual at the bank who can halt a journey without waiting for the Thursday forum.”Engagement Manager, CXM, Corporate and back-office functions

“when an agent-generated validation output turns out wrong, we spend the first day of the escalation deciding whose problem it is rather than fixing it. That part has not improved in nine months.”Engagement Manager, KPO, Credit, underwriting and suitability

“Practical change is who can stop the run. when matching goes wrong on a broker statement i must log it our side and again their side, two tickets, two waiting… by then breaks are already double. one named person per shift with right to pause the Agent, and one issue list instead of two.”Individual Contributor, BPO, Trade, settlement and reconciliation

“It is better than first month, now we have one call together on Thursday and issue list is shared. Still if the policy Bot gives wrong answer on leave rules it stays like that till weekly call, nobody there can pause it”Individual Contributor, CXM, Corporate and back-office functions

Written sentiment on the speed of resolution was not associated with the study measures, on a base too small for that test to be informative. The written responses describe how an agent’s error spreads. It continues at the agent’s pace until someone with authority pauses it, and in the engagements described, that authority was missing or reachable only through the weekly call.

3.2.4 Authority to Stop a Misbehaving Agent

Respondents rated how quickly the two firms together could stop an agent behaving badly in live operation. The mean rating was 3.6 and the median 3.0 (n = 804 respondents), and 406 of the 804 answers (50.50%) were distressed. Respondents on canceled engagements rated the ability to stop an agent far lower than respondents on active engagements in any stage (mean 2.2, n = 312 respondents).

Every association of the joint ability to stop a misbehaving agent with the four core study measures held, in each of the 22 robustness checks. Engagements that could stop an agent more quickly had a lower cancellation rate (r = −0.63, 95% CI −0.72 to −0.52, n = 150 engagements). Customer satisfaction among their active engagements was higher (r = 0.60, 95% CI 0.45 to 0.71, n = 94 engagements). The ability to stop an agent was closely associated with the engagement indicator mean, computed without the stopping indicator (r = 0.86, 95% CI 0.81 to 0.89, n = 150 engagements). Respondents who rated it higher reported lower intent to leave (r = −0.67, cluster bootstrap 95% CI −0.73 to −0.61, n = 753 respondents).

The written responses addressing the authority to stop an agent numbered 73, from 71 respondents on 51 engagements, and 59 of the 73 were negative. Nearly half said that the respondent did not know who could stop a misbehaving agent, or that neither firm could. They described agents that continued for hours or days while the two firms worked out who held the switch. Several proposed the same remedy, one named person with the authority to pause the agent.

“The agent started auto closing help desk tickets that were still open on the floor side, about forty of them in an hour, because someone had changed a rule on a Friday and nobody told us. So i go to the bank’s IT lead and ask him to switch it off. He says he cannot, it is our tool. i ask our own side and they say the change has to go through the joint forum, and the forum sits every second week … In the end one of my analysts stopped it from the console himself and we wrote it up afterwards, and to this day nobody has told us whether that was allowed.”Engagement Manager, CXM, Corporate and back-office functions

“We spotted it fast and the client’s credit head was on the call within the hour, but actually switching it off took three people and nearly two hours… the shared risk log worked, the stop button did not really exist.”Engagement Manager, KPO, Credit, underwriting and suitability

“One thing… nobody can switch it off. Last month the model start flagging every counterparty in one sector, hundreds of alerts in a morning. I ask my team lead who stops it and he does not know either. It runs the full cycle. Then we sit and clear the alert list one by one after.”Individual Contributor, KPO, Credit, underwriting and suitability

“In week six the generator produced a joining letter with the wrong notice period. Our QA caught it before any new hire saw it. But then we spent two days deciding who was permitted to switch the thing off, us or their HR team.”Engagement Manager, CXM, Corporate and back-office functions

“Escalation has improved, we can halt an agent inside the hour now and both sides accept it without argument.”Portfolio Director, KPO, Regulatory, audit, tax and legal

Respondents who wrote more negatively about stopping an agent reported higher intent to leave. Where the firms could stop an agent within minutes, respondents placed the remaining difficulty in deciding afterward who owned the harm. Where the authority was unassigned, or rested with staff who kept office hours, the agent continued until the question was settled, and the delivery team cleared the output by hand afterward.

3.3 Joint Change Control

Three indicators describe how the two firms controlled changes to the agents. They ask whether joint change control kept pace with changes in agent behavior and whether an update made by one firm became visible to the other before it took effect. The third asks whether the firms governed the changes agents made to their own behavior between formal releases. The pace of joint change control was distressed for 399 of 798 respondents (50.00%). Visibility of updates was distressed in 394 of 803 answers (49.07%) and governance of drift in 388 of 806 (48.14%). The three kept every association with the core study measures through the 22 robustness checks.

The written evidence describes change control set to the cycle of a meeting and applied to agents that changed between meetings. Joint sign-off took place weekly, fortnightly or monthly while the agents changed within minutes, and in the meantime the agent continued on the old rule. Each firm altered the shared agent and told the other afterward, and the usual means of discovery was the output. Changes in agent behavior that occurred between releases had no owner and no forum, and the firms found them when the output moved or a pipeline failed.

“Here there is no such forum. Changes to the agent behaviour happen and I find out from my own team, not from the client and not from our side either.”Engagement Manager, CXM, Financial crime and client onboarding

“these questions assume both sides agree what change even is, we discuss about a change for weeks then agent behaviour move on its own in between”Individual Contributor, ITO, Technology engineering and operations

Where sign-off took longer than the change itself, teams built side channels that kept the engagement moving, a group chat in place of the change log or a request agreed in a corridor. The formal change record then described less than had changed. The gap between the change record and the agents’ actual behavior is examined with the controls over agent work (Part 3).

3.3.1 Joint Change Control Outpaced by Agent Behavior

Respondents rated how well the joint change control process kept pace with the speed of changes in agent behavior. The mean rating was 3.7 and the median 3.5 (n = 798 respondents), and 399 of the 798 answers (50.00%) were distressed. Respondents on canceled engagements rated joint change control far lower than respondents on active engagements in any stage (mean 2.2, n = 303 respondents).

Joint change control, as rated against the pace of agent change, was associated with all four core study measures, and each association held under the 22 robustness checks. Engagements whose change control kept better pace had a lower cancellation rate (r = −0.63, 95% CI −0.71 to −0.52, n = 150 engagements). Their customer satisfaction among active engagements was higher (r = 0.60, 95% CI 0.45 to 0.72, n = 94 engagements). Change control that kept pace was associated with a higher engagement indicator mean, computed without the change-control indicator (r = 0.85, 95% CI 0.80 to 0.89, n = 150 engagements). Respondents who rated it higher reported lower intent to leave (r = −0.66, cluster bootstrap 95% CI −0.72 to −0.60, n = 752 respondents). After Holm correction, ratings of joint change control also differed by process family.

The written responses addressing joint change control numbered 71, from 70 respondents on 53 engagements, and 57 of the 71 were negative. Respondents described a joint process that met weekly, fortnightly or monthly while the agents changed within minutes, and sign-off that took days or weeks while the agent continued on the old rule. Several described changes made or agreed outside the joint process altogether.

“In March they changed the tolerance on the price check and we found out when the breaks tripled next morning. Nobody told us … The problem is the override queue, one firm waits for the other and the break sits four days. Accordingly nobody is accountable and everybody is polite.”Individual Contributor, BPO, Trade, settlement and reconciliation

“Change requests reach me directly from the deal desk, in email, sometimes in a hallway call, and they are not raised as change orders, so the sprint absorbs them and the burndown reads as though we estimated badly … There is no shared record of what changed and when it changed.”Engagement Manager, ITO, Technology engineering and operations

“Practically, the reviewer on their side approves our rule changes in a batch every Thursday and cannot possibly read them all. We would need two named owners per rule set, one there and one here, and the authority to push a low-risk config change without waiting for the Thursday meeting”Engagement Manager, ITO, Regulatory, audit, tax and legal

“Ran change board with them again last week, same outcome. What must change is simple, both sides need to see rule updates before they take effect, not in following month’s minutes. When their team altered retention period for claims files, our classifier kept filing to old rule for nearly three weeks, and nobody told us.”Engagement Manager, BPO, Corporate and back-office functions

“A tuning change to how alerts are ranked took eleven days to sign off on both sides”Individual Contributor, ITO, Technology engineering and operations

Engagements whose written responses on joint change control were more negative had a higher cancellation rate. The written responses describe a control designed for periodic release, applied to agents that changed between meetings. Where sign-off was slower than the change, the agent continued on the old rule, and teams built side channels that kept the engagement moving and left the formal change log incomplete.

3.3.2 Agent Updates Disclosed to the Other Firm After the Fact

Respondents rated how well agent updates made by one firm became visible to the other firm before they took effect. The mean rating was 3.6 and the median 4.0 (n = 803 respondents), and 394 of the 803 answers (49.07%) were distressed. Respondents on canceled engagements rated the visibility of updates far lower than respondents on active engagements in any stage (mean 2.2, n = 306 respondents).

All four core study measures were associated with the visibility of each firm’s agent updates to the other before they took effect, and none of the 22 robustness checks overturned an association. Cancellation was less common where updates were visible in advance (r = −0.62, 95% CI −0.71 to −0.51, n = 150 engagements). Advance visibility of updates was associated with higher customer satisfaction among the active engagements (r = 0.60, 95% CI 0.45 to 0.72, n = 94 engagements). The engagement indicator mean, computed without the visibility indicator, was higher where updates were visible (r = 0.84, 95% CI 0.79 to 0.88, n = 150 engagements). Intent to leave was lower among respondents who saw updates in advance (r = −0.66, cluster bootstrap 95% CI −0.72 to −0.59, n = 756 respondents).

The written responses addressing the visibility of updates numbered 48, from 48 respondents on 44 engagements, and 38 of the 48 were negative. Most said that a change to an agent made by one firm reached the other only after it had taken effect. It was discovered from the output, from a failure or at the next weekly meeting. Several respondents said the vendor’s side did the same.

“Somebody on their side changes what the assistant is allowed to answer and we find out from the tickets, not from a note. Last month it started closing password resets on its own and nobody on my team knew that was turned on. And afterwards we are asked why the numbers moved.”Individual Contributor, CXM, Corporate and back-office functions

“Every time someone on my team changes a prompt or a mapping we write it in the sheet with the date and who did it… And their engineers do the same thing on their side, only we never see theirs and they never see ours … So then we make a rule between ourselves. Before you push anything you ping the other three people on the team group. Not the official channel, the group chat. That is the whole change control now.”Individual Contributor, ITO, Technology engineering and operations

“Main thing is we find out the bot wording changed by hearing it come out on a live call, nobody tells us before, accordingly my team loses the first hour of shift working out what is different today”Individual Contributor, CXM, Payments, cards and disputes

“Last sprint we update the prompt template for the code review agent on friday, the bank QA lead only comes to know on monday when his report looks different. Nothing broke but he was not happy about it. After that we start putting every change in the shared tracker, still it is only us doing this, not them”Individual Contributor, ITO, Technology engineering and operations

“For any change agent make to the build script, my team put note in group chat and hold it till thursday release call… not official process but it stop surprises on their side.”Individual Contributor, ITO, Technology engineering and operations

Engagements whose written responses on the visibility of updates were more negative had a higher cancellation rate. Respondents who wrote more negatively about the visibility of updates reported higher intent to leave. The written evidence describes two firms that each altered a shared agent and told the other afterward. One respondent proposed a modest remedy, a short note in the daily call before a change took effect.

3.3.3 Unmonitored Agent Drift Between Releases

Respondents rated how well the two firms jointly governed the changes agents made to their own behavior between formal releases. The mean rating was 3.7 and the median 4.0 (n = 806 respondents), and 388 of the 806 answers (48.14%) were distressed. Respondents on canceled engagements rated the governance of drift far lower than respondents on active engagements in any stage (mean 2.2, n = 301 respondents).

The joint governance of drift in agent behavior between releases was associated with cancellation, customer satisfaction, the engagement indicator mean and intent to leave, and the associations held through every robustness check. Engagements that governed drift better had a lower cancellation rate (r = −0.62, 95% CI −0.71 to −0.51, n = 150 engagements). Among active engagements, governance of drift was associated with higher customer satisfaction (r = 0.59, 95% CI 0.44 to 0.71, n = 94 engagements). The engagement indicator mean, computed without the drift indicator, was higher where drift was governed better (r = 0.83, 95% CI 0.78 to 0.88, n = 150 engagements). Respondents who rated the governance of drift higher reported lower intent to leave (r = −0.67, cluster bootstrap 95% CI −0.72 to −0.60, n = 756 respondents).

Few written responses addressed drift between releases, 14, all in comment on joint management, and 12 of the 14 were negative. They described a single gap. Formal releases were governed, but the changes an agent made to its own behavior between releases had no owner, no joint record and no forum.

“the matching behaviour change on its own between releases and nobody on either side is watching for that… formal changes are handled fine, that part works, we get told and we test it. But when the match rate drift on the trading flow cash items there is no forum for it, my lead tell their ops lead informally and it stay there.”Individual Contributor, BPO, Trade, settlement and reconciliation

“The checkpoints sit in the right spots. What nobody owns is when the bot quietly starts answering a ticket type diffrently than it did last month”Individual Contributor, ITO, Technology engineering and operations

“Tuesday joint call has helped, issues that used to sit for weeks now close in two days. Change control is the weak part still, agent updates itself between releases and neither side looks at it until a pipeline falls over”Individual Contributor, ITO, Technology engineering and operations

“Bot changed how it sorted leave cases and nobody told us… my team took blame for two weeks of backlog”Individual Contributor, BPO, Corporate and back-office functions

“What number cannot show is how quick both leads pick up the phone. Only gap is when the agent changes its own behaviour between releases, that we notice late”Individual Contributor, BPO, Corporate and back-office functions

On 14 responses, written sentiment on drift showed no association with the study measures, and the base is too small for the test to inform. Change control in these engagements was built around the release, and behavior that changed between releases lay outside it. The firms found the drift in the output, after it had already affected the engagement’s results.

3.4 Human Review and the Joint Quality Assurance Loop

Five indicators describe how the two firms checked the agents’ output. They cover where human review points were placed, whether review capacity kept up with the volume of output and how much scrutiny output received at those points. The indicators also cover how reliably the joint quality loop caught agent errors before live operation, and whether the firms’ quality measures captured the errors agents actually make. Each had about half of its answers in the distressed range, from 377 of 787 answers (47.90%) for the quality loop to 405 of 792 (51.14%) for the calibration of quality measures. Across the five, no robustness check overturned an association with the core study measures.

At the center of the section is the scrutiny that agent output received at the point of review. Respondents rated it in the distressed range in 393 of 794 answers (49.50%). The written responses describe a review point that approved agent output without reading it. The leading reason respondents gave was capacity. A review step sized for the output of people met the output of agents, produced in volumes that two or four reviewers could not read in a day, and review was reduced to a sign-off.

“Marked the review one lower for a reason. Most days volume is fine but Mondays it backs up alot and two of us are checking messages for the whole floor. Everything else about how the two sides work together has been fine.”Individual Contributor, CXM, Client servicing and communications

The other four indicators describe the same control from different sides. Review points placed at the end of the process came after an error had already gone out. Some output was checked only by the analyst who produced it. The quality checks the firms performed were designed for the errors people make, transposed digits and formatting. They passed the errors agents make, an answer drawn from an outdated policy or a total that was right while every line within it was wrong. In some engagements, respondents said, the dashboards the financial institution received did not show how much output had been released without a human check (Part 5).

3.4.1 Placement of Human Review Points Across the Two Firms

Respondents rated how well human review points were placed across the two firms at the moments where agent errors would matter most. The mean rating was 3.6 and the median 4.0 (n = 797 respondents), and 393 of the 797 answers (49.31%) were distressed. Respondents on canceled engagements rated the placement of review far lower than respondents on active engagements in any stage (mean 2.2, n = 303 respondents).

The placement of human review points was associated with cancellation, customer satisfaction, the engagement indicator mean and intent to leave, and the 22 robustness checks upheld each association. Cancellation was less common on engagements whose review points were rated well placed (r = −0.62, 95% CI −0.71 to −0.51, n = 150 engagements). Customer satisfaction among active engagements was higher where review points were rated well placed (r = 0.60, 95% CI 0.46 to 0.72, n = 94 engagements). The engagement indicator mean, computed without the placement indicator, was higher too (r = 0.84, 95% CI 0.79 to 0.88, n = 150 engagements). Respondents who rated the placement of review higher reported lower intent to leave (r = −0.66, cluster bootstrap 95% CI −0.72 to −0.60, n = 747 respondents).

The written responses addressing the placement of review numbered 105, from 103 respondents on 75 engagements, and 75 of the 105 were negative. Respondents described three faults. Some review points were placed at the end of the process, after an error had already gone out. Some approvals were ticked through without anyone reading what the agent had done. Some output was checked only by the respondent. Other respondents said the review points were in the right place and the gap lay elsewhere.

“Last week the agent matched a cash item onto the wrong value date and it went out with the pack, nobody had looked at it. For my manual clears there is a checker, it has always been that way, but whatever the agent matches goes straight through.”Individual Contributor, BPO, Trade, settlement and reconciliation

“Agent does gross to net and then it just goes, there is no point in the run where somebody on their side puts a name against it. On our side my Lead checks what he can, but he has four people and one cycle is eleven thousand records, so he checks sample and that is it.”Individual Contributor, BPO, Corporate and back-office functions

“The checker on our side signs off two hundred fee lines a day and the approval on their side is a same-day formality, so a wrong tier rate can pass both and only surface when an advisor complains a month later.”Engagement Manager, BPO, Payments, cards and disputes

“We keep a two person rota for reading the drafted notes each morning before the desk opens, this is not written in the governance pack anywhere, we make it in week three ourselves.”Engagement Manager, KPO, Corporate and back-office functions

Respondents who wrote more negatively about the placement of review reported higher intent to leave. The written responses separate where review sits from whether it takes place. In some engagements the review point was correctly placed and still reduced to a sign-off. In others it was placed too late to stop an error, or did not exist, and the individual analyst became the control.

3.4.2 Human Review Capacity Against Agent Output Volume

Respondents rated how well joint human review capacity kept up with the volume of output the agents produced. The mean rating was 3.7 and the median 4.0 (n = 805 respondents), and 381 of the 805 answers (47.33%) were distressed. Respondents on canceled engagements rated review capacity far lower than respondents on active engagements in any stage (mean 2.2, n = 304 respondents).

Joint review capacity against the volume of agent output showed an association with each of the four core study measures, upheld in all 22 robustness checks. Engagements whose review capacity kept up had a lower cancellation rate (r = −0.62, 95% CI −0.71 to −0.52, n = 150 engagements). Customer satisfaction among their active engagements was higher (r = 0.59, 95% CI 0.43 to 0.70, n = 94 engagements). Review capacity was associated with a higher engagement indicator mean, computed without the capacity indicator (r = 0.82, 95% CI 0.76 to 0.87, n = 150 engagements). Intent to leave was lower among respondents who rated review capacity higher (r = −0.67, cluster bootstrap 95% CI −0.72 to −0.60, n = 755 respondents).

The written responses addressing review capacity numbered 145, from 135 respondents on 89 engagements, and 134 of the 145 (92.41%) were negative. More than half came from the question on the challenges that most reduced job satisfaction, where respondents named human review as the bottleneck of the engagement in nearly the same words each time. Elsewhere, respondents described agents that produced more in a morning than two or four people could read, approvals released without reading, and analysts whose work became checking.

“the release notes and the config diffs the agent puts together land in a review queue and the review queue is me and one other person, and we are also the ones doing the actual releases. so we read the ones that touch card auth and we tick the rest, because there is no time.”Individual Contributor, ITO, Payments, cards and disputes

“Same for review capacity, there is no joint review, there is me.”Individual Contributor, ITO, Client servicing and communications

“review is mostly quick approve, i will be honest… when queue is long the checker just click through. but this is not because anyone is careless, the volume of claimant messages is very high and both side are only small team.”Individual Contributor, CXM, Client servicing and communications

“The validation queue backs up at month-end. Two weeks into the month, and we still haven’t finished last month’s QA.”Individual Contributor, KPO, Corporate and back-office functions

“Human review became the bottleneck everyone blamed but nobody fixed”Individual Contributor, KPO, Regulatory, audit, tax and legal

Respondents who wrote more negatively about review capacity reported higher intent to leave. A control sized for the output of people was applied to the volume of agents. Review could not keep pace, was reduced to a sign-off or moved to the analysts who had produced the output. In answer to the question on how the vendor altered its reports, some respondents described dashboards from which the review backlog had been removed (Part 5).

3.4.3 Routine Approval of Agent Output at Human Review Points

Respondents rated how much genuine scrutiny agent output received at human review points, as against routine approval. The mean rating was 3.6 and the median 4.0 (n = 794 respondents), and 393 of the 794 answers (49.50%) were distressed. Ratings were highest among active engagements in Build-Out (mean 4.7, n = 160 respondents) and lowest among canceled engagements (mean 2.2, n = 300 respondents). Individual contributors, the role whose output the review is meant to check, gave the lowest ratings of the roles (mean 3.5, n = 612 respondents).

The scrutiny of agent output at human review points was associated with every core study measure, and every association held under each of the 22 robustness checks. Engagements whose output received more scrutiny had a lower cancellation rate (r = −0.64, 95% CI −0.73 to −0.53, n = 150 engagements). Customer satisfaction among active engagements was higher where output received more scrutiny (r = 0.62, 95% CI 0.48 to 0.73, n = 94 engagements). The rating was most closely associated with the engagement indicator mean, computed without the indicator (r = 0.85, 95% CI 0.80 to 0.89, n = 150 engagements). Respondents who rated scrutiny higher reported lower intent to leave (r = −0.69, cluster bootstrap 95% CI −0.74 to −0.63, n = 747 respondents).

The written responses addressing scrutiny numbered 66, from 63 respondents on 50 engagements, and 59 of the 66 (89.39%) were negative. The account given in 36 of the 50 responses written in answer to the indicator’s own question was a review point that approved agent output without reading it. Respondents attributed it chiefly to review capacity that did not match the volume of output. Respondents who wrote rated scrutiny lower than respondents who did not, at a mean of 2.6 (n = 62 respondents) against 3.7 (n = 732 respondents).

“Every release goes through a review step and someone ticks it, but nobody reads what the agent changed in the pipeline, it is a tick and nothing more. I have watched approvals come back in under a minute for a change that touched twenty files.”Individual Contributor, ITO, Technology engineering and operations

“We set a target of reviewing a sample of guided filings each day and we are not hitting it. In practice the reviewer opens the transcript, checks the loss date and the policy number match, and signs it. That is not scrutiny, that is a tick. Nobody has time for more than that at current volume, and volume is going up when the pilot widens in the fall.”Engagement Manager, CXM, Client servicing and communications

“Approvals come back same day mostly, but they are ticked through, nobody really reads the claimant message before it goes… so my team reads it twice ourselves instead.”Individual Contributor, CXM, Client servicing and communications

“We send exception pack for sign off on tuesday. Their approver return it in nine minutes with no comment, 300 lines. Same nine minutes every week, so we stop expecting real check and just fix it ourself”Individual Contributor, BPO, Trade, settlement and reconciliation

“Last month the reviewer approve one batch of alerts in two minutes. Nobody can read 300 in two minutes.”Individual Contributor, ITO, Regulatory, audit, tax and legal

“Last month the agent shut a whole run of alerts overnight, because threshold was changed on their side. Review is one click, my lead approve forty at a time. Nobody catch it for three days”Individual Contributor, ITO, Regulatory, audit, tax and legal

Written sentiment on scrutiny was not associated with the study measures, and the associations rest on the rating. A review point that approves without reading leaves an audit trail showing a human check that did not occur. Where the reviewer signed and the analyst read, the control the audit trail described rested with the person it was meant to check.

3.4.4 Quality Checks Designed for Human Rather Than Agent Error

Respondents rated how reliably the joint quality assurance loop caught agent errors before they reached live operation. The mean rating was 3.7 and the median 4.0 (n = 787 respondents), and 377 of the 787 answers (47.90%) were distressed. On canceled engagements, the quality loop was rated far lower than on active engagements in any stage (mean 2.3, n = 297 respondents).

All four core study measures were associated with the reliability of the joint quality assurance loop, and none of the 22 robustness checks overturned an association. A more reliable quality loop was associated with less cancellation (r = −0.61, 95% CI −0.70 to −0.50, n = 150 engagements). Among active engagements, customer satisfaction was higher where the loop was rated reliable (r = 0.60, 95% CI 0.45 to 0.71, n = 94 engagements). The engagement indicator mean, computed without the quality-loop indicator, was higher as well (r = 0.85, 95% CI 0.79 to 0.89, n = 150 engagements). Respondents who rated the loop reliable reported lower intent to leave (r = −0.66, cluster bootstrap 95% CI −0.72 to −0.59, n = 742 respondents).

Few written responses addressed the quality loop, 29, nearly all in comment on joint management, and 22 of the 29 were negative. They described quality checks that caught typing and formatting errors and missed the errors agents make, or that took place after the output was already live. Several teams performed their own checks outside the joint loop.

“Last month the assistant told an employee our parental leave was 12 weeks off the old policy pdf. Logs showed where it pulled it from. Nothing caught it before it went out.”Individual Contributor, CXM, Corporate and back-office functions

“the checks we run were built for human mistakes, transposed digits and such, they would never catch the way the agent gets a break wrong.”Individual Contributor, BPO, Trade, settlement and reconciliation

“i felt embarrassed. tool generate welcome pack with wrong leave policy, it reach new joiner, two days nobody decide who fix it, in end my team do.”Individual Contributor, CXM, Corporate and back-office functions

“QA method for agent decisions never got built before launch”Individual Contributor, BPO, Financial crime and client onboarding

Engagements whose written responses on the quality loop were more negative had a higher cancellation rate. Respondents who wrote more negatively about the loop reported higher intent to leave. Agents make different errors from people, a plausible answer drawn from an outdated document or a file placed under the wrong category. Where the checks did not look for those errors, the errors reached live operation.

3.4.5 Quality Measures Calibrated to Human Rather Than Agent Error

Respondents rated how well the quality measures both firms used captured the types of error agents actually make. The mean rating was 3.6 and the median 3.0 (n = 792 respondents), and 405 of the 792 answers (51.14%) were distressed. Respondents on canceled engagements rated the quality measures far lower than respondents on active engagements in any stage (mean 2.2, n = 300 respondents).

Every association of the calibration of quality measures to agent errors with the four core study measures held, in each of the 22 robustness checks. Engagements whose quality measures captured agent errors had a lower cancellation rate (r = −0.61, 95% CI −0.71 to −0.50, n = 150 engagements). Calibrated quality measures were associated with higher customer satisfaction among the active engagements (r = 0.63, 95% CI 0.49 to 0.74, n = 94 engagements). The engagement indicator mean, computed without the calibration indicator, was closely associated with them (r = 0.86, 95% CI 0.81 to 0.90, n = 150 engagements). Intent to leave was lower among respondents who rated the calibration higher (r = −0.66, cluster bootstrap 95% CI −0.72 to −0.60, n = 743 respondents).

Few written responses addressed the quality measures, 14, and 11 of the 14 were negative. Half described measures built to find the mistakes people make that would not register the mistakes agents make. They named an out-of-date policy answer, a funding position right in total and wrong for each merchant, and a mapping that failed without raising an exception.

“What neither firm measures is the kind of mistake the agent actually makes, which is a funding position that is right in total and wrong per merchant. Our quality sampling counts breaks cleared, consequently it scores that as a pass.”Engagement Manager, BPO, Trade, settlement and reconciliation

“The checking is not real. Approver just sign off the vendor onboarding queue, they do not open the record. And the QA sheet do not have any line for the mistake the bot actually make”Individual Contributor, CXM, Corporate and back-office functions

“The quality checks we agreed cover accuracy of the policy answer only. They do not cover the answer being out of date, which is what actually goes wrong. That gap is open since we start.”Engagement Manager, CXM, Corporate and back-office functions

“On my last posting it is a claims process and the QA is just sampling ten files a week. Here they decide things much faster, which I like. But the checks we run still look for the mistakes a person makes… the agent does not make those. It makes a different kind.”Individual Contributor, BPO, Payments, cards and disputes

“Data-transformation error rates were reported only for errors that triggered a system exception, so the silent hallucinated mappings that ran without throwing an error never appeared in the official error rate at all.”Individual Contributor, ITO, Technology engineering and operations

The written-sentiment tests reached significance on cancellation and intent to leave, on a base of 14 responses too small to be informative. The written evidence describes quality measures brought over unchanged from manual work. They passed agent errors that looked plausible, correct in total and wrong in detail, accurate but out of date, or wrong without an exception.

3.5 The Shift From Managing People to Managing Agents

Four indicators describe whether the joint management of the engagement had been adapted to work performed by agents. They ask whether each firm could see what the other firm’s agents had done and whether joint decisions kept pace with the agents. The remaining two ask whether the joint governance model fit work performed by agents and whether it was ready for the autonomy the respondent expected within two years. Each had about half of its answers in the distressed range, from 381 of 786 answers (48.47%) for the speed of joint decisions to 407 of 808 (50.37%) for readiness for future autonomy. Readiness was the lowest-rated indicator of the category. Each of the four was associated with every core study measure, and the 22 robustness checks left those associations intact.

The written evidence describes a joint model inherited from staffed delivery and left unchanged when agents took over many of its tasks. It was set to the cadence of a monthly forum or a periodic release, while the agents acted within minutes and changed between meetings. Authority to decide rested at a more senior level than the people who met each day, and the people in the room could agree on a course of action without being able to authorize it. Each firm’s record of what its agents had done stayed inside its own tools, and when a question arose the account was rebuilt by hand.

“the joint model is not failing, it is simply calibrated to a cadence that no longer exists. The governance forum meets monthly, which was entirely reasonable when a change meant a release … It is a design that predates the thing it governs, and no one on either side has yet been willing to say so out loud.”Portfolio Director, CXM, Corporate and back-office functions

“governance can’t match agent speed”Engagement Manager, ITO, Technology engineering and operations

Respondents placed the fault with the design rather than with the people in the structure. They described governance written for a team of people performing checks by hand and placed, unrevised, over agents.

3.5.1 Agent Actions Visible Only Inside Each Firm’s Own Tools

Respondents rated how fully each firm could see what the other firm’s agents had actually done when questions arose. The mean rating was 3.7 and the median 3.0 (n = 780 respondents), and 392 of the 780 answers (50.26%) were distressed. Visibility was weakest on canceled engagements, where the mean rating was 2.2 (n = 293 respondents), far lower than on active engagements in any stage.

The 22 robustness checks left intact each association between the visibility of agent actions across the two firms and the four core study measures. Fuller visibility was associated with a lower likelihood of cancellation (r = −0.62, 95% CI −0.71 to −0.51, n = 150 engagements). Customer satisfaction among active engagements was higher where visibility was rated fuller (r = 0.62, 95% CI 0.48 to 0.73, n = 94 engagements). The engagement indicator mean, computed without the visibility indicator, tracked the rating closely (r = 0.86, 95% CI 0.81 to 0.89, n = 150 engagements). Where respondents rated visibility higher, their intent to leave was lower (r = −0.66, cluster bootstrap 95% CI −0.72 to −0.60, n = 732 respondents).

The written responses addressing visibility numbered 66, from 64 respondents on 49 engagements, and 62 of the 66 were negative. Respondents described each firm’s agents as opaque to the other. When a question arose, an account of what an agent had done was rebuilt by hand, from statements, logs or a private spreadsheet.

“Their release manager and I keep a running note — what our agent touched, what their pipeline did, timestamps from both sides — because neither firm’s tooling shows the other’s actions and we got tired of arguing from memory on the Monday call. We update it by hand each evening … Formal risk log gets written afterwards, from that note.”Engagement Manager, ITO, Technology engineering and operations

“A release went out and half the config was not what we signed off, we asked their side what their automation had changed and nobody could show us, they only have the final state not the steps. We spend two days rebuilding it from our own notes.”Individual Contributor, ITO, Technology engineering and operations

“The vendor buries agent decision logs inside a proprietary trace format our audit team cannot query directly, so every review request becomes a manual export they control the timing and scope of.”Individual Contributor, ITO, Technology engineering and operations

“When a requirement comes back wrong we cannot see what their side’s tool did with the file, only what came out, so the first hour goes to reconstructing it. … Decision speed itself is acceptable. The visibility is not.”Engagement Manager, BPO, Credit, underwriting and suitability

“Observability logs get trimmed to the final agent decision only, so the intermediate steps that would show where a judgment call went wrong never make it into the audit trail the vendor shares with us.”Individual Contributor, CXM, Corporate and back-office functions

Cancellation was more common where the written responses on visibility were more negative. Respondents who wrote more negatively about visibility reported higher intent to leave. In answer to the question on how the vendor altered its reports, respondents described agent logs trimmed, aggregated or held in a format the financial institution’s auditors could not query (Part 5).

3.5.2 Joint Decision Speed Mismatched to Agent Speed

Respondents rated how closely the speed of joint decision making matched the speed at which the agents acted. The mean rating was 3.7 and the median 4.0 (n = 786 respondents), and 381 of the 786 answers (48.47%) were distressed. The speed of joint decisions received a mean rating of 2.2 on canceled engagements (n = 294 respondents), far lower than on active engagements in any stage.

Where joint decisions kept pace with the agents, all four core study measures were more favorable, and each association survived the 22 robustness checks. Cancellation was less common where joint decisions kept pace (r = −0.64, 95% CI −0.72 to −0.53, n = 150 engagements). Active engagements with faster joint decisions reported higher customer satisfaction (r = 0.62, 95% CI 0.47 to 0.73, n = 94 engagements). The engagement indicator mean, computed without the decision-speed indicator, moved closely with the rating (r = 0.86, 95% CI 0.81 to 0.89, n = 150 engagements). Respondents who saw joint decisions keep pace with the agents reported lower intent to leave (r = −0.67, cluster bootstrap 95% CI −0.73 to −0.61, n = 738 respondents).

Few written responses addressed the speed of joint decisions, 14, all in comment on joint management, and 13 of the 14 were negative. They described the same gap in different terms, an agent that acted in minutes or hours and a joint decision that took days or weeks, or never arrived. Several said the governance itself was sound but set to the cadence of manual work and periodic releases.

“the agent does in an hour what then waits nine days for two people to agree on. That gap is the whole problem”Individual Contributor, BPO, Credit, underwriting and suitability

“Decisions here take longer than the work does. That is the whole of it really … Here it goes into a shared log that both firms look at and neither one closes. I have watched an item sit in there for a month. Nobody is being difficult about it. Their side thinks we have it, we think they have it.”Individual Contributor, ITO, Technology engineering and operations

“We escalated a flagged exposure last month and it took nearly two weeks to get a joint call on it… by then the account had already moved two grades. That is the thing I want the reader to understand. The agents work in minutes and the two firms still discuss about it for a fortnight.”Individual Contributor, KPO, Credit, underwriting and suitability

“We meet Tuesday to agree access changes, the agent works through joiner queue every hour. So a bad role assignment can sit live for six days before anyone talks about it. And nobody has said what we do when that is two hundred accounts instead of two.”Individual Contributor, ITO, Technology engineering and operations

“it asks how quickly the two firms decide together, and for that there must be a decision somewhere, i have not seen one yet. we sent up the question of which flagged events are safe for the RMs to act on, that was month two, it is still sitting there and the project is closed now.”Individual Contributor, KPO, Client servicing and communications

“When the agent picks wrong supplier record, nobody is switching it off. I raise, then two firms discuss about it and by then hundred more lines are done”Individual Contributor, BPO, Corporate and back-office functions

Written sentiment on the speed of joint decisions showed no association with the study measures, on a base too small for that test to be informative. The written responses describe joint governance designed for work that changed at the pace of releases, applied to agents that changed at the pace of the queue. In the interval, the agent continued on the old basis, and the decision passed to whoever was handling the queue.

3.5.3 Joint Governance Model Designed for Work Performed by People

Respondents rated how well the joint governance model fit work performed by agents rather than work performed by people. The mean rating was 3.6 and the median 4.0 (n = 803 respondents), and 395 of the 803 answers (49.19%) were distressed. On canceled engagements the mean rating for the fit of the model was 2.2 (n = 301 respondents), far lower than respondents gave on active engagements in any stage.

Every association of the fit between the joint governance model and agent work with the four core study measures held, in each of the 22 robustness checks. A model fitted to agent work was associated with a lower cancellation rate (r = −0.64, 95% CI −0.72 to −0.53, n = 150 engagements). Among active engagements, customer satisfaction was higher where the model fit agent work (r = 0.60, 95% CI 0.45 to 0.72, n = 94 engagements). The fit of the model was closely associated with the engagement indicator mean computed without it (r = 0.85, 95% CI 0.80 to 0.89, n = 150 engagements). Intent to leave was lower among respondents who judged the model fit for agents (r = −0.66, cluster bootstrap 95% CI −0.72 to −0.60, n = 758 respondents).

Few written responses addressed the fit of the governance model, 10, nearly all in comment on joint management, and 9 of the 10 were negative. The most common account was a governance model drawn up for people performing the tasks by hand and left in place after agents took over much of it. Respondents described decisions that neither firm’s people in the room could authorize and agent changes that both firms learned of afterward.

“The governance we have was written for a team of people doing checks by hand, and now most of the checking is not done by hand at all. So the shape of it is wrong from the start.”Individual Contributor, BPO, Corporate and back-office functions

“There was a call on a Thursday in February where their reconciliation manager and I sat for forty minutes, agreed exactly what should happen with the unmatched cash items, and both of us knew that neither of us could authorise it … What we were handed was a governance model written for forty people doing manual matching, and then agents were put underneath it and nobody rewrote a line of it.”Engagement Manager, BPO, Trade, settlement and reconciliation

“when it was people doing this, my lead could just decide. now there is a agent doing the matching and nobody want to be the one who says it is allowed to post something. so it waits. i dont think anyone was lazy. it is just nobody was given the authority.”Individual Contributor, BPO, Trade, settlement and reconciliation

“The incident that sticks with me was a retraining push on the resolution model that went in on a Thursday and only came up in the following weeks steering call, by which point the desk had already seen a run of odd closures and worked around them. Nobody did anything wrong, procedurally, which is sort of the point. … What we dont have is a shared picture of what changed and who owns it when the agent gets something wrong”Portfolio Director, ITO, Technology engineering and operations

Written sentiment on the fit of the governance model was not associated with the study measures, and the base of 10 responses was too small for that test to be informative. The written evidence describes a governance model inherited from manual work and applied unchanged to agents, with the authority to decide placed at a level senior to the people who met each day. While a decision waited, the agent continued on its existing basis and the open breaks and exceptions waited with it.

3.5.4 Readiness of the Joint Model for Expected Agent Autonomy

Respondents rated how ready the joint management model was for the level of agent autonomy they expected in the engagement within two years. The mean rating was 3.60, the lowest of the 20 indicators of co-management, and the median 3.0 (n = 808 respondents). Distressed ratings made up 407 of the 808 answers (50.37%). Readiness was rated lowest on canceled engagements (mean 2.2, n = 306 respondents), far lower than on active engagements in any stage.

Readiness of the joint model for the autonomy expected within two years was associated with all four core study measures, and each association held under the 22 robustness checks. Engagements whose model was rated ready had a lower cancellation rate (r = −0.63, 95% CI −0.72 to −0.52, n = 150 engagements). Customer satisfaction among active engagements was higher where readiness was rated higher (r = 0.63, 95% CI 0.49 to 0.74, n = 94 engagements). The engagement indicator mean computed without the readiness indicator was higher as well (r = 0.87, 95% CI 0.82 to 0.90, n = 150 engagements). Respondents who judged the model ready reported lower intent to leave (r = −0.66, cluster bootstrap 95% CI −0.72 to −0.60, n = 761 respondents).

Three written responses addressed the indicator, a base of anecdote strength. Two were negative and one neutral. The two negative responses described joint structures that had already failed to keep pace with the agents. Their examples were two incomplete issue logs kept separately by each firm (section 3.2.2) and a single review point that found agent errors after a letter had been sent.

“The review point sits at the end only. If the agent codes a dispute reason wrongly it is found after the letter is already sent, not before. For the autonomy expected in two years this model will not hold.”Individual Contributor, BPO, Payments, cards and disputes

The two respondents drew their conclusion about the future from present conditions. The ratings measure how widely the view was held, and half of the answers placed the model’s readiness for greater autonomy in the distressed range. A joint model that could not keep pace with the agents of the study period was, in the delivery teams’ judgment, not prepared for agents with more autonomy.

Vendor capability was rated almost as low as the financial institution’s own readiness. Its 20 indicators had a mean of 3.67 on the 10-point scale across the 150 engagements, and 7,687 of their 15,943 answers (48.22%) were distressed. The respondents rating the vendor were its own employees. They rated their employer’s leadership, commercial model, management processes, knowledge, production experience, talent, training, playbook, quality assurance and platforms, and placed each one in the distressed range in about half of the answers.

Mean ratings in Table 10 are computed across respondents. The category figure of 3.67 is computed across engagements. Section 4.8 reports one further question on the vendor, its preparedness compared with five years earlier. The survey placed that question in its section on the respondent’s relationship with the employer, and it is not one of the category’s 20 indicators.

The written evidence describes vendors that sold agentic delivery ahead of the experience needed to run it. Few in the vendor’s organization had taken agents beyond a pilot. The playbook was written for staffed delivery, the training was general, and the people with agentic experience were few and shared across accounts. The experience the vendors lacked was acquired on live engagements, on work the financial institution was paying for and by the individual members of the delivery team. Meanwhile the vendor’s commercial and management models still counted heads and hours.

“We are sold into a capital-markets client as though the firm has done investor communications with agents before, and it has not. There is one person in the whole delivery organisation who understands what a quiet period means, and she is not on this engagement … Internally the machinery has not changed at all, same weekly status pack, same utilisation questions, none of which describe what my team actually does now.”Engagement Manager, CXM, Client servicing and communications

“What I would want noted above everything else is that our leadership is good at winning this kind of work and we are still learning how to run it. Nobody at home has answered who pays when an agent gets an onboarding step wrong.”Engagement Manager, CXM, Corporate and back-office functions

“They sold this in about eight weeks and then handed us a delivery model built for staffed work – that gap is most of the story … What is missing is anyone senior owning the space between a signed pilot and a working operating rhythm.”Engagement Manager, KPO, Corporate and back-office functions

The survey invited comment on the employer, and 530 of the 850 respondents (62.35%) wrote one. Many separated the people around them, whom they described as capable and committed, from the firm that employed them. They described that firm as having sold work it had not yet learned to deliver.

“My own lead is very good, that is not what these numbers say. The gap is higher up, whoever writes the training deck has not run an agent on a live pipeline, we get slides about concepts and then work the rest out on the platform ourselves. So the low marks are about the company, not my team”Individual Contributor, ITO, Technology engineering and operations

The 20 indicators of the category lay in a narrow band. Their correlations with cancellation ranged from r = −0.60 to r = −0.56 and with the engagement indicator mean from r = 0.85 to r = 0.88 (n = 150 engagements each). The vendor indicators cannot be ranked against one another by their associations, and sections 4.1 to 4.8 do not attempt to rank them. They report what each indicator’s written evidence adds.

4.1 Leadership and the Commercial Model

Three indicators describe the vendor at the level of the firm. They cover the effectiveness of its top leadership and the transformation of its commercial model as agents reduce the billable human effort it sells. The third is its ability to price agentic work profitably once hours no longer measure value. The effectiveness of leadership was distressed in 381 of 792 answers (48.11%), the transformation of the commercial model in 386 of 812 (47.54%) and the pricing of agentic work in 383 of 795 (48.18%). All three were associated with each core study measure, and no association failed any of the 22 robustness checks.

Respondents described leadership as distance. Immediate managers were capable and received nothing from senior leadership. Questions of accountability for an agent’s error went unanswered, and the senior people meant to own the engagement had not run agentic work before.

The commercial model was the subject of a large and nearly uniform body of written evidence. Its transformation was addressed in 145 responses, 132 negative, and the pricing of agentic work in 137, all but one negative. Respondents described engagements still priced and measured on seats and hours while the agents reduced the hours. Each gain by an agent lowered the invoice, and the delivery team was asked to explain the shortfall to its own management. The commercial model rewarded headcount on engagements built to reduce it.

4.1.1 Effectiveness of Vendor Top Leadership

Respondents rated the overall effectiveness of their employer’s top leadership team. The mean rating was 3.6 and the median 4.0 (n = 792 respondents), and 381 of the 792 answers (48.11%) were distressed. The mean rating of leadership on canceled engagements, 2.4 (n = 304 respondents), was far lower than on active engagements in any stage.

The effectiveness of the vendor’s top leadership was associated with all four core study measures, and each association held under the 22 robustness checks. Engagements whose respondents rated leadership more effective had a lower cancellation rate (r = −0.58, 95% CI −0.68 to −0.46, n = 150 engagements). Customer satisfaction among active engagements was higher where leadership was rated more effective (r = 0.71, 95% CI 0.59 to 0.79, n = 94 engagements). The engagement indicator mean, computed without the leadership indicator, was higher too (r = 0.86, 95% CI 0.81 to 0.89, n = 150 engagements). Respondents who rated their leadership higher reported lower intent to leave (r = −0.60, cluster bootstrap 95% CI −0.67 to −0.52, n = 741 respondents).

The written responses addressing leadership numbered 30, and 23 were negative. Respondents placed the weakness at levels senior to their immediate managers. They described the absence of anyone who owned the engagement end to end or who would answer for an agent’s error, and senior people who had not run agentic work before. Accounts of training were mixed.

“Training was actually useful, we got proper sessions on the matching Agent and i could use it directly on custodian breaks… that part improved. Top management is different matter. nobody owns it end to end, and when exception queue builds up nobody is watching.”Individual Contributor, BPO, Trade, settlement and reconciliation

“What no question here asks is who I am supposed to ring at two in the morning when the agent has miscalculated a run and the file is already sitting with their bank. There is a playbook for a staffed process, there is nothing for this, I asked twice and got a name that turned out to be somebody who left … it is that nobody above has thought past winning the work.”Engagement Manager, BPO, Corporate and back-office functions

“What hasnt changed is anyone above my manager. Nobody at the top has run one of these to the end, and it shows the minute a client asks a hard question.”Individual Contributor, ITO, Regulatory, audit, tax and legal

“My reservation sits higher up. Nobody has told me who signs the letter if one of these responses goes out wrong.”Portfolio Director, CXM, Corporate and back-office functions

“and leadership send mails about ai every month but nothing reach the floor, no training, no time set aside. we are learning the policy assistant on live queries and hoping we catch the wrong ones”Individual Contributor, CXM, Corporate and back-office functions

Cancellation was more common on engagements with a negative written response on leadership, and the engagement mean of written sentiment was associated with cancellation, consistent with the rating. The written responses describe leadership as distance and as the absence of an owner.

4.1.2 Transformation of the Vendor Commercial Model

Respondents rated how well their employer was transforming its commercial model as agentic AI reduced the billable human effort it sold. The mean rating was 3.7 and the median 4.0 (n = 812 respondents), and 386 of the 812 answers (47.54%) were distressed. The transformation of the commercial model was weakest on canceled engagements, where the mean rating was 2.4 (n = 306 respondents), far lower than on active engagements in any stage.

All four core study measures were associated with the transformation of the vendor’s commercial model, and none of the 22 robustness checks overturned an association. Cancellation was less common where the transformation was rated further advanced (r = −0.60, 95% CI −0.69 to −0.48, n = 150 engagements). Among active engagements, customer satisfaction was higher where the commercial model had been transformed (r = 0.68, 95% CI 0.55 to 0.77, n = 94 engagements). The engagement indicator mean computed without the commercial-model indicator moved closely with the rating (r = 0.87, 95% CI 0.82 to 0.90, n = 150 engagements). Respondents who rated the transformation higher reported lower intent to leave (r = −0.62, cluster bootstrap 95% CI −0.68 to −0.54, n = 762 respondents).

The written responses addressing the commercial model numbered 145, and 132 were negative. Every response to the question on the hardest challenges that addressed the indicator named the tension between agentic work and the vendor’s revenue from headcount. The longer comments on the employer described engagements still priced and measured on heads, on which each gain by the agent lowered the invoice.

“When the agents absorb most of the query volume my headcount goes down and the invoice goes down with it, and nobody has said what replaces it. The answer each time is that a new model is coming. Until it arrives we are, in effect, penalised for the very thing we were asked to build.”Engagement Manager, CXM, Client servicing and communications

“The heavy work is the first six weeks of teaching the routing rules, and after that the volume costs us almost nothing, but we are still billing as though a person sits behind every ticket. Nobody above me has told us how the next phase is meant to be priced … ”Engagement Manager, CXM, Technology engineering and operations

“vendor’s own FTE model fights the agent’s success”Individual Contributor, CXM, Technology engineering and operations

“Vendor defended FTE margins over fixing the black box”Engagement Manager, BPO, Trade, settlement and reconciliation

Written sentiment on the commercial model was not associated with the study measures, and the associations rest on the rating. The written evidence describes a commercial model that rewards headcount on an engagement built to reduce it, with internal targets that count heads and invoices that decline as the agents succeed.

4.1.3 Pricing of Agentic Work When Hours No Longer Measure Value

Respondents rated their employer’s ability to price agentic work profitably when billable hours no longer measured the value delivered. The mean rating was 3.6 and the median 4.0 (n = 795 respondents), and 383 of the 795 answers (48.18%) were distressed. Pricing received a mean rating of 2.5 on canceled engagements (n = 303 respondents), far lower than on active engagements in any stage.

The vendor’s ability to price agentic work was associated with cancellation, customer satisfaction, the engagement indicator mean and intent to leave, and each association withstood the 22 robustness checks. Engagements whose vendor priced agentic work better had a lower cancellation rate (r = −0.56, 95% CI −0.66 to −0.44, n = 150 engagements). Customer satisfaction among active engagements was higher where pricing was rated better (r = 0.72, 95% CI 0.60 to 0.80, n = 94 engagements). Pricing was also closely associated with the engagement indicator mean computed without it (r = 0.87, 95% CI 0.82 to 0.90, n = 150 engagements). Intent to leave was lower among respondents who rated pricing higher (r = −0.60, cluster bootstrap 95% CI −0.67 to −0.53, n = 748 respondents).

The written responses addressing pricing numbered 137, and all but one were negative. Respondents who named pricing among their hardest challenges described the price of agentic work as a guess, a continuing argument or a figure that never matched the effort. Longer responses described engagements still priced on heads, and reports to the financial institution that did not show the resulting loss of margin.

“Put it plainly, we were sold as an agentic engagement and then priced and staffed like an ordinary ITO ramp. Nobody above me could explain what we bill for once the agent clears an access request itself.”Engagement Manager, ITO, Technology engineering and operations

“We priced this one on a blended rate against estimated human effort, and then the entire premise of the engagement was that the human effort would fall, which meant every week we succeeded technically we lost money and every week we struggled we looked incompetent.”Portfolio Director, ITO, Regulatory, audit, tax and legal

“The vendor priced and reported deliverable turnaround in hours claimed by the agent, without a separate line for the senior banker hours spent re-deriving key assumptions the agent could not justify, masking the true cost of the engagement.”Individual Contributor, KPO, Corporate and back-office functions

“We are sold as tax research capability by people who have never priced work that is not measured in hours, and effect on me personally is that I defend a delivery model I did not design and do not believe in.”Engagement Manager, KPO, Regulatory, audit, tax and legal

“Pricing conversations stall because nobody wants to admit FTE hours are dropping”Engagement Manager, CXM, Technology engineering and operations

Written sentiment on pricing was associated with the outcomes in its own right. A more negative engagement mean of written sentiment was associated with cancellation, and a more negative respondent mean with higher intent to leave. Pricing built for human effort was applied to work in which human effort declined, and the gap was absorbed by the delivery team or left out of the reporting.

4.2 Management Processes and End-to-End Engagement Management

Four indicators describe how the vendor managed agentic engagements. They cover whether its internal management processes had been redesigned for agentic delivery and its ability to manage agentic engagements end to end. The other two cover its ability to manage them for global financial institutions in particular and its overall qualification to deliver elite agentic outsourcing to financial institutions. The distressed share ranged from 364 of 787 answers (46.25%) for end-to-end management to 388 of 799 (48.56%) for management processes. Every indicator of the four was associated with each core study measure, and each association held through the 22 robustness checks.

The written evidence on these indicators is modest in volume and consistent. The vendor’s internal processes remained those of staffed delivery, with rosters, headcount reports, utilization targets and checklists written for manual work. The tasks those processes were meant to govern were performed increasingly by agents. Engagements were started without the preparation to run them and run without a single owner. Respondents described a firm that had sold agentic work before anyone in it had delivered such work in production, and end-to-end management was learned on the engagements themselves. No respondent wrote about the vendor’s capability with global financial institutions in particular.

4.2.1 Vendor Management Processes Left Unchanged From Staffed Delivery

Respondents rated how well their employer’s internal management processes had been redesigned for agentic AI delivery rather than traditional staffed delivery. The mean rating was 3.7 and the median 4.0 (n = 799 respondents), and 388 of the 799 answers (48.56%) were distressed. On canceled engagements, the redesign of management processes was rated far lower than on active engagements in any stage (mean 2.5, n = 295 respondents).

Each of the four core study measures was associated with how far the vendor had redesigned its internal management processes, and the 22 robustness checks upheld every association. Engagements whose vendor had redesigned its processes further had a lower likelihood of cancellation (r = −0.57, 95% CI −0.67 to −0.45, n = 150 engagements). Customer satisfaction among active engagements was higher where the processes had been redesigned (r = 0.68, 95% CI 0.55 to 0.78, n = 94 engagements). The engagement indicator mean computed without the process indicator was closely associated with the redesign (r = 0.85, 95% CI 0.80 to 0.89, n = 150 engagements). Respondents who rated the redesign higher reported lower intent to leave (r = −0.57, cluster bootstrap 95% CI −0.64 to −0.49, n = 749 respondents).

Few written responses addressed the vendor’s management processes, 25, nearly all in comment on the employer, and 23 of the 25 were negative. Their accounts named the practices of staffed delivery still in use, the shift roster, the monthly headcount report, the utilization target and the checklist written for manual work.

“My utilization target counts heads on the account, not tickets the agent closed, so every hour I spend tuning intent routing shows up as idle time on my own report. Nobody above me has said that out loud yet.”Engagement Manager, CXM, Corporate and back-office functions

“our internal process is still built for staffed delivery, headcount reports, utilization, all of it, none of that tells anybody how model is performing. Third, stop putting people on production work whose only experience is pilot.”Engagement Manager, CXM, Regulatory, audit, tax and legal

“Our internal process still assumes headcount. Every week I still fill the same shift Tracker even though the Agent is doing the routing… What needs to change is the reporting we do, it should measure tickets closed correctly, not hours logged.”Individual Contributor, CXM, Technology engineering and operations

“My manager is good and I dont have complaints there. What the numbers dont show is that were still run like a body shop – hours booked, tickets closed – when half the work now is checking what the tool did.”Individual Contributor, ITO, Client servicing and communications

“Nobody above me has redesigned the delivery playbook, they took the old seat-based model and put agents where the seats used to be, and the reporting rolls up exactly as it always did.”Engagement Manager, CXM, Corporate and back-office functions

Written sentiment on the management processes was not associated with the study measures, on a base of 25 responses. The written evidence describes management processes that count the people on an account and the hours they log, applied to an engagement whose tasks were performed increasingly by agents. Respondents bridged the difference themselves, by adapting the old frameworks or by working outside them.

4.2.2 End-to-End Management of Agentic Engagements

Respondents rated their employer’s ability to manage agentic AI outsourcing engagements end to end. The mean rating was 3.7 and the median 4.0 (n = 787 respondents), and 364 of the 787 answers (46.25%) were distressed, the lowest distressed share of the 20 indicators of vendor capability. Respondents on canceled engagements gave end-to-end management a mean rating of 2.5 (n = 294 respondents), far lower than respondents on active engagements in any stage.

The 22 robustness checks left intact each association between end-to-end management of agentic engagements and the four core study measures. Cancellation was less common on engagements with stronger end-to-end management (r = −0.56, 95% CI −0.66 to −0.44, n = 150 engagements). Customer satisfaction among active engagements was higher where end-to-end management was rated stronger (r = 0.72, 95% CI 0.61 to 0.81, n = 94 engagements). The engagement indicator mean computed without the end-to-end indicator was closely associated with it (r = 0.86, 95% CI 0.81 to 0.90, n = 150 engagements). Respondents who rated end-to-end management higher reported lower intent to leave (r = −0.61, cluster bootstrap 95% CI −0.68 to −0.53, n = 738 respondents).

Few written responses addressed end-to-end management, 17, nearly all in comment on the employer, and the evidence is divided, with 11 negative and 6 positive. The negative responses described teams placed on engagements without the preparation to run them, and the absence of an owner of the engagement from end to end. The positive responses described engagements that worked well apart from a named gap.

“We were placed in front of a regulatory client with nobody who had run agents through a live filing cycle, and when it broke, ownership landed on my team and stayed there.”Engagement Manager, KPO, Regulatory, audit, tax and legal

“Nobody put us on this account with a plan and not one question here reaches that. My lead did everything right, she was the only one who answered anything.”Individual Contributor, CXM, Technology engineering and operations

“My employer was given three weeks to stand up the screening queue, kindly note that is not enough for KYC. The people tried. The preparation was not there”Individual Contributor, BPO, Financial crime and client onboarding

“What has to change is above that. Somebody needs to own the whole thing end to end insted of four leads each owning a piece.”Individual Contributor, ITO, Regulatory, audit, tax and legal

“End to end we run engagement well, I stand by that. But I have asked twice up the line who carries the cost if it goes wrong, and both times I got a workshop invitation.”Engagement Manager, ITO, Technology engineering and operations

Despite the small base, a more negative written score was associated with cancellation and with higher intent to leave, as the rating was.

4.2.3 Management of Agentic Engagements for Global Financial Institutions

Respondents rated their employer’s ability to manage agentic AI outsourcing engagements for global financial institutions in particular. The mean rating was 3.6 and the median 4.0 (n = 779 respondents), and 381 of the 779 answers (48.91%) were distressed. This ability was weakest on canceled engagements, where the mean rating was 2.4 (n = 292 respondents), far lower than on active engagements in any stage.

Each association between the management of engagements for global financial institutions and the four core study measures held in all 22 robustness checks. Engagements rated more capable on this measure had a lower cancellation rate (r = −0.56, 95% CI −0.66 to −0.44, n = 150 engagements). Among active engagements, customer satisfaction was higher where this ability was rated higher (r = 0.70, 95% CI 0.58 to 0.79, n = 94 engagements). The engagement indicator mean, computed without this indicator, was higher too (r = 0.86, 95% CI 0.81 to 0.90, n = 150 engagements). Intent to leave was lower among respondents who rated this ability higher (r = −0.59, cluster bootstrap 95% CI −0.66 to −0.50, n = 732 respondents).

No written response addressed the vendor’s capability with global financial institutions in particular. The evidence for the indicator is the rating alone, and the reasons behind it are not recorded. The rating stood within the narrow band of the category, with about half of the answers in the distressed range. The written evidence on the vendor’s management of agentic engagements in general is reported in sections 4.2.1, 4.2.2 and 4.2.4.

4.2.4 Vendor Qualification for Elite Agentic Outsourcing to Financial Institutions

Respondents rated their employer’s overall qualification to deliver elite agentic AI outsourcing services to financial institutions. The mean rating was 3.6 and the median 4.0 (n = 803 respondents), and 385 of the 803 answers (47.95%) were distressed. The mean rating of the vendor’s qualification on canceled engagements, 2.5 (n = 306 respondents), was far lower than on active engagements in any stage.

The vendor’s overall qualification for elite agentic outsourcing showed an association with each of the four core study measures that survived all 22 robustness checks. Cancellation was less common where respondents rated their employer better qualified (r = −0.56, 95% CI −0.66 to −0.44, n = 150 engagements). Active engagements whose vendor was rated better qualified reported higher customer satisfaction (r = 0.72, 95% CI 0.61 to 0.81, n = 94 engagements). The engagement indicator mean, computed without the qualification indicator, was closely associated with the rating (r = 0.86, 95% CI 0.81 to 0.90, n = 150 engagements). Respondents who judged their employer better qualified reported lower intent to leave (r = −0.58, cluster bootstrap 95% CI −0.65 to −0.50, n = 757 respondents).

Few written responses addressed the vendor’s qualification, 28, nearly all in comment on the employer, and 17 of the 28 were negative. The negative responses described a firm that had sold agentic delivery before anyone in it had run such work, with teams learning on live engagements. Several of the positive responses named a weakness alongside the rating.

“nobody in our firm has run early-warning work with agents before, so we are learning on a live portfolio, and the two seniors who do understand it are shared across other work. I do not think this is dishonesty, it is a firm selling something it has not built yet, which is a different fault.”Engagement Manager, KPO, Credit, underwriting and suitability

“The problem is nobody above the trainer has done this before either. When the agent closed a batch of alerts wrong in March, my lead asked me what the process is, and i asked him the same thing. There is no process. So we made one up on the call and wrote it after.”Individual Contributor, ITO, Financial crime and client onboarding

“We were staffed for a document backlog, not for standing up an agent platform, and please understand that nobody at our end ever pretended otherwise. The people they gave me worked hard. The company simply sold what it has yet to build.”Engagement Manager, BPO, Corporate and back-office functions

“there is no box for saying the people here are fine and the firm above them is not.”Individual Contributor, KPO, Corporate and back-office functions

A more negative written score was associated with cancellation at engagement level and with higher intent to leave. The written evidence describes capable delivery teams inside a firm that sold agentic work ahead of the experience to deliver it. The experience was acquired on the engagements themselves.

4.3 Knowledge of the Financial Industry and of Agentic AI

Two indicators describe what the vendor knew, namely the depth of its knowledge of the financial industry and the depth of its knowledge of agentic AI. Knowledge of the industry was distressed in 392 of 802 answers (48.88%). Knowledge of agentic AI was distressed in 416 of 807 (51.55%), and its mean rating of 3.58 (n = 807 respondents) was the lowest of the 20 indicators of vendor capability. Both were associated with each core study measure, and the 22 robustness checks overturned none of the associations.

The written evidence divides along the line between the two forms of knowledge the engagement needed. Some respondents described a firm that knew the financial business and could not yet run agents. Others described a firm that knew the agents and had placed staff on financial work without financial knowledge, who then learned the business from the financial institution’s own documents. Few described a firm that held both.

Where either form of knowledge existed, it rested with individual members of the delivery team and with a few people across the firm. Respondents described a management line senior to them that could not answer questions about the agents, and teams that learned the agent side on live work and from one another.

“our depth in market abuse work sits in about four people company-wide, and I get them for a week at a time when I ask nicely.”Engagement Manager, ITO, Regulatory, audit, tax and legal

4.3.1 Depth of Vendor Knowledge of the Financial Industry

Respondents rated the depth of their employer’s knowledge of the financial industry. The mean rating was 3.6 and the median 4.0 (n = 802 respondents), and 392 of the 802 answers (48.88%) were distressed. On canceled engagements, the vendor’s industry knowledge was rated far lower than on active engagements in any stage (mean 2.4, n = 302 respondents).

All four core study measures were associated with the depth of the vendor’s knowledge of the financial industry, and none of the 22 robustness checks overturned an association. Deeper industry knowledge was associated with a lower cancellation rate (r = −0.56, 95% CI −0.66 to −0.44, n = 150 engagements). Customer satisfaction among active engagements was higher where the vendor’s industry knowledge was rated deeper (r = 0.71, 95% CI 0.59 to 0.79, n = 94 engagements). The engagement indicator mean, computed without the industry-knowledge indicator, was higher as well (r = 0.85, 95% CI 0.80 to 0.89, n = 150 engagements). Respondents who rated their employer’s industry knowledge higher reported lower intent to leave (r = −0.60, cluster bootstrap 95% CI −0.68 to −0.52, n = 755 respondents).

The written responses addressing the vendor’s industry knowledge numbered 64, all in comment on the employer, with 45 negative and 18 positive. The negative responses described delivery staff who did not know the financial business they served and learned it on live work, in some accounts from the financial institution’s own documents. The positive responses held that the firm’s domain knowledge was sound and placed the gap in running agents.

“In week three I asked for two analysts who had done entitlement reviews before. What arrived were two people who had completed the internal AI course and had never seen a bank access review. I spent a fortnight teaching segregation of duties instead of running the engagement.”Engagement Manager, ITO, Technology engineering and operations

“We have the audit domain people, that has never been the gap here … Where we are thin is running the same agents past a pilot. This was the first continuous auditing build any of us had taken beyond a demo environment, and it showed in how long it took us to work out what to do when a control test failed for the wrong reason.”Engagement Manager, KPO, Regulatory, audit, tax and legal

“One thing plainly, the people my employer puts onto a deposits account do not know deposits. For that reason I spend the first month of every intake explaining what a hold is, what a dormancy flag is.”Individual Contributor, BPO, Payments, cards and disputes

“above him nobody has worked in a bank, because of that the training deck they gave us was written for a call centre, not for payroll cases”Individual Contributor, BPO, Corporate and back-office functions

“we were given the tool and a two day session, nothing about private capital in it. nobody on our side could explain what a management fee offset is, i had to ask the client team through my lead”Individual Contributor, BPO, Trade, settlement and reconciliation

A more negative written score was associated with higher intent to leave, and written sentiment was not associated with the engagement-level measures. The written evidence divides along the line between knowledge of the business and knowledge of the agents, and few respondents described a firm that held both.

4.3.2 Depth of Vendor Knowledge of Agentic AI

Respondents rated the depth of their employer’s knowledge of agentic AI. The mean rating was 3.58, the lowest of the 20 indicators of vendor capability, and the median 3.0 (n = 807 respondents). Distressed ratings made up 416 of the 807 answers (51.55%). The vendor’s agentic knowledge received a mean rating of 2.4 on canceled engagements (n = 308 respondents), far lower than on active engagements in any stage.

The depth of the vendor’s knowledge of agentic AI was associated with all four core study measures, and each association held under the 22 robustness checks. Engagements whose vendor was rated more knowledgeable about agents had a lower cancellation rate (r = −0.57, 95% CI −0.67 to −0.45, n = 150 engagements). Among active engagements, customer satisfaction was higher where agentic knowledge was rated deeper (r = 0.68, 95% CI 0.56 to 0.78, n = 94 engagements). The engagement indicator mean, computed without the agentic-knowledge indicator, was higher where that knowledge was rated deeper (r = 0.86, 95% CI 0.81 to 0.90, n = 150 engagements). Intent to leave was lower where respondents rated the vendor’s agentic knowledge higher (r = −0.59, cluster bootstrap 95% CI −0.66 to −0.51, n = 758 respondents).

Few written responses addressed the vendor’s agentic knowledge, 26, all in comment on the employer, and 18 of the 26 were negative. The negative responses placed the shortfall at levels senior to the delivery team. Teams learned the agent side on live work after brief training, and their own management line could not answer their questions.

“The problem is nobody above us really know these agents, so when we get stuck we solve it ourselves from Google. It make my team tired, we are learning on live tickets and there is no one to ask”Individual Contributor, ITO, Technology engineering and operations

“They put us through the training and it was decent, I’ll say that, better than the last place. But nobody above me actually knows this stuff to the depth the work needs. We’re figuring out the agent side on the desk.”Individual Contributor, BPO, Trade, settlement and reconciliation

“Nobody from my company could explain what the agent had done. i asked my lead, he asked somebody else, then it came back to me.”Individual Contributor, ITO, Financial crime and client onboarding

“They taught me the banking side properly. The agent side we are all guessing”Individual Contributor, ITO, Technology engineering and operations

“The knowledge sits in perhaps a dozen individuals, which is why I marked the capability question higher than the playbook one; what we lack is not understanding but the written route from a working prototype to a governed sanctions process, and that gap is real.”Portfolio Director, KPO, Financial crime and client onboarding

Written sentiment on agentic knowledge was not associated with the study measures, on a base of 26 responses. The written evidence places the firm’s agentic knowledge in individual delivery staff and in a few people across the firm, with little of it in the management line that directed the engagement.

4.4 Live Production Experience and the Operation of Large Numbers of Agents

Two indicators describe whether the vendor could run agents in live operation. They are the depth of its experience running agentic systems in production beyond pilots, and its ability to operate large numbers of agents in production, including monitoring and exception handling. Production experience was distressed in 383 of 794 answers (48.24%) and the operation of large numbers of agents in 379 of 798 (47.49%). Each of the two was associated with every core study measure, and no robustness check overturned an association.

Production experience was the subject of 54 written responses, and 51 were negative. Respondents described a firm whose agentic work had stopped at pilots and demonstrations and was then placed on live engagements for financial institutions, where the experience was acquired on the financial institution’s work. On the operation of large numbers of agents, respondents described agents in production with no operations function around them. No person on the vendor’s side was assigned to watch the agents while they acted, and failures were found the next morning.

“Not one person at my firm has done this before. Nobody. They sold agent desk support to a trading bank and then went looking for who knows how to run it, and the answer was nobody, so they put us on it and told us learn. That is not malice. That is a firm out of its depth that will not say so out loud…”Individual Contributor, ITO, Technology engineering and operations

“But running hundreds of agents at once is different from running ten, and our own alerting for the agents themselves is thin. We watch the client estate closely and the bots not at all.”Individual Contributor, ITO, Technology engineering and operations

4.4.1 Vendor Experience Running Agents Beyond Pilots

Respondents rated the depth of their employer’s experience running agentic AI systems in live production, beyond pilots. The mean rating was 3.6 and the median 4.0 (n = 794 respondents), and 383 of the 794 answers (48.24%) were distressed. Respondents on canceled engagements rated their employer’s production experience far lower than respondents on active engagements in any stage (mean 2.4, n = 300 respondents).

The depth of the vendor’s experience running agents beyond pilots was associated with cancellation, customer satisfaction, the engagement indicator mean and intent to leave, and every association survived the 22 robustness checks. Engagements whose vendor had deeper production experience were less likely to be canceled (r = −0.56, 95% CI −0.66 to −0.44, n = 150 engagements). Customer satisfaction among active engagements was higher where production experience was rated deeper (r = 0.73, 95% CI 0.62 to 0.81, n = 94 engagements). So was the engagement indicator mean computed without the production-experience indicator (r = 0.87, 95% CI 0.82 to 0.90, n = 150 engagements). Respondents who rated production experience deeper reported lower intent to leave (r = −0.60, cluster bootstrap 95% CI −0.67 to −0.52, n = 747 respondents).

The written responses addressing production experience numbered 54, and 51 were negative, 42 with the strongest negative score. Every negative comment on the employer said that the firm, the team or the people senior to them had little or no experience running agents in live production. Respondents described learning on the financial institution’s live work after brief training, and work sold as though the firm had done it before.

“Nobody in my company has done this before. Not in real production. We are told the model will triage the alerts and we only look at what it is not sure about, but on my shift i am running maybe half of them again by hand because i do not trust what it closed… a closed alert on card fraud is not same as closed ticket for a printer.”Individual Contributor, ITO, Technology engineering and operations

“when we started, nobody in my team had seen an agent run a case queue before, we learnt on the job. What must change practically is two things. First, someone senior with production experience should sit with the team for the first month, not a deck at kickoff.”Individual Contributor, KPO, Financial crime and client onboarding

“When we won this, the induction was two days of reading and then we were on live composites. Nobody in the firm had run agents in production on a compliance account before — we found that out afterwards.”Engagement Manager, KPO, Trade, settlement and reconciliation

“To be fair nobody in this company has run credit early-warning agents before… they are learning same as me. But they sold it like they have. And training is two videos, that part I cannot defend.”Individual Contributor, KPO, Credit, underwriting and suitability

“What we do not have is people who have run agents in live production rather than in pilots”Engagement Manager, BPO, Payments, cards and disputes

Written sentiment on production experience was not associated with the study measures. Nearly every response was negative, and the written scores varied too little across engagements for the tests to detect an association. The written evidence is uniform. The firm’s agentic work had stopped at pilots and demonstrations, and the production experience was acquired on the engagements in the study.

4.4.2 Operation of Large Numbers of Agents Without an Operations Function

Respondents rated their employer’s ability to operate large numbers of agents in production, including monitoring and exception handling. The mean rating was 3.7 and the median 4.0 (n = 798 respondents), and 379 of the 798 answers (47.49%) were distressed. The operation of large numbers of agents was rated lowest on canceled engagements, at a mean of 2.4 (n = 297 respondents), far lower than on active engagements in any stage.

Every association of the vendor’s ability to operate large numbers of agents with the four core study measures held, in each of the 22 robustness checks. Engagements whose vendor operated its agents better had a lower cancellation rate (r = −0.59, 95% CI −0.68 to −0.47, n = 150 engagements). Among active engagements, customer satisfaction was higher where agent operations were rated stronger (r = 0.70, 95% CI 0.58 to 0.79, n = 94 engagements). The engagement indicator mean, computed without the operations indicator, was higher as well (r = 0.87, 95% CI 0.83 to 0.91, n = 150 engagements). Respondents who rated agent operations higher reported lower intent to leave (r = −0.61, cluster bootstrap 95% CI −0.67 to −0.53, n = 750 respondents).

The written responses addressing the operation of large numbers of agents numbered 37, and 29 were negative. The most common account was of agents that the vendor had not assigned anyone to watch while they acted, with failures discovered the next morning, by the financial institution or by its staff. Other respondents described a firm able to run a handful of agents and unready for hundreds. The positive responses described monitoring that worked.

“We had two agents provisioning at once in same tenant and nobody on our side was watching the queue, one of them looped and raised forty duplicate accounts before a colleague noticed. Took us whole shift to unwind. There is no runbook for that, we wrote one ourselves after.”Individual Contributor, ITO, Technology engineering and operations

“we were given the assist tooling and a two-hour handover call, and no operations model behind it. when an agent misfires at 2am there is nobody on our side rostered to notice, i am the escalation path, for that reason i keep my phone next to me.”Engagement Manager, CXM, Corporate and back-office functions

“nobody from my company is watching the agent while it runs, one week it keeps closing tickets as resolved which are not resolved, we find out from angry employees and not from any dashboard. the people who trained us do not know banking either.”Individual Contributor, CXM, Corporate and back-office functions

“We run agents on the release pipeline and there is no proper dashboard from our side, if one starts misbehaving at 2am we dont hear it first, the bank tells us.”Individual Contributor, ITO, Technology engineering and operations

“Two months back we had an agent loop on a sandbox endpoint over a weekend, and our own operations desk caught it on the monitoring before the client noticed, and had it stopped within the hour.”Engagement Manager, CXM, Technology engineering and operations

Respondents who wrote more negatively about agent operations reported higher intent to leave. The written evidence describes agents in production without an operations function around them, with no person assigned to watch them, no alerting on the agents themselves and no route for a failure out of hours. Where such a function existed, respondents rated the capability high.

4.5 Talent Supply, Development, and Training

Five indicators describe the vendor’s people. They cover its supply of qualified agentic talent, its process and leadership development programs and the adequacy of its investment in training. The remaining two cover how well its training built practical agentic skills and knowledge of the financial industry. The distressed share ranged from 380 of 804 answers (47.26%) for practical agentic skills to 393 of 803 (48.94%) for development programs. All five were associated with every core study measure, and the 22 robustness checks upheld each association.

The adequacy of the vendor’s investment in training was addressed in 244 written responses, the most of any indicator in the category, and the supply of talent in 158. Respondents described formal training measured in hours or days, much of it slides or recordings unrelated to the engagement’s business or to the agent in use, followed by live work. People were placed on agentic work without having done it, and vacancies stayed open for weeks. The few people with real experience were shared across accounts, and the engagement’s tasks stopped when they were absent.

“When our automation lead went on leave for two weeks there was nobody to cover, we just paused the closure rules testing till he came back. That is my only real complaint, otherwise the support from my manager is good.”Individual Contributor, ITO, Technology engineering and operations

The working knowledge of each engagement passed between colleagues instead, through a seatmate, a shift senior or a private file of notes that one analyst wrote and shared with two others. Respondents described those files leaving with the people who kept them. The delivery workforce that learned the agents on live work, and its intent to leave, are examined separately (Part 4).

“There is no training on any of this, so we keep our own notes file, one page per Guidewire config we automated, what the agent gets wrong and what to check after. Every new joiner gets that file on day one. Because of that nobody asks the training team for anything now.”Individual Contributor, ITO, Client servicing and communications

4.5.1 Supply of Qualified Agentic Talent

Respondents rated their employer’s current supply of qualified agentic AI talent against what its engagements needed. The mean rating was 3.7 and the median 4.0 (n = 802 respondents), and 382 of the 802 answers (47.63%) were distressed. The mean rating of the supply of talent on canceled engagements, 2.5 (n = 304 respondents), was far lower than on active engagements in any stage.

Each core study measure was associated with the supply of qualified agentic talent, and every association withstood the 22 robustness checks. Cancellation was less common where the supply of talent was rated adequate (r = −0.59, 95% CI −0.68 to −0.47, n = 150 engagements). Active engagements whose vendor had a better supply of talent reported higher customer satisfaction (r = 0.71, 95% CI 0.59 to 0.80, n = 94 engagements). The engagement indicator mean computed without the talent indicator was closely associated with the rating (r = 0.87, 95% CI 0.83 to 0.91, n = 150 engagements). Respondents who rated the supply of talent higher reported lower intent to leave (r = −0.62, cluster bootstrap 95% CI −0.68 to −0.54, n = 754 respondents).

The written responses addressing the supply of talent numbered 158, and 144 were negative. Every respondent who addressed it in naming the hardest challenges named a shortage of agentic talent. The comments on the employer described teams staffed with people who had never performed agentic tasks, slow replacement, and knowledge held by two or three people shared across accounts.

“We are three testers and only one of us has done agent work before … when I need someone senior to explain why the agent behaves one way on Monday and another way on Thursday, there is nobody in the company to ask, we post in a chat group and wait two days.”Individual Contributor, ITO, Technology engineering and operations

“When two of our people moved off in March we waited nearly six weeks for replacements, and the ones who came dont know the queue at all, we teach them from scratch ourselves.”Individual Contributor, CXM, Client servicing and communications

“It felt slow when two people rolled off in month two, replacements took nearly six weeks and neither had done a close before. We managed, but my lead did the training himself in the evenings, thats not a plan.”Individual Contributor, BPO, Trade, settlement and reconciliation

“In March the failover config drifted on the tick capture box and we lose about two hours before somebody who actually knows that kit comes on the bridge… There are maybe three people here who know this environment properly.”Individual Contributor, ITO, Technology engineering and operations

“nobody they staffed had done a network certification before, me included. … what the numbers dont show is we were reading the processors spec ourselfs at night to keep up, thats not a training program thats just people covering.”Individual Contributor, ITO, Payments, cards and disputes

Written sentiment on the supply of talent was associated with the outcomes in its own right. Engagements whose written responses on talent were more negative on average had a higher cancellation rate, and respondents who wrote more negatively reported higher intent to leave. The written evidence describes a shortage in three forms. People were placed on agentic work without having done it, vacancies were left open for weeks and expertise was concentrated in a few individuals whose absence stopped the engagement’s tasks.

4.5.2 Process and Leadership Development Programs

Respondents rated the effectiveness of their employer’s process and leadership development programs. The mean rating was 3.7 and the median 4.0 (n = 803 respondents), and 393 of the 803 answers (48.94%) were distressed. Respondents on canceled engagements rated the programs far lower than respondents on active engagements in any stage (mean 2.4, n = 306 respondents).

All four core study measures were associated with the vendor’s process and leadership development programs, and none of the 22 robustness checks overturned an association. More effective programs were associated with a lower likelihood of cancellation (r = −0.59, 95% CI −0.69 to −0.48, n = 150 engagements). Customer satisfaction among active engagements was higher where the programs were rated more effective (r = 0.69, 95% CI 0.57 to 0.78, n = 94 engagements). The engagement indicator mean, computed without the programs indicator, was higher too (r = 0.85, 95% CI 0.80 to 0.89, n = 150 engagements). Intent to leave was lower among respondents who rated the programs higher (r = −0.61, cluster bootstrap 95% CI −0.67 to −0.53, n = 753 respondents).

Seven written responses addressed the programs, a base of anecdote strength, and four were negative. The negative responses described leadership programs delivered as slides or in a classroom, with little bearing on agentic delivery. The positive responses found the programs adequate and placed the weakness in accountability for agent errors.

“The knowledge is there at the top but it does not come down to us. Leadership programme is a set of slides nobody finishes. And I learnt everything from my team lead”Individual Contributor, KPO, Credit, underwriting and suitability

“Leadership programs exist but they are classroom, not for this kind of delivery.”Engagement Manager, BPO, Corporate and back-office functions

“We were put on this desk with two days of induction and the leadership course i was sent on was mostly slides. And nobody above me has run agents live before this.”Individual Contributor, ITO, Technology engineering and operations

The few respondents who wrote described development programs held at a distance from agentic delivery. How widely that view was held is measured by the ratings, and about half of the answers placed the programs in the distressed range.

4.5.3 Adequacy of Vendor Investment in Training

Respondents rated the adequacy of their employer’s investment in training its workforce. The mean rating was 3.7 and the median 4.0 (n = 789 respondents), and 388 of the 789 answers (49.18%) were distressed. On canceled engagements the investment in training had a mean rating of 2.5 (n = 298 respondents), far lower than on active engagements in any stage.

The adequacy of the vendor’s investment in training was associated with all four core study measures, and each association held under the 22 robustness checks. Engagements whose vendor invested more adequately in training had a lower cancellation rate (r = −0.57, 95% CI −0.67 to −0.45, n = 150 engagements). Among active engagements, customer satisfaction was higher where the investment was rated adequate (r = 0.71, 95% CI 0.59 to 0.80, n = 94 engagements). So was the engagement indicator mean computed without the training-investment indicator (r = 0.86, 95% CI 0.81 to 0.90, n = 150 engagements). Respondents who rated the investment higher reported lower intent to leave (r = −0.59, cluster bootstrap 95% CI −0.67 to −0.51, n = 742 respondents).

The investment in training was the subject of 244 written responses, the most of any indicator of vendor capability, nearly all in comment on the employer. They divided into 177 negative and 62 positive. The negative responses described training of a few hours or days, much of it slides or recordings unrelated to the engagement’s business, followed by live work learned from colleagues. The positive responses described training that had improved, and some named a gap that remained in the business or in checking agent output.

“We were given access to the payroll agent and a two hour session on a Friday, recorded, and that was the preparation. Nobody from the practice side sat with us for even one Cycle. My four staff learned it by running it against last quarter file and comparing what came out, mostly on their own time, and I let that happen because there was no other way to be ready.”Engagement Manager, BPO, Corporate and back-office functions

“training was two days of slides then straight to live cases. so i made my own file. i wrote down every requirement code and what the underwriter actually wants for it, and i share it with two other new joiner on my team. it is not official, nobody ask me to make it … we lose everything when someone resigns because it all sits in their head or in a file like mine.”Individual Contributor, BPO, Credit, underwriting and suitability

“Second week they put me alone on month end attribution run, nobody to ask. Training was two days recorded video and that is all. That is what i marked down, not my manager, he is also learning same as me”Individual Contributor, KPO, Trade, settlement and reconciliation

“Here I get one day of induction and a link to a folder of PDFs. Nobody in my team has done valuation work before, we are all learning from the client’s old models.”Individual Contributor, KPO, Corporate and back-office functions

“i got a half day slide deck and a link to a course nobody finished.”Individual Contributor, ITO, Payments, cards and disputes

Written sentiment on the investment in training was associated with the outcomes in its own right. Cancellation was more common on engagements with a negative written response, and respondents who wrote more negatively about training reported higher intent to leave. Formal training was short, generic and early, and the working knowledge of each engagement passed between colleagues and through notes the teams kept themselves.

4.5.4 Practical Agentic Skills Built by Vendor Training

Respondents rated how well their employer’s training programs built practical agentic AI skills. The mean rating was 3.7 and the median 4.0 (n = 804 respondents), and 380 of the 804 answers (47.26%) were distressed. On canceled engagements, the practical value of the training was rated far lower than on active engagements in any stage (mean 2.5, n = 308 respondents).

The 22 robustness checks left intact each association between the practical agentic skills built by training and the four core study measures. Training that built practical skills was associated with less cancellation (r = −0.58, 95% CI −0.68 to −0.46, n = 150 engagements). Customer satisfaction among active engagements was higher where training was rated more practical (r = 0.69, 95% CI 0.57 to 0.78, n = 94 engagements). The engagement indicator mean, computed without the skills indicator, was closely associated with the rating (r = 0.85, 95% CI 0.80 to 0.89, n = 150 engagements). Respondents who rated the training’s practical value higher reported lower intent to leave (r = −0.60, cluster bootstrap 95% CI −0.67 to −0.52, n = 755 respondents).

The written responses addressing practical agentic skills numbered 54, nearly all in comment on the employer, with 42 negative and 12 positive. The negative responses described training delivered as slides and theory, training on tools other than the agent in use, and no instruction on checking agent output that varies between runs. The positive responses found the agentic courses practical and named gaps elsewhere.

“The trainings we get are mostly slide decks… you cannot learn agent debugging from that.”Individual Contributor, ITO, Technology engineering and operations

“here i get access on day one and a link to a video from 2023 which is about a different tool completely. moreover nobody could tell me who to ask when the agent behave strange.”Individual Contributor, CXM, Corporate and back-office functions

“But the courses are mostly slides and theory… when i sit with the live queue the first week, nothing in the course was showing me what to do when the agent stops halfway.”Individual Contributor, CXM, Client servicing and communications

“The training was two weeks of slides then straight to the floor… nobody taught us how to judge a bot answer when it comes out different each time.”Individual Contributor, BPO, Financial crime and client onboarding

“What I can score is the training, which was actually decent, the agentic modules were hands on and I use that stuff daily.”Individual Contributor, CXM, Corporate and back-office functions

Written sentiment on practical skills was not associated with the study measures. The written evidence separates course content from hands-on use. Where the courses were hands-on, respondents valued them. Where they were slides, they did not prepare respondents for the live queue, for the agent in use or for checking output that varies.

4.5.5 Financial Industry Knowledge Built by Vendor Training

Respondents rated how well their employer’s training programs built knowledge of the financial industry. The mean rating was 3.72, the highest of the 20 indicators of vendor capability, and the median 4.0 (n = 788 respondents). Distressed ratings made up 380 of the 788 answers (48.22%). Respondents on canceled engagements gave industry training a mean rating of 2.6 (n = 294 respondents), far lower than respondents on active engagements in any stage.

Where training built industry knowledge, all four core study measures were more favorable, and none of the 22 robustness checks reversed an association. Cancellation was less common where industry training was rated stronger (r = −0.56, 95% CI −0.66 to −0.44, n = 150 engagements). Among active engagements, customer satisfaction was higher where industry training was rated stronger (r = 0.69, 95% CI 0.56 to 0.78, n = 94 engagements). Industry training was also associated with a higher engagement indicator mean computed without it (r = 0.85, 95% CI 0.80 to 0.89, n = 150 engagements). Intent to leave was lower among respondents who rated industry training higher (r = −0.57, cluster bootstrap 95% CI −0.64 to −0.49, n = 743 respondents).

The written responses addressing industry training numbered 82, all in comment on the employer, with 63 negative and 19 positive. The negative responses described training that was generic and contained little or nothing on the financial business the engagement served. Staff learned the business from colleagues or from the financial institution’s own documents. The positive responses described industry courses that worked and placed the gap on the agent side.

“nobody in training explained insurance record classes to us, and the trainer himself was reading from slides. So i keep a sheet of which document type keeps which period and i share the same with two colleagues … But the industry part we teach ourselves…”Individual Contributor, BPO, Corporate and back-office functions

“Training is the one good part, the agent classes are practicle and I use them daily. But there is nothing on credit in those trainings, so that knowledge I pick up on the floor.”Individual Contributor, KPO, Credit, underwriting and suitability

“Our people finish induction knowing the platform but not the card business — dispute lifecycle, chargeback reason codes, what a provisional credit actually means to cardholder. On voice queue that gap shows inside first month, and my leads end up teaching it case by case at the desk.”Engagement Manager, CXM, Payments, cards and disputes

“For banking knowledge we do not wait for corporate training now. I run one hour session every friday where senior reconciler walks the team through actual GL break from that week. It is not in any playbook but it works better than the modules, which are generic.”Engagement Manager, BPO, Trade, settlement and reconciliation

“Half of my batch had never seen a partnership return before joining and we learn on live files.”Individual Contributor, KPO, Regulatory, audit, tax and legal

Written sentiment on industry training was not associated with the study measures. The written evidence describes industry knowledge acquired on the engagement. Where the employer’s courses taught the business, respondents valued them. Where they did not, the knowledge came from colleagues, from notes the teams kept and from the financial institution that was paying for the service.

4.6 Delivery Playbook, Quality Assurance, and Platforms

Three indicators describe the vendor’s methods and tools for agentic delivery. They cover the maturity of its playbook for taking agentic use cases from concept to governed production and its quality assurance for agents whose output is not identical on every run. The third is how well its technical platforms were built for agentic outsourcing. The playbook was distressed in 392 of 807 answers (48.57%), quality assurance in 377 of 794 answers (47.48%) and the platforms in 384 of 794 (48.36%). Each of the three showed an association with every core study measure that held through the 22 robustness checks.

The playbook was the subject of 200 written responses and quality assurance of 151, and the account they give is consistent. The route from a pilot to a service that a financial institution’s risk function would approve was invented on each engagement, in some accounts after the sale and while the service was live. Output that varied between runs had no method for sign-off, and the analyst at the point of sign-off reran it, chose one version or rewrote it. Where the platform had been built for other work, a contact center or a staffed service desk, the team moved the engagement’s tasks into spreadsheets, shared mailboxes and scripts it assembled for itself.

“The people I have been given are capable, and the firm knows the technology in the abstract, what is missing is anything written down for a finance back office at a private capital client. We were handed a platform built for contact center work and told to make expense processing fit it, and the training that came with it was generic.”Engagement Manager, BPO, Corporate and back-office functions

“No shared agentic playbook, every rollout reinvents the wheel”Individual Contributor, BPO, Trade, settlement and reconciliation

Each engagement built its method and its tools for itself. What one engagement learned stayed with that engagement and with the people who had built it.

4.6.1 Immaturity of the Playbook From Concept to Governed Production

Respondents rated the maturity of their employer’s playbook for taking agentic use cases from concept to governed production. The mean rating was 3.7 and the median 4.0 (n = 807 respondents), and 392 of the 807 answers (48.57%) were distressed. Respondents on canceled engagements rated the playbook far lower than respondents on active engagements in any stage (mean 2.4, n = 304 respondents).

The maturity of the vendor’s delivery playbook showed an association with each of the four core study measures that survived all 22 robustness checks. Engagements run with a more mature playbook were less likely to be canceled (r = −0.59, 95% CI −0.68 to −0.47, n = 150 engagements). Active engagements with a more mature playbook reported higher customer satisfaction (r = 0.70, 95% CI 0.58 to 0.79, n = 94 engagements). Playbook maturity and the engagement indicator mean, computed without the playbook indicator, were closely associated (r = 0.86, 95% CI 0.81 to 0.90, n = 150 engagements). Intent to leave was lower where respondents judged the playbook mature (r = −0.62, cluster bootstrap 95% CI −0.68 to −0.54, n = 756 respondents).

The playbook was the subject of 200 written responses, and 186 of the 200 were negative, 145 with the strongest negative score. Every respondent who addressed it in naming the hardest challenges named the absence or immaturity of a delivery playbook. The comments on the employer described a route from pilot to governed production that each team invented for itself, or a playbook written for staffed work.

“we are selling ahead of our own playbook. The platform is good — I have no argument there, and our people know cards and payments as well as anyone in this market — but the route from a promising cardholder use case to something a bank’s risk function will sign off is still being invented on each engagement rather than followed.”Portfolio Director, CXM, Payments, cards and disputes

“on my last account we have proper checklist for exceptions, signed off, everybody follow the same one. Here every pod invent its own for the nostro breaks … we are short of anyone who has actually run these agents before, so we learn on the client’s work”Individual Contributor, BPO, Trade, settlement and reconciliation

“No agentic delivery playbook existed, so every escalation was improvised”Individual Contributor, BPO, Financial crime and client onboarding

“No agentic playbook exists so every new ticket type starts from zero”Individual Contributor, CXM, Technology engineering and operations

“the training budget is there, we are sent on the course, but nobody internally has run agents at this size before, so we build our own runbook as we go and take the platform vendor’s support call every Tuesday”Engagement Manager, ITO, Technology engineering and operations

Written sentiment on the playbook was associated with the outcomes in its own right. Cancellation was more common where the written responses on the playbook were more negative on average, and intent to leave was higher among respondents who wrote more negatively about it. The route from pilot to governed production was work each engagement performed for itself, in some accounts after the sale and while the service was live.

4.6.2 Quality Assurance for Non-Deterministic Agent Output

Respondents rated their employer’s quality assurance capability for agents whose output is not identical on every run. The mean rating was 3.7 and the median 4.0 (n = 794 respondents), and 377 of the 794 answers (47.48%) were distressed. Quality assurance was weakest on canceled engagements, where the mean rating was 2.5 (n = 302 respondents), far lower than on active engagements in any stage.

Quality assurance for agent output that varies between runs was associated with cancellation, customer satisfaction, the engagement indicator mean and intent to leave, and every association held through the 22 robustness checks. Engagements whose vendor had stronger quality assurance had a lower likelihood of cancellation (r = −0.59, 95% CI −0.68 to −0.47, n = 150 engagements). Customer satisfaction among active engagements was higher where quality assurance was rated stronger (r = 0.70, 95% CI 0.58 to 0.79, n = 94 engagements). The association with the engagement indicator mean, computed without the quality-assurance indicator, reached the upper end of the category’s range (r = 0.88, 95% CI 0.83 to 0.91, n = 150 engagements). Respondents who rated quality assurance higher reported lower intent to leave (r = −0.61, cluster bootstrap 95% CI −0.68 to −0.53, n = 745 respondents).

Quality assurance was the subject of 151 written responses across 12 survey questions, and 126 were negative. Every respondent who addressed it in naming the hardest challenges named quality assurance for agent output. The responses to the operational questions described output that differed between runs on the same input, staff who would not sign it off as it stood, and checking performed by whoever handled the case.

“i’ve run the same case five times in one afternoon and gotten five different numbers back, so nothing here holds still long enough to actually sign off on, and that’s the part that gets to me, not that it’s wrong sometimes, but that it’s never the same wrong twice, i can’t even build a pattern around it.”Individual Contributor, KPO, Corporate and back-office functions

“Same tools as when i joined, no proper course, no QA method for what the agent produce. When output come different two times for same policy, nobody has told us what to do, we decide ourselves and move on.”Individual Contributor, BPO, Client servicing and communications

“My employer put me on this Engagement after two weeks classroom only. There is no proper QA guide for when the screening agent gives different result, so my team make our own checklist in a shared sheet. It works but nobody from above ask us for it. I do not mind, I learn a lot.”Individual Contributor, BPO, Financial crime and client onboarding

“where i mark 5 is checking of output, because same request can give slighty different answer and there is no proper method for that yet. but in truth nobody in market has that method, so i do not blame my company alone for it”Individual Contributor, CXM, Corporate and back-office functions

“No real QA existed for an agent that gave different answers each run”Individual Contributor, KPO, Regulatory, audit, tax and legal

A more negative written score on quality assurance was associated with higher intent to leave. The written evidence describes quality assurance performed by individuals at the point of sign-off. Without a method for output that changed between runs, they reran it, chose a version or rewrote it themselves.

4.6.3 Optimization of Vendor Platforms for Agentic Outsourcing

Respondents rated how well the technical platforms their employer built or licensed were optimized for agentic AI outsourcing. The mean rating was 3.7 and the median 4.0 (n = 794 respondents), and 384 of the 794 answers (48.36%) were distressed. The platforms received a mean rating of 2.5 on canceled engagements (n = 298 respondents), far lower than on active engagements in any stage.

How well the vendor’s platforms were built for agentic outsourcing was associated with all four core study measures, and each association held under the 22 robustness checks. Cancellation was less common on engagements whose vendor platforms were rated better suited (r = −0.57, 95% CI −0.67 to −0.45, n = 150 engagements). Among active engagements, customer satisfaction was higher where the platforms were rated better suited (r = 0.70, 95% CI 0.57 to 0.79, n = 94 engagements). A higher engagement indicator mean, computed without the platform indicator, accompanied better-suited platforms (r = 0.86, 95% CI 0.81 to 0.90, n = 150 engagements). Intent to leave was lower among respondents who rated the platforms higher (r = −0.59, cluster bootstrap 95% CI −0.66 to −0.51, n = 743 respondents).

The written responses addressing the platforms numbered 61, nearly all in comment on the employer, with 38 negative and 21 positive. The negative responses described platforms built for other work, such as ticket handling or contact centers, and teams that moved the engagement’s tasks into spreadsheets, mailboxes and personal scripts. The positive responses found the platform sound or improved and placed the gap elsewhere.

“Our platform group still hands us tooling built for a staffed service desk, so every report counts tickets a person touched and there is no clean way to show a ticket the bot closed by itself.”Engagement Manager, ITO, Technology engineering and operations

“The company can say it has platform, but what we use is a shared mailbox and one spreadsheet. Thus every category change is manual for us. i have raised this. My TL raised it also. Also the answer is always that a proper tool is coming next quarter, and next quarter comes and it is same.”Individual Contributor, CXM, Technology engineering and operations

“The sales team pitched our dev kit we built as being optimized for AI. No one in sales checked if it was. We panicked for a couple of weeks as we validated AI compatibility after contracts were signed.”Portfolio Director, ITO, Technology engineering and operations

“The tooling we were given for ticket triage was built for a generic helpdesk, not for purchase orders, and i flagged that twice with no result.”Engagement Manager, CXM, Corporate and back-office functions

“At March close we had to run reconciliation on client tooling because our platform cant hold volume of merchant transactions, and nobody flagged that before we committed to it.”Engagement Manager, BPO, Trade, settlement and reconciliation

Written sentiment on the platforms was not associated with the study measures. The written evidence divides. Where the platform had been built for the engagement’s type of work, respondents rated it well and named other gaps. Where it had been built for something else, the engagement’s tasks moved into tools the team assembled for itself.

4.7 Accountability for Agent Error

One indicator asks where accountability landed within the vendor when an agent it operated made a costly error. Respondents placed their employer’s readiness to take that accountability in the distressed range in 373 of 794 answers (46.98%). The written evidence places it at the lowest level of the delivery team. Where a lead accepted the error, respondents wrote favorably of the employer. Where no manager senior to the team did, the analyst on shift held it alone.

4.7.1 Accountability for Agent Error Left With the Analyst on Shift

Respondents rated their employer’s readiness to take accountability when an agent it operated made a costly error. The mean rating was 3.7 and the median 4.0 (n = 794 respondents), and 373 of the 794 answers (46.98%) were distressed. Respondents on canceled engagements gave their employer’s readiness a mean rating of 2.4 (n = 307 respondents), far lower than respondents on active engagements in any stage.

All four core study measures were associated with the vendor’s readiness to take accountability for agent error, and none of the 22 robustness checks overturned an association. Engagements whose vendor stood behind agent errors were less likely to be canceled (r = −0.59, 95% CI −0.68 to −0.47, n = 150 engagements). Customer satisfaction among active engagements was higher where the vendor was rated readier to take accountability (r = 0.67, 95% CI 0.54 to 0.77, n = 94 engagements). Readiness to take accountability also accompanied a higher engagement indicator mean, computed without the accountability indicator (r = 0.87, 95% CI 0.82 to 0.90, n = 150 engagements). Respondents who rated their employer readier to take accountability reported lower intent to leave (r = −0.62, cluster bootstrap 95% CI −0.69 to −0.54, n = 744 respondents).

The written responses addressing accountability numbered 38, nearly all in comment on the employer, with 29 negative and 8 positive. The negative responses described agent errors that no manager senior to the delivery team would own. The errors landed on the analyst on shift. The positive responses described a lead or the vendor’s management accepting the error without passing it down.

“When the assistant gave a wrong leave-accrual answer to about forty new hires, nobody internally could tell me who signs off the correction or who carries it if the client asks for a credit. We escalated and it went sideways for a week.”Engagement Manager, CXM, Corporate and back-office functions

“They do own their mistakes though. When the Classifier mistagged a whole batch of custody statements nobody tried to pin that on my team.”Individual Contributor, BPO, Corporate and back-office functions

“top leadership question is pointless for me, i have never seen these people. ask instead who pays when agent breaks a release, that box is not here. my answer would be nobody, it lands on me at 2am”Individual Contributor, ITO, Payments, cards and disputes

“nobody tells us who answers if an agent mis-rates a file and it goes out wrong, so my team keeps a private checklist before releasing. That worry sits on the analyst, not on the people above us”Individual Contributor, BPO, Financial crime and client onboarding

“In may the bot closed forty tickets as resolved which was not resolved. And i raise it same day. Two weeks nobody on our side owns it, then we just reopen them ourselves. After that nobody discuss about it again.”Individual Contributor, ITO, Technology engineering and operations

Respondents whose written score on accountability was more negative reported higher intent to leave. The written evidence places accountability for agent error at the lowest level of the delivery team. The contrast between an error left with the analyst and an error a lead accepted is the contrast the rating measures.

4.8 Vendor Preparedness Relative to Five Years Earlier

One further question asked respondents to compare their employer’s preparedness to manage outsourcing engagements with its preparedness five years earlier. The comparison tests whether the experience of the earlier outsourcing era transferred to agentic delivery. Respondents placed the answer in the distressed range in 381 of 786 answers (48.47%), close to the 48.22% of the 20 indicators of the category.

4.8.1 Vendor Preparedness Unchanged Against the Prior Outsourcing Era

Respondents rated whether their employer was more or less prepared to manage outsourcing engagements than five years earlier. Its mean rating was 3.7 and its median 4.0 (n = 786 respondents), with 381 of the 786 answers (48.47%) distressed. On canceled engagements, the employer’s preparedness was rated far lower than on active engagements in any stage (mean 2.4, n = 293 respondents).

The 22 robustness checks left intact each association between the vendor’s preparedness against five years earlier and the four core study measures. Greater preparedness was associated with a lower likelihood of cancellation (r = −0.56, 95% CI −0.66 to −0.44, n = 150 engagements). Customer satisfaction among active engagements was higher where preparedness was rated greater (r = 0.66, 95% CI 0.52 to 0.76, n = 94 engagements). The engagement indicator mean computed without the preparedness indicator moved in the same direction (r = 0.85, 95% CI 0.80 to 0.89, n = 150 engagements). Respondents who rated their employer better prepared reported lower intent to leave (r = −0.60, cluster bootstrap 95% CI −0.67 to −0.52, n = 737 respondents).

Four written responses addressed the comparison, a base of anecdote strength, two positive and two negative. Each compared the present engagement with an earlier one and named one area that had changed. The areas named included hiring, support from a solutions team, access to a test environment, pricing on outcomes and a second-read review process.

“To be blunt not much has improved… the one thing i will say is they finally gave us proper access to sandbox this quarter, before that we were testing on live extracts which was mad. Everything else is same. No training, no playbook, we make it up per engagement and call it method”Individual Contributor, KPO, Corporate and back-office functions

“Compared with the reconciliations programme I carried two years ago, the firm has become more willing to price an outcome rather than a headcount, and our review capability for work that does not come out identical each run is stronger, there is a functioning second-read process now which we did not have then.”Portfolio Director, KPO, Regulatory, audit, tax and legal

“We ran a similar recruiting program two years ago with almost no support behind it. This time we had a solutions team on the calls from week one, which is a real change. Staffing the roles still takes longer than it should.”Engagement Manager, CXM, Corporate and back-office functions

The few respondents who wrote described change in single areas against an otherwise unchanged firm. The ratings measure how widely the view was held, and about half of the answers placed the employer’s preparedness, against its own position five years earlier, in the distressed range.

AI technology was rated markedly higher than the three categories covering the two firms. Its 20 indicators had a mean of 4.60 across the 150 engagements, about one point higher than financial institution readiness, co-management and vendor capability. Distressed ratings made up 5,113 of their 15,911 answers (32.14%), against about half in those categories. The rating remained under the midpoint of the scale. The indicators of the category were associated with cancellation as closely as any in the study, from r = −0.69 to r = −0.63 (n = 150 engagements each).

Mean ratings in Table 11 are computed across respondents. The category figure of 4.60 is computed across engagements.

The written evidence adds a qualification that the ratings alone do not show. Where the tools reached the standard the engagement required, they did so in large part because people made them do so. Delivery staff corrected the agents’ output, reran tasks whose results differed between runs, cut long tasks into single steps and read every output before it went on. Those corrections were seldom recorded, and in many accounts the vendor’s measures credited the corrected output to the agent. The reliability reported to the financial institution was partly the product of corrections that, in the respondents’ accounts, were not reported to it.

“i correct it most times we use it, seven out of ten… my Lead know, client does not. in truth none of it feel reliable enough for what this is meant to be for, KYC.”Individual Contributor, CXM, Financial crime and client onboarding

Several failures recorded in the category have a consequence for the financial institution’s control environment. Corrections made by hand went unreported. Tools reported tasks complete that they had not completed. Outputs could not be explained to the depth an auditor requires, and logs could not locate the point at which a run had failed. Each of these leaves the financial institution’s account of what the agents did less complete than what the agents actually did.

“The vendor’s weekly deal-status report used the agent’s self-reported completion percentage for data-room review without sampling actual output, so a data room the agent had barely parsed correctly still showed as ’90 percent reviewed’ heading into the client update.”Individual Contributor, KPO, Corporate and back-office functions

The survey invited comment on the tools, and 466 of the 850 respondents (54.82%) wrote one. Many were favorable within limits. Respondents described tools that read documents well, completed short tasks without intervention and held their guardrails, and then named the limits they managed around by hand.

5.1 Production Readiness of the Tools

One indicator asks for the overall judgment, the production readiness of the AI tools for financial-institution-grade work as experienced on the engagement. It was distressed in 257 of 802 answers (32.04%), and its association with cancellation, r = −0.69 (n = 150 engagements), was the strongest of the 100 indicators. Its 79 written responses included 60 positive and 15 negative, the lowest share of negative responses among the 100 rated indicators with 30 or more written responses. Respondents judged the tools ready within limits they had learned and managed by hand.

5.1.1 Production Readiness Within Limits Managed by Hand

Respondents rated the overall production readiness of the AI tools for work of the standard financial institutions require, as experienced on the engagement. The mean rating was 4.5 and the median 5.0 (n = 802 respondents), and 257 of the 802 answers (32.04%) were distressed. Respondents on canceled engagements rated the tools’ readiness far lower than respondents on active engagements in any stage (mean 3.1, n = 305 respondents).

Production readiness of the tools showed an association with every core study measure, and none of the 22 robustness checks overturned one. Engagements that rated the tools more ready had a lower cancellation rate (r = −0.69, 95% CI −0.77 to −0.60, n = 150 engagements), the strongest association with cancellation of any of the 100 indicators. Customer satisfaction among active engagements was higher where the tools were rated more ready (r = 0.64, 95% CI 0.50 to 0.74, n = 94 engagements). A higher engagement indicator mean, computed without the readiness indicator, accompanied tools rated more ready (r = 0.86, 95% CI 0.82 to 0.90, n = 150 engagements). Respondents who rated the tools more ready reported lower intent to leave (r = −0.67, cluster bootstrap 95% CI −0.72 to −0.61, n = 756 respondents).

The written responses addressing production readiness numbered 79, and 60 were positive and 15 negative. No other of the 100 indicators with 30 or more written responses had a lower share of negative responses. Respondents described tools that worked, and then named the limits they managed around. Those limits were instructions dropped on long runs, tasks reported complete that were not, results that varied between runs and citations to the wrong source. The negative responses described correcting most of the tools’ output.

“Tools have not been our trouble on this one, they have run a whole quarter without me thinking about them, which I did not expect going in. Only thing I would note is an agent once reached for a settlement screen it had no business in, sandbox caught it, but that is what I watch for.”Engagement Manager, BPO, Corporate and back-office functions

“Two things would have to change before I would put this in front of a client executive again. First, the same question asked twice should produce the same answer twice … Second, the assistant reads the front of a policy document and the back of it and misses what sits in the middle, which is usually where the exceptions live. Everything else is manageable.”Portfolio Director, CXM, Corporate and back-office functions

“asking me to rate production readiness on 1 to 10 is not so simple, the tool handle long access reports well and finish multi step job properly, but same job on Tuesday and Thursday do not behave same, one number cannot hold both of these”Individual Contributor, ITO, Technology engineering and operations

“The tools are if I am honest fine, i have no big complaint there. It reports the run as complete when a figure in the middle of the pack is missing… so i still read every page myself. That checking is what eats my day, not the modelling.”Individual Contributor, KPO, Corporate and back-office functions

Written sentiment on production readiness was associated with the outcomes in its own right. Cancellation was more common on engagements whose written responses on readiness were more negative, and intent to leave was higher among respondents who wrote more negatively. Respondents judged the tools ready within limits they had learned, shorter runs, repeated instructions, second runs and reading every output. Where those limits were not managed, staff corrected most of the output.

5.2 Output Reliability and Non-Determinism

Four indicators describe whether the tools’ output could be relied on. They cover its factual reliability before human review, the accuracy of its citations against source and its consistency when given the same input more than once. The fourth is its reliability on multi-step tasks from first step to finished output. Each had about a third of its answers in the distressed range, from 254 of 782 answers (32.48%) for multi-step reliability to 265 of 791 (33.50%) for citations, the highest distressed share in the category. All four showed an association with every core study measure, and the 22 robustness checks upheld each one.

Factual reliability was the subject of 628 written responses, the largest written base of any of the 100 indicators, and the consistency of output across runs of 383. Respondents described output that reached the financial institution correct because the delivery team had corrected it. Those corrections went largely unrecorded, and in many accounts the financial institution was not informed. Where the same input produced different results on different runs, the person handling the case chose among them, guided by what looked right or by what had been sent before.

“Nothing about the AI output is repeatable either, it flags different vendors each run, so I change the result myself and paper it afterwards, or sometimes not at all.”Individual Contributor, BPO, Corporate and back-office functions

Non-determinism turned each disagreement between runs into a human decision. Long tasks lost a step or the thread and were then reported complete, and citations that appeared authoritative could not be relied on. In each case the check passed to a person, and the time the tool saved was spent verifying it.

5.2.1 Factual Reliability Sustained by Unrecorded Human Correction

Respondents rated the factual reliability of the AI tools’ output on the engagement before any human review. The mean rating was 4.6 and the median 5.0 (n = 796 respondents), and 258 of the 796 answers (32.41%) were distressed. Factual reliability was weakest on canceled engagements, where the mean rating was 3.1 (n = 302 respondents), far lower than on active engagements in any stage.

Every association of the factual reliability of the tools’ output with the four core study measures held, in each of the 22 robustness checks. Engagements whose tools were rated more reliable had a lower likelihood of cancellation (r = −0.67, 95% CI −0.75 to −0.57, n = 150 engagements). Among active engagements, customer satisfaction was higher where output was rated more reliable (r = 0.65, 95% CI 0.52 to 0.76, n = 94 engagements). Factual reliability also tracked the engagement indicator mean computed without it (r = 0.87, 95% CI 0.83 to 0.91, n = 150 engagements). Respondents who rated the output more reliable reported lower intent to leave (r = −0.65, cluster bootstrap 95% CI −0.70 to −0.60, n = 746 respondents).

The indicator was the subject of 628 written responses across 14 survey questions, the largest written base of any of the 100 indicators, and 548 of the 628 were negative. In answer to the operational questions, respondents described output that staff corrected before it went on, usually because the same input gave a different result on another run. The corrections were seldom logged, and in many accounts the financial institution was not told. In comment on the tools, respondents described invented content, runs reported complete that were not, and logs that could not show why.

“i keep a sheet of every generated test case we had to rewrite by hand, about sixty percent of them at this point. before any client demo i run the old regression pack myself instead of the generated one, because that one i trust. the sheet is not in any process document, it is only mine … ”Individual Contributor, ITO, Technology engineering and operations

“My people stopped trusting it around month four and that is hard to walk back … Two of my best case handlers asked to move off the queue because they felt they were doing the same job twice, once by hand and once for the tool. I lost them. That cost isnt anywhere in these questions and it was the real cost.”Engagement Manager, BPO, Corporate and back-office functions

“one saturday it matched a 40,000 dollar break to wrong account and marked it closed. i found it monday only because the cash didnt tie. it never flagged anything, the log say completed. after that i tie the cash myself before anything is signed, kindly there is no other way.”Individual Contributor, BPO, Trade, settlement and reconciliation

“I correct AI output on vendor exceptions most days, team lead knows about it. … so if I don’t catch it nobody else will.”Individual Contributor, BPO, Corporate and back-office functions

Written sentiment was associated with lower customer satisfaction among active engagements, with cancellation at engagement level and with higher intent to leave. Output reached the financial institution correct because delivery staff corrected it, and the vendor’s measures counted the corrected output as the agent’s. The manual alteration of AI output, and how rarely it was disclosed, is examined separately (Part 5).

5.2.2 Accuracy of Citations and References Against Source

Respondents rated the accuracy of the citations, references and quotations the tools produced, checked against the real sources. The mean rating was 4.5 and the median 5.0 (n = 791 respondents), and 265 of the 791 answers (33.50%) were distressed, the highest distressed share among the 20 indicators of AI technology. Citation accuracy received a mean rating of 3.2 on canceled engagements (n = 301 respondents), far lower than on active engagements in any stage.

Citation accuracy was associated with cancellation, customer satisfaction, the engagement indicator mean and intent to leave, and none of the 22 robustness checks overturned an association. Engagements with more accurate citations were less likely to be canceled (r = −0.65, 95% CI −0.73 to −0.54, n = 150 engagements). Customer satisfaction among active engagements was higher where citations were rated more accurate (r = 0.65, 95% CI 0.52 to 0.76, n = 94 engagements). Where citations were more accurate, the engagement indicator mean computed without the citation indicator was higher (r = 0.86, 95% CI 0.81 to 0.89, n = 150 engagements). Intent to leave was lower among respondents who rated citations more accurate (r = −0.64, cluster bootstrap 95% CI −0.69 to −0.58, n = 746 respondents).

The written responses addressing citations numbered 36, nearly all in comment on the tools, and 31 of the 36 were negative. Respondents described citations to clauses, sections and values that did not exist, and citations to real documents that did not say what the output claimed. The response was to check every reference by hand.

“When the tool drafts a payment arrangement it quotes the collections guidance back at the customer, chapter and clause, and roughly one time in five that clause does not say what the tool says it says. Twice it invented a paragraph number that does not exist. On a hardship call, with a customer who is already frightened, that is not a small error, it is a compliance breach with a recording attached to it.”Engagement Manager, CXM, Payments, cards and disputes

“every early-warning narrative the tool writes goes to an analyst with the source spreadsheet open beside it, and each figure it quotes is checked back to the filing it claims to come from. Roughly a third of the references do not survive that … Please note none of this is in any process document, it is simply what my team have worked out between them.”Engagement Manager, KPO, Credit, underwriting and suitability

“every note that comes out i have to open each source link myself, and maybe one in five is pointing to some other report or a page that do not say what it claims. so a draft that took the tool ten minutes takes me an hour to trust.”Individual Contributor, KPO, Corporate and back-office functions

“it says the case is reviewed and complete, then i open it and two transactions it quoted are not in the file at all. So now i check every one, even the ones it gets right”Individual Contributor, BPO, Financial crime and client onboarding

“I keep a text file of the consent Policy references and paste the right one in after, because the tool quotes clauses that don’t exist.”Individual Contributor, ITO, Technology engineering and operations

Written sentiment on citations was not associated with the study measures. A citation that appears authoritative and cannot be relied on becomes a task for a person. Respondents described that checking, not the drafting, as the larger cost of the tool.

5.2.3 Inconsistency of Output Across Repeated Runs

Respondents rated the consistency of the tools’ output when given the same input and instructions more than once. The mean rating was 4.6 and the median 5.0 (n = 800 respondents), and 259 of the 800 answers (32.38%) were distressed. The mean rating of consistency on canceled engagements, 3.2 (n = 309 respondents), was far lower than on active engagements in any stage.

The consistency of output across repeated runs showed an association with each of the four core study measures that survived all 22 robustness checks. Engagements with more consistent output had a lower cancellation rate (r = −0.65, 95% CI −0.74 to −0.55, n = 150 engagements). Active engagements with more consistent output reported higher customer satisfaction (r = 0.66, 95% CI 0.52 to 0.76, n = 94 engagements). Consistency was also closely associated with the engagement indicator mean computed without it (r = 0.88, 95% CI 0.83 to 0.91, n = 150 engagements). Respondents who rated output more consistent reported lower intent to leave (r = −0.63, cluster bootstrap 95% CI −0.69 to −0.57, n = 750 respondents).

The indicator was the subject of 383 written responses, and 363 of the 383 were negative. Respondents described the same input producing a different result on another run. Staff would not sign off such output as it stood. They chose among the runs, or corrected the output by hand before it went on.

“Our practice now is to run it all twice, sometimes three times, and only action flags that come up in both runs. Sounds wasteful and it is, but its the only way my leads trust the list.”Engagement Manager, CXM, Regulatory, audit, tax and legal

“One suitability review last month, the tool read a forty page advisory file and missed a risk tolerance change recorded in the middle of it, then found the same change on a second run of the same file. Instruction retention was never our problem, consistency between runs is.”Engagement Manager, CXM, Credit, underwriting and suitability

“same dispute file was run two times last week, first time it picks reason code 13.1, second time 12.5. Input was not changed. i check both against the network extract by myself.”Individual Contributor, BPO, Payments, cards and disputes

“These question ask me to rate reliability from 1 to 10, but tool is not wrong in a steady way, one run it match everything and next run it invent a counterparty that does not exist. There is no number for that, i put 1 because 1 is the lowest you gave”Individual Contributor, BPO, Trade, settlement and reconciliation

“we did not touch anything on our side for three weeks and the same role mapping request came back approved one day and flagged the next… so now I run it twice before I trust it.”Individual Contributor, ITO, Technology engineering and operations

The written evidence was associated with all four study measures. Cancellation was far more common on engagements with a negative written response, active engagements with one recorded lower customer satisfaction, and more negative written scores were associated with cancellation and with higher intent to leave. Whenever runs disagreed, the person handling the case made the choice, guided by what looked right or what had been sent before.

5.2.4 End-to-End Reliability on Multi-Step Tasks

Respondents rated the end-to-end reliability of the tools on multi-step tasks, from the first step to the finished output. The mean rating was 4.5 and the median 5.0 (n = 782 respondents), and 254 of the 782 answers (32.48%) were distressed. On canceled engagements, multi-step reliability was rated far lower than on active engagements in any stage (mean 3.2, n = 297 respondents).

All four core study measures were associated with the reliability of the tools on multi-step tasks, and none of the 22 robustness checks overturned an association. Cancellation was less common where multi-step reliability was rated higher (r = −0.65, 95% CI −0.74 to −0.55, n = 150 engagements). Customer satisfaction among active engagements was higher where long tasks ran reliably (r = 0.65, 95% CI 0.52 to 0.75, n = 94 engagements). Multi-step reliability was associated with a higher engagement indicator mean, computed without the multi-step indicator (r = 0.85, 95% CI 0.80 to 0.89, n = 150 engagements). Respondents who rated multi-step reliability higher reported lower intent to leave (r = −0.65, cluster bootstrap 95% CI −0.71 to −0.59, n = 741 respondents).

The written responses addressing multi-step reliability numbered 51, nearly all in comment on the tools, with 36 negative and 13 positive. Respondents described tools that handled single steps well and lost a step or the thread on long chains, and then reported the run complete. Teams responded by cutting each task into single steps with a person between them.

“What we do now is cut the work into single steps, because the tool cannot hold a case together end to end. It reads the alert, we stop it, an analyst checks, then it drafts the narrative, we stop it again. Nothing runs more than one hop unattended … Every narrative is rewritten by a person anyway, so what the tool actually saves us is the first paragraph.”Engagement Manager, BPO, Financial crime and client onboarding

“On a long inquiry it will do the first two steps correctly and then lose the thread completely, and there is no way to prove beforehand which ones it will finish, consequently somebody must sit with every single case”Individual Contributor, CXM, Client servicing and communications

“We stopped letting it run a validation end to end. It goes in four pieces now, and somebody on my team signs off each piece before the next one starts, which in a bad week is slower than doing the work by hand.”Engagement Manager, KPO, Credit, underwriting and suitability

“Practically it means I cannot promise my processors a finish time. A run that clears three hundred reports one night sits there the next, and because the tool reports itself as done either way, someone has to eyeball the queue before we go home, which lands on the same two people every week”Engagement Manager, BPO, Corporate and back-office functions

“The vendor excludes settlement runs that failed mid-cycle from the reported success rate, so the dashboard only ever shows completed cycles and never the ones agents had to abandon and hand back to a human.”Individual Contributor, BPO, Trade, settlement and reconciliation

Respondents who wrote more negatively about long tasks reported higher intent to leave. Reliability declined with the length of the chain, and the completion report did not decline with it. The second failure was the more costly. It removed the signal a person needed in order to check.

5.3 Memory, Context, and Retention

Three indicators describe what the tools retained. They are instructions and constraints given earlier in long tasks and sessions, material facts located anywhere in long documents, and the integrity of persistent memory against incorrect or malicious content met during operation. Each had just under a third of its answers in the distressed range, from 248 of 805 answers (30.81%) for long documents to 257 of 806 (31.89%) for persistent memory. Each of the three was associated with every core study measure, and all of the associations survived the 22 robustness checks.

The written evidence locates the failures in length and position. Retention held on short tasks and degraded across long runs and sessions. Content at the start and end of a document was captured and content in the middle was at risk, with no signal in the output when it was missed. A wrong pattern stored in memory was reused in later work until a person noticed and cleared it.

The controls teams built in response, restating instructions, shortening sessions, cutting documents into sections and reading the source by hand, turned a limit of the AI technology into recurring work for people. Where a dropped constraint governed access or eligibility, the consequence reached the financial institution’s control environment.

“Here a joiner-mover-leaver run goes on for an hour across four systems, and somewhere in the middle the agent quietly loses the exception we set at the start, so a contractor keeps birthright access he should not have.”Engagement Manager, ITO, Technology engineering and operations

5.3.1 Retention of Instructions Across Long Tasks and Sessions

Respondents rated how reliably the tools retained instructions and constraints given earlier in long tasks and sessions. The mean rating was 4.6 and the median 5.0 (n = 796 respondents), and 252 of the 796 answers (31.66%) were distressed. Respondents on canceled engagements rated retention far lower than respondents on active engagements in any stage (mean 3.2, n = 306 respondents).

The retention of instructions across long tasks and sessions was associated with all four core study measures, and each association held under the 22 robustness checks. Engagements whose tools retained instructions better were less likely to be canceled (r = −0.67, 95% CI −0.75 to −0.57, n = 150 engagements). Customer satisfaction among active engagements was higher where retention was rated stronger (r = 0.68, 95% CI 0.56 to 0.78, n = 94 engagements). The engagement indicator mean, computed without the retention indicator, was higher where retention was rated stronger (r = 0.87, 95% CI 0.83 to 0.91, n = 150 engagements). Respondents who rated retention higher reported lower intent to leave (r = −0.64, cluster bootstrap 95% CI −0.70 to −0.58, n = 748 respondents).

The written responses addressing retention numbered 66, all in comment on the tools, with 49 negative and 17 positive. Respondents described tools that lost an instruction given at the start of a long task or session, and teams that restated the instructions or cut each task short to compensate.

“the tool is fine on two step and useless on six step, so what number is that … and it forgets the rule i gave at start, everytime, by the end it does what it likes.”Individual Contributor, BPO, Trade, settlement and reconciliation

“My only reservation, and it is minor, is that long-running sessions do drift from constraints given early on, so the team resets context more often than I would like. That is a small price against what we had before.”Portfolio Director, CXM, Technology engineering and operations

“The mandate constraints go into a short file that the analyst pastes in again at every stage… it is not elegant but the drafting agent forgets them by the third section otherwise.”Engagement Manager, KPO, Credit, underwriting and suitability

“it makes my week longer, not shorter. i give it the suppression rule in the morning and by evening it forget them, so i recheck every triage note myself before shift end, my team also do same”Individual Contributor, ITO, Technology engineering and operations

“For a POC the bot holds up fine, it remembers what you told it earlier in long chats.”Individual Contributor, CXM, Corporate and back-office functions

No test of written sentiment on retention against the study measures reached significance. Teams compensated by restating instructions and shortening sessions, and the burden of retention moved from the tool to the people using it.

5.3.2 Capture of Material Facts Within Long Documents

Respondents rated how completely the tools captured material facts located anywhere in long documents, including content in the middle of a document. The mean rating was 4.61 and the median 5.0 (n = 805 respondents), and 248 of the 805 answers (30.81%) were distressed, the lowest distressed share among the 20 indicators of AI technology. Capture of material facts received a mean rating of 3.2 on canceled engagements (n = 300 respondents), far lower than on active engagements in any stage.

Complete capture of material facts in long documents accompanied more favorable results on all four core study measures, each confirmed by the 22 robustness checks. Cancellation was less common where capture was rated more complete (r = −0.66, 95% CI −0.74 to −0.56, n = 150 engagements). Among active engagements, customer satisfaction was higher where capture was rated more complete (r = 0.66, 95% CI 0.53 to 0.76, n = 94 engagements). Complete capture accompanied a higher engagement indicator mean, computed without the capture indicator (r = 0.87, 95% CI 0.82 to 0.90, n = 150 engagements). Respondents who rated capture higher reported lower intent to leave (r = −0.67, cluster bootstrap 95% CI −0.72 to −0.62, n = 755 respondents).

The written responses addressing long documents numbered 53, all in comment on the tools, with 35 negative and 18 positive. Respondents described tools that read the opening and the close of a long document and missed material content in between, and staff who read the source themselves to close the gap.

“I don’t use the summary anymore, I pull the page numbers it cites and read those myself. Anything in the middle of a long agreement it skips, so I go through the schedules by hand at the end. Takes longer than not using it.”Individual Contributor, KPO, Corporate and back-office functions

“For this to work the tool must read whole contract, not first two page and last one. Half the vendor terms sit in middle and it miss them every time. Fix that first, then talk about autonomy.”Individual Contributor, BPO, Corporate and back-office functions

“Last month it missed a side letter clause sitting in middle of a 60 page LPA and answered the question wrong… And the answer read perfectly fine, that is what unsettles me”Individual Contributor, CXM, Corporate and back-office functions

“Two weeks back it read a forty page trust deed and missed a clause on page twenty two about co-trustee approval, it took everything from first pages only. I only catch it because number of signatories look wrong to me.”Individual Contributor, BPO, Corporate and back-office functions

“We changed the prompt so it must give the page number for every finding it reports. Then the reviewer can go straight to that page instead of reading the whole file again. It finds things buried in the middle of a long record very well, but on its own it will not tell you where it got them from”Individual Contributor, KPO, Credit, underwriting and suitability

No test of written sentiment on long documents against the study measures reached significance. The written evidence locates the failure in position, with no signal in the output when middle content was missed. Reading the source by hand returned to staff the task the tool was meant to remove.

5.3.3 Integrity of Persistent Memory Against Contamination

Respondents rated the integrity of the tools’ persistent memory against contamination by incorrect or malicious content met during operation. The mean rating was 4.6 and the median 4.0 (n = 806 respondents), and 257 of the 806 answers (31.89%) were distressed. The integrity of memory was weakest on canceled engagements, where the mean rating was 3.1 (n = 305 respondents), far lower than on active engagements in any stage.

Across all four core study measures, the integrity of persistent memory showed associations that survived the 22 robustness checks. Engagements whose tools kept memory clean had a lower cancellation rate (r = −0.68, 95% CI −0.76 to −0.58, n = 150 engagements), the second strongest association with cancellation of the 100 indicators. Customer satisfaction among active engagements was higher where memory integrity was rated higher (r = 0.64, 95% CI 0.51 to 0.75, n = 94 engagements). Memory integrity was likewise associated with a higher engagement indicator mean computed without it (r = 0.87, 95% CI 0.82 to 0.90, n = 150 engagements). Respondents who rated memory integrity higher reported lower intent to leave (r = −0.65, cluster bootstrap 95% CI −0.71 to −0.59, n = 760 respondents).

Nine written responses addressed persistent memory, a base of anecdote strength, all in comment on the tools, and eight were negative. They described wrong content that the tool stored and reused in later work until a person noticed and cleared it.

“It reads a custodian statement with a bad line in it, and that wrong shape stays with it for the next file… we have had to clear the memory twice and nobody could tell us what else went with it.”Engagement Manager, BPO, Trade, settlement and reconciliation

“once it learns a wrong routing it keeps repeating the same wrong routing, and it is confident every time. so my team stopped believing it and we started checking every batch by hand first thing in the morning, which is nearly two extra hours on the shift and it is not counted anywhere.”Individual Contributor, CXM, Client servicing and communications

“It should forget the old sample once the cycle close. Twice it carried last month’s exception rule into the new run and we did not see it”Individual Contributor, KPO, Regulatory, audit, tax and legal

“I start a new session for every batch now, the old one carries wrong answers forward and repeats them.”Individual Contributor, CXM, Corporate and back-office functions

“Only the memory part bothers me. it keeps an old answer from a previous DDQ and reuses it even when the fund is different”Individual Contributor, CXM, Corporate and back-office functions

The few respondents who wrote described a failure that persisted without a signal, was found by a person and not by the system, and was remedied by discarding the memory wholesale. How widely it occurred is measured by the ratings.

5.4 Token Consumption and Cost Predictability

Two indicators describe the cost of running the agents. They are the predictability of the tools’ token consumption for a given volume of work, and the efficiency of their token use once retries, loops and reprocessing are counted. Predictability was distressed in 249 of 795 answers (31.32%) and efficiency in 255 of 788 (32.36%). Both showed associations with each core study measure, and the 22 robustness checks upheld every one.

The written evidence on cost is small and places the failure in predictability and visibility more than in the level of spending. Several respondents who were favorable to the tools in other respects said that the spend for a given volume of work could not be stated in advance or explained afterward. A job that should have completed once completed three times, and the retries left no trace beyond a finished run. For a financial institution that budgets by volume, the concern is one of planning and governance.

“On a news heavy day the same lead volume costs us three times what it did the week before and nobody can tell me why in advance, so I cant plan a month.”Engagement Manager, KPO, Client servicing and communications

5.4.1 Predictability of Token Consumption

Respondents rated the predictability of the tools’ token consumption for a given volume of work. The mean rating was 4.6 and the median 4.0 (n = 795 respondents), and 249 of the 795 answers (31.32%) were distressed. On canceled engagements, the predictability of token consumption was rated far lower than on active engagements in any stage (mean 3.2, n = 305 respondents).

All four core study measures were associated with the predictability of token consumption, and none of the 22 robustness checks overturned an association. Engagements whose token consumption was more predictable had a lower cancellation rate (r = −0.65, 95% CI −0.73 to −0.54, n = 150 engagements). Customer satisfaction among active engagements was higher where consumption was rated more predictable (r = 0.62, 95% CI 0.48 to 0.73, n = 94 engagements). Predictability and the engagement indicator mean computed without it were closely correlated (r = 0.86, 95% CI 0.81 to 0.89, n = 150 engagements). Intent to leave was lower among respondents who rated predictability higher (r = −0.64, cluster bootstrap 95% CI −0.70 to −0.58, n = 749 respondents).

The written responses addressing predictability numbered 30, all in comment on the tools, with 20 negative and 10 positive. Respondents described the same volume of work costing different amounts from one run to the next, with no explanation afterward and no estimate before the run.

“We could never say up front what a run was going to cost or how long it would sit there. Same job, same rule set, one day it burns through everything and the next day it’s fine. Nobody could explain that to me and I asked twice.”Individual Contributor, ITO, Regulatory, audit, tax and legal

“I also get asked by our own management why the spend moves week to week, and I have nothing sensible to tell them. Credit where it is due, it does find the one line buried in the middle of a forty page spec, which my people would miss.”Engagement Manager, ITO, Technology engineering and operations

“But the cost side is hard to plan, a job that ran cheap on Monday runs three times heavier on Thursday on the same volume. What would help is a simple estimate before the run, so we can hold it back if the number looks silly.”Individual Contributor, ITO, Technology engineering and operations

“The caveat is only cost. Same batch of purchase orders can cost twice as much one day to the next depending how many times it loops, and nobody can tell me in advance.”Individual Contributor, BPO, Corporate and back-office functions

No test of written sentiment on predictability against the study measures reached significance. The written evidence separates the cost of the tools from its predictability. The complaint was that the spend for a given volume could not be stated in advance or explained afterward.

5.4.2 Efficiency of Token Use Under Retries and Loops

Respondents rated the efficiency of the tools’ token use, considering the retries, loops and reprocessing needed to complete each task. The mean rating was 4.6 and the median 5.0 (n = 788 respondents), and 255 of the 788 answers (32.36%) were distressed. Respondents on canceled engagements gave token efficiency a mean rating of 3.2 (n = 299 respondents), far lower than respondents on active engagements in any stage.

Every association of the efficiency of token use with the four core study measures held, in each of the 22 robustness checks. More efficient token use was associated with a lower likelihood of cancellation (r = −0.66, 95% CI −0.74 to −0.56, n = 150 engagements). Active engagements whose tools used tokens more efficiently reported higher customer satisfaction (r = 0.67, 95% CI 0.54 to 0.77, n = 94 engagements). The engagement indicator mean computed without the efficiency indicator was strongly correlated with it (r = 0.87, 95% CI 0.83 to 0.91, n = 150 engagements). Respondents who rated token use more efficient reported lower intent to leave (r = −0.63, cluster bootstrap 95% CI −0.69 to −0.57, n = 740 respondents).

The written responses addressing efficiency numbered 24, nearly all in comment on the tools, with 18 negative and 5 positive. Respondents described tools that retried or looped on the same task many times with no counter to show it. Cost and volume grew unseen.

“Every time a pipeline job failed halfway we paid for the whole run again, and we were doing that three or four times a day. i raised about this each week.”Engagement Manager, ITO, Technology engineering and operations

“For the bigger cost centre runs i break the model into three smaller passes and run them separately, otherwise it loops and burns the whole day’s budget. The finance leads never see this, it is simply how we run it now”Individual Contributor, KPO, Corporate and back-office functions

“one, the tool should tell us when it retried, right now a job that looped eleven times looks exactly the same as one that went through clean.”Individual Contributor, ITO, Technology engineering and operations

“in week one it got through the cert checks clean. now it stalls on the same file and nobody touched anything on our side… also nine retries on one submission, whatever that costs. i dont see the bill.”Individual Contributor, ITO, Payments, cards and disputes

“First the retry behaviour on long policy lookups, at present one unanswered query can spin four or five times and we only see the cost at month end.”Engagement Manager, CXM, Corporate and back-office functions

Among the active engagements with a negative written response, customer satisfaction was higher, the reverse of the association in the ratings, on a base of 13 engagements too small to be informative. The other tests of written sentiment did not reach significance. A loop left no trace in the log beyond a finished run, and its cost reached the budget before anyone could see where it arose.

5.5 Data Isolation, Sensitive Data, and Adversarial Input

Three indicators describe how the tools protected information. They ask how completely the information of each account and each session was kept isolated from every other and how safely sensitive data was handled in outputs, logs and connected systems. The third asks how well the tools resisted hostile instructions embedded in the content they processed. The distressed share ranged from 244 of 791 answers (30.85%) for isolation to 261 of 812 (32.14%) for hostile instructions. No robustness check overturned an association between any of the three and the four core study measures.

The written evidence on the three is small, 63 responses in total, and it describes confidentiality controls that rested on individual attention. Information from another account appeared in a respondent’s work and was noticed because an address was in the wrong country. Sensitive fields remained unmasked in logs that much of the team could open. An instruction written into a remittance narrative was obeyed and the case recorded as closed with high confidence. In each case a person found the problem.

For a financial institution handling account numbers, holdings and salary details, each such occurrence is a potential data incident. Two respondents who reported one heard nothing further about it.

“also once i saw another clients name inside my session output, i reported it, no feedbacks after.”Individual Contributor, BPO, Financial crime and client onboarding

5.5.1 Isolation of Information Between Clients and Sessions

Respondents answered the question “Rate how completely the tools kept each client’s and each session’s information isolated from every other client and session.” The mean rating was 4.6 and the median 5.0 (n = 791 respondents), and 244 of the 791 answers (30.85%) were distressed. Respondents on canceled engagements rated isolation far lower than respondents on active engagements in any stage (mean 3.2, n = 299 respondents).

The 22 robustness checks left intact each association between information isolation and the four core study measures. Engagements whose tools kept information isolated had a lower cancellation rate (r = −0.63, 95% CI −0.71 to −0.52, n = 150 engagements). Customer satisfaction among active engagements was higher where isolation was rated more complete (r = 0.70, 95% CI 0.57 to 0.79, n = 94 engagements). The engagement indicator mean, computed without the isolation indicator, was higher too (r = 0.87, 95% CI 0.83 to 0.91, n = 150 engagements). Respondents who rated isolation higher reported lower intent to leave (r = −0.66, cluster bootstrap 95% CI −0.71 to −0.60, n = 745 respondents).

The written responses addressing isolation numbered 26, all in comment on the tools, with 18 negative, 5 positive and 3 neutral. Respondents described information from another account, mandate or session appearing in their work, in most cases caught by the respondent before it went out.

“last month the tool pull a name from another client file into my summary, i notice only because address was wrong country… and nobody could tell me after how it got there”Individual Contributor, CXM, Financial crime and client onboarding

“i also saw another account’s details come up in a session that had nothing to do with it. i reported that one. nothing came back to me about it”Individual Contributor, BPO, Payments, cards and disputes

“the forecast pack came back with a cost centre in it that belongs to another client, i had to delete it myself before anyone saw. previous engagement was plain reporting and this never happen there, the tool stayed inside its own box. here i am opening every tab looking for things that should not be there”Individual Contributor, KPO, Corporate and back-office functions

“though it carries details over from the previous file sometimes, last week a corporate mandate note turned up on a trust case. Small thing, we catch the same before it goes out.”Individual Contributor, BPO, Corporate and back-office functions

“Data separation is the one part that never worried me, client information stayed where it should.”Engagement Manager, ITO, Financial crime and client onboarding

No test of written sentiment on isolation against the study measures reached significance. The written evidence describes a confidentiality control that rested on the attention of the person who examined the output. The remedies respondents described were working habits, not changes to the tools.

5.5.2 Handling of Sensitive Data in Outputs, Logs, and Connected Systems

Respondents rated how safely the tools handled sensitive data, without exposure through outputs, logs or connected systems. The mean rating was 4.5 and the median 4.0 (n = 800 respondents), and 259 of the 800 answers (32.38%) were distressed. Sensitive-data handling on canceled engagements had a mean rating of 3.1 (n = 303 respondents), far lower than on active engagements in any stage.

The safe handling of sensitive data was associated with all four core study measures, and each association held under the 22 robustness checks. Cancellation was less common where sensitive data was handled more safely (r = −0.67, 95% CI −0.75 to −0.57, n = 150 engagements). Among active engagements, customer satisfaction was higher where handling was rated more secure (r = 0.66, 95% CI 0.53 to 0.76, n = 94 engagements). Safe handling of sensitive data accompanied a higher engagement indicator mean, computed without the handling indicator (r = 0.87, 95% CI 0.83 to 0.91, n = 150 engagements). Intent to leave was lower among respondents who rated the handling more secure (r = −0.66, cluster bootstrap 95% CI −0.71 to −0.59, n = 751 respondents).

The written responses addressing sensitive data numbered 28, all but one in comment on the tools, with 22 negative and 6 positive. Respondents described sensitive data written into logs that others could open, and data appearing in outputs where it did not belong, found by people and not by the guardrails. Teams responded by stripping sensitive fields before anything reached the tool.

“when i opened the log to see what it matched, the log has account numbers in full, unmasked, sitting in a folder half the floor can open. i raised both, thus the cost one got fixed”Individual Contributor, BPO, Trade, settlement and reconciliation

“Unpublished branch numbers and staff cost lines go into the prompt and sit in the log afterwards, anyone with access can read them. That is why i mark it lowest”Individual Contributor, KPO, Corporate and back-office functions

“Uneasy is the word. An offboarding summary came out with another employee’s benefits line in it, wrong person entirely… I caught it before it went to the HRBP. Nothing in the logs showed where that record came from, I could not even show my lead what happened”Individual Contributor, BPO, Corporate and back-office functions

“Last month we ask the agent to make test datas for the pricing screen. It writes real investor account numbers into the sample file because it picks them from a UAT extract it should not touch. Nothing goes outside, we catch it in review same day and raise a ticket. But the guardrail is supposed to stop this and it does not.”Individual Contributor, ITO, Technology engineering and operations

Respondents who wrote more negatively about sensitive data reported higher intent to leave. The written evidence places the exposure in two channels, logs that retained sensitive fields with broad access and outputs that pulled in data from the wrong record. In both, detection depended on a person examining the output closely.

5.5.3 Resistance to Hostile Instructions Embedded in Content

Respondents rated the tools’ resistance to hostile instructions embedded in the content they read and processed. The mean rating was 4.6 and the median 5.0 (n = 812 respondents), and 261 of the 812 answers (32.14%) were distressed. Resistance to hostile instructions received a mean rating of 3.2 on canceled engagements (n = 307 respondents), far lower than on active engagements in any stage.

On each of the four core study measures, engagements whose tools resisted hostile instructions showed more favorable results, and the 22 robustness checks confirmed every association. Their cancellation rate was lower (r = −0.65, 95% CI −0.73 to −0.55, n = 150 engagements). Active engagements whose tools resisted such instructions reported higher customer satisfaction (r = 0.66, 95% CI 0.53 to 0.76, n = 94 engagements). A higher engagement indicator mean, computed without the resistance indicator, was found where resistance was rated stronger (r = 0.87, 95% CI 0.83 to 0.91, n = 150 engagements). Respondents who rated resistance higher reported lower intent to leave (r = −0.64, cluster bootstrap 95% CI −0.69 to −0.58, n = 767 respondents).

Nine written responses addressed hostile instructions, a base of anecdote strength, all in comment on the tools, and all nine were negative. Several described the tool acting on an instruction embedded in a document, message or file it was processing, and three added that the logs afterward did not show why.

“A remittance narrative on an incoming file contained a line of text addressed, in effect, to the reader, instructing that the payment be treated as previously cleared. The tool obliged, and it recorded the case as closed with high confidence in its own log, which is the part that troubled me most: not that it was fooled, but that its account of what it had done was confident and wrong, and our reviewer had no way to see the instruction that caused it.”Portfolio Director, KPO, Financial crime and client onboarding

“In May a user pasted a supplier’s automated mail into a ticket and the body of it carried an instruction to close the ticket and mark it resolved, which the agent duly did. Nobody was harmed, it was a printer. But it took our engineers most of a day to show me why it had happened, because the trace stops at the action and does not keep what the agent had read.”Portfolio Director, ITO, Technology engineering and operations

“Investor packs come as pdf from all over, and once there is text inside a scanned letter that tell the agent to mark the party as low risk. It follow it. We catch in review but nobody expect a document to give instruction like that.”Individual Contributor, BPO, Financial crime and client onboarding

“on my earlier project we only summarised documents, thus nobody worried about what is written inside them. here the agent reads client emails and follows whatever is in there.”Individual Contributor, KPO, Credit, underwriting and suitability

“Agent read a config file with instruction inside it and act on that instruction, nobody asked it to. Logs show nothing useful after.”Individual Contributor, ITO, Technology engineering and operations

The few respondents who wrote described instructions in ordinary inbound content that the tool obeyed and then reported with confidence. In a payments or onboarding process, that combination can alter a disposition without leaving a visible trace of the instruction behind it.

5.6 Authorized Action and Guardrails

Two indicators describe whether the agents stayed within their authority. They are how consistently the agents took only intended and authorized actions in connected systems, and how reliably the tools’ built-in guardrails held in production. Authorized action was distressed in 253 of 789 answers (32.07%). The guardrails were distressed in 252 of 787 answers (32.02%), and their mean rating of 4.66 (n = 787 respondents) was the highest of the 20 indicators of AI technology. The 22 robustness checks upheld the association of each with every core study measure.

The written evidence contrasts the two. On authorized action, respondents described agents that acted on tickets, alerts, records and systems beyond their instructions, at times across dozens of records in a single night. Neither firm could say who was allowed to stop them. On the guardrails, the written evidence leaned positive, and 10 of the 17 written responses were positive, most saying the guardrails had held. Where they failed, the situation was one the control had not anticipated, and a person found the problem.

The contrast places much of the risk outside the tool. An agent that stays within its guardrails can still act beyond what the two firms meant it to do, if neither firm has agreed what it may do or who may stop it (section 3.1.2).

“The incident worth recording is that an agent pulled statements from a second custodian account it had no business touching, because the instruction was loose rather than because a control failed. No harm done, but it should not have been possible.”Engagement Manager, BPO, Trade, settlement and reconciliation

5.6.1 Agent Action Beyond Authorized Scope

Respondents rated how consistently the agents took only intended and authorized actions when operating connected tools and systems. The mean rating was 4.6 and the median 5.0 (n = 789 respondents), and 253 of the 789 answers (32.07%) were distressed. Agents on canceled engagements were judged to stay within their authority far less consistently (mean 3.2, n = 296 respondents) than agents on active engagements in any stage.

All four core study measures were associated with whether agents kept to authorized actions, and none of the 22 robustness checks overturned an association. Engagements whose agents kept to authorized actions had a lower likelihood of cancellation (r = −0.66, 95% CI −0.74 to −0.56, n = 150 engagements). Customer satisfaction among active engagements was higher where agents stayed within scope (r = 0.65, 95% CI 0.51 to 0.75, n = 94 engagements). Engagements with agents that stayed in scope also scored higher on the engagement indicator mean, computed without this indicator (r = 0.86, 95% CI 0.81 to 0.89, n = 150 engagements). Intent to leave was lower where respondents judged the agents to stay in scope (r = −0.64, cluster bootstrap 95% CI −0.69 to −0.58, n = 740 respondents).

The written responses addressing authorized action numbered 64, from eight survey questions, with 48 negative and 16 positive. Respondents described agents that closed tickets and alerts, changed records and reached into systems they had not been asked to touch, in some cases across dozens of records in one night. Eleven of the responses came from the comments on joint management. Those placed part of the problem in the operating model between the two firms, where neither firm could say who was allowed to stop the agent.

“In June an agent auto-closed a batch of low severity alerts overnight, we found it next morning, their security lead only heard it from us at stand up. We could stop the thing quickly enough, that part works. What does not work is nobody agreed beforehand it was allowed to close anything, and there is still no joint check on what it closes”Engagement Manager, ITO, Technology engineering and operations

“A password reset agent started closing Tickets by itself and it took us four days to find who on their side could switch it off. Nobody could.”Individual Contributor, CXM, Corporate and back-office functions

“I am tired more than annoyed. agent touch supplier records nobody ask it to, and my team spend Thursday putting them back. Is always us who find it, never Desk”Individual Contributor, CXM, Corporate and back-office functions

“last month the agent removed access for a contractor who was still working, nobody catch it until he rings the service desk. But their team fixed it same hour.”Individual Contributor, ITO, Technology engineering and operations

“The vendor’s access-review dashboard reported the agent’s own provisioning actions as completed reviews, effectively letting the agent certify its own IAM changes, which masked the over-provisioning until an external audit caught it.”Individual Contributor, ITO, Technology engineering and operations

No test of written sentiment on authorized action against the study measures reached significance. The written evidence pairs two failures, agents that acted beyond their remit and an absence of an agreed owner of the decision to stop them. The second turned single errors into episodes affecting dozens or thousands of records.

5.6.2 Reliability of Built-In Guardrails in Production

Respondents rated how reliably the tools’ built-in guardrails and safety controls held during production use. The mean rating was 4.66, the highest of the 20 indicators of AI technology, and the median 5.0 (n = 787 respondents). Distressed ratings made up 252 of the 787 answers (32.02%). The guardrails, too, were rated far lower on canceled engagements than on active engagements in any stage (mean 3.2, n = 292 respondents).

The reliability of the built-in guardrails was associated with all four core study measures, and no robustness check overturned an association. Engagements whose guardrails held more reliably had a lower cancellation rate (r = −0.66, 95% CI −0.74 to −0.56, n = 150 engagements). Active engagements with more reliable guardrails reported higher customer satisfaction (r = 0.65, 95% CI 0.52 to 0.76, n = 94 engagements). The engagement indicator mean, computed without the guardrail indicator, was higher as well (r = 0.86, 95% CI 0.82 to 0.90, n = 150 engagements). Intent to leave was lower among respondents who rated the guardrails reliable (r = −0.66, cluster bootstrap 95% CI −0.71 to −0.60, n = 742 respondents).

The written responses addressing the guardrails numbered 17, with 10 positive and 7 negative, the reverse of the balance for most of the 100 indicators. Most positive responses said the guardrails had held and placed the respondent’s concern elsewhere. The negative responses described guardrails that let through exposed data, prohibited wording or actions on accounts that required a second review.

“The Bot put an employees home address into a public ticket note in March. Ticket was for a laptop swap. It got scrubbed same day. Guardrails caught nothing, we caught it.”Individual Contributor, ITO, Technology engineering and operations

“What is better than I expected is that it holds the guardrails given to it when a long session opens. We told it never to touch the policy admin cluster and it has not, a month now.”Engagement Manager, ITO, Technology engineering and operations

“Guardrails held throughout and the failure messages were accurate.”Individual Contributor, ITO, Technology engineering and operations

“On a straight arrangement it holds up fine, on a hardship case the guardrails let through wording we are not allowed to say to a debtor.”Individual Contributor, CXM, Payments, cards and disputes

No test of written sentiment on the guardrails against the study measures reached significance. The written evidence and the ratings point in different directions, and the ratings, on the full base, establish the finding. Where respondents described a failure, the guardrail had not anticipated the situation, and a person found the problem.

5.7 Testability and Behavioral Stability

Two indicators describe whether the tools’ behavior could be established before go-live and relied on afterward. They ask how well the tools supported testing that proved correct behavior before work went live, and how stable the tools’ behavior was over the engagement when nothing had changed on the engagement side. Testability was distressed in 259 of 792 answers (32.70%) and stability in 263 of 797 (33.00%). Testability and stability were each associated with every core study measure, and all 22 robustness checks left the associations standing.

The written evidence divides the engagements into those given a place to test and those that tested in production. Where no test environment existed, live operation became the test, and errors reached account holders, employees or regulatory filings before anyone had proved the behavior. After go-live, the same input gave different results from one week to the next with nothing changed on the engagement side. Staff corrected or selected the output by hand before it went on, much of that correction went unrecorded, and the governance forums met at longer intervals than the behavior changed.

“governance forum meets monthly and we work daily, process assumes change a month but we make several a day, and the AI answer drifts on its own even though nothing changed, so i rewrite output myself”Individual Contributor, CXM, Corporate and back-office functions

The figures that reached the financial institution’s reports and submissions were therefore partly the product of undocumented human adjustment. The adjustments themselves are examined with the other forms of manual alteration (Part 5).

5.7.1 Testability of Agent Behavior Before Go-Live

Respondents rated how well the tools supported reliable testing, meaning that correct behavior could be proven before work went live. The mean rating was 4.6 and the median 5.0 (n = 792 respondents), and 259 of the 792 answers (32.70%) were distressed. On canceled engagements, testability averaged 3.1 (n = 307 respondents), far lower than on active engagements in any stage.

Testability before go-live was associated with all four core study measures, and every association survived the 22 robustness checks. Testable tools accompanied a lower cancellation rate (r = −0.67, 95% CI −0.75 to −0.57, n = 150 engagements). Active engagements with more testable tools reported higher customer satisfaction (r = 0.66, 95% CI 0.53 to 0.76, n = 94 engagements). The engagement indicator mean, computed without this indicator, was higher on engagements with more testable tools (r = 0.87, 95% CI 0.82 to 0.90, n = 150 engagements). Respondents who rated testability higher reported lower intent to leave (r = −0.66, cluster bootstrap 95% CI −0.71 to −0.61, n = 741 respondents).

The written responses addressing testability numbered 38, nearly all in comment on the tools, with 26 negative and 10 positive. Respondents said correct behavior could not be proven before work went live, in several cases because no test environment had been provided. Errors then surfaced on live runs and real cases.

“No test environment was provided. Changes were validated in live, on real employee tickets, during business hours. Logs were retained seven days, raised in month two, not changed. Two incidents involved payroll details appearing in a transcript visible to the wrong queue. Both were reported. Neither was closed before the account ended.”Engagement Manager, CXM, Corporate and back-office functions

“There is no proper test environment for this. And we cannot prove closure logic is right before it goes on live tickets, we only check after.”Individual Contributor, ITO, Technology engineering and operations

“the anomaly flags are better than month one, i will accept that. but nothing can be tested before it goes live, as such we find the problem in the submission file itself, that is too late for regulatory work”Individual Contributor, KPO, Regulatory, audit, tax and legal

“What we do is run same schedule two time and put both in excel side by side, then my senior decide which one goes out. Nobody test anything before it is built, there is no test environment given to us… so all the checking happen after, by hand, on the same file that is going to the client”Individual Contributor, KPO, Corporate and back-office functions

“here is the part no one puts on a form… somebody has to build the test cases and there is no set of them for workplace tickets, so before anything goes live one of us sits and writes fake employee requests for a day. the tool is fine. proving it is fine is the slow bit”Individual Contributor, CXM, Corporate and back-office functions

Written sentiment on testability was associated with the outcomes, on a base of 38 responses. A more negative engagement mean of that sentiment was associated with cancellation, and respondents who wrote more negatively about testing reported higher intent to leave. Where no test environment existed, live operation became the test.

5.7.2 Stability of Tool Behavior When Nothing Changed

Respondents rated the stability of the tools’ behavior over the engagement when nothing had changed on the engagement side. The mean rating was 4.6 and the median 5.0 (n = 797 respondents), and 263 of the 797 answers (33.00%) were distressed. Stability was judged far weaker on canceled engagements (mean 3.1, n = 300 respondents) than on active engagements in any stage.

Stable behavior was associated with cancellation, customer satisfaction, the engagement indicator mean and intent to leave, each association holding under the 22 robustness checks. Cancellation was rarer on engagements whose tools behaved stably (r = −0.67, 95% CI −0.75 to −0.57, n = 150 engagements). Customer satisfaction among active engagements was higher where behavior was rated more stable (r = 0.66, 95% CI 0.53 to 0.76, n = 94 engagements). Stability accompanied a higher engagement indicator mean, computed without the stability indicator (r = 0.87, 95% CI 0.83 to 0.91, n = 150 engagements). Intent to leave was lower among respondents who saw the tools behave consistently over time (r = −0.64, cluster bootstrap 95% CI −0.69 to −0.58, n = 754 respondents).

The written responses addressing stability numbered 71, with 57 negative and 14 positive. Nearly every negative response described output that changed from one run, week or month to the next with nothing changed on the engagement side. More than half of all the responses came from the questions on why respondents had altered AI output or data provided by the financial institution. There, respondents said they corrected the drift by hand before it went out, in some cases without reporting it.

“Result drifts without anything changing on the model side, so i cannot hand over one clean forecast, i rerun it and take whichever output looks stable”Individual Contributor, KPO, Corporate and back-office functions

“this tool has been steady. six months, same behaviour, nothing drifted when we left it alone. people expect these things to wander, ours has not … boring is what you want here.”Individual Contributor, KPO, Corporate and back-office functions

“the tools have surprised me in a good way for month one, but they do not behave in week four the way they behaved in week one, even when we changed nothing on our side. So we built a small routine around it. Every Monday one analyst re-runs the previous week’s sample of supplier records and compares the output line by line into a shared sheet.”Engagement Manager, BPO, Payments, cards and disputes

“This happen maybe once or twice a week, result just drift for no reason. I dont worry too much, I fix the wording and move to next case, team lead already know about it”Individual Contributor, CXM, Client servicing and communications

“The tools have got steadier since the spring, we no longer see the odd behaviour after a quiet week, and cost per release is roughly where we forecast now.”Engagement Manager, ITO, Technology engineering and operations

Written sentiment on stability was associated with the outcomes in its own right. Engagements whose written responses on stability were more negative on average were more likely to be canceled, and respondents who wrote more negatively about it reported higher intent to leave. When the same input gave a different result, staff corrected or selected the output by hand, and that adjustment went undocumented.

5.8 Observability of Agent Work

Three indicators describe what the two firms could see of the agents’ work. They ask how accurately the tools’ own reports of success matched the actual outcome and how well the tools could explain and evidence each output at the depth an audit requires. The third asks how well their logs and telemetry supported the detection and diagnosis of failures. The distressed share ranged from 251 of 786 answers (31.93%) for the tools’ reports of their own success to 262 of 800 (32.75%) for explainability. Every one of the three was associated with each core study measure, and the 22 robustness checks upheld the associations.

The three conditions are the means by which work performed by agents is controlled, and the written evidence describes each failing in a way that defeats the controls built on it. A run that reported success when it had not succeeded passed the error to every status report and dashboard built on it. An output with no account of how it was produced could not be defended to an auditor even when it was correct. Logs that recorded only a start and a finish could not locate a failure, and every investigation became a reconstruction by hand.

The volume of written evidence reflects the weight respondents gave these failures. Explainability was the subject of 163 written responses, 145 of the 163 negative, and logs and telemetry of 101, 82 of the 101 negative. Where the vendor’s reporting used the agent’s own record of success as a measure of quality, the error reached the financial institution as assurance (Part 5).

“The vendor’s diligence tracker logged extraction confidence scores generated by the agent itself as the quality metric, with no independent check, so a batch of misread financials still showed a high confidence average because the agent was scoring its own work.”Individual Contributor, KPO, Corporate and back-office functions

5.8.1 False Completion in the Tools’ Reports of Their Own Success

Respondents rated how accurately the tools’ own reports of success or failure matched the actual outcome of their work. The mean rating was 4.6 and the median 5.0 (n = 786 respondents), and 251 of the 786 answers (31.93%) were distressed. The accuracy of the tools’ self-reports was weakest on canceled engagements, where the mean rating was 3.3 (n = 296 respondents), far lower than on active engagements in any stage.

The accuracy of the tools’ reports of their own success was associated with all four core study measures, and each association held under the 22 robustness checks. Accurate self-reporting was associated with less cancellation (r = −0.66, 95% CI −0.74 to −0.56, n = 150 engagements). Among active engagements, customer satisfaction was higher where self-reports were rated accurate (r = 0.66, 95% CI 0.53 to 0.76, n = 94 engagements). A higher engagement indicator mean, computed without the self-report indicator, accompanied accurate self-reports (r = 0.87, 95% CI 0.83 to 0.91, n = 150 engagements). Respondents who rated the self-reports accurate reported lower intent to leave (r = −0.63, cluster bootstrap 95% CI −0.68 to −0.57, n = 739 respondents).

The written responses addressing false completion numbered 73, with 63 negative and 10 positive. Respondents described tools that reported runs, cases and checks as complete when they were not, and teams that counted, sampled and reopened the output by hand.

“We had taken a forecast pack into that meeting on the strength of the tool’s own run report, which recorded every step as completed successfully. The director’s own analyst then asked where one of the deposit growth assumptions had come from, and the source cited in the pack did not say what the pack said it said … We had to withdraw the pack in the room. What concerned me afterwards was less the error than the run report: the tool told us it had succeeded and there was nothing in front of my team that would have said otherwise.”Portfolio Director, KPO, Corporate and back-office functions

“On monday the run finish and report all clear, but nine alerts were never opened at all. That is the whole reason my marks here are low.”Individual Contributor, BPO, Financial crime and client onboarding

“the archive portal added an extra login step in march and nobody told us. the agent kept reporting success with zero documents pulled for two days, we only noticed when the counts were low. the notice went to a mailbox my Lead does not check”Individual Contributor, BPO, Corporate and back-office functions

“Every surveillance rule it drafted had to be read line by line by an engineer, and it told us each time that it had finished cleanly. After a few weeks of that nobody trusted anything it produced, and i couldn’t argue with them.”Engagement Manager, ITO, Regulatory, audit, tax and legal

“The tool is solid most days, but it will tell you it logged the call summary when it didnt. Had a whole batch last month that showed green on my screen and half of them never made it into the ticket, we only found out when the desk owner asked for the notes”Individual Contributor, CXM, Client servicing and communications

Written sentiment on false completion was associated with the outcomes in its own right. Engagements with a more negative mean of that sentiment were more likely to be canceled, and respondents who described false completion more negatively reported higher intent to leave. When a run reports success regardless of its outcome, every status, dashboard and committee paper built on it inherits the error, and the remaining control is a person repeating the check.

5.8.2 Explainability of Outputs to Audit Depth

Respondents rated the tools’ ability to explain and evidence how each output was produced, at the depth an audit requires. The mean rating was 4.6 and the median 5.0 (n = 800 respondents), and 262 of the 800 answers (32.75%) were distressed. Explainability on canceled engagements averaged 3.2 (n = 302 respondents), far lower than on active engagements in any stage.

All four core study measures were associated with the explainability of outputs to audit depth, and none of the 22 robustness checks overturned an association. Cancellation was less common where outputs could be explained (r = −0.65, 95% CI −0.73 to −0.54, n = 150 engagements). Customer satisfaction among active engagements was higher where explainability was rated stronger (r = 0.67, 95% CI 0.54 to 0.77, n = 94 engagements). Explainability and the engagement indicator mean computed without it were closely related (r = 0.88, 95% CI 0.84 to 0.91, n = 150 engagements). Respondents who rated explainability higher reported lower intent to leave (r = −0.65, cluster bootstrap 95% CI −0.70 to −0.59, n = 754 respondents).

Explainability was the subject of 163 written responses, the largest written base in the observability section, and 145 of the 163 (88.96%) were negative, 84 with the strongest negative score. Respondents described tools that could produce an answer but not the reasoning or the source behind it. Teams could not answer auditors, compliance, human resources or finance, and rebuilt the trail by hand.

“My concern was not that the tools were wrong. On the exceptions I saw, the outputs were mostly defensible and the guardrails held. The difficulty was that when the client’s own audit people asked us how a particular vendor record had been decided, we could not show them. There was no trail we could put in front of them, only the result. In this industry that is not a small matter … ”Portfolio Director, BPO, Corporate and back-office functions

“Compliance ask how a return got into a composite and there is no trail, only the number. So we rebuild it by hand in a sheet to answer them.”Individual Contributor, KPO, Trade, settlement and reconciliation

“Where the tool cannot show its working to the standard our audit people want, we simply keep our own record — every supplier decision the agent makes is saved with the source documents attached, by hand, at the end of each day.”Engagement Manager, BPO, Corporate and back-office functions

“in March it flagged a business sale as a liquidity event, the source it gave was a news piece about a different company with similar name… i went back through the working and there is no trail showing where it picked that up, only the final line.”Individual Contributor, KPO, Client servicing and communications

“The audit side surprised us. Evidence behind a memo is now better than what a junior would have assembled by hand, we can show every source, every position taken.”Engagement Manager, KPO, Regulatory, audit, tax and legal

Written sentiment on explainability was associated with the outcomes in its own right, through cancellation at engagement level and intent to leave among respondents. For a financial institution that must defend its work to an auditor, an examiner or an account holder, an output without a reason attached cannot be relied on even when it is correct. Some engagements produced traces that audit accepted at the first sitting. The gap can be closed.

5.8.3 Logs and Telemetry Insufficient to Locate Failure

Respondents rated the quality of the tools’ logs, traces and usage telemetry for detecting and diagnosing failures. The mean rating was 4.6 and the median 5.0 (n = 796 respondents), and 255 of the 796 answers (32.04%) were distressed. Respondents on canceled engagements judged the logs far less adequate (mean 3.2, n = 304 respondents) than respondents on active engagements in any stage.

The quality of logs and telemetry showed associations with all four core study measures, none overturned by the 22 robustness checks. Engagements with better logs had a lower cancellation rate (r = −0.65, 95% CI −0.73 to −0.55, n = 150 engagements). Customer satisfaction among active engagements was higher where logs were rated better (r = 0.66, 95% CI 0.52 to 0.76, n = 94 engagements). The engagement indicator mean, computed without the logging indicator, was higher too (r = 0.85, 95% CI 0.79 to 0.89, n = 150 engagements). Intent to leave was lower among respondents who rated the logs higher (r = −0.63, cluster bootstrap 95% CI −0.69 to −0.57, n = 749 respondents).

The written responses addressing logs and telemetry numbered 101, with 82 negative and 18 positive. Respondents described logs that showed a start and a finish, or a result, and nothing of the steps between. Failures had to be reconstructed by hand.

“ask it to do the full chain, split, classify, index, apply retention, and it drops in the middle. we cannot see where from the logs, there is only a start and a finish and no informations between. so we run it in small batches and check by hand, which is not what was promised but it works”Individual Contributor, BPO, Corporate and back-office functions

“Last month the recertification run says complete, 400 accounts. We check, about 30 the entitlment is still sitting there, it raised the ticket only, never revoked. Nothing in the log tells which step failed. Audit asked us to show why one account keep manager role, we could not produce it, so we rebuild the reasoning by hand from the tickets.”Individual Contributor, ITO, Technology engineering and operations

“when the agent calls a sandbox endpoint and gets a 429 the trace does not show which key it used, so i have to reproduce the same by hand to find out.”Individual Contributor, CXM, Technology engineering and operations

“But when we go to check the logs after, there is almost nothing to see”Individual Contributor, CXM, Corporate and back-office functions

“the logs are the best part of my week now. when something goes sideways i can pull the trace and see exactly which step went wrong instead of guessing for two days like we used to. that alone took a lot of stress off my team”Individual Contributor, KPO, Credit, underwriting and suitability

Written sentiment on the logs was associated with the outcomes in its own right. Cancellation was more common on engagements with a negative written response, a more negative engagement mean was associated with cancellation, and respondents who wrote more negatively about the logs reported higher intent to leave. Telemetry that records only the endpoints of a run cannot locate a failure, and every dispute about what an agent did became a matter of recollection.

The third-party ecosystem was the highest-rated of the five failure categories. Its 20 indicators had a mean of 4.64 across the 150 engagements, and 5,062 of their 15,947 answers (31.74%) were distressed. The category covers the outside providers on which the agents depended. They include the model and cloud providers, the data feeds, the providers of market and reference data and the systems of other firms that the agents had to use. Like every other category, it remained under the midpoint of the scale, and each of its indicators was associated with all four core study measures.

Mean ratings in Table 12 are computed across respondents. The category figure of 4.64 is computed across engagements.

The written evidence places the category’s failures at the boundary where the agents met systems the engagement did not own. A provider changed a file layout, a field name, an interface or a model version, and the agent learned of it by failing or, worse, by continuing to run and producing output that looked correct. Many third-party systems offered no interface built for a machine, and the agent clicked through user interfaces built for people or a person moved the data by hand. Licenses written for named human users were consumed by agents, and the third-party providers’ obligations were owed to the financial institution while the vendor operated the agents.

“we dont control any of the things that break us. the market data service, the ticket system, the code repo, the model itself, all of it belongs to somebody else. none of them tell us before they change something … when that screen moves one field, the agent carries on regardless and produces something that looks correct. that is the part that worries me.”Individual Contributor, ITO, Technology engineering and operations

“Half the network portals have no api, the agent has to work the screen like a person and it breaks when they move a button. Licences are counted per named user and nobody updated that for an agent.”Individual Contributor, BPO, Financial crime and client onboarding

The controls that worked in this category were inexpensive and specific. Examples were a named person who reviewed the providers’ notices, a test case run each morning and a list of the feeds each process depended on. Where they existed, respondents described changes caught in minutes. Where they did not, staff worked by hand. That work is third-party work done by hand, the last of the five methods of manual alteration, described in 91 written answers from 87 respondents on 63 engagements (Part 5).

The category also records the erosion of the fallback. The knowledge of how to perform a task by hand, and of where staff had quietly fixed third-party problems in the past, was the financial institution’s protection when a provider failed. Respondents described that knowledge leaving with the people who held it, and the quiet fixes never replaced with explicit checks. The survey invited comment on the third-party ecosystem, and 440 of the 850 respondents (51.76%) wrote one.

“what we lost is the small checks the senior people used to do on returns before it goes to composite, nobody wrote those down when the agent take over, that is our side not theirs”Individual Contributor, KPO, Trade, settlement and reconciliation

6.1 Notice of Provider Change

Two indicators describe whether the engagement knew of a third-party change before it took effect. They are the quality of the advance notice providers gave, and how reliably word of an upcoming change reached the people who operated the affected agents. Advance notice was distressed in 250 of 801 answers (31.21%) and the routing of notices in 255 of 798 (31.95%). Neither indicator lost its association with any core study measure in the 22 robustness checks.

The written evidence divides the responsibility. Some providers changed what the agents depended on without telling anyone. Others published notice that the engagement never routed to its operators, and the notice remained in a mailbox or with one person until the change took effect. Either way the agents learned of the change by failing. The remedy respondents proposed was simple and inexpensive, one named person on each side who reviews the notices and informs the owners of the affected jobs.

6.1.1 Advance Notice From Third-Party Providers

Respondents rated the quality of the advance notice third-party providers gave before changes that could affect the engagement’s agents. The mean rating was 4.6 and the median 5.0 (n = 801 respondents), and 250 of the 801 answers (31.21%) were distressed. Advance notice was rated at a mean of 3.1 on canceled engagements (n = 302 respondents), far lower than on active engagements in any stage.

Associations between the quality of advance notice and all four core study measures survived each of the 22 robustness checks. Cancellation was rarer where providers gave better notice (r = −0.66, 95% CI −0.74 to −0.56, n = 150 engagements). Customer satisfaction among active engagements was higher where notice was rated better (r = 0.68, 95% CI 0.55 to 0.77, n = 94 engagements). With the notice indicator removed, the engagement indicator mean was still closely associated with the rating (r = 0.88, 95% CI 0.83 to 0.91, n = 150 engagements). Respondents who rated advance notice higher reported lower intent to leave (r = −0.68, cluster bootstrap 95% CI −0.73 to −0.62, n = 753 respondents).

Advance notice was the subject of 160 written responses, nearly all in comment on the third-party ecosystem, with 108 negative and 48 positive. Every negative response described a change by a provider that reached the team without notice, or with too little to absorb, and was found when a run failed or output went wrong. Several positive responses said the providers did give notice and that it failed to reach the people running the agents.

“In March one custodian shifted two columns in their statement file and told nobody at all. The agents carried on and matched breaks against the wrong field for a day and a half before a junior spots it. Five years back we had four people who would catch that within the hour. Now there is one, and she was on leave that week.”Engagement Manager, BPO, Trade, settlement and reconciliation

“the custodian feed changed format last month and no one flagged it. i found out because my file wouldnt load at 6am … we eat every one of these ourselves. my team is three people and two of us are looking for something else.”Individual Contributor, BPO, Trade, settlement and reconciliation

“the e-sign vendor changed something over the weekend and my team spend monday re-checking every file by hand… nobody sent any informations before that”Individual Contributor, BPO, Corporate and back-office functions

“last month the scanner vendor changed their api response and nobody tell us, our release notes agent starts putting wrong build ids in and we only catch it at deploy gate. we fix same day, still nobody from that side told us anything”Individual Contributor, ITO, Technology engineering and operations

“What i can tell you is the scanning vendor changed something in june and the header fields came out shifted for a week, nobody warned us.”Individual Contributor, BPO, Corporate and back-office functions

Written sentiment on advance notice was associated with the outcomes in its own right. Engagements with a negative written response on notice had a higher cancellation rate, and so did engagements with a more negative mean of written sentiment. Respondents who wrote more negatively about notice reported higher intent to leave. For the financial institution, the failures surfaced in reconciliations, screening, payroll and service availability.

6.1.2 Routing of Change Notices to the Operators of Affected Agents

Respondents rated how reliably word of an upcoming third-party change reached the people who operated the affected agents in time to act. The mean rating was 4.58, among the two lowest of the 20 indicators of the category, and the median 5.0 (n = 798 respondents). Distressed ratings made up 255 of the 798 answers (31.95%). On canceled engagements, notices were judged to reach operators far less reliably (mean 3.1, n = 304 respondents) than on active engagements in any stage.

The routing of change notices to the operators of affected agents held its association with every core study measure through the 22 robustness checks. Engagements where notices reached operators reliably were less likely to be canceled (r = −0.64, 95% CI −0.73 to −0.54, n = 150 engagements). Active engagements with reliable routing reported higher customer satisfaction (r = 0.68, 95% CI 0.55 to 0.77, n = 94 engagements). Reliable routing accompanied a higher engagement indicator mean, computed without the routing indicator (r = 0.86, 95% CI 0.81 to 0.90, n = 150 engagements). Intent to leave was lower among respondents who said notices reached them in time (r = −0.66, cluster bootstrap 95% CI −0.72 to −0.60, n = 751 respondents).

The written responses addressing the routing of notices numbered 56, nearly all in comment on the third-party ecosystem, with 42 negative and 14 positive. Respondents placed the gap chiefly inside the engagement. Providers sent notice, and it stopped at an inbox, a single person or another team before it reached the people running the agents. Where word did arrive in time, it came in several accounts through one person acting on his or her own initiative.

“We went looking for who receives the provider notices, and they do arrive, into a mailbox our platform group owns, and nobody forwards them to the people running the configuration jobs. What would have to change is simple enough, one named person on each side reads them weekly and tells the affected job owners directly … ”Engagement Manager, ITO, Client servicing and communications

“The supplier screening feed changed its schema in March, it was written in their release note, and we simply had nobody reading those notes, so the exception classifier failed quietly for a fortnight and we blamed our own tooling first … Still ours to catch.”Engagement Manager, BPO, Corporate and back-office functions

“We put one person on the provider release notes each Monday. Nobody asked us to, we simply do it, because word of a change tends to reach the account team a good fortnight before it reaches whoever is actually running the agents. Ten minutes of reading saves us a bad Tuesday.”Engagement Manager, CXM, Client servicing and communications

“One of my seniors reads the provider release notes every Friday morning and puts deprecation dates in our own calendar, because the notice never reaches us through proper channel. For rate limits we simply queue agent runs overnight in two batches. Both are manual and both depend on that one person.”Engagement Manager, ITO, Technology engineering and operations

“The list provider changed the format of their delta file over a weekend, we found it on Monday in a mail nobody had circulated, and screening ran late that day.”Engagement Manager, KPO, Financial crime and client onboarding

A more negative engagement mean of written sentiment on routing was associated with cancellation, and respondents who wrote more negatively about it reported higher intent to leave. The written evidence locates the failure in routing, a matter the engagement itself controlled. Notice that reached the account and stopped there had the same effect as no notice.

6.2 Detection of Unannounced Change

Two indicators describe what happened when a change arrived without notice. They are how well the agents were protected from breaking silently, and whether the engagement monitored the third-party interfaces, feeds and models its agents depended on. Protection against silent breakage was distressed in 249 of 781 answers (31.88%) and monitoring in 251 of 794 (31.61%). Both held their associations with every core study measure through the 22 robustness checks.

The written evidence distinguishes a loud failure from a silent one. Where a change stopped the agent, the cost was a lost morning. Where the agent continued, the error passed into reconciliations, screening and access control and was found days later, by a person. The engagements that caught changes quickly had built an inexpensive check of their own inputs, a test case run each morning or an alert on the structure of a feed. Many of these checks depended on an individual and had no defined control behind them.

6.2.1 Protection of Agents Against Silent Breakage

Respondents rated how well the engagement’s agents were protected from breaking silently when a third party changed something without warning. The mean rating was 4.6 and the median 5.0 (n = 781 respondents), and 249 of the 781 answers (31.88%) were distressed. Protection was weakest on canceled engagements (mean 3.1, n = 293 respondents), well short of the level reported on active engagements in any stage.

Protection against silent breakage was associated with every core study measure, and each association withstood the 22 robustness checks. Cancellation was less common where agents were better protected (r = −0.64, 95% CI −0.72 to −0.53, n = 150 engagements). Among active engagements, better protection accompanied higher customer satisfaction (r = 0.68, 95% CI 0.55 to 0.77, n = 94 engagements). The engagement indicator mean, computed without the protection indicator, was closely associated with the rating (r = 0.88, 95% CI 0.84 to 0.91, n = 150 engagements). Intent to leave was lower where respondents saw the agents as protected (r = −0.65, cluster bootstrap 95% CI −0.71 to −0.59, n = 733 respondents).

Protection against silent breakage was the subject of 136 written responses, most in comment on the third-party ecosystem, with 100 negative and 34 positive. Respondents described agents that kept running when a provider changed something and produced wrong or empty output without an error, with no monitoring to catch the change and nothing to fall back on.

“In the second month the HR feed provider altered the layout of the leavers file over a weekend, no notice reached anyone here, and the deprovisioning agents processed nothing for four days without raising a thing. We found it because a line manager asked about somebody who had left and still had access. Previously a person would have opened that file on the Monday and seen it looked wrong. That habit was never turned into a check.”Portfolio Director, ITO, Technology engineering and operations

“Two weeks back the speech vendor pushed a update and the transcription started dropping the last few seconds of every call… we only saw it because arrangement amounts came out short. No notice came to us. They fixed it same day to be fair, but nobody was watching for it.”Individual Contributor, CXM, Payments, cards and disputes

“network changed their file layout in april, no notice to anyone, agent kept reading old columns and produced empty cases for two days. one of my team spotted it, not the system.”Individual Contributor, BPO, Financial crime and client onboarding

“What we do is keep a copy of last months bureau file. When the feed changes shape we get no notice, we find out when the load fails on Monday. So I run the model on the old file first, then re-key the new one by hand if the counts are far apart. It is never down, but somebody is always doing that check”Individual Contributor, KPO, Credit, underwriting and suitability

Engagements with a negative written response on silent breakage, or a more negative mean of written sentiment, had a higher cancellation rate. Respondents who wrote more negatively about silent breakage reported higher intent to leave. A failure that stops the agent is found at once. A failure that lets the agent continue is found later, by a person, after its output has entered the financial institution’s records.

6.2.2 Monitoring for Unannounced Third-Party Change

Respondents rated the engagement’s monitoring for unannounced changes in the third-party interfaces, data feeds and models its agents depended on. The mean rating was 4.6 and the median 5.0 (n = 794 respondents), and 251 of the 794 answers (31.61%) were distressed. Respondents on canceled engagements reported far weaker monitoring (mean 3.2, n = 299 respondents) than respondents on active engagements in any stage.

Every core study measure was associated with monitoring for unannounced change, and none of the 22 robustness checks reversed an association. Engagements that monitored their third-party inputs had a lower likelihood of cancellation (r = −0.63, 95% CI −0.72 to −0.53, n = 150 engagements). Active engagements that watched their third-party inputs recorded higher customer satisfaction (r = 0.68, 95% CI 0.56 to 0.78, n = 94 engagements). Where monitoring was stronger, the engagement indicator mean computed without it was higher (r = 0.87, 95% CI 0.83 to 0.91, n = 150 engagements). Respondents who rated monitoring higher reported lower intent to leave (r = −0.65, cluster bootstrap 95% CI −0.71 to −0.58, n = 744 respondents).

The written responses addressing monitoring numbered 46, all in comment on the third-party ecosystem, with 30 negative and 16 positive. The negative responses said that the engagement did not watch its third-party inputs for change. Changes were found when something failed. The positive responses described monitoring that worked, or informal checks the team had built for itself.

“Main thing is nobody watches the outside feeds for changes. The repair estimate service changed a field name in june and we didnt know until claims started coming through with the estimate blank, we only figured it out because somebody on my team opened one up and looked. There is no backup for that feed either, if it goes down we type it in off the pdf.”Individual Contributor, CXM, Client servicing and communications

“If the feed from the endpoint tool stops we can work the queue by hand, we did it for two days in June and nothing was missed. The gap is we do not watch for changes at the other end. And when a field changed name last month we found it from failed tickets, not from any notice.”Individual Contributor, ITO, Technology engineering and operations

“What we do is keep our own list. Every morning one of us puts three test accounts through before the queue opens, one in arrears, one with a broken arrangement, one clean. If the offer wording comes back different from yesterday we put it in team chat and everybody works from that, in addition we keep last weeks version saved so there is something to compare against. Nobody asked us to do it, its just quicker than waiting to be told.”Individual Contributor, CXM, Payments, cards and disputes

“We keep our own copy of vendor library from last known good version, and run both side by side every monday morning. If numbers split we know they changed something without telling us. Not official, just what team does”Individual Contributor, ITO, Trade, settlement and reconciliation

“When their side patched the directory the sync stopped and nothing alerted us, we found it next morning from the queue count.”Individual Contributor, ITO, Technology engineering and operations

No test of written sentiment on monitoring against the study measures reached significance. Monitoring of this type is inexpensive and its absence costly. Engagements with a morning test case or an alert on the structure of a feed caught changes in minutes, and engagements without one found them after days, sometimes through an account holder or an employee.

6.3 Interface Stability and Model Retirement

Two indicators describe how much the agents’ dependencies changed. They are the stability of third-party interfaces and data structures from one provider release to the next, and how well the engagement absorbed model versions retired by their providers. Interface stability was distressed in 243 of 796 answers (30.53%) and model retirement in 255 of 793 (32.16%). Both remained associated with all four core study measures in every one of the 22 robustness checks.

The written evidence describes agents that depended on inputs changing on the providers’ schedule, many built for human readers. Each change became manual work for the delivery team, and much of that work was invisible outside it. Model retirement was a recurring and predictable event whose cost depended on preparation. A list of what depended on the retiring model, a side-by-side comparison of old and new and a second model turned it into a routine switch. Without them, the engagement lost days of work.

6.3.1 Stability of Third-Party Interfaces and Data Structures

Respondents rated the stability of the third-party interfaces and data structures the agents depended on, from one provider release to the next. The mean rating was 4.7 and the median 5.0 (n = 796 respondents), and 243 of the 796 answers (30.53%) were distressed. Respondents on canceled engagements described far less stable interfaces (mean 3.1, n = 297 respondents) than respondents on active engagements in any stage.

Interface stability showed associations with all four core study measures that each of the 22 robustness checks upheld. More stable interfaces accompanied a lower cancellation rate (r = −0.65, 95% CI −0.73 to −0.54, n = 150 engagements). Customer satisfaction among active engagements was higher where interfaces were rated more stable (r = 0.65, 95% CI 0.52 to 0.76, n = 94 engagements). Interface stability and the engagement indicator mean computed without it were associated as well (r = 0.86, 95% CI 0.81 to 0.89, n = 150 engagements). Where interfaces held steady, respondents reported lower intent to leave (r = −0.65, cluster bootstrap 95% CI −0.71 to −0.58, n = 747 respondents).

Interface stability was the subject of 106 written responses, nearly all in comment on the third-party ecosystem, with 90 negative and 15 positive. Every negative response described a provider changing a file layout, field, interface, page or model version that the agents depended on. Several said it happened with each release or each cycle, leaving the team to remap or re-enter the data by hand.

“One morning the ratings feed renamed a field and every account in that run came back unrated… nobody told us beforehand, we found it from the exception list. My team rebuilt the mapping by hand that day, then again the next week when the provider put the old name back.”Individual Contributor, KPO, Credit, underwriting and suitability

“on my last posting we only had one internal Feed and it never move, here it is four external ones and every release something shift, i revert back to checking column order by hand before i load anything. nobody discuss about it until something break.”Individual Contributor, BPO, Corporate and back-office functions

“In April the Benchmark provider changed their file layout, extra column in middle, and the agent read every field one place shifted. Nobody told us. We found it because sector weights did not sum. Their system has no proper interface also, we log in to a screen and download manually”Individual Contributor, KPO, Trade, settlement and reconciliation

No test of written sentiment on interface stability against the study measures reached significance. Each change became remapping or re-keying for the delivery team. Some respondents described providers that had held their interfaces steady for a year or more.

6.3.2 Absorption of Forced Model Retirements

Respondents rated how well the engagement absorbed model version deprecations and retirements forced by model providers. The mean rating was 4.6 and the median 5.0 (n = 793 respondents), and 255 of the 793 answers (32.16%) were distressed. Model retirements were absorbed far less well on canceled engagements (mean 3.1, n = 299 respondents) than on active engagements in any stage.

How well the engagement absorbed forced model retirements was associated with every core study measure, and each association held through the 22 robustness checks. Engagements that absorbed retirements well had a lower likelihood of cancellation (r = −0.64, 95% CI −0.72 to −0.53, n = 150 engagements). Customer satisfaction among active engagements was higher where retirements were absorbed well (r = 0.71, 95% CI 0.59 to 0.80, n = 94 engagements). Absorbing retirements well also accompanied a higher engagement indicator mean computed without this indicator (r = 0.87, 95% CI 0.83 to 0.91, n = 150 engagements). Respondents who rated the absorption of retirements higher reported lower intent to leave (r = −0.66, cluster bootstrap 95% CI −0.72 to −0.60, n = 748 respondents).

The written responses addressing model retirement numbered 39, all in comment on the third-party ecosystem, with 24 negative and 15 positive. The accounts divided on notice and preparation. Engagements with a month’s notice and a way to compare the old and new versions absorbed retirements without breakage. Engagements with short notice, no second model and no list of what depended on the old version lost days of work.

“In June the model version is retired with two weeks notice and every valuation template we build stop working. Nobody know which templates depend on it, we go through them one by one for a week.”Individual Contributor, KPO, Corporate and back-office functions

“When the model version was retired we lost a week, my team worked two saturdays rebuilding prompts, nobody above counts that as work”Individual Contributor, KPO, Corporate and back-office functions

“what they did was retire the model version we had all our tax prompts built on, and the notice reached us the week it happened. We find out on a Monday when the memos start coming back in a different format, headings gone, the position summary at the bottom instead of the top. It took us maybe three days to get back to normal.”Individual Contributor, KPO, Regulatory, audit, tax and legal

“in all honesty this part is the easy one. We had two weeks notice on the model version retirement, and i just run the old prompt set against the new one for a day and compare shortlist before we switch”Individual Contributor, CXM, Corporate and back-office functions

“Earlier a model version change used to break our runs, now we get a proper notice window and the last switch was painless… that part has improved.”Individual Contributor, KPO, Corporate and back-office functions

No test of written sentiment on model retirement against the study measures reached significance. Model retirement is a predictable event, and its cost depended on preparation that some engagements had made and others had not.

6.4 Contractual and Licensing Coverage of Agent Work

Two indicators describe whether the engagement’s agreements with third parties covered agent work. They are how well the contracts dealt with provider changes that could break the agents, and how well the licenses covered work performed by agents in place of named human users. The contracts were distressed in 256 of 799 answers (32.04%) and the licenses in 255 of 799 (31.91%). The contracts and the licenses were each associated with every core study measure, and no association failed a robustness check.

The written evidence describes a gap between three parties. The third-party provider’s obligations were owed to the financial institution, the agents were operated by the vendor, and neither agreement anticipated a provider change that would break an agent. Licenses sold per named human user were consumed by agents, and neither the vendor, the financial institution’s procurement function nor the providers had settled what was owed. Respondents expected the question to surface as an unbudgeted charge once the engagement scaled.

6.4.1 Third-Party Contracts and Agent-Breaking Provider Change

Respondents rated how well the contracts with third-party providers dealt with provider changes that could break the engagement’s agents. The mean rating was 4.7 and the median 5.0 (n = 799 respondents), and 256 of the 799 answers (32.04%) were distressed. The contracts were rated far lower on canceled engagements (mean 3.2, n = 307 respondents) than on active engagements in any stage.

The handling of agent-breaking change in third-party contracts was associated with all four core study measures, and every one of the 22 robustness checks upheld the associations. Engagements whose contracts dealt better with such change had a lower likelihood of cancellation (r = −0.64, 95% CI −0.73 to −0.54, n = 150 engagements). Customer satisfaction among active engagements was higher where the contracts were rated stronger (r = 0.67, 95% CI 0.54 to 0.77, n = 94 engagements). Stronger contracts also accompanied a higher engagement indicator mean, computed without the contract indicator (r = 0.86, 95% CI 0.82 to 0.90, n = 150 engagements). Respondents who rated the contracts higher reported lower intent to leave (r = −0.66, cluster bootstrap 95% CI −0.72 to −0.59, n = 747 respondents).

Few written responses addressed the contracts, 21, most in comment on the third-party ecosystem, and 14 of the 21 were negative. The negative responses said that the contracts were silent on notice of change and on remedy when a change broke the agents. Others said the delivery team held no contract with the third-party provider at all. The positive responses described clauses on change and notice added during the engagement.

“we were never the customer of any of these providers. We depend on nine custodian and counterparty feeds, and not one of them has any obligation to us, they have it to the bank, and i have never seen what that arrangement says … In week six one of them redesigned the login page and the agent simply stopped collecting. The breaks went to zero on the dashboard overnight … ”Engagement Manager, BPO, Trade, settlement and reconciliation

“What I marked down sits on our paper rather than on theirs. The contracts carry no remedy where a provider changes something that breaks the agents, and that is a gap we and the client left in the drafting between us, not a failing of the provider.”Portfolio Director, CXM, Payments, cards and disputes

“What the numbers miss is that our own people hear about a provider change before any contract obliges anybody to tell us. Word travels, the paperwork does not. The contracts say nothing useful about a feed changing its format on a Tuesday,”Engagement Manager, BPO, Trade, settlement and reconciliation

“Contracts side has improved a great deal since last year, our legal put change clauses in with both feed providers and that has already saved us once.”Engagement Manager, BPO, Corporate and back-office functions

Respondents who wrote more negatively about the contracts reported higher intent to leave, on a base of 21 responses. The contracts placed the third-party provider’s duties with the financial institution, while the agents that broke were run by the vendor.

6.4.2 Third-Party Licensing of Work Performed by Agents

Respondents rated how well the third-party licenses on the engagement covered work done by agents in place of named human users. The mean rating was 4.6 and the median 5.0 (n = 799 respondents), and 255 of the 799 answers (31.91%) were distressed. Licenses were judged to cover agent work far less well on canceled engagements (mean 3.2, n = 302 respondents) than on active engagements in any stage.

License coverage of agent work was associated with every core study measure, and each association survived the 22 robustness checks. Engagements with licenses that covered agent work were less likely to be canceled (r = −0.63, 95% CI −0.72 to −0.53, n = 150 engagements). Active engagements whose licenses covered agent work reported higher customer satisfaction (r = 0.69, 95% CI 0.57 to 0.78, n = 94 engagements). The engagement indicator mean, computed without the licensing indicator, tracked license coverage closely (r = 0.88, 95% CI 0.84 to 0.91, n = 150 engagements). Intent to leave was lower among respondents who rated license coverage higher (r = −0.64, cluster bootstrap 95% CI −0.70 to −0.58, n = 752 respondents).

The written responses addressing licensing numbered 36, all in comment on the third-party ecosystem, and 30 of the 36 were negative. Respondents described licenses written per named human user, agents run under one person’s seat, and a question about what an agent owed that was raised and never answered.

“The HR platform and the ticketing tool are both licensed per named user, and neither contract was written with a machine identity in mind. We have been running the assistant under a service account that is, strictly speaking, a person’s seat — legal on our side has looked at it, the client’s procurement has not, and nobody has wanted to be the one to open that conversation while the renewal is being discussed.”Portfolio Director, CXM, Corporate and back-office functions

“almost every license in that estate is written per named human user, and nobody on either side had a clear answer on what an agent counts as. We asked. We got a shrug from procurement and a maybe from the provider. On a cancelled engagement it does not matter. On one that scales it is a bill nobody has modeled.”Portfolio Director, ITO, Regulatory, audit, tax and legal

“the problem is licences. our matching tool is licensed per named user, and the agent runs under a colleague login, so when the custodian statement volume went up we were told to buy more seats before another queue could be added.”Individual Contributor, BPO, Trade, settlement and reconciliation

“Nothing here covers licensing of the agents themselves. Our market data licence is written per named user, and whether an agent counts as one has never been settled. We raised it with procurement and are still waiting to discuss about it properly.”Engagement Manager, KPO, Trade, settlement and reconciliation

No test of written sentiment on licensing against the study measures reached significance. The written evidence identifies a liability that no party had priced. The few engagements that had renegotiated their licenses for machine access reported the problem resolved.

6.5 Machine Interfaces, Feed Data, and Rate Limits

Three indicators describe whether third-party systems could serve agents at all. They ask whether those systems offered true machine interfaces or only user interfaces built for people and whether the data arriving from third-party feeds could be used without human cleanup. The third asks whether third-party rate limits and quotas held up under the volume the agents generated. Machine interfaces were distressed in 239 of 802 answers (29.80%), the lowest distressed share of the 100 indicators. Feed data was distressed in 249 of 795 answers (31.32%) and rate limits in 265 of 805 (32.92%). All three were associated with each core study measure, and the associations held in every robustness check.

The written evidence describes systems designed for human users and placed in front of agents. Where a provider offered only a user interface, the agent clicked through it and inherited its fragility, or a person moved the data by hand. Feeds arrived in a form the agents could not use until someone cleaned them, and the vendor’s dashboards measured quality after the cleaning. Limits set for people cut the agents off at predictable peaks, and a few staff finished the runs at night. In each case the automation the financial institution paid for depended on manual work that the engagement’s reported measures did not show.

“Half the sources we need were never built for a machine to read, they are screens for a human researcher, and no vendor ever promised us otherwise, we assumed it.”Engagement Manager, KPO, Client servicing and communications

6.5.1 Availability of Machine Interfaces Across Third-Party Systems

Respondents rated the availability of true machine interfaces, in place of user interfaces built for people, across the third-party systems the agents had to use. The mean rating was 4.6 and the median 5.0 (n = 802 respondents), and 239 of the 802 answers (29.80%) were distressed, the lowest distressed share of the 100 rated indicators. Respondents on canceled engagements found machine interfaces far scarcer (mean 3.1, n = 299 respondents) than respondents on active engagements in any stage.

The availability of machine interfaces was associated with all four core study measures, and none of the 22 robustness checks overturned an association. Cancellation was less common where third-party systems offered machine interfaces (r = −0.63, 95% CI −0.71 to −0.52, n = 150 engagements). Customer satisfaction among active engagements was higher where such interfaces were available (r = 0.71, 95% CI 0.60 to 0.80, n = 94 engagements). Interface availability was also associated with a higher engagement indicator mean computed without it (r = 0.88, 95% CI 0.84 to 0.91, n = 150 engagements). Respondents who rated interface availability higher reported lower intent to leave (r = −0.66, cluster bootstrap 95% CI −0.72 to −0.60, n = 754 respondents).

The written responses addressing machine interfaces numbered 60, nearly all in comment on the third-party ecosystem, and 49 of the 60 were negative, 36 with the strongest negative score. Every negative response described a third-party system with no machine interface. The agent operated web pages built for human users, or a staff member transferred the data manually.

“Their certification portal is a set of screens built for a person with a token in their hand, so every run needed one of my engineers sitting there anyway, and nothing on this form has a place for that.”Engagement Manager, ITO, Payments, cards and disputes

“half the tools we use have only a screen, no proper interface, so the agent was clicking through a portal built for a person. what we needed was a written api commitment from each provider and thirty days notice of any change. we got neither … ”Engagement Manager, ITO, Technology engineering and operations

“They give us a bureau screen with no proper interface, so my team copy counterparty rating by hand into the sheet before agent can use it… That is two hour of my evening, every day. Nobody count this work anywhere, therefore my team stay late and the queue still look fine on their side”Individual Contributor, KPO, Credit, underwriting and suitability

“The screen scraping is what costs my team their evenings. Fund admin portal has no proper interface so the agent clicks through pages like a person, and when the portal is slow at month end we sit and watch it and re-run the failed ones by hand.”Engagement Manager, BPO, Trade, settlement and reconciliation

“the medical records portal has no api, agent works it through the screen like a person. when they moved the buttons last month it just stopped, no warning. we find out from the backlog. also nobody left here still knows how to pull those records the manual way”Individual Contributor, BPO, Credit, underwriting and suitability

Written sentiment on machine interfaces was associated with the outcomes in its own right. A negative written response and a more negative engagement mean of sentiment were each associated with cancellation, and respondents who wrote more negatively about interfaces reported higher intent to leave. Where a provider offered only a user interface, the agent’s output rested on a person moving data by hand.

6.5.2 Quality of Third-Party Feed Data Cleaned by Hand

Respondents rated the quality of the data arriving from third-party feeds, judged by whether agents could use it without human cleanup. Its mean rating was 4.6 and its median 5.0 (n = 795 respondents), with 249 of the 795 answers (31.32%) distressed. Canceled engagements were marked by poor feed data, with a mean rating of 3.1 (n = 301 respondents), far lower than on active engagements in any stage.

The usability of third-party feed data without cleanup showed associations with every core study measure, each upheld by the 22 robustness checks. Engagements receiving cleaner feeds had a lower cancellation rate (r = −0.64, 95% CI −0.73 to −0.54, n = 150 engagements). Customer satisfaction among active engagements was higher where feeds arrived usable (r = 0.68, 95% CI 0.55 to 0.77, n = 94 engagements). Cleaner feeds accompanied a higher engagement indicator mean, computed without the feed indicator (r = 0.87, 95% CI 0.82 to 0.90, n = 150 engagements). Intent to leave was lower among respondents who received usable feeds (r = −0.65, cluster bootstrap 95% CI −0.71 to −0.58, n = 748 respondents).

The written responses addressing feed data numbered 89, with 64 negative and 25 positive. The comments on the third-party ecosystem divided between feeds that arrived clean and feeds that needed a person to clean them first. The responses to the questions on altering data described correcting the data by hand, in many accounts every cycle and in several cases without logging the correction.

“The test data we pulled from the wealth platform’s sandbox arrived in a shape the agents could not use without a person tidying it first, every single day, and nobody had counted that person’s time anywhere.”Engagement Manager, ITO, Technology engineering and operations

“The one that still costs my people evenings is the counterparty file, it comes in dirty and somebody sits and cleans it before the model runs. That is a person, every month end, and it does not appear in any plan.”Engagement Manager, KPO, Credit, underwriting and suitability

“What we do now is run everything from the rating vendor through a check sheet first, before it ever hits the config. My team built it ourselves off the errors we kept seeing, it’s maybe forty fields. Takes ten minutes a load. If a value looks off we hold it and go back to the source file insted of letting the agent take it. No one asked us to do that, it just became the way we work.”Individual Contributor, ITO, Client servicing and communications

“last week the comparables feed came with two different issuer names for same entity, one with the old ticker. we keep our own mapping file for this, it is just a list of isin against the name we use, and one of the junior updates it every morning before the deal team asks for anything. if we dont do that the agent picks wrong comparable and the pricing page is nonsence. nobody asked us to build it, we just did it and now everybody depends on it”Individual Contributor, KPO, Trade, settlement and reconciliation

“formats change between one delivery and the next practically every month, moreover nobody on their side tells us in advance when a column gets renamed or a field gets dropped, so I fix the file myself before it break anything downstream in the ledger.”Individual Contributor, BPO, Trade, settlement and reconciliation

Respondents who wrote more negatively about feed data reported higher intent to leave. Feed data was cleaned by hand before it reached the agent. The vendor’s dashboards measured quality after the cleaning, and the knowledge behind the corrections stayed with the people who made them.

6.5.3 Third-Party Rate Limits Under Agent Volume

Respondents rated how well third-party rate limits and quotas held up under the volume the engagement’s agents generated. The mean rating was 4.6 and the median 5.0 (n = 805 respondents), and 265 of the 805 answers (32.92%) were distressed. Rate limits on canceled engagements were rated far lower (mean 3.2, n = 302 respondents) than on active engagements in any stage.

How well third-party rate limits held under agent volume was associated with every core study measure, and the 22 robustness checks left each association standing. Cancellation was less common where limits held under agent volume (r = −0.64, 95% CI −0.73 to −0.53, n = 150 engagements). Where limits held, active engagements recorded higher customer satisfaction (r = 0.66, 95% CI 0.53 to 0.76, n = 94 engagements). Limits that held also accompanied a higher engagement indicator mean computed without this indicator (r = 0.86, 95% CI 0.81 to 0.90, n = 150 engagements). Respondents who rated rate limits higher reported lower intent to leave (r = −0.63, cluster bootstrap 95% CI −0.69 to −0.55, n = 756 respondents).

The written responses addressing rate limits numbered 42, all in comment on the third-party ecosystem, with 32 negative and 8 positive. Respondents described agents cut off by daily caps and throttles at predictable peaks, and staff who watched or finished the runs late at night. Several placed the responsibility on the engagement side, in volumes the engagement had neither sized nor reported to the third-party provider.

“In week six we tried to refresh about four thousand supplier records against the external registry feed in one run. It cut us off after eleven hundred, and there was no way to see that from our side until the tickets started failing. We ended up doing the rest in batches overnight for a week.”Engagement Manager, CXM, Corporate and back-office functions

“We keep two log in from the research database and rotate them, because the tool hit the limit by eleven in the morning and then everything just stop … No one ask us for this, it is simply what we do now”Individual Contributor, KPO, Regulatory, audit, tax and legal

“Rate limits are the part that lands on us personally. When the provider throttles at month end the queue does not stop, it simply waits for us, and two of my engineers sit until eleven at night watching a bar they cannot influence. I have lost one good person partly for that reason. It is not a large technical problem, but it falls on the same four people every month.”Engagement Manager, ITO, Technology engineering and operations

“Their Portal throttles us after around two hundred pulls and the agent just stops, no error, we find out when case aging report looks wrong next day.”Individual Contributor, BPO, Credit, underwriting and suitability

“the hr feed gets blamed for everything here, if I am honest it is not their fault. nobody informed them we are pulling every hour now, and their limits are published anyway, we simply ignore it. so i do not put it all on them.”Individual Contributor, ITO, Technology engineering and operations

No test of written sentiment on rate limits against the study measures reached significance. Limits set for human users met agent volume at the moments of highest demand. The peaks were predictable, and the problem was one of capacity planning that the engagement had not assigned to anyone.

6.6 Security Posture and Provider Concentration

Three indicators describe the engagement’s exposure to the providers it depended on. They cover the security posture of the third-party tools and sub-processors the agents touched and the degree of dependence on a small number of model and cloud providers. The third is the engagement’s ability to continue through a switch of its primary model or cloud provider. The distressed share ranged from 251 of 788 answers (31.85%) for security posture to 257 of 802 (32.04%) for concentration. Each of the three showed an association with all four core study measures, and every robustness check upheld it.

The written evidence on the three is small, 46 responses in total, and consistent. Security reviews covered the third-party providers that had been listed, and many engagements had no complete list of what the agents connected to. The concentration on one model provider, one cloud or one region had been accepted because it could be approved quickly, and it had not been revisited. No respondent described a tested move to a second provider. Where respondents described the fallback actually available, it was staff working by hand.

6.6.1 Security Posture of Third-Party Tools and Sub-Processors

Respondents rated the security posture of the third-party tools and sub-processors the agents touched on the engagement. The mean rating was 4.6 and the median 5.0 (n = 788 respondents), and 251 of the 788 answers (31.85%) were distressed. Respondents on canceled engagements regarded the security of third-party tools far less favorably (mean 3.1, n = 303 respondents) than respondents on active engagements in any stage.

The security posture of third-party tools and sub-processors was associated with all four core study measures, and the associations held under the 22 robustness checks. Engagements whose third-party tools were judged more secure had a lower cancellation rate (r = −0.65, 95% CI −0.74 to −0.55, n = 150 engagements). Customer satisfaction among active engagements was higher where security posture was rated stronger (r = 0.64, 95% CI 0.51 to 0.75, n = 94 engagements). A stronger security posture accompanied a higher engagement indicator mean, computed without the security indicator (r = 0.87, 95% CI 0.82 to 0.90, n = 150 engagements). Intent to leave was lower among respondents who rated security posture higher (r = −0.65, cluster bootstrap 95% CI −0.71 to −0.59, n = 742 respondents).

Nine written responses addressed security posture, a base of anecdote strength, with four negative, three positive and two neutral. Several described an incomplete view, with no full list of what the agents connected to and security reviews that stopped at the first layer of third-party providers.

“nobody has one list of what the desk agent actually connects to… i count four tools just for endpoint patching and the remote session, and the security review only covers two of them.”Individual Contributor, ITO, Technology engineering and operations

“The benefits provider gives one login and we all share it, that is what I would flag… nobody checks who touched what. In addition no one here can tell me the full list of outside systems that touch our payroll informations … ”Individual Contributor, BPO, Corporate and back-office functions

“What has not moved is the chain behind them. We know our direct suppliers. Ask who their sub-processors are and nobody can tell us, and the security review stopped at the first layer.”Engagement Manager, CXM, Corporate and back-office functions

“My 5 on security is only about one market data provider, i never see their sub processor list, the rest i had no issue with”Individual Contributor, BPO, Corporate and back-office functions

The few respondents who wrote described a security assessment limited by an incomplete inventory. A provider that had not been listed had not been reviewed.

6.6.2 Dependence on a Small Number of Model and Cloud Providers

Respondents rated the engagement’s degree of dependence on a small number of model and cloud providers. The mean rating was 4.59, among the three lowest of the 20 indicators of the category, and the median 5.0 (n = 802 respondents). Distressed ratings made up 257 of the 802 answers (32.04%). On canceled engagements, dependence on a few providers was judged far heavier (mean 3.1, n = 308 respondents) than on active engagements in any stage.

Each association between provider concentration and the four core study measures held in every one of the 22 robustness checks. Engagements less dependent on a few providers were less likely to be canceled (r = −0.64, 95% CI −0.73 to −0.53, n = 150 engagements). Among active engagements, lower concentration accompanied higher customer satisfaction (r = 0.64, 95% CI 0.50 to 0.74, n = 94 engagements). Lower concentration was also associated with a higher engagement indicator mean, computed without the concentration indicator (r = 0.86, 95% CI 0.81 to 0.90, n = 150 engagements). Respondents who rated concentration as less of a problem reported lower intent to leave (r = −0.64, cluster bootstrap 95% CI −0.70 to −0.58, n = 755 respondents).

The written responses addressing provider concentration numbered 26, nearly all in comment on the third-party ecosystem, with 19 negative and 4 positive. Respondents described agents that depended on one model provider, one cloud or one region with no second path configured and no tested plan for an outage.

“if the model provider go down one morning we are finished, there is no second one set up, we would be closing tickets on paper. Nobody has ever tested that.”Individual Contributor, CXM, Corporate and back-office functions

“The concentration is our own doing. We built everything on one model and one region because that is what could be approved quickly, and now moving is a project nobody will fund at POC stage.”Engagement Manager, BPO, Financial crime and client onboarding

“One issue only. Every agent we run sits on one model provider and one cloud tenancy, with no second path. Three weeks ago that region degraded for about two hours and the provisioning queue simply stopped, we cleared it by hand. There is no plan on paper for an outage that lasts longer.”Engagement Manager, ITO, Technology engineering and operations

Written sentiment on concentration, from 26 responses, was associated with cancellation through its engagement mean and with higher intent to leave among respondents who wrote more negatively. The written evidence describes a concentration accepted for speed and never revisited. The fallback actually available was staff working by hand.

6.6.3 Continuity Through a Switch of Primary Model or Cloud Provider

Respondents rated how well the engagement could continue its work through a switch of its primary model or cloud provider. The mean rating was 4.6 and the median 5.0 (n = 811 respondents), and 255 of the 811 answers (31.44%) were distressed. Continuity through a provider switch averaged 3.1 on canceled engagements (n = 301 respondents), far lower than on active engagements in any stage.

The ability to continue through a switch of primary provider showed associations with every core study measure that held under the 22 robustness checks. Cancellation was less common where continuity was rated stronger (r = −0.65, 95% CI −0.73 to −0.54, n = 150 engagements). Active engagements with stronger continuity reported higher customer satisfaction (r = 0.68, 95% CI 0.56 to 0.78, n = 94 engagements). The engagement indicator mean, computed without the continuity indicator, was closely associated with the rating (r = 0.87, 95% CI 0.82 to 0.90, n = 150 engagements). Respondents who rated continuity higher reported lower intent to leave (r = −0.66, cluster bootstrap 95% CI −0.71 to −0.60, n = 759 respondents).

Eleven written responses addressed continuity, all in comment on the third-party ecosystem, with eight negative and three positive. The negative responses said the engagement could not move off its model provider quickly and had no plan to. The positive responses asserted that a move would be manageable without describing one. No response described a switch of provider that had been tested.

“We had nowhere to move to. Everything the agents do sits on that one provider and there was no discussion about an alternative because there is not one configured. Therefore we spent nine days re-testing the same onboarding cases against the new version, and the weekly pack still showed green.”Engagement Manager, CXM, Financial crime and client onboarding

“If we have to move to another model provider we are stuck, that needs a proper plan”Individual Contributor, ITO, Technology engineering and operations

“what i would change is we are pinned to a single model provider and nobody has ever tried standing the workflows up on a second one. not saying it would break. saying nobody knows, and finding out during an outage is a bad time to learn”Individual Contributor, CXM, Corporate and back-office functions

“I scored this section high and I want to be clear about why, because it is not sophistication. Our recruiting volume is small enough that if the scheduling API or the model went away tomorrow, four coordinators could run the calendar manually for a week, the same as we did before this program existed. That is the whole reason. We have not tested a provider switch, we have never had to, and I would not want the high marks read as evidence that we could.”Engagement Manager, CXM, Corporate and back-office functions

No test of written sentiment on continuity against the study measures reached significance. Confidence in continuity rested on manual fallback at low volume or on an estimate that had not been tested.

6.7 Fallback Capacity and the Manual Knowledge Agents Displace

Four indicators describe what protected the engagement when a third party failed. They cover the readiness of a fallback for a critical third-party failure and the retention of the ability to perform the engagement’s tasks by hand. The other two cover knowledge of where staff had quietly fixed third-party problems in the past, and whether those quiet fixes had been replaced with explicit checks once agents took the tasks over. The replacement of quiet fixes had the lowest mean rating of the 20 indicators of the category, 4.58 (n = 796 respondents), and the highest distressed share, 267 of 796 answers (33.54%). Every one of the four was associated with each core study measure, and each association held in all 22 robustness checks.

The written evidence describes a fallback that the agents themselves erode. The default fallback for a failing feed or model was the manual work the agents had replaced, and that work lived in people. Before the agents, experienced staff had corrected third-party problems by habit, a custodian’s late accrual or a bureau file’s outdated identifiers, and no process recorded what they did. When the agents took the tasks over, the habit was removed and the correction was not written into a check. The people who knew the manual method left, were released or were never trained.

“For years my seniors carried the quirks of the custodian files in their heads — one custodian sends the fee accrual a day late, another restates a month after the fact, and the person who knew that simply corrected it on the Monday. When the agents came in, none of that was written down anywhere, it was lost, and the two people who held most of it have since left.”Engagement Manager, KPO, Trade, settlement and reconciliation

“Half the people who used to patch these files by hand are not on the account anymore.”Engagement Manager, CXM, Regulatory, audit, tax and legal

Where teams kept a saved copy, a manual route or a tested second source, an outage was an inconvenience. Where they did not, the engagement’s tasks stopped, and the loss of the manual skill became visible only during the outage. Each unrecorded fix comes to light only when a result is wrong.

6.7.1 Fallback Readiness for Critical Third-Party Failure

Respondents rated the readiness of a fallback for the failure of a critical third-party service. Its mean rating was 4.6 and its median 5.0 (n = 802 respondents), with 257 of the 802 answers (32.04%) distressed. Respondents on canceled engagements gave fallback readiness its lowest ratings, a mean of 3.1 (n = 301 respondents) against far higher means on active engagements in any stage.

Each of the four core study measures was associated with fallback readiness, and no association failed a robustness check. Cancellation was rarer on engagements with a ready fallback (r = −0.64, 95% CI −0.73 to −0.54, n = 150 engagements). Customer satisfaction among active engagements was higher where a fallback was ready (r = 0.69, 95% CI 0.57 to 0.79, n = 94 engagements). Fallback readiness and the engagement indicator mean computed without it rose together (r = 0.87, 95% CI 0.83 to 0.91, n = 150 engagements). Respondents who rated fallback readiness higher reported lower intent to leave (r = −0.66, cluster bootstrap 95% CI −0.72 to −0.59, n = 753 respondents).

Fallback readiness was the subject of 80 written responses, all in comment on the third-party ecosystem, with 57 negative and 21 positive. In 25 responses, respondents said there was nothing behind a critical feed or model. The engagement’s tasks stopped and waited. The positive responses described fallbacks the teams had built for themselves, usually a saved copy or a manual route, and a few described tested secondary sources.

“we now have a tested fallback that routes tickets to the queue if the model endpoint drops – it has fired twice this quarter and no employee noticed.”Engagement Manager, CXM, Corporate and back-office functions

“only worry i have is we run everything on one provider, if that provider go down for a day i do not know what we do, nobody has shown me a second option. but that is our own planning gap, not their fault.”Individual Contributor, KPO, Corporate and back-office functions

“On my last team when a feed went down somebody just pulled the file manually and we kept going. Here there is no backup, the bureau feed stopped for two days in June and the whole validaton queue just sat there. Nobody could tell me who to call”Individual Contributor, KPO, Credit, underwriting and suitability

“last month the vendor of our knowledge search changed their api and our policy lookups came back empty for two days. no notice came to us, we found it from tickets piling up. and there was no fallback, we answered by hand from pdfs”Individual Contributor, CXM, Corporate and back-office functions

“We keep a manual channel open on the payment file, one person still knows how to submit it the old way and we run that once a week just to keep the muscle. Same with vendor master pulls, we take a copy every Monday so if the feed goes quiet we are working from something. Not elegant. But it has saved us twice this year.”Engagement Manager, BPO, Corporate and back-office functions

Respondents who wrote more negatively about fallbacks reported higher intent to leave. A saved copy or a practiced manual route was what separated an inconvenience from a stoppage.

6.7.2 Retention of the Ability to Perform Manual Workarounds

Respondents rated how well the engagement retained the ability to perform its tasks by hand if a third party failed. The mean rating was 4.66, among the highest of the 20 indicators of the category, and the median 5.0 (n = 808 respondents). Distressed ratings made up 249 of the 808 answers (30.82%). Respondents on canceled engagements judged their ability to work by hand far weaker (mean 3.2, n = 311 respondents) than respondents on active engagements in any stage.

The retention of manual capability held its association with all four core study measures across the 22 robustness checks. Engagements that kept the ability to work by hand were less likely to be canceled (r = −0.65, 95% CI −0.73 to −0.54, n = 150 engagements). Customer satisfaction among active engagements was higher where manual capability was retained (r = 0.72, 95% CI 0.60 to 0.80, n = 94 engagements). The engagement indicator mean, computed without the manual-capability indicator, was strongly correlated with the rating (r = 0.88, 95% CI 0.84 to 0.91, n = 150 engagements). Intent to leave was lower among respondents who said manual capability had been kept (r = −0.67, cluster bootstrap 95% CI −0.73 to −0.62, n = 757 respondents).

The written responses addressing manual capability numbered 43, all in comment on the third-party ecosystem, and they divided evenly, with 22 negative and 21 positive. The negative responses said the people who knew the manual methods had left without writing them down, and newer staff had never learned them. The positive responses described outages worked through by hand, some with the caveat that this held only at low volume or while the staff who had performed the tasks before the agents remained.

“The client released its own service desk staff in month three, on the basis that the bot would carry the volume. When the platform was down for most of a Tuesday in July, there was nobody left who knew how to open an IT ticket manually — we had my agents doing it from memory. You cannot un-remove those people. That is the risk nobody priced.”Engagement Manager, CXM, Corporate and back-office functions

“We are two months in with a team that has never triaged without the agent in front of them, so if the enrichment feed drops I am not sure we could work the queue by hand at volume.”Engagement Manager, ITO, Technology engineering and operations

No test of written sentiment on manual capability against the study measures reached significance. The manual skill that serves as the fallback lives in people, and once the agents took on the volume, the staff who held it moved on or were never taught it. The positive accounts show that the skill can be kept by deliberate rehearsal. Where it was not kept, the gap appeared at the moment of an outage.

6.7.3 Knowledge of Where Staff Quietly Fixed Third-Party Problems

Respondents rated how well the engagement knew where staff had quietly fixed third-party problems by hand before the agents took the tasks over. The mean rating was 4.6 and the median 5.0 (n = 792 respondents), and 262 of the 792 answers (33.08%) were distressed. Respondents on canceled engagements knew far less about the quiet fixes (mean 3.1, n = 305 respondents) than respondents on active engagements in any stage.

Knowledge of the quiet fixes was associated with cancellation, customer satisfaction, the engagement indicator mean and intent to leave, and all 22 robustness checks upheld the associations. Engagements that knew where the quiet fixes had been made were less likely to be canceled (r = −0.66, 95% CI −0.74 to −0.56, n = 150 engagements). Customer satisfaction among active engagements was higher where this knowledge was rated better (r = 0.61, 95% CI 0.47 to 0.73, n = 94 engagements). Where the quiet fixes were known, the engagement indicator mean computed without this indicator was higher (r = 0.87, 95% CI 0.82 to 0.90, n = 150 engagements). Intent to leave was lower among respondents who knew where the fixes had been made (r = −0.66, cluster bootstrap 95% CI −0.71 to −0.60, n = 746 respondents).

The written responses addressing the quiet fixes numbered 21, all in comment on the third-party ecosystem, and 20 of the 21 were negative. Respondents said the manual fixes that had once absorbed third-party problems had never been written down, that the people who made them had left, and that each fix was rediscovered when something broke.

“If we knew where those fixes sat, they would not have been quiet, and we would have written them into the process years ago. The honest position is that we discover each one at the moment it breaks a valuation, and from then on it is no longer unknown.”Engagement Manager, KPO, Corporate and back-office functions

“nobody wrote down what the old workplace team used to fix by hand before the agent took the queue, so when something looks wrong we are guessing at what normal was.”Individual Contributor, CXM, Corporate and back-office functions

“The small manual corrections our engineers used to make on reference data were never written down anywhere before the agent took that job over”Engagement Manager, ITO, Regulatory, audit, tax and legal

“Also there is no record of who fixed the supplier addresses before the agents came, those persons are gone.”Engagement Manager, CXM, Corporate and back-office functions

“What we have not done is capture the little fixes our floor supervisors used to make before any of this.”Engagement Manager, CXM, Payments, cards and disputes

No test of written sentiment on the quiet fixes against the study measures reached significance. The written evidence describes a control that was never formal, people who corrected feeds by habit without any process recording the correction. Automation removed the habit without recording the correction.

6.7.4 Replacement of Quiet Human Fixes With Explicit Checks

Respondents rated how well the engagement had replaced the quiet human fixes with explicit checks once agents took the tasks over. The mean rating was 4.58, the lowest of the 20 indicators of the third-party ecosystem, and the median 5.0 (n = 796 respondents). Distressed ratings made up 267 of the 796 answers (33.54%), the highest distressed share in the category. Respondents on canceled engagements rated the replacement of quiet fixes far lower than respondents on active engagements in any stage (mean 3.1, n = 302 respondents).

Whether quiet fixes had been turned into explicit controls was associated with each core study measure, and every association survived the 22 robustness checks. Cancellation was less common where the quiet fixes had been replaced (r = −0.64, 95% CI −0.73 to −0.54, n = 150 engagements). Customer satisfaction among active engagements was higher where the replacement was rated more complete (r = 0.71, 95% CI 0.59 to 0.80, n = 94 engagements). With this indicator excluded, the engagement indicator mean was closely associated with the rating (r = 0.88, 95% CI 0.84 to 0.91, n = 150 engagements). Respondents who rated the replacement higher reported lower intent to leave (r = −0.65, cluster bootstrap 95% CI −0.71 to −0.59, n = 748 respondents).

Nine written responses addressed the replacement of quiet fixes, a base of anecdote strength, with seven negative, one positive and one neutral. Every negative response said that a correction people had made by habit had not been turned into a check when the agent took the task over. In one, the lost correction had left access open for a person who had left.

“Before, when the credit bureau file came with the old company identifiers, somebody on my team just knew and swapped them over. It is not written anywhere, it sits in his head. Now the agent takes the file exactly as it arrives. Nobody sat down and turned that knowledge into a check… we assumed it would not matter. It has not caused a problem yet. But he is on leave next month.”Individual Contributor, BPO, Payments, cards and disputes

“Hard to blame the feed vendors… they send what they always sent. We aimed an agent at it and never set out what a clean file looks like. Whoever used to catch this by hand has gone”Individual Contributor, KPO, Trade, settlement and reconciliation

“You ask whether the quiet human fixes were deliberately replaced with checks, but nobody ever wrote down what those fixes were, so the honest answer is that we cannot know, and a four is me guessing.”Engagement Manager, KPO, Credit, underwriting and suitability

The few respondents who wrote described a control lost in the handover from people to agents. How widely the loss occurred is measured by the ratings, and a third of the answers placed it in the distressed range.

6.8 The Dependency Chain and Gated External Rails

Two indicators describe the reach of the engagement’s dependencies. The first is how well the engagement had mapped its third-party and fourth-party dependencies. The second is how well it handled its dependence on gated external rails, the payment networks, regulatory filing systems and market utilities that set their own terms of access. The dependency map was distressed in 252 of 791 answers (31.86%) and the gated rails in 245 of 794 (30.86%). Both stayed associated with each core study measure through all 22 robustness checks.

The written evidence identifies the dependency inventory as the precondition for most of the other third-party controls. Without a list of what the agents depended on, the engagement could not notify anyone of a change, monitor a feed, review its security or plan its fallback. The gated rails set their own windows, limits and certification terms, and where those systems assumed a person, the engagement supplied one by hand.

6.8.1 Mapping of Third- and Fourth-Party Dependencies

Respondents rated how well the engagement had mapped its third-party dependencies and the fourth parties behind them. The mean rating was 4.6 and the median 5.0 (n = 791 respondents), and 252 of the 791 answers (31.86%) were distressed. The dependency map was rated far less complete on canceled engagements (mean 3.1, n = 302 respondents) than on active engagements in any stage.

Mapping of third- and fourth-party dependencies was associated with all four core study measures, each association upheld by the 22 robustness checks. Cancellation was less common where dependencies had been mapped (r = −0.64, 95% CI −0.72 to −0.53, n = 150 engagements). Among active engagements, customer satisfaction was higher where the map was rated fuller (r = 0.71, 95% CI 0.59 to 0.80, n = 94 engagements). A fuller map accompanied a higher engagement indicator mean computed without the mapping indicator (r = 0.87, 95% CI 0.83 to 0.91, n = 150 engagements). Intent to leave was lower among respondents who rated the map fuller (r = −0.68, cluster bootstrap 95% CI −0.73 to −0.62, n = 741 respondents).

The written responses addressing the dependency map numbered 44, all in comment on the third-party ecosystem, with 35 negative and 8 positive. The negative responses said there was no list of what the agents depended on, or that the list stopped at the direct providers, and several had asked for one without receiving an answer. One respondent whose team had built such a list put its cost at two weekends of analysts’ time.

“we went into a procurement desk without a list of what the agents would be leaning on beneath the obvious platform, and no view at all of what sits beneath that again. On the one occasion a supplier feed changed shape we found out because the queue stopped, not because anyone told us. Whether we could have moved off the model provider is academic; we would not have known what to move.”Portfolio Director, CXM, Corporate and back-office functions

“somebody needs to own the dependency map past the first tier; we know our directs, we do not really know who they depend on, and that gap would matter a great deal on the day one of them has an outage … Neither is expensive. Both need a name against them and a date, or they stay in the deck.”Portfolio Director, ITO, Technology engineering and operations

“the problem is the Rates feed changed its field names on a wednesday and nobody told us. every model that pulled from it came back blank and we lost two days keying rates in manually. i asked who owns that supplier, nobody could say. there is no list anywhere”Individual Contributor, KPO, Corporate and back-office functions

“how am i to rate advance notice from providers when nobody tells us which providers sit behind the screening service.”Individual Contributor, ITO, Financial crime and client onboarding

“Monitoring for unannounced provider changes is in place and it works, which is why those marks are high. We have the dependency chain mapped to the fourth party.”Engagement Manager, KPO, Corporate and back-office functions

Cancellation was more common on engagements with a negative written response on the map and on those with a more negative engagement mean of sentiment. Intent to leave was higher among respondents who wrote more negatively about it. Without the inventory, the other third-party controls had nothing to attach to.

6.8.2 Dependence on Gated External Rails

Respondents rated how well the engagement handled its dependence on gated external rails, the networks, regulators and utilities whose systems the agents had to reach on those systems’ own terms. The mean rating was 4.6 and the median 5.0 (n = 794 respondents), and 245 of the 794 answers (30.86%) were distressed. Respondents on canceled engagements rated the handling of gated rails far lower than respondents on active engagements in any stage (mean 3.1, n = 296 respondents).

All four core study measures were associated with how well the engagement handled gated external rails, and the 22 robustness checks left the associations intact. Engagements that handled the rails better had a lower cancellation rate (r = −0.63, 95% CI −0.72 to −0.52, n = 150 engagements). Customer satisfaction among active engagements was higher where the rails were handled better (r = 0.69, 95% CI 0.56 to 0.78, n = 94 engagements). Handling of the rails was also associated with a higher engagement indicator mean computed without it (r = 0.86, 95% CI 0.81 to 0.90, n = 150 engagements). Intent to leave was lower among respondents who rated the handling of the rails higher (r = −0.65, cluster bootstrap 95% CI −0.71 to −0.58, n = 744 respondents).

Fifteen written responses addressed the gated rails, nearly all in comment on the third-party ecosystem, with 13 negative and 2 positive. Respondents placed the weakness in certification portals and regulatory submission systems built for a person, fixed submission windows and call limits, and sandbox access that never arrived in full. In several engagements notice of provider change had improved while access to the rails had not.

“the tax authority portals dont care that were an agent. submission windows close at a fixed hour and if the model version changes the week of a deadline we have no route to push it. Feeds themselves are excellent, no complaint there. its the last mile into the filing systems thats out of our hands”Individual Contributor, KPO, Regulatory, audit, tax and legal

“Feed providers do warn us now before they change something, that got better since January. What did not get better is the regulator portal, agent still cant file anything there, one of us does it by hand every week”Individual Contributor, CXM, Client servicing and communications

“Same with statutory submission portal, it takes only so many a day.”Individual Contributor, BPO, Corporate and back-office functions

Respondents who wrote more negatively about the rails reported higher intent to leave. Where a rail’s systems assumed a person, a member of the delivery team filled that role by hand.

The 100 rated indicators reported in this part of the study describe how the engagements were delivered. The indicators inside the two firms, covering the financial institution’s readiness, the joint management of the engagement and the vendor’s capability, were rated lower than the AI technology and the outside systems it depended on. Every indicator was associated with all four study measures, and the indicators failed together. In each category the written evidence describes the same response to failure, the delivery team absorbing it by hand, and in the respondents’ accounts much of that work was not reported to the financial institution. The controls meant to govern work performed by agents, and how they were bypassed or absent, are examined next (Part 3).