Third-Party Ecosystem

The third-party ecosystem was the highest-rated of the five failure categories. Its 20 indicators had a mean of 4.64 across the 150 engagements, and 5,062 of their 15,947 answers (31.74%) were distressed. The category covers the outside providers on which the agents depended. They include the model and cloud providers, the data feeds, the providers of market and reference data, and the systems of other firms that the agents had to use. Like every other category, it remained under the midpoint of the scale, and each of its indicators was associated with all four core study measures.

Mean ratings in Table 2.7 are computed across respondents. The category figure of 4.64 is computed across engagements.

Respondents described difficulties at interfaces with systems outside the engagement’s direct control. Accounts included changes to provider formats or models, systems lacking machine interfaces, uncertainty about licenses for agent use, and divided contractual responsibilities. These findings concern the coordination of dependencies as well as the underlying technology.

“we dont control any of the things that break us. the market data service, the ticket system, the code repo, the model itself, all of it belongs to somebody else. none of them tell us before they change something … when that screen moves one field, the agent carries on regardless and produces something that looks correct. that is the part that worries me.” -Individual Contributor, ITO, Technology engineering and operations

“Half the network portals have no api, the agent has to work the screen like a person and it breaks when they move a button. Licences are counted per named user and nobody updated that for an agent.” -Individual Contributor, BPO, Financial crime and client onboarding

Respondents identified practices they considered useful, including assigned responsibility for provider notices, routine test cases, and dependency inventories. The study did not measure their implementation costs or independently evaluate their effectiveness. Part 5 examines third-party tasks performed manually, described in 91 written answers from 87 respondents on 63 engagements.

The category also addresses fallback capability and the retention of knowledge previously applied during manual work. The survey invited comments on the third-party ecosystem. 440 of 850 respondents (51.76%) provided one.

“what we lost is the small checks the senior people used to do on returns before it goes to composite, nobody wrote those down when the agent take over, that is our side not theirs” -Individual Contributor, KPO, Trade, settlement and reconciliation

1 Notice of Provider Change

Provider notice could be inadequate at its source or fail during internal routing before reaching the operators of affected agents. Two indicators distinguish the advance notice supplied by third-party providers from the routing of that notice to affected operators. A provider can communicate a change without the information reaching the people responsible for adapting the engagement.

1.1 Advance Notice From Third-Party Providers

Indicator C181 · Rate the quality of advance notice third-party providers give before changes that can affect the agents on this engagement · Full statistical analysis ›

Accounts of provider change distinguished inadequate advance notice from notices that had been supplied but failed to reach the people operating affected agents. The mean rating was 4.6 and the median 5.0 (n = 801 respondents). Advance notice was rated at a mean of 3.1 on canceled engagements (n = 302 respondents), lower than on active engagements in any stage.

Associations between the quality of advance notice and all four core study measures survived each of the applicable robustness checks. Cancellation was rarer where providers gave better notice (r = −0.66, 95% CI −0.74 to −0.56, n = 150 engagements). Customer satisfaction among active engagements was higher where notice was rated better (r = 0.68, 95% CI 0.55 to 0.77, n = 94 engagements). With the notice indicator removed, the engagement indicator mean was still closely associated with the rating (r = 0.88, 95% CI 0.83 to 0.91, n = 150 engagements). Respondents who rated advance notice higher reported lower intent to leave (r = −0.68, cluster bootstrap 95% CI −0.73 to −0.62, n = 753 respondents). Distressed ratings came from 250 of the 801 answers (31.21%), and the engagement mean was distressed on 36 of the 150 engagements.

Advance notice was the subject of 160 written responses, nearly all in comment on the third-party ecosystem, with 108 negative and 48 positive. Every negative response described a change by a provider that reached the team without notice, or with too little to absorb, and was found when a run failed or output went wrong. Several positive responses said the providers did give notice and that it failed to reach the people running the agents.

“In March one custodian shifted two columns in their statement file and told nobody at all. The agents carried on and matched breaks against the wrong field for a day and a half before a junior spots it. Five years back we had four people who would catch that within the hour. Now there is one, and she was on leave that week.” -Engagement Manager, BPO, Trade, settlement and reconciliation

“the custodian feed changed format last month and no one flagged it. i found out because my file wouldnt load at 6am … we eat every one of these ourselves. my team is three people and two of us are looking for something else.” -Individual Contributor, BPO, Trade, settlement and reconciliation

“the e-sign vendor changed something over the weekend and my team spend monday re-checking every file by hand… nobody sent any informations before that” -Individual Contributor, BPO, Corporate and back-office functions

“last month the scanner vendor changed their api response and nobody tell us, our release notes agent starts putting wrong build ids in and we only catch it at deploy gate. we fix same day, still nobody from that side told us anything” -Individual Contributor, ITO, Technology engineering and operations

“What i can tell you is the scanning vendor changed something in june and the header fields came out shifted for a week, nobody warned us.” -Individual Contributor, BPO, Corporate and back-office functions

Written sentiment on advance notice was associated with the outcomes in its own right. Engagements with a negative written response on notice had a higher cancellation rate, and so did engagements with a more negative mean of written sentiment. Respondents who wrote more negatively about notice reported higher intent to leave. For the financial institution, the failures surfaced in reconciliations, screening, payroll, and service availability.

1.2 Routing of Change Notices to the Operators of Affected Agents

Indicator C182 · Rate how reliably word of an upcoming third-party change reaches the people who operate the affected agents in time to act · Full statistical analysis ›

Third-party change notices were reported to stop at individual inboxes or other teams before reaching the personnel who needed to act. The mean rating was 4.58, among the two lowest of the 20 indicators of the category, and the median 5.0 (n = 798 respondents). On canceled engagements, notices were judged to reach operators less reliably (mean 3.1, n = 304 respondents) than on active engagements in any stage.

The routing of change notices to the operators of affected agents held its association with every core study measure through the applicable robustness checks. Engagements where notices reached operators reliably had lower observed cancellation rates (r = −0.64, 95% CI −0.73 to −0.54, n = 150 engagements). Active engagements with reliable routing reported higher customer satisfaction (r = 0.68, 95% CI 0.55 to 0.77, n = 94 engagements). Reliable routing accompanied a higher engagement indicator mean, computed without the routing indicator (r = 0.86, 95% CI 0.81 to 0.90, n = 150 engagements). Intent to leave was lower among respondents who said notices reached them in time (r = −0.66, cluster bootstrap 95% CI −0.72 to −0.60, n = 751 respondents). Distressed ratings came from 255 of the 798 answers (31.95%), and the engagement mean was distressed on 37 of the 150 engagements.

The written responses addressing the routing of notices numbered 56, nearly all in comment on the third-party ecosystem, with 42 negative and 14 positive. Respondents placed the gap chiefly inside the engagement. Providers sent notice, and it stopped at an inbox, a single person, or another team before it reached the people running the agents. Where word did arrive in time, it came in several accounts through one person acting on their own initiative.

“We went looking for who receives the provider notices, and they do arrive, into a mailbox our platform group owns, and nobody forwards them to the people running the configuration jobs. What would have to change is simple enough, one named person on each side reads them weekly and tells the affected job owners directly … ” -Engagement Manager, ITO, Client servicing and communications

“The supplier screening feed changed its schema in March, it was written in their release note, and we simply had nobody reading those notes, so the exception classifier failed quietly for a fortnight and we blamed our own tooling first … Still ours to catch.” -Engagement Manager, BPO, Corporate and back-office functions

“We put one person on the provider release notes each Monday. Nobody asked us to, we simply do it, because word of a change tends to reach the account team a good fortnight before it reaches whoever is actually running the agents. Ten minutes of reading saves us a bad Tuesday.” -Engagement Manager, CXM, Client servicing and communications

“One of my seniors reads the provider release notes every Friday morning and puts deprecation dates in our own calendar, because the notice never reaches us through proper channel. For rate limits we simply queue agent runs overnight in two batches. Both are manual and both depend on that one person.” -Engagement Manager, ITO, Technology engineering and operations

“The list provider changed the format of their delta file over a weekend, we found it on Monday in a mail nobody had circulated, and screening ran late that day.” -Engagement Manager, KPO, Financial crime and client onboarding

More negative engagement-level sentiment about notice routing was associated with cancellation. More negative respondent-level sentiment was associated with higher intent to leave. Accounts identified internal distribution as a separate weakness from the adequacy of the provider’s original notice.

2 Detection of Unannounced Change

Some third-party changes were detected only after failures, while others left agents producing incorrect results without an immediate error signal. Two indicators assess protection against undetected failures and monitoring for unannounced changes. They distinguish the resilience of agent operation from the engagement’s capacity to identify changes in its dependencies.

Respondents distinguished interruptions that immediately revealed a problem from continued operation that produced incorrect results. They also described test cases and feed-structure alerts intended to identify changes. The evidence does not establish comparative detection times or implementation costs across engagements.

2.1 Protection Against Undetected Failures

Indicator C178 · Rate how well the agents on this engagement are protected from breaking silently when a third party changes something without warning · Full statistical analysis ›

Some unannounced third-party changes were reported to leave agents processing incorrect or empty outputs without an error signal. The mean rating was 4.6 and the median 5.0 (n = 781 respondents). Protection was weakest on canceled engagements (mean 3.1, n = 293 respondents), well short of the level reported on active engagements in any stage.

Protection against silent breakage was associated with every core study measure, and each association withstood the applicable robustness checks. Cancellation was less common where agents were better protected (r = −0.64, 95% CI −0.72 to −0.53, n = 150 engagements). Among active engagements, better protection accompanied higher customer satisfaction (r = 0.68, 95% CI 0.55 to 0.77, n = 94 engagements). The engagement indicator mean, computed without the protection indicator, was closely associated with the rating (r = 0.88, 95% CI 0.84 to 0.91, n = 150 engagements). Intent to leave was lower where respondents saw the agents as protected (r = −0.65, cluster bootstrap 95% CI −0.71 to −0.59, n = 733 respondents). Distressed ratings came from 249 of the 781 answers (31.88%), and the engagement mean was distressed on 36 of the 150 engagements.

Protection against silent breakage was the subject of 136 written responses, most in comment on the third-party ecosystem, with 100 negative and 34 positive. Respondents described agents that kept running when a provider changed something and produced wrong or empty output without an error, with no monitoring to catch the change and nothing to fall back on.

“In the second month the HR feed provider altered the layout of the leavers file over a weekend, no notice reached anyone here, and the deprovisioning agents processed nothing for four days without raising a thing. We found it because a line manager asked about somebody who had left and still had access. Previously a person would have opened that file on the Monday and seen it looked wrong. That habit was never turned into a check.” -Portfolio Director, ITO, Technology engineering and operations

“Two weeks back the speech vendor pushed a update and the transcription started dropping the last few seconds of every call… we only saw it because arrangement amounts came out short. No notice came to us. They fixed it same day to be fair, but nobody was watching for it.” -Individual Contributor, CXM, Payments, cards and disputes

“network changed their file layout in april, no notice to anyone, agent kept reading old columns and produced empty cases for two days. one of my team spotted it, not the system.” -Individual Contributor, BPO, Financial crime and client onboarding

“What we do is keep a copy of last months bureau file. When the feed changes shape we get no notice, we find out when the load fails on Monday. So I run the model on the old file first, then re-key the new one by hand if the counts are far apart. It is never down, but somebody is always doing that check” -Individual Contributor, KPO, Credit, underwriting and suitability

Negative written responses and more negative engagement-level sentiment about undetected failures were associated with higher cancellation rates. More negative respondent-level sentiment was associated with higher intent to leave. Accounts described incorrect outputs reaching institutional processes before staff identified the underlying change.

2.2 Monitoring for Unannounced Third-Party Change

Indicator C183 · Rate the engagement’s monitoring for unannounced changes in the third-party APIs, data feeds, and models its agents depend on · Full statistical analysis ›

Accounts of monitoring for unannounced third-party changes ranged from rapid detection through test cases or alerts to discovery only after a failure. The mean rating was 4.6 and the median 5.0 (n = 794 respondents). Respondents on canceled engagements reported weaker monitoring (mean 3.2, n = 299 respondents) than respondents on active engagements in any stage.

Every core study measure was associated with monitoring for unannounced change, and none of the applicable robustness checks reversed an association. Engagements that monitored their third-party inputs had a lower observed cancellation rate (r = −0.63, 95% CI −0.72 to −0.53, n = 150 engagements). Active engagements that watched their third-party inputs recorded higher customer satisfaction (r = 0.68, 95% CI 0.56 to 0.78, n = 94 engagements). Where monitoring was stronger, the engagement indicator mean computed without it was higher (r = 0.87, 95% CI 0.83 to 0.91, n = 150 engagements). Respondents who rated monitoring higher reported lower intent to leave (r = −0.65, cluster bootstrap 95% CI −0.71 to −0.58, n = 744 respondents). Distressed ratings came from 251 of the 794 answers (31.61%), and the engagement mean was distressed on 37 of the 150 engagements.

The written responses addressing monitoring numbered 46, all in comment on the third-party ecosystem, with 30 negative and 16 positive. The negative responses said that the engagement did not watch its third-party inputs for change. Changes were found when something failed. The positive responses described monitoring that worked, or informal checks the team had built for itself.

“Main thing is nobody watches the outside feeds for changes. The repair estimate service changed a field name in june and we didnt know until claims started coming through with the estimate blank, we only figured it out because somebody on my team opened one up and looked. There is no backup for that feed either, if it goes down we type it in off the pdf.” -Individual Contributor, CXM, Client servicing and communications

“If the feed from the endpoint tool stops we can work the queue by hand, we did it for two days in June and nothing was missed. The gap is we do not watch for changes at the other end. And when a field changed name last month we found it from failed tickets, not from any notice.” -Individual Contributor, ITO, Technology engineering and operations

“What we do is keep our own list. Every morning one of us puts three test accounts through before the queue opens, one in arrears, one with a broken arrangement, one clean. If the offer wording comes back different from yesterday we put it in team chat and everybody works from that, in addition we keep last weeks version saved so there is something to compare against. Nobody asked us to do it, its just quicker than waiting to be told.” -Individual Contributor, CXM, Payments, cards and disputes

“We keep our own copy of vendor library from last known good version, and run both side by side every monday morning. If numbers split we know they changed something without telling us. Not official, just what team does” -Individual Contributor, ITO, Trade, settlement and reconciliation

“When their side patched the directory the sync stopped and nothing alerted us, we found it next morning from the queue count.” -Individual Contributor, ITO, Technology engineering and operations

Written sentiment about monitoring showed no statistically significant association with the study measures. Some respondents described rapid detection through test cases or alerts. Others reported delayed discovery. These accounts illustrate possible monitoring benefits without establishing a measured cost-benefit relationship.

3 Interface Stability and Model Retirement

Provider releases and model retirements were reported to require remapping, manual processing, or transition preparation that was not consistently available. Two indicators assess interface stability across provider releases and the engagement’s ability to accommodate model retirement. Both concern changes originating outside the engagement, but they require different preparation and transition arrangements.

3.1 Stability of Third-Party Interfaces and Data Structures

Indicator C187 · Rate the stability of the third-party APIs and data structures the agents depend on, from one provider release to the next · Full statistical analysis ›

Changes to third-party interfaces and data structures were reported to require manual remapping or data entry after provider releases. The mean rating was 4.7 and the median 5.0 (n = 796 respondents). Respondents on canceled engagements described less stable interfaces (mean 3.1, n = 297 respondents) than respondents on active engagements in any stage.

Interface stability showed associations with all four core study measures that each of the applicable robustness checks upheld. More stable interfaces accompanied a lower cancellation rate (r = −0.65, 95% CI −0.73 to −0.54, n = 150 engagements). Customer satisfaction among active engagements was higher where interfaces were rated more stable (r = 0.65, 95% CI 0.52 to 0.76, n = 94 engagements). Interface stability and the engagement indicator mean computed without it were associated as well (r = 0.86, 95% CI 0.81 to 0.89, n = 150 engagements). Where interfaces held steady, respondents reported lower intent to leave (r = −0.65, cluster bootstrap 95% CI −0.71 to −0.58, n = 747 respondents). Distressed ratings came from 243 of the 796 answers (30.53%), and the engagement mean was distressed on 36 of the 150 engagements.

Interface stability was the subject of 106 written responses, nearly all in comment on the third-party ecosystem, with 90 negative and 15 positive. Every negative response described a provider changing a file layout, field, interface, page, or model version that the agents depended on. Several said it happened with each release or each cycle, leaving the team to remap or re-enter the data by hand.

“One morning the ratings feed renamed a field and every account in that run came back unrated… nobody told us beforehand, we found it from the exception list. My team rebuilt the mapping by hand that day, then again the next week when the provider put the old name back.” -Individual Contributor, KPO, Credit, underwriting and suitability

“on my last posting we only had one internal Feed and it never move, here it is four external ones and every release something shift, i revert back to checking column order by hand before i load anything. nobody discuss about it until something break.” -Individual Contributor, BPO, Corporate and back-office functions

“In April the Benchmark provider changed their file layout, extra column in middle, and the agent read every field one place shifted. Nobody told us. We found it because sector weights did not sum. Their system has no proper interface also, we log in to a screen and download manually” -Individual Contributor, KPO, Trade, settlement and reconciliation

Written sentiment about interface stability showed no statistically significant association with the study measures. Negative accounts described manual remapping or data entry following changes. Some positive accounts described interfaces that had remained stable for a year or more.

3.2 Management of Provider-Initiated Model Retirement

Indicator C185 · Rate how well the engagement absorbs model version deprecations and retirements forced by model providers · Full statistical analysis ›

Respondents described more manageable model retirements where advance notice, dependency information, and a way to compare replacement versions were available. The mean rating was 4.6 and the median 5.0 (n = 793 respondents). Model retirements were absorbed less well on canceled engagements (mean 3.1, n = 299 respondents) than on active engagements in any stage.

How well the engagement absorbed forced model retirements was associated with every core study measure, and each association held through the applicable robustness checks. Engagements that absorbed retirements well had a lower observed cancellation rate (r = −0.64, 95% CI −0.72 to −0.53, n = 150 engagements). Customer satisfaction among active engagements was higher where retirements were absorbed well (r = 0.71, 95% CI 0.59 to 0.80, n = 94 engagements). Absorbing retirements well also accompanied a higher engagement indicator mean computed without this indicator (r = 0.87, 95% CI 0.83 to 0.91, n = 150 engagements). Respondents who rated the absorption of retirements higher reported lower intent to leave (r = −0.66, cluster bootstrap 95% CI −0.72 to −0.60, n = 748 respondents). Distressed ratings came from 255 of the 793 answers (32.16%), and the engagement mean was distressed on 37 of the 150 engagements.

The written responses addressing model retirement numbered 39, all in comment on the third-party ecosystem, with 24 negative and 15 positive. The accounts divided on notice and preparation. Engagements with a month's notice and a way to compare the old and new versions absorbed retirements without breakage. Engagements with short notice, no second model, and no list of what depended on the old version lost days of work.

“In June the model version is retired with two weeks notice and every valuation template we build stop working. Nobody know which templates depend on it, we go through them one by one for a week.” -Individual Contributor, KPO, Corporate and back-office functions

“When the model version was retired we lost a week, my team worked two saturdays rebuilding prompts, nobody above counts that as work” -Individual Contributor, KPO, Corporate and back-office functions

“what they did was retire the model version we had all our tax prompts built on, and the notice reached us the week it happened. We find out on a Monday when the memos start coming back in a different format, headings gone, the position summary at the bottom instead of the top. It took us maybe three days to get back to normal.” -Individual Contributor, KPO, Regulatory, audit, tax and legal

“in all honesty this part is the easy one. We had two weeks notice on the model version retirement, and i just run the old prompt set against the new one for a day and compare shortlist before we switch” -Individual Contributor, CXM, Corporate and back-office functions

“Earlier a model version change used to break our runs, now we get a proper notice window and the last switch was painless… that part has improved.” -Individual Contributor, KPO, Corporate and back-office functions

Written sentiment about model retirement showed no statistically significant association with the study measures. Respondents associated manageable transitions with notice, dependency information, and comparison of replacement models, but the study did not isolate the effect of those preparations.

i raised this in stand up many times, nothing change. What must change is simple. Somebody has to tell us before the Processor changes their api. We got no notice at all, not once. One morning the sandbox key stop working and i spend whole day finding out why. Nobody in my team knew, and person who knows is not in our Meeting anyway. Second, model version. They retire the version we build on and give us two weeks. Two weeks for pipeline that took two months. i had to revert back and rewrite the prompts, also half the parsing broke. This is not in any plan. Third, network access. We never got proper access to payment network sandbox, so we are testing on sample file somebody exported in march. Sample file is not real traffic. Everything we validated is on old data. So, notice before change, longer window on version retire, real access. That is all. Nothing here is complicated, it just needs somebody to own it. Nobody owns it. That is why it went the way it did.”

INDIVIDUAL CONTRIBUTOR · ITO, Technology engineering and operations

4 Contractual and Licensing Coverage of Agent Work

Respondents reported gaps in contractual responsibility for disruptive provider changes and uncertainty about licenses written for named human users. Two indicators assess contractual coverage of provider changes and licensing coverage of work performed by agents. The analysis concerns respondents’ assessments of these arrangements, not a legal review of the agreements.

Accounts described divided responsibilities among the financial institution, delivery vendor, and third-party provider. Respondents also raised uncertainty about applying licenses for named human users to automated activity.

4.1 Contractual Coverage of Disruptive Provider Changes

Indicator C184 · Rate how well the contracts with third-party providers deal with provider changes that can break the agents on this engagement · Full statistical analysis ›

Accounts of third-party contracts identified gaps between the firm holding the contract and the vendor operating agents affected by provider changes. The mean rating was 4.7 and the median 5.0 (n = 799 respondents). The contracts were rated lower on canceled engagements (mean 3.2, n = 307 respondents) than on active engagements in any stage.

The handling of provider changes that disrupt agent operation in third-party contracts was associated with all four core study measures, and every one of the applicable robustness checks upheld the associations. Engagements whose contracts dealt better with such change had a lower observed cancellation rate (r = −0.64, 95% CI −0.73 to −0.54, n = 150 engagements). Customer satisfaction among active engagements was higher where the contracts were rated stronger (r = 0.67, 95% CI 0.54 to 0.77, n = 94 engagements). Stronger contracts also accompanied a higher engagement indicator mean, computed without the contract indicator (r = 0.86, 95% CI 0.82 to 0.90, n = 150 engagements). Respondents who rated the contracts higher reported lower intent to leave (r = −0.66, cluster bootstrap 95% CI −0.72 to −0.59, n = 747 respondents). Distressed ratings came from 256 of the 799 answers (32.04%), and the engagement mean was distressed on 41 of the 150 engagements.

Few written responses addressed the contracts, 21, most in comment on the third-party ecosystem, and 14 of the 21 were negative. The negative responses said that the contracts were silent on notice of change and on remedy when a change broke the agents. Others said the delivery team held no contract with the third-party provider at all. The positive responses described clauses on change and notice added during the engagement.

“we were never the customer of any of these providers. We depend on nine custodian and counterparty feeds, and not one of them has any obligation to us, they have it to the bank, and i have never seen what that arrangement says … In week six one of them redesigned the login page and the agent simply stopped collecting. The breaks went to zero on the dashboard overnight … ” -Engagement Manager, BPO, Trade, settlement and reconciliation

“What I marked down sits on our paper rather than on theirs. The contracts carry no remedy where a provider changes something that breaks the agents, and that is a gap we and the client left in the drafting between us, not a failing of the provider.” -Portfolio Director, CXM, Payments, cards and disputes

“What the numbers miss is that our own people hear about a provider change before any contract obliges anybody to tell us. Word travels, the paperwork does not. The contracts say nothing useful about a feed changing its format on a Tuesday,” -Engagement Manager, BPO, Trade, settlement and reconciliation

“Contracts side has improved a great deal since last year, our legal put change clauses in with both feed providers and that has already saved us once.” -Engagement Manager, BPO, Corporate and back-office functions

More negative written sentiment about contracts was associated with higher intent to leave. With only 21 responses, this association is exploratory. Accounts described perceived gaps between the party holding the third-party contract and the vendor operating the affected agents.

4.2 Third-Party Licensing of Work Performed by Agents

Indicator C195 · Rate how well third-party licenses on this engagement cover work done by agents rather than named human users · Full statistical analysis ›

Licenses written for named human users left respondents uncertain about permission and charges for work performed by agents. The mean rating was 4.6 and the median 5.0 (n = 799 respondents). Licenses were judged to cover agent work less well on canceled engagements (mean 3.2, n = 302 respondents) than on active engagements in any stage.

License coverage of agent work was associated with every core study measure, and each association survived the applicable robustness checks. Engagements with licenses that covered agent work had lower observed cancellation rates (r = −0.63, 95% CI −0.72 to −0.53, n = 150 engagements). Active engagements whose licenses covered agent work reported higher customer satisfaction (r = 0.69, 95% CI 0.57 to 0.78, n = 94 engagements). The engagement indicator mean, computed without the licensing indicator, tracked license coverage closely (r = 0.88, 95% CI 0.84 to 0.91, n = 150 engagements). Intent to leave was lower among respondents who rated license coverage higher (r = −0.64, cluster bootstrap 95% CI −0.70 to −0.58, n = 752 respondents). Distressed ratings came from 255 of the 799 answers (31.91%), and the engagement mean was distressed on 34 of the 150 engagements.

The written responses addressing licensing numbered 36, all in comment on the third-party ecosystem, and 30 of the 36 were negative. Respondents described licenses written per named human user, agents run under one person's seat, and a question about what an agent owed that was raised and never answered.

“The HR platform and the ticketing tool are both licensed per named user, and neither contract was written with a machine identity in mind. We have been running the assistant under a service account that is, strictly speaking, a person's seat — legal on our side has looked at it, the client's procurement has not, and nobody has wanted to be the one to open that conversation while the renewal is being discussed.” -Portfolio Director, CXM, Corporate and back-office functions

“almost every license in that architecture is written per named human user, and nobody on either side had a clear answer on what an agent counts as. We asked. We got a shrug from procurement and a maybe from the provider. On a cancelled engagement it does not matter. On one that scales it is a bill nobody has modeled.” -Portfolio Director, ITO, Regulatory, audit, tax and legal

“the problem is licences. our matching tool is licensed per named user, and the agent runs under a colleague login, so when the custodian statement volume went up we were told to buy more seats before another queue could be added.” -Individual Contributor, BPO, Trade, settlement and reconciliation

“Nothing here covers licensing of the agents themselves. Our market data licence is written per named user, and whether an agent counts as one has never been settled. We raised it with procurement and are still waiting to discuss about it properly.” -Engagement Manager, KPO, Trade, settlement and reconciliation

Written sentiment about licensing showed no statistically significant association with the study measures. Respondents identified uncertainty about permissions and potential charges. These accounts do not establish an actual liability. A few described renegotiated arrangements that resolved their concerns.

Half the third party consent sources have no API at all, so we screen scrape them. Literally someone on my team has a script that clicks through the provider's web console every night and dumps it to csv. If they change a button we find out in the morning when the file is empty. And nobody has ever asked whether we're allowed to do that, or whether their licence covers a Bot logging in. We just do it, because otherwise nothing loads.”

INDIVIDUAL CONTRIBUTOR · ITO, Technology engineering and operations

5 Machine Interfaces, Feed Data, and Rate Limits

Unavailable machine access, unusable feed data, and processing limits were reported to require supplementary manual work. Three indicators assess the availability of machine interfaces, the usability of incoming feed data, and the adequacy of rate limits and quotas. These capabilities affect whether agents can use third-party systems at the required volume without supplementary manual processing.

Respondents described manual data transfer, feed preparation, and intervention when processing limits were reached. Part 5 examines whether this work was reflected in reported performance.

“Half the sources we need were never built for a machine to read, they are screens for a human researcher, and no vendor ever promised us otherwise, we assumed it.” -Engagement Manager, KPO, Client servicing and communications

5.1 Availability of Machine Interfaces Across Third-Party Systems

Indicator C186 · Rate the availability of true machine interfaces, rather than screens built for humans, across the third-party systems the agents must use · Full statistical analysis ›

Unavailable machine interfaces, meaning programmatic access to third-party systems, left some engagements relying on interfaces designed for people or manual data transfer. The survey described these as “true machine interfaces.” The mean rating was 4.6 and the median 5.0 (n = 802 respondents). The mean on canceled engagements was 3.1 (n = 299 respondents), below those for active engagements in each stage.

The availability of machine interfaces was associated with all four core study measures, and none of the applicable robustness checks overturned an association. Cancellation was less common where third-party systems offered machine interfaces (r = −0.63, 95% CI −0.71 to −0.52, n = 150 engagements). Customer satisfaction among active engagements was higher where such interfaces were available (r = 0.71, 95% CI 0.60 to 0.80, n = 94 engagements). Interface availability was also associated with a higher engagement indicator mean computed without it (r = 0.88, 95% CI 0.84 to 0.91, n = 150 engagements). Respondents who rated interface availability higher reported lower intent to leave (r = −0.66, cluster bootstrap 95% CI −0.72 to −0.60, n = 754 respondents). Distressed ratings came from 239 of the 802 answers (29.80%), the lowest observed distressed share among the 100 rated indicators, and the engagement mean was distressed on 40 of the 150 engagements.

The written responses addressing machine interfaces numbered 60, nearly all in comment on the third-party ecosystem, and 49 of the 60 were negative, 36 with the strongest negative score. Every negative response described a third-party system with no machine interface. The agent operated web pages built for human users, or a staff member transferred the data manually.

“Their certification portal is a set of screens built for a person with a token in their hand, so every run needed one of my engineers sitting there anyway, and nothing on this form has a place for that.” -Engagement Manager, ITO, Payments, cards and disputes

“half the tools we use have only a screen, no proper interface, so the agent was clicking through a portal built for a person. what we needed was a written api commitment from each provider and thirty days notice of any change. we got neither … ” -Engagement Manager, ITO, Technology engineering and operations

“They give us a bureau screen with no proper interface, so my team copy counterparty rating by hand into The study before agent can use it… That is two hour of my evening, every day. Nobody count this work anywhere, my team stay late and the queue still look fine on their side” -Individual Contributor, KPO, Credit, underwriting and suitability

“The screen scraping is what costs my team their evenings. Fund admin portal has no proper interface so the agent clicks through pages like a person, and when the portal is slow at month end we sit and watch it and re-run the failed ones by hand.” -Engagement Manager, BPO, Trade, settlement and reconciliation

“the medical records portal has no api, agent works it through the screen like a person. when they moved the buttons last month it just stopped, no warning. we find out from the backlog. also nobody left here still knows how to pull those records the manual way” -Individual Contributor, BPO, Credit, underwriting and suitability

Negative written responses and more negative engagement-level sentiment about machine interfaces were associated with cancellation. More negative respondent-level sentiment was associated with higher intent to leave. Accounts described both agent interaction with human-oriented interfaces and manual transfer of data where machine access was unavailable.

5.2 Usability of Third-Party Feed Data

Indicator C188 · Rate the quality of the data arriving from third-party feeds, judged by whether agents can use it without human cleanup · Full statistical analysis ›

Some third-party feed data required manual correction before agent use, with the intervention not consistently logged or reflected in reporting. Its mean rating was 4.6 and its median 5.0 (n = 795 respondents). Canceled engagements were marked by poor feed data, with a mean rating of 3.1 (n = 301 respondents), lower than on active engagements in any stage.

The usability of third-party feed data without cleanup showed associations with every core study measure, each upheld by the applicable robustness checks. Engagements receiving cleaner feeds had a lower cancellation rate (r = −0.64, 95% CI −0.73 to −0.54, n = 150 engagements). Customer satisfaction among active engagements was higher where feeds arrived usable (r = 0.68, 95% CI 0.55 to 0.77, n = 94 engagements). Cleaner feeds accompanied a higher engagement indicator mean, computed without the feed indicator (r = 0.87, 95% CI 0.82 to 0.90, n = 150 engagements). Intent to leave was lower among respondents who received usable feeds (r = −0.65, cluster bootstrap 95% CI −0.71 to −0.58, n = 748 respondents). Distressed ratings came from 249 of the 795 answers (31.32%), and the engagement mean was distressed on 35 of the 150 engagements.

The written responses addressing feed data numbered 89, with 64 negative and 25 positive. The comments on the third-party ecosystem divided between feeds that arrived clean and feeds that needed a person to clean them first. The responses to the questions on altering data described correcting the data by hand, in many accounts every cycle and in several cases without logging the correction.

“The test data we pulled from the wealth platform's sandbox arrived in a shape the agents could not use without a person tidying it first, every single day, and nobody had counted that person's time anywhere.” -Engagement Manager, ITO, Technology engineering and operations

“The one that still costs my people evenings is the counterparty file, it comes in dirty and somebody sits and cleans it before the model runs. That is a person, every month end, and it does not appear in any plan.” -Engagement Manager, KPO, Credit, underwriting and suitability

“What we do now is run everything from the rating vendor through a check sheet first, before it ever hits the config. My team built it ourselves off the errors we kept seeing, it's maybe forty fields. Takes ten minutes a load. If a value looks off we hold it and go back to the source file insted of letting the agent take it. No one asked us to do that, it just became the way we work.” -Individual Contributor, ITO, Client servicing and communications

“last week the comparables feed came with two different issuer names for same entity, one with the old ticker. we keep our own mapping file for this, it is just a list of isin against the name we use, and one of the junior updates it every morning before the deal team asks for anything. if we dont do that the agent picks wrong comparable and the pricing page is nonsence. nobody asked us to build it, we just did it and now everybody depends on it” -Individual Contributor, KPO, Trade, settlement and reconciliation

“formats change between one delivery and the next practically every month, moreover nobody on their side tells us in advance when a column gets renamed or a field gets dropped, so I fix the file myself before it break anything downstream in the ledger.” -Individual Contributor, BPO, Trade, settlement and reconciliation

More negative written sentiment about feed data was associated with higher intent to leave. Some accounts described quality measures applied after manual cleaning and corrections known principally to the staff performing them. These reporting practices are examined in Part 5.

5.3 Third-Party Rate Limits Under Agent Volume

Indicator C189 · Rate how well third-party rate limits and quotas hold up under the volume the agents on this engagement generate · Full statistical analysis ›

Respondents described third-party rate limits interrupting agent runs and staff completing work outside normal hours when planned volumes exceeded available capacity. The mean rating was 4.6 and the median 5.0 (n = 805 respondents). Rate limits on canceled engagements were rated lower (mean 3.2, n = 302 respondents) than on active engagements in any stage.

Ratings of the adequacy of third-party quotas for agent workloads were associated with all four core measures under the applicable robustness checks. Higher ratings were associated with lower cancellation (r = −0.64, 95% CI −0.73 to −0.53, n = 150 engagements), higher active-engagement customer satisfaction (r = 0.66, 95% CI 0.53 to 0.76, n = 94 engagements), and a higher engagement indicator mean excluding this item (r = 0.86, 95% CI 0.81 to 0.90, n = 150 engagements). Higher respondent ratings were associated with lower intent to leave (r = −0.63, cluster bootstrap 95% CI −0.69 to −0.55, n = 756 respondents). Distressed ratings came from 265 of the 805 answers (32.92%), and the engagement mean was distressed on 36 of the 150 engagements.

The written responses addressing rate limits numbered 42, all in comment on the third-party ecosystem, with 32 negative and 8 positive. Respondents described agents cut off by daily caps and throttles at predictable peaks, and staff who watched or finished the runs late at night. Several placed the responsibility on the engagement side, in volumes the engagement had neither sized nor reported to the third-party provider.

“In week six we tried to refresh about four thousand supplier records against the external registry feed in one run. It cut us off after eleven hundred, and there was no way to see that from our side until the tickets started failing. We ended up doing the rest in batches overnight for a week.” -Engagement Manager, CXM, Corporate and back-office functions

“We keep two log in from the research database and rotate them, because the tool hit the limit by eleven in the morning and then everything just stop … No one ask us for this, it is simply what we do now” -Individual Contributor, KPO, Regulatory, audit, tax and legal

“Rate limits are the part that lands on us personally. When the provider throttles at month end the queue does not stop, it simply waits for us, and two of my engineers sit until eleven at night watching a bar they cannot influence. I have lost one good person partly for that reason. It is not a large technical problem, but it falls on the same four people every month.” -Engagement Manager, ITO, Technology engineering and operations

“Their Portal throttles us after around two hundred pulls and the agent just stops, no error, we find out when case aging report looks wrong next day.” -Individual Contributor, BPO, Credit, underwriting and suitability

“the hr feed gets blamed for everything here, if I am honest it is not their fault. nobody informed them we are pulling every hour now, and their limits are published anyway, we simply ignore it. so i do not put it all on them.” -Individual Contributor, ITO, Technology engineering and operations

Written sentiment about rate limits showed no statistically significant association with the study measures. Accounts suggest gaps in capacity planning and communication of expected volumes. They do not establish that all interruptions were predictable or that responsibility was universally unassigned.

Compared with the payments engagement I governed before this one, the third-party position here is considerably more comfortable; there we sat behind a single screening vendor with no realistic alternative, whereas here the list providers can be swapped and the volumes have never troubled anyone's limits. What is less comfortable, and I did not appreciate it until we ran the transition, is how much of the old quality came from two long-serving analysts who simply knew which feed to distrust on a Monday.”

PORTFOLIO DIRECTOR · KPO, Financial crime and client onboarding

6 Security Posture and Provider Concentration

Incomplete dependency visibility and reliance on a small number of model or cloud providers raised concerns about security assessment and continuity. Three indicators assess third-party security, management of dependence on a small number of model and cloud providers, and continuity during a provider switch. The concentration indicator measures how well dependence is managed, not the number of providers or the degree of concentration itself.

The 46 written responses describe incomplete dependency inventories, concentrated provision, and limited evidence of tested alternatives. These accounts provide context for the ratings but do not constitute an independent security or resilience assessment.

6.1 Security of Third-Party Tools and Subprocessors

Indicator C180 · Rate the security posture of the third-party tools, platforms, and sub-processors the engagement’s agents connect to · Full statistical analysis ›

The small written-answer set on third-party security identified incomplete dependency inventories and reviews limited to direct providers. The mean rating was 4.6 and the median 5.0 (n = 788 respondents). Respondents on canceled engagements regarded the security of third-party tools less favorably (mean 3.1, n = 303 respondents) than respondents on active engagements in any stage.

The security posture of third-party tools and sub-processors was associated with all four core study measures, and the associations held under the applicable robustness checks. Engagements whose third-party tools were judged more secure had a lower cancellation rate (r = −0.65, 95% CI −0.74 to −0.55, n = 150 engagements). Customer satisfaction among active engagements was higher where security posture was rated stronger (r = 0.64, 95% CI 0.51 to 0.75, n = 94 engagements). A stronger security posture accompanied a higher engagement indicator mean, computed without the security indicator (r = 0.87, 95% CI 0.82 to 0.90, n = 150 engagements). Intent to leave was lower among respondents who rated security posture higher (r = −0.65, cluster bootstrap 95% CI −0.71 to −0.59, n = 742 respondents). Distressed ratings came from 251 of the 788 answers (31.85%), and the engagement mean was distressed on 36 of the 150 engagements.

Nine written responses addressed security posture: four negative, three positive, and two neutral. Several described incomplete inventories or reviews limited to the first layer of providers. This small sample supports illustrative observations only.

“nobody has one list of what the desk agent actually connects to… i count four tools just for endpoint patching and the remote session, and the security review only covers two of them.” -Individual Contributor, ITO, Technology engineering and operations

“The benefits provider gives one login and we all share it, that is what I would flag… nobody checks who touched what. In addition no one here can tell me the full list of outside systems that touch our payroll informations … ” -Individual Contributor, BPO, Corporate and back-office functions

“What has not moved is the chain behind them. We know our direct suppliers. Ask who their sub-processors are and nobody can tell us, and the security review stopped at the first layer.” -Engagement Manager, CXM, Corporate and back-office functions

“My 5 on security is only about one market data provider, i never see their sub processor list, the rest i had no issue with” -Individual Contributor, BPO, Corporate and back-office functions

The accounts identify dependency visibility as a potential limitation on security assessment. They do not establish that every unlisted provider had escaped review through other processes.

6.2 Management of Provider Concentration

Indicator C193 · Rate how well the engagement manages its dependence on a small number of model and cloud providers · Full statistical analysis ›

Accounts of provider concentration described dependence on a small number of model or cloud providers without a configured alternative or tested outage plan. The mean rating was 4.59, among the three lowest in the category, and the median 5.0 (n = 802 respondents). The mean on canceled engagements was 3.1 (n = 308 respondents), below the means for active engagements in each stage. Lower scores indicate weaker management of dependence, not necessarily greater concentration.

The management-of-dependence rating was associated with all four core study measures, with the associations retained under the applicable robustness checks. Higher ratings were associated with a lower cancellation rate (r = −0.64, 95% CI −0.73 to −0.53, n = 150 engagements), higher customer satisfaction among active engagements (r = 0.64, 95% CI 0.50 to 0.74, n = 94 engagements), and a higher engagement indicator mean excluding this item (r = 0.86, 95% CI 0.81 to 0.90, n = 150 engagements). Higher respondent ratings were associated with lower intent to leave (r = −0.64, cluster bootstrap 95% CI −0.70 to −0.58, n = 755 respondents). Distressed ratings came from 257 of the 802 answers (32.04%), and the engagement mean was distressed on 41 of the 150 engagements.

The written responses addressing provider concentration numbered 26, nearly all in comment on the third-party ecosystem, with 19 negative and 4 positive. Respondents described agents that depended on one model provider, one cloud, or one region with no second path configured and no tested plan for an outage.

“if the model provider go down one morning we are finished, there is no second one set up, we would be closing tickets on paper. Nobody has ever tested that.” -Individual Contributor, CXM, Corporate and back-office functions

“The concentration is our own doing. We built everything on one model and one region because that is what could be approved quickly, and now moving is a project nobody will fund at POC stage.” -Engagement Manager, BPO, Financial crime and client onboarding

“One issue only. Every agent we run sits on one model provider and one cloud tenancy, with no second path. Three weeks ago that region degraded for about two hours and the provisioning queue simply stopped, we cleared it by hand. There is no plan on paper for an outage that lasts longer.” -Engagement Manager, ITO, Technology engineering and operations

More negative written sentiment about concentration was associated with cancellation and higher intent to leave. These findings are exploratory because the qualitative sample contains 26 responses. Accounts described concentrated provision retained after initial approval and reliance on manual processing as a fallback.

6.3 Continuity Through a Switch of Primary Model or Cloud Provider

Indicator C194 · Rate how well this engagement could keep operating if it had to switch its primary model or cloud provider · Full statistical analysis ›

Written accounts of continuity through a model or cloud provider switch expressed expectations rather than describing a tested transition. The mean rating was 4.6 and the median 5.0 (n = 811 respondents). Continuity through a provider switch averaged 3.1 on canceled engagements (n = 301 respondents), lower than on active engagements in any stage.

The ability to continue through a switch of primary provider showed associations with every core study measure that held under the applicable robustness checks. Cancellation was less common where continuity was rated stronger (r = −0.65, 95% CI −0.73 to −0.54, n = 150 engagements). Active engagements with stronger continuity reported higher customer satisfaction (r = 0.68, 95% CI 0.56 to 0.78, n = 94 engagements). The engagement indicator mean, computed without the continuity indicator, was closely associated with the rating (r = 0.87, 95% CI 0.82 to 0.90, n = 150 engagements). Respondents who rated continuity higher reported lower intent to leave (r = −0.66, cluster bootstrap 95% CI −0.71 to −0.60, n = 759 respondents). Distressed ratings came from 255 of the 811 answers (31.44%), and the engagement mean was distressed on 42 of the 150 engagements.

Eleven written responses addressed continuity, all in comment on the third-party ecosystem, with eight negative and three positive. The negative responses said the engagement could not move off its model provider quickly and had no plan to. The positive responses asserted that a move would be manageable without describing one. No response described a switch of provider that had been tested.

“We had nowhere to move to. Everything the agents do sits on that one provider and there was no discussion about an alternative because there is not one configured. we spent nine days re-testing the same onboarding cases against the new version, and the weekly pack still showed green.” -Engagement Manager, CXM, Financial crime and client onboarding

“If we have to move to another model provider we are stuck, that needs a proper plan” -Individual Contributor, ITO, Technology engineering and operations

“what i would change is we are pinned to a single model provider and nobody has ever tried standing the workflows up on a second one. not saying it would break. saying nobody knows, and finding out during an outage is a bad time to learn” -Individual Contributor, CXM, Corporate and back-office functions

“I scored this section high and I want to be clear about why, because it is not sophistication. Our recruiting volume is small enough that if the scheduling API or the model went away tomorrow, four coordinators could run the calendar manually for a week, the same as we did before this program existed. That is the whole reason. We have not tested a provider switch, we have never had to, and I would not want the high marks read as evidence that we could.” -Engagement Manager, CXM, Corporate and back-office functions

Written sentiment about continuity showed no statistically significant association with the study measures. The 11 responses described expectations of continuity rather than documented tests of a provider switch.

the one thing here is the watchlist file, when provider add new format column nothing shout about it, the screening just start passing name it should stop. we catch it in monitoring after two days that time. rest of this section is fine, we can move provider if we have to”

INDIVIDUAL CONTRIBUTOR · ITO, Financial crime and client onboarding

7 Fallback Capacity and Retention of Operational Knowledge

Accounts of third-party failure distinguished retained manual capability from knowledge lost through staff departures or undocumented corrections. Four indicators assess fallback readiness, retention of manual capability, knowledge of previously informal corrections, and replacement of those corrections with explicit controls. They address whether automation preserves the operational knowledge needed when a dependency fails.

Respondents described manual processes as a fallback for unavailable feeds or models. Some accounts also described the loss of experienced staff or undocumented corrections after automation. These observations suggest a possible resilience gap. They do not establish that agent deployment itself caused the loss of knowledge.

“For years my seniors carried the quirks of the custodian files in their heads — one custodian sends the fee accrual a day late, another restates a month after the fact, and the person who knew that simply corrected it on the Monday. When the agents came in, none of that was written down anywhere, it was lost, and the two people who held most of it have since left.” -Engagement Manager, KPO, Trade, settlement and reconciliation

“Half the people who used to patch these files by hand are not on the account anymore.” -Engagement Manager, CXM, Regulatory, audit, tax and legal

7.1 Fallback Readiness for Critical Third-Party Failure

Indicator C179 · Rate the quality of the fallback the engagement has ready when a critical third-party feed, platform, or model stops working · Full statistical analysis ›

Accounts of critical third-party service failure distinguished work that stopped from continuity supported by saved data, manual procedures, or alternative sources. Its mean rating was 4.6 and its median 5.0 (n = 802 respondents). Respondents on canceled engagements gave fallback readiness its lowest ratings, a mean of 3.1 (n = 301 respondents) against higher means on active engagements in any stage.

Each of the four core study measures was associated with fallback readiness, and no association failed a robustness check. Cancellation was rarer on engagements with a ready fallback (r = −0.64, 95% CI −0.73 to −0.54, n = 150 engagements). Customer satisfaction among active engagements was higher where a fallback was ready (r = 0.69, 95% CI 0.57 to 0.79, n = 94 engagements). Fallback readiness and the engagement indicator mean computed without it rose together (r = 0.87, 95% CI 0.83 to 0.91, n = 150 engagements). Respondents who rated fallback readiness higher reported lower intent to leave (r = −0.66, cluster bootstrap 95% CI −0.72 to −0.59, n = 753 respondents). Distressed ratings came from 257 of the 802 answers (32.04%), and the engagement mean was distressed on 39 of the 150 engagements.

Fallback readiness was the subject of 80 written responses, all in comment on the third-party ecosystem, with 57 negative and 21 positive. In 25 responses, respondents said there was nothing behind a critical feed or model. The engagement's tasks stopped and waited. The positive responses described fallbacks the teams had built for themselves, usually a saved copy or a manual route, and a few described tested secondary sources.

“we now have a tested fallback that routes tickets to the queue if the model endpoint drops – it has fired twice this quarter and no employee noticed.” -Engagement Manager, CXM, Corporate and back-office functions

“only worry i have is we run everything on one provider, if that provider go down for a day i do not know what we do, nobody has shown me a second option. but that is our own planning gap, not their fault.” -Individual Contributor, KPO, Corporate and back-office functions

“On my last team when a feed went down somebody just pulled the file manually and we kept going. Here there is no backup, the bureau feed stopped for two days in June and the whole validaton queue just sat there. Nobody could tell me who to call” -Individual Contributor, KPO, Credit, underwriting and suitability

“last month the vendor of our knowledge search changed their api and our policy lookups came back empty for two days. no notice came to us, we found it from tickets piling up. and there was no fallback, we answered by hand from pdfs” -Individual Contributor, CXM, Corporate and back-office functions

“We keep a manual channel open on the payment file, one person still knows how to submit it the old way and we run that once a week just to keep the muscle. Same with vendor master pulls, we take a copy every Monday so if the feed goes quiet we are working from something. Not elegant. But it has saved us twice this year.” -Engagement Manager, BPO, Corporate and back-office functions

More negative written sentiment about fallback arrangements was associated with higher intent to leave. Respondents described saved data, practiced manual procedures, and alternative sources as ways to maintain operations during an outage.

7.2 Retention of the Ability to Perform Manual Workarounds

Indicator C190 · Rate the engagement’s ability to still perform by hand the manual workarounds that once absorbed third-party quirks and breakage · Full statistical analysis ›

Respondents described both retained capacity to work by hand and gaps following the departure of experienced staff or insufficient training of newer staff. The mean rating was 4.66, among the highest of the 20 indicators of the category, and the median 5.0 (n = 808 respondents). Respondents on canceled engagements judged their ability to work by hand weaker (mean 3.2, n = 311 respondents) than respondents on active engagements in any stage.

The retention of manual capability held its association with all four core study measures across the applicable robustness checks. Engagements that kept the ability to work by hand had lower observed cancellation rates (r = −0.65, 95% CI −0.73 to −0.54, n = 150 engagements). Customer satisfaction among active engagements was higher where manual capability was retained (r = 0.72, 95% CI 0.60 to 0.80, n = 94 engagements). The engagement indicator mean, computed without the manual-capability indicator, was strongly correlated with the rating (r = 0.88, 95% CI 0.84 to 0.91, n = 150 engagements). Intent to leave was lower among respondents who said manual capability had been kept (r = −0.67, cluster bootstrap 95% CI −0.73 to −0.62, n = 757 respondents). Distressed ratings came from 249 of the 808 answers (30.82%), and the engagement mean was distressed on 38 of the 150 engagements.

The written responses addressing manual capability numbered 43, all in comment on the third-party ecosystem, and they divided evenly, with 22 negative and 21 positive. The negative responses said the people who knew the manual methods had left without writing them down, and newer staff had never learned them. The positive responses described outages worked through by hand, some with the caveat that this held only at low volume or while the staff who had performed the tasks before the agents remained.

“The client released its own service desk staff in month three, on the basis that the bot would carry the volume. When the platform was down for most of a Tuesday in July, there was nobody left who knew how to open an IT ticket manually — we had my agents doing it from memory. You cannot un-remove those people. That is the risk nobody priced.” -Engagement Manager, CXM, Corporate and back-office functions

“We are two months in with a team that has never triaged without the agent in front of them, so if the enrichment feed drops I am not sure we could work the queue by hand at volume.” -Engagement Manager, ITO, Technology engineering and operations

Written sentiment about manual capability showed no statistically significant association with the study measures. Accounts described both retained skills and gaps associated with staff departures or inadequate training. Positive accounts suggest that deliberate practice may support continuity, although its effect was not tested.

7.3 Knowledge of Informal Third-Party Corrections

Indicator C191 · Rate how completely the engagement knows where humans used to quietly fix third-party data and interface problems before agents took over · Full statistical analysis ›

Undocumented informal corrections to third-party problems were reported to be rediscovered after failures when experienced staff had left. The mean rating was 4.6 and the median 5.0 (n = 792 respondents). Respondents on canceled engagements knew less about the informal corrections (mean 3.1, n = 305 respondents) than respondents on active engagements in any stage.

Knowledge of the informal corrections was associated with cancellation, customer satisfaction, the engagement indicator mean, and intent to leave, and all applicable robustness checks upheld the associations. Engagements that knew where the informal corrections had been made had lower observed cancellation rates (r = −0.66, 95% CI −0.74 to −0.56, n = 150 engagements). Customer satisfaction among active engagements was higher where this knowledge was rated better (r = 0.61, 95% CI 0.47 to 0.73, n = 94 engagements). Where the informal corrections were known, the engagement indicator mean computed without this indicator was higher (r = 0.87, 95% CI 0.82 to 0.90, n = 150 engagements). Intent to leave was lower among respondents who knew where the fixes had been made (r = −0.66, cluster bootstrap 95% CI −0.71 to −0.60, n = 746 respondents). Distressed ratings came from 262 of the 792 answers (33.08%), and the engagement mean was distressed on 39 of the 150 engagements.

The written responses addressing the informal corrections numbered 21, all in comment on the third-party ecosystem, and 20 of the 21 were negative. Respondents said the manual fixes that had once absorbed third-party problems had never been written down, that the people who made them had left, and that each fix was rediscovered when something broke.

“If we knew where those fixes sat, they would not have been quiet, and we would have written them into the process years ago. The honest position is that we discover each one at the moment it breaks a valuation, and from then on it is no longer unknown.” -Engagement Manager, KPO, Corporate and back-office functions

“nobody wrote down what the old workplace team used to fix by hand before the agent took the queue, so when something looks wrong we are guessing at what normal was.” -Individual Contributor, CXM, Corporate and back-office functions

“The small manual corrections our engineers used to make on reference data were never written down anywhere before the agent took that job over” -Engagement Manager, ITO, Regulatory, audit, tax and legal

“Also there is no record of who fixed the supplier addresses before the agents came, those persons are gone.” -Engagement Manager, CXM, Corporate and back-office functions

“What we have not done is capture the little fixes our floor supervisors used to make before any of this.” -Engagement Manager, CXM, Payments, cards and disputes

Written sentiment about informal corrections showed no statistically significant association with the study measures. The 21 responses describe knowledge that had not been documented before changes in staffing or task allocation.

7.4 Replacement of Informal Corrections With Explicit Controls

Indicator C192 · Rate how deliberately the quiet human fixes for third-party quirks were replaced with explicit checks when agents took over the work · Full statistical analysis ›

Replacement of informal human corrections with explicit checks received the lowest mean rating among third-party ecosystem indicators. The mean rating was 4.58, the lowest of the 20 indicators of the third-party ecosystem, and the median 5.0 (n = 796 respondents). Respondents on canceled engagements rated the replacement of informal corrections lower than respondents on active engagements in any stage (mean 3.1, n = 302 respondents).

Whether informal corrections had been turned into explicit controls was associated with each core study measure, and every association survived the applicable robustness checks. Cancellation was less common where the informal corrections had been replaced (r = −0.64, 95% CI −0.73 to −0.54, n = 150 engagements). Customer satisfaction among active engagements was higher where the replacement was rated more complete (r = 0.71, 95% CI 0.59 to 0.80, n = 94 engagements). With this indicator excluded, the engagement indicator mean was closely associated with the rating (r = 0.88, 95% CI 0.84 to 0.91, n = 150 engagements). Respondents who rated the replacement higher reported lower intent to leave (r = −0.65, cluster bootstrap 95% CI −0.71 to −0.59, n = 748 respondents). Distressed ratings came from 267 of the 796 answers (33.54%), the highest distressed share in the category, and the engagement mean was distressed on 40 of the 150 engagements.

Nine written responses addressed the replacement of informal corrections: seven negative, one positive, and one neutral. Negative accounts described corrections that had not been incorporated into explicit checks when agents assumed the task. One account linked a missed correction to access remaining available after an employee’s departure. These observations are illustrative only.

“Before, when the credit bureau file came with the old company identifiers, somebody on my team just knew and swapped them over. It is not written anywhere, it sits in his head. Now the agent takes the file exactly as it arrives. Nobody sat down and turned that knowledge into a check… we assumed it would not matter. It has not caused a problem yet. But he is on leave next month.” -Individual Contributor, BPO, Payments, cards and disputes

“Hard to blame the feed vendors… they send what they always sent. We aimed an agent at it and never set out what a clean file looks like. Whoever used to catch this by hand has gone” -Individual Contributor, KPO, Trade, settlement and reconciliation

“You ask whether the quiet human fixes were deliberately replaced with checks, but nobody ever wrote down what those fixes were, so the honest answer is that we cannot know, and a four is me guessing.” -Engagement Manager, KPO, Credit, underwriting and suitability

The structured ratings assess respondents’ perceptions of control replacement. They do not measure the frequency of the specific incidents described in the nine written responses.

8 The Dependency Chain and Gated External Rails

Incomplete dependency inventories and restricted external access were reported to leave delivery teams without a full view of required systems or a fully automated route through them. Two indicators assess dependency mapping and the management of restricted-access external infrastructure, including payment networks, regulatory filing systems, and market utilities. The latter are described in the survey as gated external rails.

Dependency information can support notification, monitoring, security review, and fallback planning. The analysis also considers external systems whose access conditions require certification, fixed submission windows, or human participation.

8.1 Mapping of Third- and Fourth-Party Dependencies

Indicator C196 · Rate how completely the engagement has mapped every third and fourth party its agents depend on, down the full chain · Full statistical analysis ›

Some reported dependency inventories stopped at direct third-party providers or omitted the systems on which agents relied. The mean rating was 4.6 and the median 5.0 (n = 791 respondents). The dependency map was rated less complete on canceled engagements (mean 3.1, n = 302 respondents) than on active engagements in any stage.

Mapping of third- and fourth-party dependencies was associated with all four core study measures, each association upheld by the applicable robustness checks. Cancellation was less common where dependencies had been mapped (r = −0.64, 95% CI −0.72 to −0.53, n = 150 engagements). Among active engagements, customer satisfaction was higher where the map was rated fuller (r = 0.71, 95% CI 0.59 to 0.80, n = 94 engagements). A fuller map accompanied a higher engagement indicator mean computed without the mapping indicator (r = 0.87, 95% CI 0.83 to 0.91, n = 150 engagements). Intent to leave was lower among respondents who rated the map fuller (r = −0.68, cluster bootstrap 95% CI −0.73 to −0.62, n = 741 respondents). Distressed ratings came from 252 of the 791 answers (31.86%), and the engagement mean was distressed on 36 of the 150 engagements.

The written responses addressing the dependency map numbered 44, all in comment on the third-party ecosystem, with 35 negative and 8 positive. The negative responses said there was no list of what the agents depended on, or that the list stopped at the direct providers, and several had asked for one without receiving an answer. One respondent whose team had built such a list put its cost at two weekends of analysts' time.

“we went into a procurement desk without a list of what the agents would be leaning on beneath the obvious platform, and no view at all of what sits beneath that again. On the one occasion a supplier feed changed shape we found out because the queue stopped, not because anyone told us. Whether we could have moved off the model provider is academic; we would not have known what to move.” -Portfolio Director, CXM, Corporate and back-office functions

“somebody needs to own the dependency map past the first tier; we know our directs, we do not really know who they depend on, and that gap would matter a great deal on the day one of them has an outage … Neither is expensive. Both need a name against them and a date, or they stay in the deck.” -Portfolio Director, ITO, Technology engineering and operations

“the problem is the Rates feed changed its field names on a wednesday and nobody told us. every model that pulled from it came back blank and we lost two days keying rates in manually. i asked who owns that supplier, nobody could say. there is no list anywhere” -Individual Contributor, KPO, Corporate and back-office functions

“how am i to rate advance notice from providers when nobody tells us which providers sit behind the screening service.” -Individual Contributor, ITO, Financial crime and client onboarding

“Monitoring for unannounced provider changes is in place and it works, which is why those marks are high. We have the dependency chain mapped to the fourth party.” -Engagement Manager, KPO, Corporate and back-office functions

Negative written responses and more negative engagement-level sentiment about dependency mapping were associated with higher cancellation rates. More negative respondent-level sentiment was associated with higher intent to leave. Accounts suggest that incomplete inventories complicated coordination of other third-party controls.

8.2 Dependence on Gated External Rails

Indicator C197 · Rate how well the engagement manages agent dependence on gated external rails such as payment networks and regulatory submission systems · Full statistical analysis ›

Restricted-access external networks, regulators, and utilities were reported to retain steps that people had to perform, even where agentic delivery depended on them. The mean rating was 4.6 and the median 5.0 (n = 794 respondents). Respondents on canceled engagements rated the handling of gated rails lower than respondents on active engagements in any stage (mean 3.1, n = 296 respondents).

All four core study measures were associated with how well the engagement handled gated external rails, and the applicable robustness checks left the associations intact. Engagements that handled the rails better had a lower cancellation rate (r = −0.63, 95% CI −0.72 to −0.52, n = 150 engagements). Customer satisfaction among active engagements was higher where the rails were handled better (r = 0.69, 95% CI 0.56 to 0.78, n = 94 engagements). Handling of the rails was also associated with a higher engagement indicator mean computed without it (r = 0.86, 95% CI 0.81 to 0.90, n = 150 engagements). Intent to leave was lower among respondents who rated the handling of the rails higher (r = −0.65, cluster bootstrap 95% CI −0.71 to −0.58, n = 744 respondents). Distressed ratings came from 245 of the 794 answers (30.86%), and the engagement mean was distressed on 38 of the 150 engagements.

Fifteen written responses addressed the gated rails, nearly all in comment on the third-party ecosystem, with 13 negative and 2 positive. Respondents placed the weakness in certification portals and regulatory submission systems built for a person, fixed submission windows and call limits, and sandbox access that never arrived in full. In several engagements notice of provider change had improved while access to the rails had not.

“the tax authority portals dont care that were an agent. submission windows close at a fixed hour and if the model version changes the week of a deadline we have no route to push it. Feeds themselves are excellent, no complaint there. its the last mile into the filing systems thats out of our hands” -Individual Contributor, KPO, Regulatory, audit, tax and legal

“Feed providers do warn us now before they change something, that got better since January. What did not get better is the regulator portal, agent still cant file anything there, one of us does it by hand every week” -Individual Contributor, CXM, Client servicing and communications

“Same with statutory submission portal, it takes only so many a day.” -Individual Contributor, BPO, Corporate and back-office functions

More negative written sentiment about restricted-access infrastructure was associated with higher intent to leave. This result is exploratory because the qualitative sample contains 15 responses. Accounts described staff completing steps that external systems required a person to perform.

Across the 100 indicators, respondents rated financial institution readiness, co-management, and vendor capability less favorably than AI technology and the third-party ecosystem. The indicators were correlated with one another and associated with all four study measures. These relationships do not identify separate causes of failure. Part 3 synthesizes the governance implications. Part 5 examines the manual interventions described by delivery team members.

Notice from the major providers has improved markedly over the last year, to the point where we now plan around deprecations rather than react to them, and our handling of the gated rails, the payroll rails in particular, is as good as I have seen on any account. What has not moved is our answer to the concentration question. If we had to move providers, I could not tell a client with a straight face how long that would take, and we have been saying that for three quarters now.”

PORTFOLIO DIRECTOR · CXM, Corporate and back-office functions