Source: https://akholi.com/outsourcing-2026-agentic-ai-tier-1-banks/part-3-governance-oversight-control/vendor-preparation-and-accountability-in-the-control-environment/
Author: Phil Hatch
Publisher: Akholi

Vendor Preparation and Accountability in the Control Environment

The 2026 Report · Part 3

# Vendor Preparation and Accountability in the Control Environment

Vendor playbooks, quality assurance, and accountability for agent error

By Phil Hatch · Akholi · Version 1.8.2

Complete study · PDFPart 3 · PDFStudy overview / Part 3 / Vendor Preparation and Accountability in the Control Environment

The vendor's playbook and accountability ratings were associated with governance across the engagement. Both concern the vendor's ability to make agent work governable, through a route into controlled production and responsibility for costly errors. The governance, oversight, and control score summarizes the wider reported environment. Negative adjusted coefficients changed markedly when other vendor capability indicators were removed from the comparison mean, making model sensitivity central to interpretation.

Each indicator was associated with the score computed without it at r = 0.78 and 0.79 and with governance in the other four failure categories at r = 0.78 (n = 150 engagements). After adjustment for the mean of the other 53 indicators, the two ties turned inverse and significant, at partial r = −0.38 and −0.46, the second-strongest and strongest inverse results of the 47 governance indicators. The 53 include the vendor's other 18 capability indicators. With that block omitted from the comparison mean, the two partial r moved to 0.04 and −0.03, while leaving out any other category's block left them between −0.41 and −0.29. The change in estimates shows sensitivity to the vendor capability ratings in the comparison mean. It limits a distinct-effect interpretation of the negative coefficients. Table 3.10 sets out both indicators against these measures.

Table 3.10. The two vendor governance indicators against reported governance weakness across the engagement

| Indicator | A score of 3 or below among respondents rating it 1 to 3 | r with score excluding item (95% CI) | r with the other four categories | Adjusted partial r (p) | Negative written answers |
| --- | --- | --- | --- | --- | --- |
| Maturity of the Playbook From Concept to Governed Production | 232 of 392 (59.18%) | 0.78 (0.71 to 0.84) | 0.78 | −0.38 (p < 0.001) | 186 of 200 |
| Readiness to Take Accountability for a Costly Agent Error | 240 of 373 (64.34%) | 0.79 (0.72 to 0.84) | 0.78 | −0.46 (p < 0.001) | 29 of 38 |

Note: Rating-band and item-to-score results use the score excluding the tested indicator. Cross-category results exclude its entire category and use 150 engagements. Partial r accounts for the mean of the 53 indicators outside the score. Adjusted-correlation p values are nominal. The ordinary respondent-level band tests do not account for clustering within engagements, including where Monte Carlo calculation handles sparse cells. Their fractions remain descriptive. Zero observations are not proof of no risk.

> We were so busy fixing pre-prod issues that we never thought about how to do the actual work.” ENGAGEMENT MANAGER · KPO, Corporate and back-office functions

## 1 Delivery Playbook, Quality Assurance, and Platforms

The playbook rating connects preparation for governed production with the wider score. Its observed relationship was positive, while the main adjusted result was negative and became near zero under the category-removal comparison. The substantive question remains whether the vendor has a usable process for establishing controls before and during live operation. The sensitivity precludes interpreting the negative coefficient as a benefit of weaker preparation.

### 1.1 Maturity of the Playbook From Concept to Governed Production

Indicator C126 · Rate the maturity of your employer's playbook for taking agentic use cases from concept to governed production · Full statistical analysis ›

A less mature vendor playbook was associated with low scores, but its adjusted relationship depended strongly on the comparison measure. The substantive question is how the vendor takes agentic work from concept into governed production. A defined playbook can connect preparation, testing, responsibility, and ongoing operation. The negative conditional coefficient cannot be read as evidence that an immature playbook improves control, given the sensitivity to other vendor capability items.

Figure 3.53. Maturity of the Playbook From Concept to Governed Production: observed association and adjustment sensitivity.

Scroll horizontally to view the complete figure.

A less mature vendor playbook was associated with low scores, but its adjusted relationship depended strongly on the comparison measure.

Scroll horizontally to view the complete comparison.

| Comparison | Correlation r | Estimate and analysis base |
| --- | --- | --- |
| Unadjusted cross-category r | | r = 0.78 150 engagements |
| Adjusted, full comparison mean | | r = −0.38 150 engagements |
| Adjusted, excluding other vendor items | | r = 0.04 150 engagements |
| | −1 0 +1 | |

Exact comparison data

Unadjusted cross-category r

Exact chart data

| Series | Point index | X | Y |
| --- | --- | --- | --- |
| Reported association | 0 | 0.77504553003440457 | 1 |

Adjusted, full comparison mean

Exact chart data

| Series | Point index | X | Y |
| --- | --- | --- | --- |
| Reported association | 0 | -0.38185558726053881 | 1 |

Adjusted, excluding other vendor items

Exact chart data

| Series | Point index | X | Y |
| --- | --- | --- | --- |
| Reported association | 0 | 3.6341658180504735E-2 | 1 |

Note: r measures signed linear association from −1 to +1. Partial r describes the adjusted association under the model specified in the section. Adjusted p values are nominal. The comparisons do not establish causal effects.

Correlations use 150 engagements. The alternative mean removes the stated category items. Estimates are descriptive model comparisons, with nominal inference in the text.

Of the 392 respondents who rated the playbook 1 to 3, 232 had a score of 3 or below. Among the 396 who rated it 4 to 7, 34 (8.59%) still did, and none of the 19 who rated it 8 to 10 did (chi-square = 237.80, p < 0.001, Cramér's V = 0.54).

The playbook was associated with the score computed without it at r = 0.78 (95% CI 0.71 to 0.84, n = 150 engagements) and with governance in the other four categories at r = 0.78. After adjustment for the mean of the other 53 indicators, the partial r was −0.38 (p < 0.001). The result held when the financial institution readiness block was omitted from the comparison mean (−0.30, p < 0.001), when the AI technology block left it (−0.34, p < 0.001), and when the third-party block left it (−0.29, p < 0.001). It disappeared only when the vendor capability block was omitted from the comparison mean, at 0.04 (p = 0.660). Respondents who rated the playbook distressed recorded a score of 2.87 (n = 392), against 5.09 (n = 415) among the rest (cluster-robust p < 0.001).

The association was observed within the canceled engagements at r = 0.66 and across the active stages at 0.70 in Build-Out, 0.65 in Transition/Cut-Over, and 0.81 in Production. After Holm adjustment, it was significant in 86 of the 87 segment levels tested. Within role it stood at 0.78 among individual contributors, 0.75 among engagement managers, and 0.68 among portfolio directors. Among individual contributors alone, 61.25% (196 of 320) of those who rated it 1 to 3 had a score of 3 or below.

The playbook drew 200 written answers from 184 respondents on 110 engagements, and 186 were negative, 75 from canceled and 111 from active engagements. Of the 186, only 14 (7.53%) described one of the 26 forms of a control bypassed or not working, most often unclear liability or regulatory accountability (7). The answers described a framework that was absent rather than bypassed. Teams were placed on live work with a deck and two days of training, wrote runbooks at night while backlogs grew, and governed the work through habits learned by watching the agents.

> “There was no playbook. I got two days of training and a deck. And when the fails queue went wrong it came back on me, not on anyone who built it.” -Individual Contributor, BPO, Trade, settlement and reconciliation

> “What we have settled into, rather than waiting for the tooling to improve, is a set of habits. Long submissions, and a loss run of forty pages is not unusual, are split into sections before anything is read, because material facts sitting in the middle of a document were being passed over often enough that we stopped trusting a single pass. Each account is worked in its own workspace, opened fresh and closed at the end of the day, so that nothing from one carrier's file can find its way into another's. And where an agent has to touch a system that writes rather than reads, one of my seniors approves the action by hand. None of this is in the playbook. It is what four months of watching the thing has taught us, and hence my middling scores rather than poor ones.” -Engagement Manager, KPO, Credit, underwriting and suitability

Part 2 (Vendor Capability, section 6.1) reports the association of the playbook with the four study measures.

## 2 Accountability for Agent Error

Vendor accountability for agent error was associated with the wider score, but its conditional estimate depended on the adjustment measure. This control concerns responsibility for the consequences of agent action, beyond routine task completion. The evidence supports examining that responsibility while resisting a causal interpretation of the negative adjusted coefficient.

### 2.1 Readiness to Take Accountability for a Costly Agent Error

Indicator C130 · Rate your employer's readiness to take accountability when an agent it operates makes a costly error · Full statistical analysis ›

Limited vendor readiness to accept accountability for a costly agent error was associated with low scores. This concerns responsibility when the vendor-operated agent causes a material problem, beyond the existence of a delivery process. The observed positive association and the negative adjusted coefficient answer different questions. The latter largely disappeared when the other vendor capability items were removed from the comparison mean, limiting a separate-effect interpretation.

Figure 3.54. Readiness to Take Accountability for a Costly Agent Error: observed association and adjustment sensitivity.

Scroll horizontally to view the complete figure.

Limited vendor readiness to accept accountability for a costly agent error was associated with low scores.

Scroll horizontally to view the complete comparison.

| Comparison | Correlation r | Estimate and analysis base |
| --- | --- | --- |
| Unadjusted cross-category r | | r = 0.78 150 engagements |
| Adjusted, full comparison mean | | r = −0.46 150 engagements |
| Adjusted, excluding other vendor items | | r = −0.03 150 engagements |
| | −1 0 +1 | |

Exact comparison data

Unadjusted cross-category r

Exact chart data

| Series | Point index | X | Y |
| --- | --- | --- | --- |
| Reported association | 0 | 0.78082692313353674 | 1 |

Adjusted, full comparison mean

Exact chart data

| Series | Point index | X | Y |
| --- | --- | --- | --- |
| Reported association | 0 | -0.45982686437805298 | 1 |

Adjusted, excluding other vendor items

Exact chart data

| Series | Point index | X | Y |
| --- | --- | --- | --- |
| Reported association | 0 | -3.2293038957112427E-2 | 1 |

Note: r measures signed linear association from −1 to +1. Partial r describes the adjusted association under the model specified in the section. Adjusted p values are nominal. The comparisons do not establish causal effects.

Correlations use 150 engagements. The alternative mean removes the stated category items. Estimates are descriptive model comparisons, with nominal inference in the text.

Of the 373 respondents who rated the vendor's readiness 1 to 3, 240 had a score of 3 or below, a share within the co-management range. Among the 396 who rated it 4 to 7, 32 (8.08%) still did, and none of the 25 who rated it 8 to 10 did (chi-square = 283.42, p < 0.001, Cramér's V = 0.60).

The indicator was associated with the score computed without it at r = 0.79 (95% CI 0.72 to 0.84, n = 150 engagements) and with governance in the other four categories at r = 0.78. After adjustment for the mean of the other 53 indicators, the partial r was −0.46 (p < 0.001). It stayed inverse with the financial institution readiness block omitted from the comparison mean (−0.41), with the AI technology block left out (−0.38), and with the third-party block left out (−0.32), each at p < 0.001. With the vendor capability block left out it stood at −0.03 (p = 0.696). Respondents who rated it distressed recorded a score of 2.75 (n = 373), against 5.05 (n = 421) among the rest (cluster-robust p < 0.001).

The association was observed within the canceled engagements at r = 0.69 and across the active stages at 0.71 in Build-Out, 0.66 in Transition/Cut-Over, and 0.77 in Production. After Holm adjustment, it was significant in 86 of the 87 segment levels tested. Within role it stood at 0.79 among individual contributors, 0.75 among engagement managers, and 0.68 among portfolio directors. Among individual contributors alone, 65.06% (203 of 312) of those who rated it 1 to 3 had a score of 3 or below.

Thirty-eight respondents on 32 engagements wrote about the vendor's readiness to take accountability, one answer each, and 29 of the answers were negative, 9 from canceled and 20 from active engagements. Of the 29, 11 (37.93%) described one of the 26 forms of a control bypassed or not working, most often unclear liability or regulatory accountability (7) and problems raised with no response (6). The answers described leadership that fell silent after an error and responsibility that came to rest on the individual contributor on shift.

> “Playbook for taking a use case into production does exist, and it is better than what we had two years ago; training on payments and card scheme rules is genuinely useful. Where I could not defend them is accountability. When model produced a summary that missed an indemnity cap and client raised it, our leadership went quiet for the better part of two weeks. That silence did more damage than the error.” -Engagement Manager, KPO, Regulatory, audit, tax and legal

> “Two of us have covered nights since day one because tier 2 doesnt pick up. And when the agent closed out a real intrusion alert as noise, nobody upstairs put their name on it. That falls on the analyst who was on shift. Thats me some weeks. I dont sleep great” -Individual Contributor, ITO, Technology engineering and operations

Part 2 (Vendor Capability, section 7.1) reports the association of the vendor's readiness to take accountability with the four study measures.

← Financial Institution Readiness and the Governance of Agentic WorkTechnical Controls and Governance Across the Engagement →