Technical safeguards were associated with the broader control environment, even though their low-rating share was smaller than in the management categories. The eight technology indicators had 2,052 low ratings among 6,362 ratings. The governance, oversight, and control score relates those assessments to controls elsewhere in the engagement. The main adjusted model did not detect a separate positive cross-category relationship, with sensitivity to the composition of the comparison measure.
Each technology indicator correlated with governance in the other four categories at r = 0.77–0.81 (n = 150). Under the main adjustment, partial r ranged from −0.09 to 0.08 and none met nominal p < 0.05. Item-to-score correlations were 0.65–0.70 in Build-Out and 0.78–0.82 in Production. This cross-sectional ordering is descriptive and does not establish a stronger relationship once the same agents enter live operation.
A distressed technology rating was rarer than a distressed management rating and associated with a larger conditional low-score share. Between 251 and 262 respondents rated a technology control 1 to 3, against 377 to 407 for each co-management indicator, and 74.90% to 78.54% of them had a score of 3 or below. Ratings of 4 to 7 still carried a score of 3 or below for 13.21% to 16.42% of respondents, against 2.65% to 5.30% for the co-management indicators. Table 3.11 sets out the eight indicators against these measures.
Indicator | A score of 3 or below among respondents rating it 1 to 3 | r with score excluding item (95% CI) | r with the other four categories | Adjusted partial r (p) | Negative written answers |
|---|---|---|---|---|---|
Resistance to Hostile Embedded Instructions | 205 of 261 (78.54%) | 0.86 (0.81 to 0.90) | 0.81 | 0.08 (p = 0.310) | 9 of 9 |
Safe Handling of Sensitive Data | 194 of 259 (74.90%) | 0.84 (0.79 to 0.88) | 0.79 | −0.09 (p = 0.295) | 22 of 28 |
Consistency of Intended and Authorized Agent Action | 192 of 253 (75.89%) | 0.83 (0.77 to 0.87) | 0.78 | −0.03 (p = 0.745) | 48 of 64 |
Reliability of Built-In Guardrails in Production | 193 of 252 (76.59%) | 0.85 (0.79 to 0.89) | 0.80 | 0.06 (p = 0.488) | 7 of 17 |
Proof of Correct Behavior Before Work Goes Live | 195 of 259 (75.29%) | 0.85 (0.80 to 0.89) | 0.80 | 0.05 (p = 0.527) | 26 of 38 |
Accuracy of the Tools' Own Success and Failure Reports | 189 of 251 (75.30%) | 0.84 (0.79 to 0.88) | 0.79 | −0.04 (p = 0.623) | 63 of 73 |
Explanation and Evidence of Outputs at Audit Depth | 197 of 262 (75.19%) | 0.86 (0.81 to 0.89) | 0.81 | 0.03 (p = 0.739) | 145 of 163 |
Quality of Logs, Traces, and Usage Telemetry | 191 of 255 (74.90%) | 0.82 (0.76 to 0.87) | 0.77 | −0.03 (p = 0.702) | 82 of 101 |
Note: Rating-band and item-to-score results use the score excluding the tested indicator. Cross-category results exclude its entire category and use 150 engagements. Partial r accounts for the mean of the 53 indicators outside the score. Adjusted-correlation p values are nominal. The ordinary respondent-level band tests do not account for clustering within engagements, including where Monte Carlo calculation handles sparse cells. Their fractions remain descriptive. Zero observations are not proof of no risk.
In April we took a batch of eighty investigations to the bank's financial crime committee and the tool's own run summary said all eighty had completed cleanly. Six had not, they had stopped part way and reported success. Nothing came of it because our own reviewers caught it first, but i had to explain to their head of compliance why a machine's account of its own work is not evidence it did the work. That conversation set the programme back a month.”
PORTFOLIO DIRECTOR · BPO, Financial crime and client onboarding
1 Data Isolation, Sensitive Data, and Adversarial Input
Resistance to hostile input and safe handling of sensitive data connect technical boundaries with governance of agent action. Both were associated with the wider score, while the main adjusted estimates were not clearly different from zero. These results concern reported tool properties. They do not independently establish security performance or remove the need to explain how the controls operated in the engagement.
1.1 Resistance to Hostile Embedded Instructions
Weak resistance to hostile embedded instructions was associated with low scores. The control concerns instructions placed inside content an agent reads, which may divert behavior from the authorized task. That mechanism connects a technical property with the governance boundary over agent action. The study records respondents' assessments of resistance, rather than results from an independent security test, and the adjusted comparison did not detect a clear separate association.
Scroll horizontally to view the complete figure.
Weak resistance to hostile embedded instructions was associated with low scores. | ||||||||||||||||
Exact chart data
|
Note: The score excludes this indicator. Labels give the low-score numerator and the answering base. Zero observations in a band do not establish absence of risk.
Of the 261 respondents who rated resistance 1 to 3, 205 had a score of 3 or below. Among the 477 who rated it 4 to 7, 63 (13.21%) did, the lowest such share of the eight, and none of the 74 who rated it 8 to 10 did (chi-square = 365.79, p < 0.001, Cramér's V = 0.67, the largest conditional share of the 47 governance indicators).
Resistance was associated with the score computed without it at r = 0.86 (95% CI 0.81 to 0.90, n = 150 engagements) and with governance in the other four categories at r = 0.81, the joint highest of the eight. After adjustment for the mean of the other 53 indicators, the partial r was 0.08 and was not significant (p = 0.310). Respondents who rated it distressed recorded a score of 2.29 (n = 261), against 4.79 (n = 551) among the rest (cluster-robust p < 0.001).
The association ranged across the active-stage groups from 0.69 in Build-Out to 0.74 in Transition/Cut-Over and 0.81 in Production, and held within the canceled engagements at r = 0.72, the strongest of the eight. After Holm adjustment, it was significant in 84 of the 87 segment levels tested. Within role it stood at 0.81 among individual contributors, 0.83 among engagement managers, and 0.77 among portfolio directors. Among individual contributors alone, 80.18% (174 of 217) of those who rated it 1 to 3 had a score of 3 or below.
Nine respondents on 9 engagements wrote about hostile instructions, 8 of them on active engagements, and all nine answers were negative. None described one of the 26 forms of a control bypassed or not working, and each described the technical control itself failing. Respondents reported instructions embedded in a remittance narrative, a scanned letter, the footnotes of a circular, and the text of a ticket that the tools obeyed, in one case recording the case as closed with high confidence.
“Last month a circular we fed in had instruction lines in the footnotes, tool followed the same. Logs are good otherwise, I could see exactly what it ran” -Individual Contributor, KPO, Regulatory, audit, tax and legal
“need the Tool to read runbook from one locked source… it pick up any text in ticket and follow it.” -Individual Contributor, ITO, Technology engineering and operations
Part 2 (AI Technology, section 5.3) reports the association of resistance with the four study measures.
1.2 Safe Handling of Sensitive Data
Low ratings for safe handling of sensitive data were associated with low scores. The indicator concerns exposure through outputs, logs, or connected systems, bringing the evidence generated by agent work within the control boundary. Its associations vary with the score and adjustment used. The nonsignificant estimates do not establish either a distinct effect or the absence of an operational data-protection problem.
Scroll horizontally to view the complete figure.
Low ratings for safe handling of sensitive data were associated with low scores. | ||||||||||||||||
Exact chart data
|
Note: The score excludes this indicator. Labels give the low-score numerator and the answering base. Zero observations in a band do not establish absence of risk.
For the score excluding the sensitive-data item, the adjusted estimate was partial r = 0.11 (p = 0.179). Against governance in the other four categories, its partial r after that adjustment was −0.09 (p = 0.295), the lowest of the eight. Scores were 3 or below for 194 of 259 (74.90%) at indicator ratings 1–3. The corresponding counts were 71 of 471 (15.07%) at indicator ratings 4–7 and 0 of 70 (0.00%) at indicator ratings 8–10. The reported band results were chi-square = 308.02, p < 0.001, Cramér's V = 0.62.
The indicator was associated with the score computed without it at r = 0.84 (95% CI 0.79 to 0.88, n = 150 engagements) and with governance in the other four categories at r = 0.79. Respondents who rated it distressed recorded a score of 2.35 (n = 259), against 4.76 (n = 541) among the rest (cluster-robust p < 0.001).
The association ranged across the active-stage groups from 0.68 in Build-Out to 0.82 in Production and held within the canceled engagements at r = 0.70. After Holm adjustment, it was significant in 85 of the 87 segment levels tested. Within role it stood at 0.81 among individual contributors, 0.84 among engagement managers, and 0.73 among portfolio directors. Among individual contributors alone, 75.69% (165 of 218) of those who rated it 1 to 3 had a score of 3 or below.
Twenty-eight respondents on 27 engagements wrote about sensitive data, and 22 of the answers were negative, 13 from canceled and 9 from active engagements. Three of the 22 described one of the 26 forms of a control bypassed or not working, two of them problems raised with no response. Respondents described account numbers and salary lines written in full into logs that much of the delivery team could open, and records from one account file that appeared in a summary of another.
“the tool put client salary lines into a log i could read… that should not be possible.” -Individual Contributor, KPO, Corporate and back-office functions
“it needs to stop pulling account numbers into the chat log for a start. and nobody should be letting it touch a live settlement without me reading it first” -Individual Contributor, BPO, Trade, settlement and reconciliation
Part 2 (AI Technology, section 5.2) reports the association of sensitive data handling with the four study measures.
2 Authorized Action and Guardrails
Authorized action and production guardrails were associated with the wider score. The first concerns the scope of delegated action, while the second concerns whether restrictions held during use. Their adjusted estimates did not establish distinct relationships under the main comparison mean. Sensitivity of the authorized-action result illustrates why a conditional null should not become a conclusion that technical controls have no role of their own.
2.1 Consistency of Intended and Authorized Agent Action
Inconsistent restriction of agent activity to intended and authorized actions was associated with low scores. The control function is the boundary between delegated work and action outside the delegation. The adjusted cross-category estimate was near zero under the main comparison mean, but positive in the category-removal sensitivity check. This model dependence limits a claim that authorization has no relationship of its own, without establishing a causal effect.
Scroll horizontally to view the complete figure.
Scroll horizontally to view the complete comparison.
| Comparison | Correlation r | Estimate and analysis base |
|---|---|---|
Unadjusted cross-category r | r = 0.78 | |
Adjusted, full comparison mean | r = −0.03 | |
Adjusted, excluding other AI technology items | r = 0.29 | |
−10+1 |
Exact comparison data
Exact chart data
| Series | Point index | X | Y |
|---|---|---|---|
| Reported association | 0 | 0.77696885546463523 | 1 |
Exact chart data
| Series | Point index | X | Y |
|---|---|---|---|
| Reported association | 0 | -2.6867112977836706E-2 | 1 |
Exact chart data
| Series | Point index | X | Y |
|---|---|---|---|
| Reported association | 0 | 0.2900666646748038 | 1 |
Note: r measures signed linear association from −1 to +1. Partial r describes the adjusted association under the model specified in the section. Adjusted p values are nominal. The comparisons do not establish causal effects.
Correlations use 150 engagements. The alternative mean removes the stated category items. Estimates are descriptive model comparisons, with nominal inference in the text.
Of the 253 respondents who rated authorized action 1 to 3, 192 had a score of 3 or below. Among the 474 who rated it 4 to 7, 69 (14.56%) did, and none of the 62 who rated it 8 to 10 did (chi-square = 313.56, p < 0.001, Cramér's V = 0.63).
Authorized action was associated with the score computed without it at r = 0.83 (95% CI 0.77 to 0.87, n = 150 engagements) and with governance in the other four categories at r = 0.78. After adjustment for the mean of the other 53 indicators, the partial r was −0.03 (p = 0.745). Respondents who rated it distressed recorded a score of 2.31 (n = 253), against 4.78 (n = 536) among the rest (cluster-robust p < 0.001). Removing the other 12 AI technology items from the comparison mean changed partial r to 0.29 (nominal p < 0.001), illustrating model sensitivity.
The descriptive stage correlations ranged from 0.65 in Build-Out to 0.71 in Transition/Cut-Over and 0.78 in Production and held within the canceled engagements at r = 0.71. After Holm adjustment, it was significant in 86 of the 87 segment levels tested. Within role it stood at 0.81 among individual contributors, 0.75 among engagement managers, and 0.70 among portfolio directors. Among individual contributors alone, 78.20% (165 of 211) of those who rated it 1 to 3 had a score of 3 or below.
Authorized action tracked the manual alteration of AI outputs. Of the respondents who rated it 1 to 3, 66.39% (158 of 238) altered AI outputs, against 18.76% (88 of 469) at ratings of 4 to 7 and 8.06% (5 of 62) at ratings of 8 to 10 (chi-square p < 0.001).
Authorized action drew 64 written answers from 62 respondents on 50 engagements, and 48 were negative, 19 from canceled and 29 from active engagements. Of the 48, 7 (14.58%) described one of the 26 forms of a control bypassed or not working, most often problems raised with no response (3). Respondents described agents that pulled records outside the batch they were given, touched queues and accounts beyond their scope, and in one account did so through a loose instruction rather than a failed control.
“it hit the corporate card feed and pulled transactions for a cardholder that was not in the batch. nobody told it to. that happened twice in january. on cost, it will loop on the same receipt eight or nine times before it gives up, and there is no counter anywhere that shows that. i found it by watching the runtime go up. logs show it finished. thats all they show” -Individual Contributor, BPO, Corporate and back-office functions
“it gives same answer more often than before, that part improved. but it still touches queues it has no business touching, and nothing is logged.” -Individual Contributor, CXM, Technology engineering and operations
Part 2 (AI Technology, section 6.1) reports the association of authorized action with the four study measures.
2.2 Reliability of Built-In Guardrails in Production
Less reliable built-in guardrails were associated with low scores. The indicator concerns whether safeguards held during production use, linking designed restrictions with reported operation. Its findings should be distinguished from the presence of a documented safety policy and from authorized-action ratings. The main adjusted comparison did not detect a clear separate association, and the relatively less negative written answers do not independently verify guardrail effectiveness.
Scroll horizontally to view the complete figure.
Less reliable built-in guardrails were associated with low scores. | ||||||||||||||||
Exact chart data
|
Note: The score excludes this indicator. Labels give the low-score numerator and the answering base. Zero observations in a band do not establish absence of risk.
Of the 252 respondents who rated guardrails 1 to 3, 193 had a score of 3 or below, the second-highest share of the eight. Among the 454 who rated them 4 to 7, 69 (15.20%) did, and none of the 81 who rated them 8 to 10 did (chi-square = 320.05, p < 0.001, Cramér's V = 0.64).
Guardrails were associated with the score computed without them at r = 0.85 (95% CI 0.79 to 0.89, n = 150 engagements) and with governance in the other four categories at r = 0.80. After adjustment for the mean of the other 53 indicators, the partial r was 0.06 (p = 0.488). Respondents who rated them distressed recorded a score of 2.35 (n = 252), against 4.78 (n = 535) among the rest (cluster-robust p < 0.001).
The association ranged across the active-stage groups from 0.66 in Build-Out to 0.80 in Production and held within the canceled engagements at r = 0.71. After Holm adjustment, it was significant in 85 of the 87 segment levels tested. Within role it stood at 0.82 among individual contributors and 0.79 among engagement managers and portfolio directors. Among individual contributors alone, 78.74% (163 of 207) of those who rated them 1 to 3 had a score of 3 or below.
Seventeen respondents on 14 engagements wrote about guardrails, and 7 of the answers were negative, all from active engagements. One of the 7 linked the failure to regulatory or compliance exposure. Respondents reported guardrails that now held while the tool lost instructions within a flow, and guardrails that stayed the weak point of a tool whose matching had improved.
“Guardrails hold, thats better than it was. The fnol flow still loses what I told it two steps back.” -Individual Contributor, CXM, Client servicing and communications
“the matching itself got much better over the six months, it runs a full statement end to end now and when it says it cleared the break it really did clear it. the guardrails are the weak part still, it will happily action something on a client cash account that i would want a second pair of eyes on” -Individual Contributor, BPO, Trade, settlement and reconciliation
Part 2 (AI Technology, section 6.2) reports the association of guardrails with the four study measures.
3 Testability and Behavioral Stability
Testability connects assurance before live work with the broader score. The indicator asks whether correct behavior can be established through testing, while the later output-consistency analysis concerns variation during repeated work. Together they distinguish prospective assurance from observed stability. A favorable rating remains a report about capability, not evidence that all relevant behavior was tested successfully.
3.1 Proof of Correct Behavior Before Work Goes Live
Limited support for testing correct behavior before live operation was associated with low scores. This connects the ability to examine agent work before deployment with the wider reported control environment. A favorable rating describes perceived testability, not proof that all relevant behavior was tested or that correctness was established. The adjusted result qualifies the association without resolving that operational assurance question.
Scroll horizontally to view the complete figure.
Limited support for testing correct behavior before live operation was associated with low scores. | ||||||||||||||||
Exact chart data
|
Note: The score excludes this indicator. Labels give the low-score numerator and the answering base. Zero observations in a band do not establish absence of risk.
Of the 259 respondents who rated testability 1 to 3, 195 had a score of 3 or below. Among the 466 who rated it 4 to 7, 74 (15.88%) did, and none of the 67 who rated it 8 to 10 did (chi-square = 299.62, p < 0.001, Cramér's V = 0.62).
Testability was associated with the score computed without it at r = 0.85 (95% CI 0.80 to 0.89, n = 150 engagements) and with governance in the other four categories at r = 0.80. After adjustment for the mean of the other 53 indicators, the partial r was 0.05 (p = 0.527). Respondents who rated it distressed recorded a score of 2.34 (n = 259), against 4.75 (n = 533) among the rest (cluster-robust p < 0.001).
The association ranged across the active-stage groups from 0.67 in Build-Out to 0.73 in Transition/Cut-Over and 0.80 in Production and held within the canceled engagements at r = 0.71. After Holm adjustment, it was significant in 86 of the 87 segment levels tested. Within role it stood at 0.81 among individual contributors and engagement managers and 0.74 among portfolio directors. Among individual contributors alone, 76.82% (169 of 220) of those who rated it 1 to 3 had a score of 3 or below.
Thirty-eight respondents on 33 engagements wrote about testability, and 26 of the answers were negative, 12 from canceled and 14 from active engagements. Three of the 26 described one of the 26 forms of a control bypassed or not working. Respondents described the absence of any test environment, validation performed on live cases, and a reviewer's own reading as the only proof that an agent's work was correct.
“Whoever reads this, the main thing is we cannot prove anything works before it goes live, there is no real test here, only my eyes” -Individual Contributor, ITO, Regulatory, audit, tax and legal
“there was no way to test properly before going live. we found the misses later from the false positive report. i would ask you to note that is the main thing here” -Individual Contributor, ITO, Financial crime and client onboarding
Part 2 (AI Technology, section 7.1) reports the association of testability with the four study measures.
4 Observability of Agent Work
Observability connects agent work with the evidence available to oversee it. Tool reports, output explanations, and logs were associated with the wider score, while tool-reported success also correlated with the accuracy of reporting supplied to the financial institution. These measures concern different parts of the evidence chain. Their alignment does not independently verify the completeness or truth of the underlying record.
“Main thing is it report success when it has not done the thing. Pipeline log say green, the artefact is not in the repo… that is the one that cost us most” -Individual Contributor, ITO, Technology engineering and operations
4.1 Accuracy of the Tools' Own Success and Failure Reports
Inaccurate tool reports of success and failure were associated with low scores. This creates a governance concern because oversight depends on evidence that corresponds to the work performed. A system can report successful completion while producing an incorrect or absent result. The relationship with reporting provided to the financial institution links this technical evidence problem with the information used to oversee delivery.
Scroll horizontally to view the complete figure.
Inaccurate tool reports of success and failure were associated with low scores. | ||||||||||||||||
Exact chart data
|
Note: The score excludes this indicator. Labels give the low-score numerator and the answering base. Zero observations in a band do not establish absence of risk.
Among the 251 respondents rating tool-report accuracy at 1–3, 189 had a score of 3 or below (75.30%). The corresponding counts were 78 of 475 at ratings 4–7 (16.42%) and one of 60 at ratings 8–10. The ordinary band test was chi-square = 283.75 (p < 0.001, Cramér's V = 0.60), subject to the stated clustering limitation. The reporting-accuracy correlation of 0.68 addresses a separate relationship.
The tools' reports were associated with the score computed without them at r = 0.84 (95% CI 0.79 to 0.88, n = 150 engagements) and with governance in the other four categories at r = 0.79. After adjustment for the mean of the other 53 indicators, the partial r was −0.04 (p = 0.623). Respondents who rated them distressed recorded a score of 2.30 (n = 251), against 4.74 (n = 535) among the rest (cluster-robust p < 0.001).
The association ranged across the active-stage groups from 0.67 in Build-Out to 0.81 in Production and held within the canceled engagements at r = 0.69. After Holm adjustment, it was significant in 86 of the 87 segment levels tested. Within role it stood at 0.79 among individual contributors, 0.81 among engagement managers, and 0.78 among portfolio directors. Among individual contributors alone, 77.25% (163 of 211) of those who rated them 1 to 3 had a score of 3 or below.
The tools' own reports tracked reporting alteration. Of the respondents who rated them 1 to 3, 82.87% (208 of 251) described reporting alteration, against 46.32% (220 of 475) at ratings of 4 to 7 and 15.00% (9 of 60) at ratings of 8 to 10 (chi-square p < 0.001).
The indicator drew 73 written answers from 73 respondents on 58 engagements, and 63 were negative, 27 from canceled and 36 from active engagements. Of the 63, 10 (15.87%) described one of the 26 forms of a control bypassed or not working, most often change discovered only through failure (5) and problems raised with no response (4). Respondents described runs reported as complete in which items had never been processed, and teams that stopped trusting the status and counted the output themselves.
“The job status is the main issue. Tool reports the load as completed, then we check the target table and half the rows are not there. This happened on four occasions in my time here. Logs do not show anything useful, only that the step finished. So we stopped trusting the status and started counting rows ourselves each morning” -Individual Contributor, ITO, Technology engineering and operations
“To be fair, by month six the behaviour had settled — it stopped drifting week to week, and a composite run today looks much like the same run last month. What never improved is the reporting it does on itself. It would finish a batch of disclosures and mark it complete when three of the composites had not been built at all, and the logs give you nothing to work back from. We only found those by hand, on the day the pack was due.” -Engagement Manager, KPO, Trade, settlement and reconciliation
Part 2 (AI Technology, section 8.1) reports the association of the tools' reports with the four study measures, and Part 5 (Leading Dynamics of Manual Alteration, section 6.8) their association with manual alteration.
4.2 Explanation and Evidence of Outputs at Audit Depth
Weak explanation and evidence of how outputs were produced were associated with low scores. The indicator concerns whether agent work can be reconstructed at the depth an audit requires. This is distinct from whether an output appears correct or a dashboard reports success. The ratings describe perceived auditability, with no independent finding that an audit standard or legal obligation was met or breached.
Scroll horizontally to view the complete figure.
Weak explanation and evidence of how outputs were produced were associated with low scores. | ||||||||||||||||
Exact chart data
|
Note: The score excludes this indicator. Labels give the low-score numerator and the answering base. Zero observations in a band do not establish absence of risk.
Of the 262 respondents who rated explanation 1 to 3, 197 had a score of 3 or below. Among the 476 who rated it 4 to 7, 66 (13.87%) did, and none of the 62 who rated it 8 to 10 did (chi-square = 320.91, p < 0.001, Cramér's V = 0.63).
Explanation was associated with the score computed without it at r = 0.86 (95% CI 0.81 to 0.89, n = 150 engagements) and with governance in the other four categories at r = 0.81. After adjustment for the mean of the other 53 indicators, the partial r was 0.03 (p = 0.739). Respondents who rated it distressed recorded a score of 2.34 (n = 262), against 4.79 (n = 538) among the rest (cluster-robust p < 0.001).
The association ranged across the active-stage groups from 0.70 in Build-Out to 0.77 in Transition/Cut-Over and 0.81 in Production and held within the canceled engagements at r = 0.71. After Holm adjustment, it was significant in 85 of the 87 segment levels tested. Within role it stood at 0.82 among individual contributors, 0.80 among engagement managers, and 0.87 among portfolio directors. Among individual contributors alone, 76.82% (169 of 220) of those who rated it 1 to 3 had a score of 3 or below.
Explanation drew 163 written answers from 152 respondents on 90 engagements, and 145 were negative, 65 from canceled and 80 from active engagements. Only 9 of the 145 (6.21%) described one of the 26 forms of a control bypassed or not working, so the answers recorded a control that the tools could not supply rather than one the delivery team set aside. Respondents described compliance and audit questions answered by rebuilding the trail manually, outputs that could be corrected and not explained, and citations to clauses that did not exist.
“It cannot show where a number came from… i asked for the working and got a summary only. On a long custodian statement it skips whatever sits in the middle pages” -Individual Contributor, BPO, Trade, settlement and reconciliation
“Checked our exception log before answering. There is no rating for this, when it files document to wrong class we cannot show why afterwards. We can correct it, we cannot explain it, and for records work explaining is half the job.” -Engagement Manager, BPO, Corporate and back-office functions
Part 2 (AI Technology, section 8.2) reports the association of explanation with the four study measures, and Part 5 (Leading Dynamics of Manual Alteration, section 6.5) its association with manual alteration.
4.3 Quality of Logs, Traces, and Usage Telemetry
Weak logs, traces, and usage telemetry were associated with low scores. These records provide the evidence needed to detect a problem and examine what an agent did. Their governance significance lies in observability and diagnosis, rather than the volume of records generated. The separate joint comparison with output consistency examines how two reported limitations occur together and does not establish that one causes or amplifies the other.
Scroll horizontally to view the complete figure.
Weak logs, traces, and usage telemetry were associated with low scores. | ||||||||||||||||
Exact chart data
|
Note: The score excludes this indicator. Labels give the low-score numerator and the answering base. Zero observations in a band do not establish absence of risk.
Of the 255 respondents who rated logs and telemetry 1 to 3, 191 had a score of 3 or below, the same share as for sensitive data handling. Among the 466 who rated them 4 to 7, 75 (16.09%) did, and none of the 75 who rated them 8 to 10 did (chi-square = 297.73, p < 0.001, Cramér's V = 0.61).
Logs were associated with the score computed without them at r = 0.82 (95% CI 0.76 to 0.87, n = 150 engagements), the lowest of the eight, and with governance in the other four categories at r = 0.77. After adjustment for the mean of the other 53 indicators, the partial r was −0.03 (p = 0.702). Respondents who rated them distressed recorded a score of 2.36 (n = 255), against 4.76 (n = 541) among the rest (cluster-robust p < 0.001).
The association ranged across the active-stage groups from 0.66 in Build-Out to 0.72 in Transition/Cut-Over and 0.79 in Production. After Holm adjustment, it was significant in 85 of the 87 segment levels tested. Within role it stood at 0.80 among individual contributors and 0.76 among engagement managers and fell to 0.61 among portfolio directors. Among individual contributors alone, 77.93% (166 of 213) of those who rated them 1 to 3 had a score of 3 or below.
Logs and output-consistency ratings were strongly correlated across engagements (r = 0.94, n = 150). Their joint comparison, reported under The Fit Between Governance Processes and the Pace of Agent Work, uses only respondents with both numeric answers. It describes the coincidence of reported limitations and does not establish that either caused the other.
Logs drew 101 written answers from 97 respondents on 73 engagements, and 82 were negative, 47 from canceled and 35 from active engagements. Of the 82, 9 (10.98%) described one of the 26 forms of a control bypassed or not working. Respondents described logs that recorded a start and a finish and nothing between, and failures whose point of origin could not be found.
“report job runs, fails halfway, runs again from start. Same job three times in one morning is normal here, furthermore nobody can tell me why because there is no trace, only a status of failed. Logs from Platform show start time and end time and nothing between. When it does finish, numbers are mostly right, ill give it that. But i cant prove anything to my lead, i can only re-run it and show him the third one.” -Individual Contributor, KPO, Corporate and back-office functions
“Last month I gave it the recertification batch, two hundred accounts. Around ninety it forgot the rule on contractors and started passing them through… and nothing in the logs tells me where it turned” -Individual Contributor, ITO, Technology engineering and operations
Part 2 (AI Technology, section 8.3) reports the association of logs with the four study measures, and Part 5 (Leading Dynamics of Manual Alteration, section 6.6) their association with manual alteration.
In fairness to the providers, these are proper machine interfaces, not screens we scrape, and the claims feeds arrive clean enough that the agents can work them without us in the middle. Where we are genuinely thin is our own fallback. If a model provider went dark on a Tuesday morning we would be running claim status by hand, and half the desk has never worked that way.”
ENGAGEMENT MANAGER · CXM, Client servicing and communications