I have altered AI outputs

Of the 828 respondents who answered, 278 (33.57%) answered True and 550 (66.43%) answered False. At engagement level, the answer was associated with all four study measures. Engagements whose respondents most often said they had altered AI outputs were canceled more often (76.19% of 42 engagements against 22.00% of 100), and recorded lower customer satisfaction among active engagements, a lower engagement indicator mean and higher intent to leave.

After Holm correction, the share answering True differed by engagement status and lifecycle stage (chi-square 37.84, Holm p < 0.001, Cramer's V 0.52): the share answering True was highest among respondents on canceled engagements (54.19%, n = 310 respondents) and lowest among respondents on active engagements in Transition/Cut-Over (18.52%, n = 243 respondents). It also differed by respondent role (chi-square 107.09, Holm p < 0.001, Cramer's V 0.36): the share answering True was highest among individual contributors (42.46%, n = 650 respondents) and lowest among portfolio directors (0.00%, n = 31 respondents). Outsourcing model, institution type, process family, class of work and vendor type showed no difference.

The 407 written responses came from 276 respondents on 116 engagements, and 399 were negative. The respondents who answered C214, on why they altered the outputs, described an agent that gave a different answer to the same input from one run to the next, so they chose the version that looked right and corrected it before it moved on. Those who commented under C216 described the alterations as routine, known to the team lead, unrecorded and largely out of the financial institution's sight.

Of the 828 respondents who answered, 278 (33.57%) answered True and 550 (66.43%) answered False. The non-answers were I do not know (2) and I prefer to not answer (16). Each engagement was assigned the answer given by the largest number of its respondents. True was the assigned answer on 42 engagements and False on 100, and 8 engagements were tied and carry no assigned answer.

The primary responses numbered 361, written in answer to C214, C216, C208, C210, C202 and C204, the written-answer questions of the indicator's survey section. The remaining 46 responses, 11.30% of the total, were secondary and came from 4 other questions. That secondary share ranked 1st from the highest among the 13 Part 5 indicators with 30 or more written responses, against a median of 2.00%. Respondents who wrote both a primary and a secondary response numbered 18. Negative responses made up 98.61% of primary responses (356 of 361) and 93.48% of secondary responses (43 of 46).

The 181 responses to C214 gave the reasons for altering the outputs. In 74 responses, respondents said the same input produced a different answer on another run, or that reruns did not match. In 57, they described choosing among the runs: the version that looked right, matched the prior run or matched what the team expected. Respondents said they could not sign off, or put their name to, an output that would not reproduce in 11 responses. Change approval slower than the work, a ticket that took longer than the fix, or a deadline that passed while the team waited appeared in 31 responses across all questions. Eleven responses said leadership had told the team to make the output work and not raise the problem, and 11 responses to C214 placed part of the problem with a vendor, feed, tool version or third-party platform outside the team's control.

The 161 responses to C216 described how the alterations were handled. Respondents used words of frequency, such as every day, most weeks or almost every ticket, in 95 responses, and 8 gave a share out of ten. The team lead or manager appeared in 41 responses, in 38 as someone who knew or was told. Across all questions, respondents said the financial institution was not told, or that they did not know whether it was, in 15 responses, and described alterations that were unlogged, unreported or made quietly in 41. In 9 responses to C216, respondents described the quality assurance score as low, and in 15 they said they caught the errors themselves before the output left the team. Tiredness appeared in 14 responses across all questions, and 5 noted that the engagement had been canceled regardless.

Responses to C177, C156 and C135 that also addressed the indicator described an agent that lost its instructions over a long session or failed on a long document, logs that did not show what the agent had done, and no one with authority to switch it off. Three responses described a side list, private list or own tracker of corrections. Four positive responses said the respondent altered outputs only rarely.

Of the 407 responses, 399 (98.03%, 95% CI 96.61% to 99.26%) were negative and 111 (27.27%) carried the strongest negative score of −10. Positive responses numbered 8 (1.97%), and no response was neutral. The mean written score was −6.2 (95% CI −6.4 to −5.9, n = 407 responses), and the median was −5.0.

Each share is computed on the respondents who answered in that row. Rows with fewer than 10 respondents report counts only.

Segment families recorded at engagement level were tested on engagements, and respondent role and vendor type on respondents. After Holm correction, the share answering True differed by engagement status and lifecycle stage (chi-square 37.84, Holm p < 0.001, Cramer's V 0.52): the share answering True was highest among respondents on canceled engagements (54.19%, n = 310 respondents) and lowest among respondents on active engagements in Transition/Cut-Over (18.52%, n = 243 respondents). It also differed by respondent role (chi-square 107.09, Holm p < 0.001, Cramer's V 0.36): the share answering True was highest among individual contributors (42.46%, n = 650 respondents) and lowest among portfolio directors (0.00%, n = 31 respondents). Outsourcing model, institution type, process family, class of work and vendor type showed no difference.

Written sentiment differed by engagement status and lifecycle stage across the 116 engagements with a written response (Kruskal-Wallis H 9.64, df 3, p = 0.022). Of the groups with 10 or more responses, responses from canceled engagements were the most negative (mean −6.5, 242 responses) and responses from active engagements in Build-Out the least negative (mean −5.2, 49 responses).

No test across respondent roles was run, because fewer than two roles had 30 or more written responses. Individual contributors wrote 394 responses, engagement managers 12 and portfolio directors 1.

Cancellation rate is the share of engagements canceled by the study date, measured at engagement level (150 engagements, 56 canceled). Canceled engagements were rated after cancellation. The association is therefore a descriptive contrast and not a forecast of cancellation. Each test compares the engagements by their assigned answer, within the study set and within each segment.

Of the engagements whose respondents most often answered True, 32 of 42 (76.19%) were canceled, against 22 of 100 (22.00%) of those whose respondents most often answered False (chi-square 36.86, p < 0.001, Cramer's V 0.51, n = 142 engagements). The difference reached significance and held after Holm correction in 5 of the 5 segment levels tested.

Customer satisfaction is the percentage recorded against each engagement. The primary estimate uses the 94 active engagements.

Among active engagements, customer satisfaction had a mean of 54.7% (n = 10 engagements) where respondents most often answered True and 65.3% (n = 78 engagements) where they most often answered False (Welch F 11.44, p = 0.005, omega squared 0.072, n = 88 engagements). The difference reached significance and held after Holm correction in 0 of the 0 segment levels tested.

The engagement indicator mean is the mean of an engagement's ratings across the rated indicators, measured at engagement level. The indicator is not a rated indicator and does not enter the mean.

The engagement indicator mean was 2.7 (n = 42 engagements) on engagements whose respondents most often answered True and 4.5 (n = 100 engagements) on those whose respondents most often answered False (Welch F 87.54, p < 0.001, omega squared 0.297, n = 142 engagements). The difference reached significance and held after Holm correction in 6 of the 6 segment levels tested.

Intent to leave is the answer to C019, “How likely are you to leave your current job in the next 12 months?”, rated 1 to 10 by up to 798 respondents. For the indicator, the comparison uses the mean intent to leave of each engagement.

The engagement mean of intent to leave was 8.7 (n = 42 engagements) on engagements whose respondents most often answered True and 6.0 (n = 100 engagements) on those whose respondents most often answered False (Welch F 79.99, p < 0.001, omega squared 0.259, n = 142 engagements). The difference reached significance and held after Holm correction in 5 of the 6 segment levels tested.

Two comparisons set the engagements with at least one negative response against all other engagements. Of the 35 other engagements, 34 had no written response addressing the indicator. These two comparisons therefore do not isolate written sentiment. Of the 115 engagements with a negative response, 54 (46.96%) were canceled, against 2 of the 35 other engagements (5.71%). The difference reached significance (Fisher exact p < 0.001). Customer satisfaction among active engagements differed between the two groups (mean 60.84% on 61 engagements against 70.12% on 33, Welch t −3.95, p < 0.001).

Two correlations test written sentiment directly. Across the 116 engagements with a written response, a more negative engagement mean written score came with cancellation (point-biserial r −0.26, p = 0.005). Across 254 respondents, a more negative respondent mean written score came with higher intent to leave (Spearman rho −0.27, p < 0.001). Of the four comparisons, all reached significance at 0.05.

Steiger's z compares two correlations. A categorical indicator yields an effect size across its levels and not a correlation, so the pairs of measures were not compared by test. On omega squared, the association was strongest with the engagement indicator mean (0.297) and weakest with customer satisfaction (0.072).

Answers clustered within engagements (ICC 0.23). Clustering reduced the 846 respondent answers to the equivalent of about 412 independent answers. Respondent-level intervals therefore used the cluster bootstrap. Respondent-level tests used standard errors that allow for clustering by engagement.

No association was equivalent to zero. Each of the four differed from zero. The equivalence bound is an omega squared of 0.090, the study's threshold for a negligible association expressed as variance explained.

All nine checks held. A check holds when its test falls on the same side of 0.05 as the primary test. The checks replace Welch's F with a Kruskal-Wallis test, repeat the tests on active engagements alone, and repeat them at respondent level with a Wald F clustered by engagement.

Each of the four measures was computed from data the indicator does not contain. Each result is therefore a finding and not an arithmetic consequence.

A categorical indicator has no signed correlation with the study measures and therefore no profile against the study norm. The comparison by answer appears in the four association sections.

Among the 13 Part 5 indicators with 30 or more written responses, the indicator's share of negative responses, 98.03%, ranked 7th from the highest, against a median of 98.03%. Its mean written score, −6.2, ranked 10th from the most negative, against a median of −6.3.

The indicator most often present in the same responses was C157, “Rate the factual reliability of the AI tools' outputs on this engagement before any human review.”, in 313 of the 407 responses (76.90%). C159, “Rate the consistency of the tools' outputs when given the same input and instructions more than once.” (175 responses), C212, “I discussed the AI output alteration with internal management” (54 responses), C213, “I or my management discussed AI output alteration with the client” (28 responses) and C175, “Rate the stability of the tools' behavior over the engagement when nothing changed on the engagement side.” (26 responses) followed.

The most frequent theme was “Human rework absorbs the failure”, in 323 responses (79.36%). The second was “Occurs every cycle or most days”, in 93 responses.

Of the respondents who answered, 33.57% said they had altered AI outputs. Engagements whose respondents most often said so were canceled more often, recorded lower customer satisfaction and a lower engagement indicator mean, and their respondents reported higher intent to leave. Every association held under each of the robustness checks.

The written evidence describes the alteration of outputs as the delivery team's answer to an agent that would not give the same answer twice, approval cycles slower than the work, and an instruction to keep the output moving. The alterations were made routinely, known within the team and rarely reported to the financial institution. The associations do not establish that the alterations produced the outcomes. The written responses describe them as one response to the same instability and the same gaps in change control that the Part 5 ratings record.

The evidence is the reported experience of the vendor's delivery teams, collected in a single cross-section. The associations reported here are not causal. The answer is the respondent's own report of an alteration and not a record of it.

Each engagement carries the answer given by the largest number of its respondents, and tied engagements are excluded from the engagement-level tests.

Shares of written sentiment are shares of responses and not of the study set of respondents. The written score takes five fixed values, so its mean and median are coarse. Responses clustered within engagements (407 responses on 116 engagements). Intervals for written sentiment therefore used a bootstrap by engagement.

The written responses are counted by sets of words stated in each rule. A response that described a reason in other words is not counted, so each count is a floor.