Appendix A defines the study’s principal terms using the Study Taxonomy. Survey questions and respondent quotations retain their original wording, including terminology that differs from the report’s usage.
Term | Definition |
|---|---|
Financial institution | Tier 1 firm buying the outsourced service, of 10 types that include insurers and private equity and private capital firms. |
Vendor | Outsourcing firm delivering the service and employing every respondent. |
The two firms | Financial institution and the vendor considered together, where a weakness belongs to their relationship rather than to either firm alone. |
Model provider | Supplier of the underlying AI models and the platforms they run on. |
Third-party provider | Any other supplier to the engagement, such as a supplier of tools, data feeds, or platforms. |
Engagement | One contracted piece of outsourced work performed by one vendor for one financial institution. The engagement is the primary unit of the study. |
Agent | One autonomous AI system operating within an engagement. |
Agentic AI outsourcing | Outsourcing in which agents perform part of the contracted work alongside the vendor's delivery team. |
Study set | 150 engagements and 850 respondents the study covers. |
Respondent | A member of the vendor's delivery team who answered the survey about one engagement. |
Individual contributor | Vendor role that performs the engagement's tasks. The study set holds 667. |
Engagement manager | Vendor role that runs a single engagement. The study set holds 150, one on each engagement. |
Portfolio director | Vendor role that oversees several engagements and their engagement managers. The study set holds 31. |
Customer management executive | Vendor role that owns the relationship with the financial institution. The study set holds two, reported as counts only. |
Delivery team | All vendor staff on one engagement, across roles. |
Delivery workforce | All 850 respondents across the study set. |
Financial institution staff | People employed by the financial institution who work with the engagement, including named subject-matter experts. |
Outsourcing model | Form of outsourcing under which an engagement was contracted, customer experience management (CXM), information technology outsourcing (ITO), business process outsourcing (BPO) or knowledge process outsourcing (KPO). |
Class of work | Whether an engagement served a line of business the financial institution runs or a corporate or back-office function. |
Process family | One of eight groups of use cases. Each engagement was assigned to one process family from the use case it recorded. |
Type of financial institution | One of 10 types recorded for each engagement, from asset management to private equity and private capital. |
Vendor type | Respondent's employer type. The five types are the IT services major, the BPO specialist, the consulting firm, the pure-play agentic AI firm, and the boutique or specialist provider. |
Build-Out | First stage measured, covering planning, team build-out, and knowledge transfer. |
Transition/Cut-Over | Second stage measured, covering development, proof of concept, and engagements on hold. |
Production | Third stage measured, covering engagements in production and delivery. |
Canceled | An engagement terminated as of the study date. |
Active | An engagement still running as of the study date. |
Failure category | One of five groups of 20 rated indicators, each describing the conditions of an engagement by where a weakness lies. |
Financial institution readiness | Conditions inside the financial institution, covering its data, its processes, its legacy systems, the skills of its people, its governance, and its responsiveness. |
Vendor capability | Conditions inside the vendor, covering leadership, industry knowledge, agentic AI knowledge, the delivery playbook, platforms, talent, training, and the commercial model. |
Co-management | Conditions arising from the two firms working together, covering decision rights, joint management of risks and issues, change control, human review, and the fit of governance. |
AI technology | Conditions of the agentic tools themselves, covering reliability, non-determinism, memory, context, security, authorized action, token economics, and observability. |
Third-party ecosystem | Conditions arising from third-party providers and the chain of dependencies beneath them. |
Indicator | A survey question, engagement field or study measure analyzed in its own companion analysis. |
Rated indicator | One of the 100 indicators, 20 in each failure category, that make up the engagement indicator mean. |
10-point scale | Scale from 1 to 10 on which respondents rated the indicators and the other rated questions. |
Rating | A respondent's answer from 1 to 10 on a rated question. |
Favorable-direction indicator | An indicator on which favorability increases across the 10-point scale, where 1 is the most unfavorable rating and 10 the most favorable. |
Favorable-direction conversion | Presentation of an unfavorable-direction indicator’s rating in the favorable direction, made by subtracting each rating from 11, so that a higher figure denotes the more favorable outcome. |
Unfavorable-direction indicator | An indicator on which 1 is the most favorable rating and 10 the least favorable. The eight are C019, C021, C023, C025, C039, C051, C053 and C054, among them intent to leave and cancellation likelihood. |
Question without a favorable end | A rated question with neither a favorable nor an unfavorable end. The eight are C027, C028, C066, C067, C068, C203, C209 and C215. |
Distressed | A rating at the bad end of the question's own scale, 3 or below where a low rating is bad and 8 or above where a high rating is bad. The rule applies to an engagement's mean rating in the same way. |
Indicators distressed | Share of an engagement's ratings on the rated indicators that are distressed, computed for each respondent, averaged within the engagement and then across engagements. |
Indicator quintile | One of five bands of 30 engagements, ranked by engagement indicator mean. |
Indicator class | Level at which an indicator was recorded, as respondent level, engagement attribute, or perceived fact. |
Engagement attribute | One fact about the engagement, repeated across its respondents and never counted as independent observations. |
Perceived fact | A fact about the engagement as the respondent understood it, such as the asset size of the financial institution. An answer of “I do not know” is reported as a finding. |
Structured answers | Answers to the 195 structured survey questions, 172 numeric and 23 categorical. The study holds 149,619. |
Written answers | Answers to the study's 16 open questions. The study holds 8,918. |
Written response | A written answer about the respondent's own engagement that addresses a given indicator. Non-answers and answers about another engagement are excluded. |
Primary and secondary response | A primary response answered the open question of the indicator's own survey section. A secondary response answered another open question and addressed the indicator in its content. |
Sentiment score | Score of a written response on five values, from −10, a subject described at or near its worst, through −5, 0 and +5 to +10, at or near its best. |
Theme | A recurring point in the written answers that no survey question captures, such as the frequency of manual work. |
Emotional state | A state the respondent reported of himself or herself in a written answer, such as stressed, tired, or frustrated. |
Non-answer | Answer “I do not know” or “I prefer not to answer,” counted separately and excluded from a mean. |
Four core study measures | Cancellation rate, customer satisfaction, engagement indicator mean, and intent to leave, applied to every indicator in that order. |
Cancellation rate | Share of a named group of engagements canceled by the study date, computed at engagement level. |
Customer satisfaction | Satisfaction figure recorded against each engagement on a 100-point scale, as reported by the delivery team. The measure is never abbreviated. |
Engagement indicator mean | Mean of an engagement's ratings on the 100 rated indicators, computed for each respondent and averaged within the engagement. |
Intent to leave | A respondent's rating, from 1 to 10, of how likely he or she is to leave the current job in the next 12 months. |
Measured return | Return on an engagement, recorded for the 30 engagements in Production and printed with that base. |
Cancellation likelihood | Delivery team's rating, from 1 to 10, of how likely the financial institution was to cancel an active engagement. |
High risk | An active engagement whose mean cancellation likelihood rating was 7 or above. |
Moderate risk | An active engagement whose mean cancellation likelihood rating was 5 or above and below 7. |
Canceled or at risk | Canceled engagements together with the active engagements at high or moderate risk. |
Manual alteration | A change made by hand by the delivery team to financial institution material, agent output, or reporting, counted by five methods. |
Manual alteration of financial-institution-provided data | Changing by hand the data the financial institution supplied to the engagement. |
Manual alteration of financial institution processes | Changing by hand a process of the financial institution, such as deciding a process step the financial institution had not decided. |
Manual alteration of AI outputs | Adjusting by hand the output of an agent before it was used or sent. |
Reporting alteration | Vendor's alteration of the performance reports and dashboards the financial institution received. |
Third-party work done by hand | Delivery team performing by hand the tasks a third-party system did not deliver, such as keying in data a broken feed did not supply. |
Internal disclosure | Telling the vendor's own management about a manual alteration. |
Financial institution disclosure | Respondent or the respondent's management discussing a manual alteration with the financial institution. |
Reported experience | Respondents' account of their own engagements, the evidence class of the whole study. |
Engagement level | A figure computed with one value per engagement, on a base of up to 150 engagements. |
Respondent level | A figure computed across respondents, whose answers cluster within engagements, on a base of up to 850 respondents. |
Base | Count of respondents, answers, or engagements on which a figure was computed, being those who answered rather than those who were asked. |
Confidence interval | Range around an estimate that contains the true value at a stated level of confidence, 95% unless stated otherwise. |
p value | Probability of a result at least as extreme as the one observed if no association existed. |
Pearson's r | Correlation between two measured quantities, from −1 to 1. |
Spearman's rho | A correlation computed on the ranks of the values, used as a check on Pearson's r. |
Chi-square test | A test of whether a split across categories departs from what independence would produce. |
Welch's analysis of variance | A test of whether a mean differs across the levels of an attribute, without assuming equal spread across the levels. |
Holm correction | An adjustment of each p value in a family of tests for the number of tests in the family. |
False discovery rate | Expected share of false positives among the results that pass the test, controlled across the 300 partial tests. |
Detection floor | Smallest correlation the tests could reliably detect, 0.23 at 150 engagements, a test level of 0.05, and 80% power. |
Equivalence bound | Bound of 0.30 in absolute value of r, below which an association is treated as negligible. |
Equivalence test | A pair of one-sided tests of whether an association lies within the equivalence bound. |
Held | State of an association that had p below 0.05, survived correction for multiple comparisons, and kept its sign and its p below 0.05 under the robustness checks. |
Did not hold | State of an association that had p below 0.05 on the primary test but failed correction or one or more robustness checks. |
Null equivalent to zero | A result that had p of 0.05 or above and whose interval lay within the equivalence bound. |
Inconclusive null | A result that had p of 0.05 or above and could not be shown equivalent to zero. |
Clean estimate | An association computed against customer satisfaction among the 94 active engagements, where no cancellation had occurred for a respondent to react to. |
Descriptive contrast | An association computed against realized cancellation across all 150 engagements, whose canceled engagements were rated after cancellation. |
Leave-one-out | Recomputation of the engagement indicator mean without the indicator under test. |
Partial test | A test of whether an indicator was associated with a measure beyond what the engagement indicator mean accounts for. |
Intraclass correlation (ICC) | Share of the variance in an indicator that lies between engagements rather than within them. |
Design effect | Factor by which clustering within engagements inflates variance, equal to one plus the ICC multiplied by the mean cluster size less one. |
Effective sample size | Number of respondents who answered divided by the design effect, the number of independent answers the clustered answers are worth. |
Cluster bootstrap | An interval computed by resampling whole engagements rather than respondents. The answers from one engagement stay together. |
Robustness check | A repeat of a primary test on a subset of the study set or under another method. |
Study norm | Median association of the study's indicators with each of the four core study measures, against which each indicator is compared. |
Thin base | A base under 30, printed with its count and without a confidence interval. |
Anecdote strength | A base under 10, reported as counts only and never as a rate. |
Companion analysis | Full statistical analysis of one indicator, engagement field, study measure, or derived analysis. The study holds 216. |