Study Data · Part 1 · Indicator Deep-Dive Analysis

Reported return on the engagement: ROI %

Indicator ANALYSIS:measured_return · By Phil Hatch · Akholi · October 9, 2026

Indicator Specification

Table 1. Indicator specificationTable
FieldValue
Indicator codeNone, a derived analysis
Question as askedReported return on the engagement: ROI %
Failure categoryNot one of the 100 rated indicators
Place in the surveyNot a survey item
Indicator classDerived from other indicators
Indicator typeDerived analysis
Response scaleNot applicable, a derived analysis
Answering baseAs stated in Analysis Results
Non-answersNot applicable

Summary of Findings

Respondents reported a return on 17 of the 30 engagements in production. Among the engagements with a reported return, the return on the engagement had a mean close to its median (mean 24.7, n = 17 engagements, median 23.0). The rule printed in Analysis Results recorded all 10 canceled engagements in production at zero return. Active engagements in production (n = 20 engagements) had a mean return (mean 21.0), and 3 of them were recorded at zero. Among engagements with a reported return, higher returns came with higher customer satisfaction (r = 0.71, 95% CI 0.35 to 0.89, n = 17 engagements) and lower intent to leave (r = −0.71, 95% CI −0.89 to −0.34, n = 17 engagements). The association between return and the engagement indicator mean did not hold under test (r = 0.47, 95% CI −0.01 to 0.78, p 0.056, n = 17 engagements).

Analysis Results

The figures in this section were computed for the study and placed by code from the statistics record.

Table 2. Analysis resultsTable
ItemValue
Unit of analysisEngagement
Rulea
Engagements in production30
Production engagements canceled10
Returns recorded outside production0
Returns recorded in production, zero included30
Reported a return above zero17
Mean return of those reporting one, percent24.7
Production engagements with a zero return13
Of those, canceled10
Active production engagements with a zero return3
Base under 30 (thin base)Yes
Note: a R-008: a return’s base is the engagements in production, and a zero is an engagement that has not generated a return.

Active production engagements, the return.

Table 3. Analysis results: active production engagements, measured returnTable
ItemValue
n20
Mean21.0
SD14.7
Median20.0
Min0.0
Max47.0

Engagements that reported a return above zero, the return.

Table 4. Analysis results: engagements that reported a return above zero, measured returnTable
ItemValue
n17
Mean24.7
SD12.6
Median23.0
Min6.0
Max47.0

The return against the measures.

Table 5a. Analysis results: measured return against the study measures, part 1 of 2Table
MeasureBasenrR squared95% CI
Customer satisfactionEngagements that reported a return170.710.500.35 to 0.89
Engagement indicator meanEngagements that reported a return170.470.22−0.01 to 0.78
Intent to leaveEngagements that reported a return17−0.710.50−0.89 to −0.34
Table 5b. Analysis results: measured return against the study measures, part 2 of 2Table
MeasuretdfpTOST pEquivalent to zeroPower at the observed effectSmallest detectable r (MDE)
Customer satisfaction3.91150.0010.985No0.910.63
Engagement indicator mean2.07150.0560.776No0.480.63
Intent to leave−3.85150.0020.983No0.910.63

Distribution of Recorded Values

This block is a derived analysis. Its own results are in Analysis Results. The blocks of its input indicators carry the response distribution of each input.

Descriptive Statistics by Segment

This block is a derived analysis. Its own results are in Analysis Results. The blocks of its input indicators carry the segment breakdown of each input.

Tests of Difference Between Segments

This block is a derived analysis. Its own results are in Analysis Results. The blocks of its input indicators carry the tests across segments for each input.

Association with Cancellation Rate

This block is a derived analysis. Its own results are in Analysis Results. The blocks of its input indicators carry each input’s association with this measure.

Association with Customer Satisfaction

This block is a derived analysis. Its own results are in Analysis Results. The blocks of its input indicators carry each input’s association with this measure.

Association with the Engagement Indicator Mean

This block is a derived analysis. Its own results are in Analysis Results. The blocks of its input indicators carry each input’s association with this measure.

Association with Intent to Leave

This block is a derived analysis. Its own results are in Analysis Results. The blocks of its input indicators carry each input’s association with this measure.

Comparison of Associations Across the Four Measures

This block is a derived analysis. Its own results are in Analysis Results. The blocks of its input indicators carry each input’s comparison across the four measures.

Clustering and Effective Sample Size

This block is a derived analysis. Its own results are in Analysis Results. The blocks of its input indicators carry the clustering of each input.

Equivalence Tests of Null Results

This block is a derived analysis. Its own results are in Analysis Results. The blocks of its input indicators carry each input’s equivalence test.

Robustness Checks

This block is a derived analysis. Its own results are in Analysis Results. The blocks of its input indicators carry each input’s robustness checks.

Measure Independence and Part-Whole Correction

This block is a derived analysis. Its own results are in Analysis Results. The blocks of its input indicators carry each input’s identity check.

Comparison with the Study Norm

This block is a derived analysis. Its own results are in Analysis Results. The blocks of its input indicators carry each input’s profile.

Interpretation of the Data

The zeros in production came mostly from canceled engagements, 10 of 13. The positive association with customer satisfaction (t 3.91, df 15, p 0.001) and the negative association with intent to leave (t −3.85, df 15, p 0.002) both held under test.

Limitations

The evidence is reported experience: what the delivery team said about its own engagement, gathered in one cross-section. The figures showed associations, not causes. Indicators about the financial institution were reported at second hand by vendor staff. Cancellation is a descriptive contrast, because canceled engagements were rated after they ended.

The associations rested on a small group of engagements with a reported return (n = 17 engagements). The test against the engagement indicator mean ran at low power (power at the observed effect 0.48).

Definitions

Table 6. DefinitionsTable
TermDefinition
Indicator classDerived from other indicators: computed by the study from other recorded fields, as Analysis Results sets out.
Failure categoryOne of the five failure categories of the Study Taxonomy, section 3. Only the 100 rated indicators carry one, and the indicator is not among them.
DirectionEach result is read in the direction of the measure it reports.
Engagement indicator meanThe mean of an engagement’s 100 indicator ratings.

Sources and Provenance

Table 7. Sources and provenanceTable
ItemValue
Study dataStudy Data.xlsx, sha256 219eba0457274d04
Statistics enginecompute_statistics.py, sha256 738319eef764602b
Statistics recordComputed Statistics.json, record ANALYSIS:measured_return, sha256 e405f65a43e37a6e
ComputedOctober 8, 2026, 01:38 UTC
TemplateIndicator Statistical Analysis Template, version 1.1 of 2026-10-06
Figures placed by script86 of 86

Continue the report

Executive summary →
Overview

Executive Summary

Four findings from 150 engagements

Read →
Introduction

Introduction

Scope, evidence, and how the report is organized

Read →
Part 1

Engagement Outcomes

Poor results on every measure, and what separates them

Read →
Part 2

The Five Failure Categories

Where failure concentrates, category by category

Read →
Part 3

Governance, Oversight, and Control

Controls that existed in name only

Read →
Part 4

The Delivery Workforce

Early in their careers, dissatisfied, and many intending to leave

Read →
Part 5

Manual Alteration

How the delivery team altered the work to sustain it

Read →
Part 6

Executive Action Plan

Seven immediate actions and a first-year plan

Read →
Appendices

Appendices

Definitions, indicator inventory, method, and disclosures

Read →
Data

Study Data

Deep-dive analyses of the study indicators, published in stages

Read →