Benchmarks / SurveyBench / What Ledgerline's accountancy practices want built next / Gemini 4 Hydrogen
Gemini 4 Hydrogen: What Ledgerline's accountancy practices want built next
The decision: which one improvement to build next half-year: AI categorisation of bank transactions, more reliable bank feeds (fewer broken connections), a client portal for requesting documents, or bulk Making Tax Digital submissions. The brief and the data →
- Verdict
- ✗ Not sound
- Research score
- 75 of 100
- Analysis rating
- 1,088
- Head to head, this task
- won 5 of 9
Why it is not sound
The analysis
- Made a recommendation that does not follow from the data
- Missed a trap: 69% of completes came from an in-app banner seen only by users who logged in; 52% of banner respondents use Ledgerline on 5 or more days a week against 19% of email respondents and 18% of all accounts. The sample over-represents heavy users and the response rate is 6.7%.
Scorecard
Numbers 11 of 11 right
| Question | Its answer | True | |
|---|---|---|---|
| What percentage of all respondents ranked AI categorisation of bank transactions first? | ✓ | 37.8% | 37.8% |
| Among respondents on a paid plan, what percentage ranked more reliable bank feeds first? | ✓ | 48.3% | 48.3% |
| What is the mean rank of more reliable bank feeds across all respondents (1 = highest priority, 4 = lowest), to two decimal places? | ✓ | 1.99 | 1.99 |
| What percentage of all respondents ranked AI categorisation of bank transactions last (4th)? | ✓ | 42.2% | 42.2% |
| What percentage of all respondents said yes, their practice would pay £8 per user per month for AI categorisation? | ✓ | 27.7% | 27.7% |
| What is the 95% margin of error, in percentage points, on the unweighted share of all respondents who ranked AI categorisation first? | ✓ | 4.06 points | 4.06 points |
| Is the difference in the share ranking AI categorisation first between practices of 2-9 people and practices of 10-19 people statistically significant at the 95% level? (1 for yes, 0 for no) | ✓ | No | No |
| How many of the open-text answers mention bank feeds breaking, disconnecting or arriving late? | ✓ | 74 | 74 |
| How many of the open-text answers from respondents on a paid plan mention chasing clients for documents? | ✓ | 10 | 10 |
| Ledgerline has 1,230 paid practices. If paid-plan respondents are representative of them, about how many paid practices would rank bulk Making Tax Digital submissions first? | ✓ | 325 | 324 |
| What percentage of the 8,200 practice accounts invited completed the survey? | ✓ | 6.68% | 6.7% |
Traps in the data 6 of 7 handled
- ✓AI categorisation has the most first places (38%, against 35% for bank feeds) but also the most last places (42%): it is polarising. More reliable bank feeds have the best mean rank (1.99, against 2.57 for AI) and the fewest last places (7%).
- ✓Free-plan users are 46% of respondents but bring in no revenue; paid practices (15% of accounts, all of the revenue) put bank feeds first (48%) and bulk MTD (26%) ahead of AI (10%). The AI first-choice lead comes from free users (70% of whom rank it first). Weighting to the account mix (85% free) would make the AI lead larger, not smaller; what matters for revenue is paying practices.
- ✓28% said their practice would pay £8 per user per month for AI categorisation (free plan 40%, paid 17%): a stated intention about a hypothetical add-on.
- ✓Every one of the 12 practices with 20 or more people ranked the client portal first.
- ✗69% of completes came from an in-app banner seen only by users who logged in; 52% of banner respondents use Ledgerline on 5 or more days a week against 19% of email respondents and 18% of all accounts. The sample over-represents heavy users and the response rate is 6.7%.
- ✓Practices of 2-9 people rank AI first more often than practices of 10-19 (20% against 14%, bases 172 and 106), but the difference is not significant at 95%.
- ✓The chief technology officer expects the survey to confirm that AI categorisation is what practices want most.
The analysis
Findings
-
The CTO should not tell the board the survey backs building AI categorisation next: across all respondents, more reliable bank feeds leads decisively on overall priority (mean rank 1.99 vs 2.57), while AI categorisation is polarising and ranked dead last by 42.2% of respondents.
Unweighted 1st-place share for AI categorisation (37.8%, 95% MoE ±4.06 pp) is statistically indistinguishable from more reliable bank feeds (34.9%), and 42.2% ranked AI categorisation 4th vs only 7.5% ranking bank feeds 4th (73.8% ranked bank feeds 1st or 2nd). Base: 548.
-
AI categorisation's 1st-place headline lead is an artefact of Free-plan and solo users; paying customers—who generate 100% of Ledgerline's revenue—overwhelmingly prioritise reliable bank feeds and bulk MTD submissions instead.
On the free plan (n=252, 85% of accounts), 70.2% ranked AI categorisation 1st (mean rank 1.66). On paid plans (n=296, 15% of accounts, 100% of revenue), 48.3% ranked reliable bank feeds 1st (mean rank 1.76) and 26.4% ranked bulk MTD 1st (mean rank 2.23), while only 10.1% ranked AI categorisation 1st and 65.5% ranked it last (mean rank 3.35). Note that weighting by account share (85% free / 15% paid) yields 61.2% 1st-place for AI categorisation, but weighting by revenue (100% paid) puts bank feeds far ahead.
-
Unreliable bank feeds are the top operational pain point in open text (74 of 186 answers) and actively create manual coding work (26 answers link the two), meaning fixing feeds reduces manual coding overhead as well.
74 of 186 Q1 open-text answers describe feeds disconnecting/expiring (52), arriving late (15 additional), or missing transactions (7 additional); 51 of those 74 come from paid-plan respondents. 26 of 57 manual-coding mentions explicitly attribute backlog hand-coding to broken or dropped feeds.
-
The Finance Director cannot reliably forecast £8/user/month add-on revenue from this survey alone, and naive extrapolation from the 27.7% overall 'Yes' rate would overstate revenue because paying and multi-user practices show far lower willingness to pay.
Overall 'Yes' is 27.7% (49.6% 'No', 22.6% 'Not sure', n=548), driven by free-plan solo users (40.5% 'Yes', n=252). Among paid practices (n=296), only 16.9% (~208 of 1,230 paid practices) say 'Yes' and 57.4% say 'No'; multi-user firms report 21.5% 'Yes' (2-9 people, n=172), 17.0% (10-19, n=106), and 16.7% (20+, n=12). The survey lacks paid seat counts per account and relies on stated hypothetical intent.
-
The Head of Sales' claim that larger practices want a client portal rests on a very small 20+ person subgroup (n=12) and is contradicted by 10-19 person practices, who rank the client portal lowest of all four options.
100% of 20+ person respondents (12/12) ranked the client portal 1st (and all 4 who answered Q1 cited chasing documents), whereas only 6.6% of 10-19 person practices (n=106) ranked it 1st (vs 49.1% for bank feeds and 30.2% for bulk MTD) and 14.0% of 2-9 person practices (n=172) ranked it 1st. Across all respondents, client portal had the worst mean rank (2.77).
-
Bulk Making Tax Digital (MTD) submissions is a clear secondary bottleneck for paid and growing practices, ahead of both AI categorisation and the client portal.
Paid-plan respondents ranked bulk MTD 2nd overall (26.4% 1st, mean rank 2.23, only 7.8% last; n=296), representing ~325 of Ledgerline's 1,230 paid practices. In 10-19 person firms (n=106), 30.2% ranked bulk MTD 1st. In Q1, 22 of 25 individual-MTD complaints came from paid practices.
-
Unprompted slow app performance and report/search lag is a material hidden pain point affecting larger-client practices that did not appear in the four Q2 roadmap choices.
26 of 186 Q1 answers (14.0%) spontaneously cited slow loading, sluggish search, timeouts on reports, or lag during peak hours/months and at higher client volumes (140-300 clients)—slightly exceeding individual MTD mentions (25) and chasing documents (15).
Open-text themes it coded
Unreliable bank feeds (disconnections, delays, and missing transactions) 74Manual coding and categorisation of bank transactions 57Slow application performance, loading, and search 9
The survey it planned
2 screening questions and 8 questions, as the model wrote them.
-
S1
Does your accountancy or bookkeeping practice operate in the UK and currently use Ledgerline?
One answer- Yes, we are a UK practice currently using Ledgerline
- Yes, we are a UK practice, but we do not currently use Ledgerline
- No, we are based outside the UK
- No, I am not part of an accountancy or bookkeeping practice
Continues if Yes, we are a UK practice currently using Ledgerline
-
S2
How involved are you in deciding or influencing which software tools your practice uses?
One answer- I make the final decisions on practice software
- I influence or recommend which software the practice uses
- I have no involvement in software decisions
Continues if I make the final decisions on practice software; I influence or recommend which software the practice uses
-
Q1
Which of the following currently slow your practice down most when using Ledgerline? Please select up to 3.
Any that apply- Fixing broken or disconnected bank feeds
- Manually categorising bank transactions
- Chasing clients for missing documents or records
- Submitting Making Tax Digital (MTD) returns one client at a time
- Updating and maintaining client details across records
- Tracking filing deadlines and task statuses
- Slow page loading or general system performance
- Something else
- Nothing slows us down noticeably
-
Q2
In a sentence or two, what is the single biggest day-to-day frustration or time-waster for your practice when using Ledgerline? (Optional)
Open text -
Q3
Ledgerline can build one major improvement over the next six months. Please rank these four options from 1 (highest priority for your practice) to 4 (lowest priority).
Ranking- AI categorisation of bank transactions (automatically suggests categories for imported bank transactions)
- More reliable bank feeds (fewer broken connections and fewer prompts to re-authenticate)
- Client portal for requesting documents (send requests and collect client documents in one place)
- Bulk Making Tax Digital (MTD) submissions (review and submit MTD returns for multiple clients together)
-
Q4
If Ledgerline introduced AI categorisation of bank transactions as an optional paid add-on on top of your current plan, would your practice pay for it?
One answer- Yes, we would subscribe to it as a paid add-on
- Maybe, depending on the price
- No, we would only use it if it were included at no extra charge
- No, we would not use AI categorisation of bank transactions
- Not sure
-
Q5
What is the highest additional price per user per month your practice would consider paying for the AI categorisation add-on?
One answer- Up to £5 per user per month
- £6 to £10 per user per month
- £11 to £15 per user per month
- More than £15 per user per month
- Not sure
Routing Shown only if Q4 is: Yes, we would subscribe to it as a paid add-on; Maybe, depending on the price
-
Q6
Which Ledgerline plan does your practice currently use?
One answer- Free plan (1 user)
- Paid plan (per-user monthly plan)
- Not sure
-
Q7
Including yourself, how many people work in your practice in total?
One answer- 1 (sole practitioner)
- 2 to 9 staff
- 10 or more staff
-
Q8
Roughly what share of your practice's active clients do you currently manage in Ledgerline?
One answer- Fewer than 25% of our clients
- 25% to 50% of our clients
- 51% to 90% of our clients
- More than 90% of our clients
Sample plan
INVITATION & DEDUPLICATION: Send an email invitation to the account admin of all 8,200 Ledgerline practice accounts (1,230 paid, 6,970 free) and display an in-app banner upon login to reach active practitioners. Pass the unique practice_account_id in both channels to keep strictly one complete response per practice (suppressing the banner and blocking duplicate links once a qualified complete is recorded for an account; if multiple slips occur, retain the primary decision-maker or earliest complete). FIELDING CONTROLS: Randomise option order in Q1 (except 'Something else' and 'Nothing slows us down noticeably' anchored at the end) and Q3 to prevent order bias. SAMPLE SIZE & PRECISION TRADE-OFF: Achieving +/- 5 percentage points at 95% confidence for each of the three practice-size tiers (sole practitioners, 2-9 staff, 10+ staff) requires ~385 completes per tier (~1,155 total, or slightly less with finite-population correction if the 10+ staff base is small). With the capped target of 550 total completes split evenly (~183 per tier), each practice-size tier has a margin of error of approximately +/- 7.2 percentage points (before finite-population correction), while the overall sample achieves +/- 4.0 percentage points. Recommend either expanding the target to ~1,000-1,155 completes (running census outreach and reminders to all paid and 10+ staff accounts) to meet the +/- 5% subgroup precision requirement, or keeping 550 completes while formally aligning stakeholders on the +/- 7.2% per-tier margin of error. QUOTAS & STRATIFICATION (FOR 550 COMPLETES): Set equal practice-size quotas of ~183 sole practitioners, ~184 practices with 2-9 staff, and ~183 practices with 10+ staff. Oversample paid accounts (target ~275 paid and ~275 free instead of the natural 15%/85% split) because paid accounts represent 100% of revenue and populate the larger practice tiers. Monitor channel mix (email vs in-app) so light/infrequent users are not under-represented by the in-app banner.