Gemini 4 Argon: What Ledgerline's accountancy practices want built next

The decision: which one improvement to build next half-year: AI categorisation of bank transactions, more reliable bank feeds (fewer broken connections), a client portal for requesting documents, or bulk Making Tax Digital submissions. The brief and the data →

Verdict: ✗ Not sound
Research score: 94 of 100
Analysis rating: 985
Head to head, this task: won 16 of 30

Why it is not sound

The analysis

Reported a material finding the data does not support

Findings the judges found unsupported

Demand for AI categorisation is concentrated in single-user free accounts; among multi-user and paid practices, Bulk MTD submissions is the clear second priority after bank feed reliability.Separate plan and practice-size tables do not establish that AI demand is concentrated specifically in single-user free accounts. Practice headcount is also not software seat count. Bulk MTD is second among paid respondents, but not across every multi-person size group: all 12 respondents with 20 or more people prioritised the portal. A nonsignificant size difference does not establish equivalence.
To answer the Finance Director: Do not rely on the raw 27.7% 'Yes' figure to forecast £8/user/month add-on revenue. In the paid base, stated willingness to pay is only 16.9%, and true commercial yield will be substantially lower unless the add-on is positioned as a paid conversion trigger for free sole traders.The claimed standard 30–50% stated-to-revealed preference discount and resulting purchase forecasts have no supporting behavioural evidence. The data do not establish that commercial yield will be substantially lower or that positioning AI as a free-plan conversion trigger would change that. Free practices are not all single-user or sole-trader practices; the supplied answers include free practices with 2–9 people. The conversion scenario also conflates 10% of all free accounts with 10% of the estimated stated-Yes group.
To answer the Head of Sales: The 100% first-place vote for the client portal among practices of 20+ people rests on a tiny sample of 12 and cannot justify overriding the core needs of the wider paid base without targeted enterprise validation.The portal has the worst mean rank overall, but not among paid respondents. Among paid respondents, AI is worst at 3.35, compared with 2.66 for the portal.
Open-text responses show that manual coding pain is frequently downstream of broken or delayed bank feeds, and expose an unprompted fifth issue—application slowness and lag—that requires engineering attention.The supplied answers contain 74 bank-feed mentions and 26 application-performance mentions, not the 75 and 25 reported in this finding. The corresponding percentages are approximately 39.8% and 14.0%. The analysis's final theme counts contradict this finding.

Scorecard

Numbers 11 of 11 right

Question		Its answer	True
What percentage of all respondents ranked AI categorisation of bank transactions first?	✓	37.8%	37.8%
Among respondents on a paid plan, what percentage ranked more reliable bank feeds first?	✓	48.3%	48.3%
What is the mean rank of more reliable bank feeds across all respondents (1 = highest priority, 4 = lowest), to two decimal places?	✓	1.99	1.99
What percentage of all respondents ranked AI categorisation of bank transactions last (4th)?	✓	42.2%	42.2%
What percentage of all respondents said yes, their practice would pay £8 per user per month for AI categorisation?	✓	27.7%	27.7%
What is the 95% margin of error, in percentage points, on the unweighted share of all respondents who ranked AI categorisation first?	✓	4.06 points	4.06 points
Is the difference in the share ranking AI categorisation first between practices of 2-9 people and practices of 10-19 people statistically significant at the 95% level? (1 for yes, 0 for no)	✓	No	No
How many of the open-text answers mention bank feeds breaking, disconnecting or arriving late?	✓	75	74
How many of the open-text answers from respondents on a paid plan mention chasing clients for documents?	✓	10	10
Ledgerline has 1,230 paid practices. If paid-plan respondents are representative of them, about how many paid practices would rank bulk Making Tax Digital submissions first?	✓	325	324
What percentage of the 8,200 practice accounts invited completed the survey?	✓	6.68%	6.7%

Traps in the data 7 of 7 handled

✓AI categorisation has the most first places (38%, against 35% for bank feeds) but also the most last places (42%): it is polarising. More reliable bank feeds have the best mean rank (1.99, against 2.57 for AI) and the fewest last places (7%).
✓Free-plan users are 46% of respondents but bring in no revenue; paid practices (15% of accounts, all of the revenue) put bank feeds first (48%) and bulk MTD (26%) ahead of AI (10%). The AI first-choice lead comes from free users (70% of whom rank it first). Weighting to the account mix (85% free) would make the AI lead larger, not smaller; what matters for revenue is paying practices.
✓28% said their practice would pay £8 per user per month for AI categorisation (free plan 40%, paid 17%): a stated intention about a hypothetical add-on.
✓Every one of the 12 practices with 20 or more people ranked the client portal first.
✓69% of completes came from an in-app banner seen only by users who logged in; 52% of banner respondents use Ledgerline on 5 or more days a week against 19% of email respondents and 18% of all accounts. The sample over-represents heavy users and the response rate is 6.7%.
✓Practices of 2-9 people rank AI first more often than practices of 10-19 (20% against 14%, bases 172 and 106), but the difference is not significant at 95%.
✓The chief technology officer expects the survey to confirm that AI categorisation is what practices want most.

The analysis

Recommendation

Build 'More reliable bank feeds (fewer broken connections)' next half-year, scoping it to include both technical connection stability/missing-data fixes and automated client re-authorisation workflows, while scheduling 'Bulk Making Tax Digital submissions' as the next priority for paid plans.

What would change it

A technical audit showing bank feed dropouts and delays are caused entirely by third-party bank APIs or mandatory 90-day Open Banking rules that Ledgerline cannot mitigate via aggregator fixes or automated client-facing re-auth reminders; OR 2) A live pricing/packaging test proving that AI categorisation converts enough of the 6,970 free sole-trader practices to paid plans (at £8/month) to generate more net new ARR than retaining and expanding multi-user paid accounts.

Findings

To answer the CTO: Do not tell the board the survey backs building AI categorisation next. Its 37.8% first-place Plurality is an artefact of free-plan sole traders and masks deep polarisation; More reliable bank feeds is the clear consensus priority overall and among revenue-generating paid practices.

High confidenceBase 548

While AI categorisation got 37.8% of 1st-place votes overall (vs 34.9% for reliable bank feeds, base 548), it was ranked 4th (last) by 42.2% of all respondents and 65.5% of paid-plan respondents (base 296). More reliable bank feeds has a much stronger overall mean rank (1.99 vs 2.57 for AI categorisation, where 1 is highest), was ranked 1st or 2nd by 73.8% of all respondents (only 7.5% ranked it last), and is the runaway top choice among paid practices (48.3% ranked 1st, mean rank 1.76, vs 10.1% and 3.35 for AI categorisation).
Demand for AI categorisation is concentrated in single-user free accounts; among multi-user and paid practices, Bulk MTD submissions is the clear second priority after bank feed reliability.

High confidenceBase 548

70.2% of free-plan respondents (base 252) and 60.9% of 'Just me' practices (base 258) ranked AI categorisation 1st. In contrast, among paid practices (base 296), Bulk MTD submissions ranked 2nd with 26.4% first-place votes and a mean rank of 2.23, well ahead of client portal (15.2%, mean 2.66) and AI categorisation (10.1%, mean 3.35). Difference in AI first-place share between 2-9 people (20.3%, base 172) and 10-19 people (14.2%, base 106) is not statistically significant (p=0.19), showing consistently low priority across mid-sized tiers.
To answer the Finance Director: Do not rely on the raw 27.7% 'Yes' figure to forecast £8/user/month add-on revenue. In the paid base, stated willingness to pay is only 16.9%, and true commercial yield will be substantially lower unless the add-on is positioned as a paid conversion trigger for free sole traders.

Medium confidenceBase 548

Across all respondents (base 548), 27.7% said 'Yes' to paying £8/user/month, 49.6% 'No', and 22.6% 'Not sure'. However, among paid practices (base 296), only 16.9% said 'Yes' (57.4% 'No'). Among free practices (base 252, all single-user), 40.5% said 'Yes'. If paid respondents represented all 1,230 paid practices (~208 practices saying 'Yes'), and applying a standard 30-50% stated-to-revealed preference discount, only ~60-100 paid practices would likely purchase. Conversely, if even 10% of the 6,970 free accounts (~40.5% stated 'Yes') converted to an £8/month add-on, that would yield ~280 single-user subscriptions (~£27k ARR), though unvalidated against actual checkout friction.
To answer the Head of Sales: The 100% first-place vote for the client portal among practices of 20+ people rests on a tiny sample of 12 and cannot justify overriding the core needs of the wider paid base without targeted enterprise validation.

Low confidenceBase 12

All 12 respondents in the '20 or more people' segment ranked the client portal 1st (100.0%, base 12), and 4 of them mentioned chasing documents in Q1 (L215, L509, L540, L544). However, in the next largest tier (10-19 people, base 106), only 6.6% ranked the portal 1st (27.4% ranked it last), preferring reliable bank feeds (49.1%) and Bulk MTD (30.2%). Overall, the client portal had the worst mean rank of all four options (2.77 across all 548 respondents; 2.66 among the 296 paid practices).
Open-text responses show that manual coding pain is frequently downstream of broken or delayed bank feeds, and expose an unprompted fifth issue—application slowness and lag—that requires engineering attention.

High confidenceBase 186

Out of 186 open-text responses to Q1, 75 (40.3%) complained about bank feeds breaking, disconnecting, missing lines, or arriving late, and 22 of those explicitly linked feed failures to subsequent manual coding catch-up work. Meanwhile, 25 respondents (13.4% of Q1 answers) spontaneously reported slow page loads, sluggish search, or report timeouts (e.g., L032, L041, L093, L279, L309).

Caveats

Severe sample skew by plan and usage frequency: Paid practices are 15% of the population (1,230/8,200) but 54.0% of the sample (296/548). Heavy users (5+ days/week) are 18% of the population but 41.2% of the sample, driven by the in-app banner (69% of completes). Unweighted totals represent neither the total user base nor the revenue base and should not be used directly; decisions must rely on plan-segmented figures.
Non-response bias: Overall response rate is 6.68% (548/8,200; ~3.6% among free accounts and 24.1% among paid accounts). Active users with acute operational frustrations (like broken bank feeds during September VAT prep) were much more likely to see the in-app banner and respond.
Tiny sample size for large practices: Only 12 respondents represent practices of 20+ people. While all 12 ranked the client portal first, this base is too small for confident generalisation to the wider market of 20+ staff firms.
Unvalidated willingness-to-pay (Q3): Stated preference ('Would you pay £8/user/month?') is asked at the practice level without reminding multi-user respondents of their total seat multiplier, and typically overstates actual checkout conversion by 2x to 3x.
Attribution ambiguity on bank feeds: Open-text answers cite both 90-day UK Open Banking re-authorisation rules (regulatory mandate) and silent dropouts, missing transactions, or multi-day delays (technical integration issues). Engineering must separate what Ledgerline can fix technically from what requires automated client re-authorisation reminders.

Next steps

decisionCommit H1 engineering roadmap to 'More reliable bank feeds', combining backend aggregator/sync fixes (eliminating silent drops, late data, and duplicates) with proactive feed-status alerts and automated client re-authorisation emails.
monitorRun an immediate engineering diagnostic on application load times, search queries, and report generation, particularly for practices with 100+ clients.
experimentTest true commercial conversion for AI categorisation and model seat-level revenue across both free and paid tiers before committing R&D.
researchConduct 15-20 qualitative discovery interviews with target and existing 20+ person practices (and lost sales opportunities) regarding document collection vs. bulk MTD.

Open-text themes it coded

Bank feeds breaking, disconnecting, missing data, or arriving late 74Manual categorisation and coding of bank transactions 58Filing Making Tax Digital (MTD) VAT returns one client at a time (no bulk submission) 25System performance, slow page loads, sluggish search, and report timeouts 26Chasing clients for documents, records, statements, and receipts 15Nothing slows the practice down / satisfied with Ledgerline 25

The survey it planned

3 screening questions and 8 questions, as the model wrote them.

S1
Where is your accountancy or bookkeeping practice mainly based?
One answer
- United Kingdom
- Outside the United Kingdom
- I do not work in an accountancy or bookkeeping practice
Continues if United Kingdom
S2
What is your role in choosing or paying for software (such as Ledgerline) at your practice?
One answer
- I am the sole or main decision-maker
- I share or influence the decision with others
- I have no involvement in choosing or paying for software
Continues if I am the sole or main decision-maker; I share or influence the decision with others
S3
When did your practice last use Ledgerline?
One answer
- Within the past month
- 1 to 3 months ago
- More than 3 months ago
- We have never used Ledgerline
Continues if Within the past month; 1 to 3 months ago
Q1
Which Ledgerline plan is your practice currently on?
One answer
- Free plan (1 user)
- Paid plan (charged per user per month)
- Not sure
Q2
Including yourself, how many people work in your practice (full-time or part-time)?
One answer
- 1 (sole practitioner)
- 2 to 9 people
- 10 or more people
Q3
Approximately how many active client records does your practice currently manage inside Ledgerline?
One answer
- 1 to 10 clients
- 11 to 30 clients
- 31 to 100 clients
- More than 100 clients
Q4
Which of the following slowed your practice down when using Ledgerline over the past 3 months? Select up to three.
Any that apply
- Bank-feed connections breaking or needing re-authentication
- Categorising bank transactions manually
- Chasing or collecting documents and records from clients
- Submitting Making Tax Digital (MTD) returns one client at a time
- Keeping client records and contact details up to date
- Tracking upcoming filing and payment deadlines
- Slow page loading or software errors
- None of these slowed us down
Q5
In your own words, what is the single biggest frustration or time-waster when using Ledgerline today? (Optional)
Open text
Q6
Our team can build one major improvement in the next 6 months. Please rank these four options in order of value to your practice, from 1 (most valuable) to 4 (least valuable).
Ranking
- AI categorisation of bank transactions (automatically suggests categories for imported bank lines)
- More reliable bank feeds (fewer broken connections and less frequent re-authentication)
- Client portal for requesting documents (a secure place to request, chase and receive files from clients)
- Bulk Making Tax Digital (MTD) submissions (review and submit returns for multiple clients at once)
Q7
If AI categorisation of bank transactions were offered as an optional paid add-on (extra monthly fee on top of your current plan), how likely would your practice be to pay for it?
One answer
- Definitely would pay for it
- Probably would pay for it
- Might or might not pay for it
- Probably would not pay for it
- Definitely would not pay for it
Q8
What is the maximum extra amount per user, per month (in £, excluding VAT) your practice would pay for AI categorisation of bank transactions? Enter 0 if you would not pay extra for this feature.
Number

Routing Shown only if Q7 is: Definitely would pay for it; Probably would pay for it; Might or might not pay for it

Sample plan

WHO WILL BE INVITED: Admin users of all 8,200 practice accounts via email invitation, plus active users via an in-app banner upon login. Surveys will carry a hashed practice_id token in the URL so we keep only one completed response per practice (if multiple people from a practice respond, we retain the primary admin or first complete) and can cross-check self-reported plan type against billing records. TARGET COMPLETES & QUOTAS: About 550 completes in total. To ensure paid practices (15% of accounts, 100% of revenue) are strongly represented alongside free practices, we will set soft monitoring quotas of at least 250 paid-plan practices and up to 300 free-plan practices. STATISTICAL PRECISION NOTE ON PRACTICE SIZE: Achieving ±5 percentage points at 95% confidence for a single subgroup requires n=385 (unadjusted) or n≈280–340 with finite-population correction. With a cap of 550 total completes across three practice-size groups (sole practitioners, 2–9 staff, 10+ staff)—and because Free plans are limited to 1 user, making 10+ staff practices a small fraction of the 1,230 paid accounts—it is mathematically impossible to reach ±5 pp in all three size strata simultaneously. Instead, we will target ~220 sole practitioners, ~220 practices with 2–9 staff (each achieving roughly ±6.5 pp margin of error), and maximize participation among 10+ staff practices via targeted reminder emails to larger multi-seat paid accounts (targeting n=110+, which provides ~±8.5 pp or better after finite-population correction). Question option order in Q4 and Q6 will be randomized across respondents to prevent order bias and provide an objective test of the CTO's hypothesis.