Gemini 3.1 Pro Preview: What Ledgerline's accountancy practices want built next

The decision: which one improvement to build next half-year: AI categorisation of bank transactions, more reliable bank feeds (fewer broken connections), a client portal for requesting documents, or bulk Making Tax Digital submissions. The brief and the data →

Verdict: ✗ Not sound
Research score: 74 of 100
Analysis rating: 810
Head to head, this task: won 4 of 10

Why it is not sound

The analysis

Reported a material finding the data does not support
Made a recommendation that does not follow from the data
Missed a trap: AI categorisation has the most first places (38%, against 35% for bank feeds) but also the most last places (42%): it is polarising. More reliable bank feeds have the best mean rank (1.99, against 2.57 for AI) and the fewest last places (7%).
Missed a trap: 28% said their practice would pay £8 per user per month for AI categorisation (free plan 40%, paid 17%): a stated intention about a hypothetical add-on.
Proposed a mistaken next step: Forecast AI add-on revenue from the share who said they would pay £8 per user per month

Findings the judges found unsupported

AI categorisation is highly desired by Free plan users, but has very low direct revenue potential as an add-on.Hypothetical willingness to pay does not establish very low or negligible add-on revenue. Revenue would depend on actual conversion, user counts and demand among both paid and free practices.
Our largest practices unanimously demand a client portal for requesting documents.All 12 responding practices with 20 or more people ranked the portal first, but this does not establish unanimous demand among Ledgerline's largest practices generally. The later small-base caveat appropriately limits the result, but the finding itself states it too broadly.

Scorecard

Numbers 11 of 11 right

Question		Its answer	True
What percentage of all respondents ranked AI categorisation of bank transactions first?	✓	37.8%	37.8%
Among respondents on a paid plan, what percentage ranked more reliable bank feeds first?	✓	48.3%	48.3%
What is the mean rank of more reliable bank feeds across all respondents (1 = highest priority, 4 = lowest), to two decimal places?	✓	1.99	1.99
What percentage of all respondents ranked AI categorisation of bank transactions last (4th)?	✓	42.2%	42.2%
What percentage of all respondents said yes, their practice would pay £8 per user per month for AI categorisation?	✓	27.7%	27.7%
What is the 95% margin of error, in percentage points, on the unweighted share of all respondents who ranked AI categorisation first?	✓	4.06 points	4.06 points
Is the difference in the share ranking AI categorisation first between practices of 2-9 people and practices of 10-19 people statistically significant at the 95% level? (1 for yes, 0 for no)	✓	No	No
How many of the open-text answers mention bank feeds breaking, disconnecting or arriving late?	✓	74	74
How many of the open-text answers from respondents on a paid plan mention chasing clients for documents?	✓	10	10
Ledgerline has 1,230 paid practices. If paid-plan respondents are representative of them, about how many paid practices would rank bulk Making Tax Digital submissions first?	✓	325	324
What percentage of the 8,200 practice accounts invited completed the survey?	✓	6.68%	6.7%

Traps in the data 5 of 7 handled

✗AI categorisation has the most first places (38%, against 35% for bank feeds) but also the most last places (42%): it is polarising. More reliable bank feeds have the best mean rank (1.99, against 2.57 for AI) and the fewest last places (7%).
✓Free-plan users are 46% of respondents but bring in no revenue; paid practices (15% of accounts, all of the revenue) put bank feeds first (48%) and bulk MTD (26%) ahead of AI (10%). The AI first-choice lead comes from free users (70% of whom rank it first). Weighting to the account mix (85% free) would make the AI lead larger, not smaller; what matters for revenue is paying practices.
✗28% said their practice would pay £8 per user per month for AI categorisation (free plan 40%, paid 17%): a stated intention about a hypothetical add-on.
✓Every one of the 12 practices with 20 or more people ranked the client portal first.
✓69% of completes came from an in-app banner seen only by users who logged in; 52% of banner respondents use Ledgerline on 5 or more days a week against 19% of email respondents and 18% of all accounts. The sample over-represents heavy users and the response rate is 6.7%.
✓Practices of 2-9 people rank AI first more often than practices of 10-19 (20% against 14%, bases 172 and 106), but the difference is not significant at 95%.
✓The chief technology officer expects the survey to confirm that AI categorisation is what practices want most.

The analysis

Recommendation

Prioritise fixing bank feeds and building bulk Making Tax Digital submissions. Direct the CTO not to rely on the unsegmented overall 37.8% vote for AI categorisation, because 65.5% of Paid users actively rejected it by ranking it last. Direct the Finance Director that £8/month add-on revenue for AI will be negligible, as only 16.9% of Paid users would pay for it. Advise the Head of Sales that while the largest accounts (20+ staff) unanimously want a client portal, building it now would ignore the core workflow failures—broken feeds and single MTD submissions—that are frustrating the vast majority of our current revenue-generating base.

What would change it

We would change this decision if the company's primary strategic objective pivots strictly to either capturing upmarket enterprise firms (which would justify prioritising the client portal immediately) or attempting to convert our free 'Just me' users into paying users via premium AI features, rather than retaining and satisfying our current paying subscriber base.

Findings

Paid customers, who generate all revenue, urgently need more reliable bank feeds and strongly reject AI categorisation.

High confidenceBase 296

48.3% of Paid plan users ranked more reliable bank feeds first (mean rank 1.76), while 65.5% ranked AI categorisation last.
AI categorisation is highly desired by Free plan users, but has very low direct revenue potential as an add-on.

High confidenceBase 548

70.2% of Free plan users ranked AI categorisation first. However, only 16.9% of Paid users said they would pay £8/month for it.
Our largest practices unanimously demand a client portal for requesting documents.

Medium confidenceBase 12

100% of practices with 20 or more people ranked the client portal as their top priority, and open text confirmed excessive time spent chasing client paperwork.
System sluggishness and slow page loads are a significant, unprompted pain point degrading the user experience.

High confidenceBase 26

26 respondents complained in open-text about slow loading times, lagging dashboards, and reports timing out.

Caveats

The unweighted aggregate numbers are highly skewed. Paid users (15% of the population) are heavily overrepresented in the survey (54%), meaning any overall averages hide extreme differences between segments.
Because an in-app banner heavily drove responses, daily active users (41.2% in survey vs 18% in population) are overrepresented, potentially inflating the visibility of high-frequency workflow pains like system sluggishness.
The 100% preference for the client portal among the largest practices (20 or more people) is based on a very small sample size of 12 respondents, so it should be used cautiously as a directional signal.

Next steps

decisionCommit engineering resources to stabilise bank feeds and investigate disconnection root causes.
experimentDesign and test a prototype for bulk Making Tax Digital submissions.
researchConduct qualitative interviews with practices of 20+ people on their document collection workflows.
researchInvestigate application performance and diagnose sluggish load times.

Open-text themes it coded

Bank feed unreliability (disconnects, broken links, delayed transactions) 74Time-consuming manual categorization of transactions 52Inefficiency of single Making Tax Digital submissions (need for bulk filing) 25Chasing clients for documents, statements, and paperwork 15System sluggishness and performance issues 26Satisfied with product / No complaints 25

The survey it planned

2 screening questions and 6 questions, as the model wrote them.

S1
Are you involved in making or influencing decisions about the software your practice uses?
One answer
- Yes
- No
Continues if Yes
S2
How many staff members, including yourself, work at your practice?
One answer
- Sole practitioner
- 2-9 staff
- 10 or more staff
Continues if Sole practitioner; 2-9 staff; 10 or more staff
Q1
Which Ledgerline plan is your practice currently on?
One answer
- Free plan (1 user)
- Paid plan (per user per month)
Q2
How heavily does your practice rely on Ledgerline day-to-day?
One answer
- Lightly (we use it occasionally for specific tasks)
- Moderately (we use it regularly every week)
- Heavily (we use it daily; it is core to our operations)
Q3
Thinking about your day-to-day work, which of the following currently slows your practice down the most when using Ledgerline? (Select up to two)
Any that apply
- Managing and entering client records
- Tracking filing deadlines
- Fixing broken or disconnected bank feeds
- Categorising bank transactions
- Chasing clients for missing documents
- Submitting Making Tax Digital (MTD) returns
- Other
Q4
Ledgerline is planning a major update. Please rank the following potential improvements in order of how much they would benefit your practice (1 = Most beneficial, 4 = Least beneficial).
Ranking
- AI categorisation of bank transactions
- More reliable bank feeds (fewer broken connections)
- A client portal for requesting documents
- Bulk Making Tax Digital (MTD) submissions
Q5
If 'AI categorisation of bank transactions' was available as an optional add-on, how likely would your practice be to pay an additional monthly fee for it?
Scale

Scale 1-5: Very unlikely to Very likely
Q6
What is the main reason for your rating regarding paying for an AI categorisation add-on?
Open text

Sample plan

Targeting 550 completes total. Invitation via in-app banner and an email to the admin of every practice account. To prevent duplicates, responses will be restricted to one per practice account ID. To get as close as possible to the requested +/- 5 percentage points precision per practice size, we will set strictly equal quotas of 183 completes each for Sole practitioners, 2-9 staff, and 10+ staff. (Note: A sample of 183 yields approximately +/- 7.2% margin of error at 95% confidence; reaching exactly 5% per subgroup would require exceeding the 550 completion constraint to ~1,152 completes).