Benchmarks / SurveyBench / What Ledgerline's accountancy practices want built next / Kimi K3.1 Pro
Kimi K3.1 Pro: What Ledgerline's accountancy practices want built next
The decision: which one improvement to build next half-year: AI categorisation of bank transactions, more reliable bank feeds (fewer broken connections), a client portal for requesting documents, or bulk Making Tax Digital submissions. The brief and the data →
- Verdict
- ✗ Not sound
- Research score
- 77 of 100
- Analysis rating
- 608
- Head to head, this task
- won 11 of 14
Why it is not sound
The analysis
- Made a recommendation that does not follow from the data
Scorecard
Numbers 11 of 11 right
| Question | Its answer | True | |
|---|---|---|---|
| What percentage of all respondents ranked AI categorisation of bank transactions first? | ✓ | 37.8% | 37.8% |
| Among respondents on a paid plan, what percentage ranked more reliable bank feeds first? | ✓ | 48.3% | 48.3% |
| What is the mean rank of more reliable bank feeds across all respondents (1 = highest priority, 4 = lowest), to two decimal places? | ✓ | 1.99 | 1.99 |
| What percentage of all respondents ranked AI categorisation of bank transactions last (4th)? | ✓ | 42.2% | 42.2% |
| What percentage of all respondents said yes, their practice would pay £8 per user per month for AI categorisation? | ✓ | 27.7% | 27.7% |
| What is the 95% margin of error, in percentage points, on the unweighted share of all respondents who ranked AI categorisation first? | ✓ | 4.1 points | 4.06 points |
| Is the difference in the share ranking AI categorisation first between practices of 2-9 people and practices of 10-19 people statistically significant at the 95% level? (1 for yes, 0 for no) | ✓ | No | No |
| How many of the open-text answers mention bank feeds breaking, disconnecting or arriving late? | ✓ | 74 | 74 |
| How many of the open-text answers from respondents on a paid plan mention chasing clients for documents? | ✓ | 10 | 10 |
| Ledgerline has 1,230 paid practices. If paid-plan respondents are representative of them, about how many paid practices would rank bulk Making Tax Digital submissions first? | ✓ | 325 | 324 |
| What percentage of the 8,200 practice accounts invited completed the survey? | ✓ | 6.7% | 6.7% |
Traps in the data 7 of 7 handled
- ✓AI categorisation has the most first places (38%, against 35% for bank feeds) but also the most last places (42%): it is polarising. More reliable bank feeds have the best mean rank (1.99, against 2.57 for AI) and the fewest last places (7%).
- ✓Free-plan users are 46% of respondents but bring in no revenue; paid practices (15% of accounts, all of the revenue) put bank feeds first (48%) and bulk MTD (26%) ahead of AI (10%). The AI first-choice lead comes from free users (70% of whom rank it first). Weighting to the account mix (85% free) would make the AI lead larger, not smaller; what matters for revenue is paying practices.
- ✓28% said their practice would pay £8 per user per month for AI categorisation (free plan 40%, paid 17%): a stated intention about a hypothetical add-on.
- ✓Every one of the 12 practices with 20 or more people ranked the client portal first.
- ✓69% of completes came from an in-app banner seen only by users who logged in; 52% of banner respondents use Ledgerline on 5 or more days a week against 19% of email respondents and 18% of all accounts. The sample over-represents heavy users and the response rate is 6.7%.
- ✓Practices of 2-9 people rank AI first more often than practices of 10-19 (20% against 14%, bases 172 and 106), but the difference is not significant at 95%.
- ✓The chief technology officer expects the survey to confirm that AI categorisation is what practices want most.
The analysis
Findings
-
The CTO should not tell the board that the survey, by itself, backs building AI next: AI leads the narrow first-preference headline in the unweighted sample, but bank-feed reliability has much broader support across the full rankings.
Across all 548 respondents, AI received 37.8% of first places versus 34.9% for bank feeds, a lead of only 2.9 percentage points. However, 42.2% ranked AI last and its mean rank was 2.57; only 7.5% ranked bank feeds last, their mean rank was 1.99, and 73.8% put them first or second versus 47.3% for AI. The simple 95% sampling margin on AI's first-place share is approximately ±4.1 points. A paired test of the AI-versus-bank difference requires respondent-level data and is not supplied.
-
For a decision intended to protect and grow value for the practices that currently generate revenue, reliable bank feeds are the best-supported investment; AI demand is concentrated among free-plan and solo respondents and is weak among paid respondents.
Among 296 paid-plan respondents, bank feeds were first for 48.3%, with mean rank 1.76 and only 5.1% ranking them last. Bulk MTD followed at 26.4% first and mean rank 2.23, then the portal at 15.2% and 2.66. AI was first for only 10.1%, last for 65.5%, and had the worst mean rank at 3.35. By contrast, among 252 free-plan respondents, AI was first for 70.2% with mean rank 1.66; among 258 solo respondents it was first for 60.9%. Plan and practice size cannot be disentangled with the supplied tables.
-
The open-text evidence makes feed reliability an upstream operational problem, not merely a stated preference: AI categorisation would not itself repair missing, late or duplicated data, expired connections or repeated client re-authorisation.
Of 186 open-text answers, 74 (39.8%) mention feed disconnection, re-authorisation, or late or missing bank data, while 56 (30.1%) mention manual coding or categorisation. Twenty-five answers mention both feeds and manual coding, commonly describing a feed failure followed by a manual backlog. Performance was mentioned in 26 answers, one-by-one MTD filing in 25, document chasing in 15, and 25 reported no slowdown. These are self-selected qualitative mentions, not population prevalence estimates.
-
The finance director cannot be given a defensible AI add-on revenue forecast from this survey; only an explicitly assumption-led illustration is possible.
Overall, 27.7% said yes, 49.6% no and 22.6% not sure. Among paid respondents, only 16.9% said yes, equivalent to 50 of 296; 57.4% said no and 25.7% were unsure. Among free respondents, 40.5% said yes, but that is neither an actual purchase nor a free-to-paid conversion rate. Scaling the paid yes share to 1,230 paid practices gives approximately 208 practices. If every one bought exactly one retained add-on seat, the arithmetic would be about £1.66k MRR or £20k ARR. That is not a forecast: actual revenue must be multiplied by the unknown number of adopting seats and adjusted for observed conversion and retention. Q5 workforce bands are not Ledgerline seat counts.
-
Bulk MTD is a credible runner-up for paid customers and should be sized on effort and operational value rather than discarded.
Among paid respondents, 26.4% ranked bulk MTD first and its mean rank was 2.23, ahead of both the portal and AI on those measures. If paid respondents were representative, 26.4% of 1,230 paid practices would be approximately 325 practices ranking it first. Twenty-five of the 186 open-text answers describe one-by-one MTD filing, including 22 from paid respondents. Among 106 respondents from practices of 10–19 people, 30.2% ranked bulk MTD first.
-
The client portal is a strong hypothesis for the very largest respondents, but it is not yet evidence that portal should be the company-wide priority for winning larger firms.
All 12 respondents from practices with 20 or more people ranked the portal first. All four such respondents who supplied open text—L215, L509, L540 and L544, all on paid plans—described chasing documents. Nevertheless, the base is only 12. Among the much larger 10–19-person segment, base 106, just 6.6% ranked the portal first, versus 49.1% for bank feeds and 30.2% for bulk MTD. The survey also covers current practices, not larger prospects, lost deals or buying committees.
-
The all-account headline is highly sensitive to weighting, so decision-makers must distinguish an all-account engagement objective from a paid-revenue objective.
The sample is 46.0% free and 54.0% paid, whereas the account population is 85% free and 15% paid. It also has 41.2% using Ledgerline on five or more days, versus 18% in the population. As an arithmetic sensitivity check only, weighting solely on plan would produce approximately 61.2% AI-first (0.85 × 70.2% + 0.15 × 10.1%) and 23.4% bank-first. That is not a validated population estimate because usage and channel self-selection remain, but it demonstrates why neither the unweighted headline nor an unqualified board claim should be treated as population truth.
Open-text themes it coded
Bank-feed reliability and recovery failures: disconnections, re-authorisation, or late, missing or duplicated bank data 74Manual bank-transaction coding or categorisation consumes substantial time 56
The survey it planned
2 screening questions and 8 questions, as the model wrote them.
-
S1
Which of the following best describes the practice you are responding about?
One answer- A UK accountancy or bookkeeping practice that currently uses Ledgerline (free or paid plan)
- A UK accountancy or bookkeeping practice that does not currently use Ledgerline
- An accountancy or bookkeeping practice based outside the UK, whether or not it uses Ledgerline
- I am not responding on behalf of an accountancy or bookkeeping practice
Continues if A UK accountancy or bookkeeping practice that currently uses Ledgerline (free or paid plan)
-
S2
How are you involved in decisions about the software this practice uses?
One answer- I make the final decisions about software
- I share in making decisions or influence them by evaluating or recommending software
- I do not make or influence software decisions
Continues if I make the final decisions about software; I share in making decisions or influence them by evaluating or recommending software
-
Q1
Thinking about your practice's work in Ledgerline over the last 3 months, what has slowed the practice down the most? Describe the task or problem, with a recent example if you can. If nothing has slowed it down, write Nothing; if you do not know, write Don't know.
Open text -
Q2
In the last 3 months, which, if any, of the following have slowed down your practice's work involving Ledgerline? Select all that apply, leaving anything that does not apply to your practice unselected.
Any that apply- Manually categorising, reviewing or correcting bank transactions
- Bank feeds disconnecting, failing to import transactions, or needing reconnection
- Requesting, chasing or receiving documents from clients
- Preparing or making Making Tax Digital (MTD) submissions for clients one at a time
- Keeping client records up to date
- Tracking deadlines or outstanding work
- Finding information or moving between clients in Ledgerline
- Ledgerline loading slowly, crashing or being unavailable
- Something else not listed above
- Nothing has slowed my practice down in work involving Ledgerline in the last 3 months (select this on its own)
- Don't know (select this on its own)
-
Q3
Suppose Ledgerline could deliver only one of these four improvements in the next six months. Rank all four from 1 (most valuable to your practice) to 4 (least valuable), using each rank once. Rank them even if some are not relevant to your practice.
Ranking- AI categorisation of bank transactions — Ledgerline suggests categories for you to review or correct before accepting
- More reliable bank feeds — fewer broken connections, failed imports and reconnection tasks
- A client portal — request documents from clients and receive them in Ledgerline
- Bulk Making Tax Digital (MTD) submissions — complete submissions for multiple clients in one workflow
-
Q4
AI categorisation would suggest categories for bank transactions, for you to review or correct before accepting. Imagine it were available as an optional add-on, charged per user per month in addition to your current plan. Based on what you know today, which statement best describes your practice?
One answer- I would expect my practice to pay for it as an add-on
- My practice might pay for it, depending on the price and how well it works
- My practice would use it only if it were included at no extra cost
- My practice would not use it, even at no extra cost
- I don't know enough to say
-
Q5
What is the most your practice would be willing to pay for this AI categorisation add-on for each user per month, excluding VAT? Enter your best estimate in pounds, using numbers only. Enter 0 if you would not pay anything. If you do not know, leave the field blank.
NumberRouting Shown only if Q4 is: I would expect my practice to pay for it as an add-on; My practice might pay for it, depending on the price and how well it works
-
Q6
How many people work in your practice, including you and any regular contractors? Select one.
One answer- 1 — sole practitioner
- 2–9 staff
- 10 or more staff
-
Q7
In a typical week, roughly how many hours in total do all people in your practice spend using Ledgerline? Include everyone in the practice and enter your best estimate in hours. Enter 0 if no one uses it in a typical week.
Number -
Q8
Which parts of Ledgerline has your practice used in the last 3 months? Select all that apply.
Any that apply- Client records
- Deadline tracking
- Bank feeds
- Making Tax Digital (MTD) submissions
- None of these (select this on its own)
- Don't know (select this on its own)
Sample plan
Use the practice account as the sampling unit. Email the admin of all 8,200 accounts and show the in-app banner to users in practices without an eligible complete, with the account ID carried through both routes. Screen everyone, append free/paid plan from billing and 90-day usage telemetry where available, and use Q6 to monitor size quotas. De-duplicate before analysis, preferring a final decision-maker over an influencer and otherwise keeping the first eligible complete. Randomise Q2's substantive options and Q3's four improvements, keeping none/don't-know options last, and soft-launch to verify typical completion within 5 minutes and that ranking works on mobile. Monitor nonresponse and paid-practice representation; oversample paid practices with targeted reminders if needed for O3. Weight any all-practice estimates to known account-frame distributions, at minimum plan and also size if available; if size weights cannot be constructed, label overall figures as sample estimates rather than population estimates. The ~550-complete target cannot guarantee the requested precision. For a proportion near 50%, ±5 percentage points at 95% confidence requires approximately 385 unweighted simple-random-sample responses in each size band. Recommended quotas are 385 sole practitioners, 385 practices with 2–9 staff and 385 practices with 10 or more staff, totalling 1,155 completes, before design effects; increase these if weighting reduces effective sample size, and recalculate small bands using their actual populations and the finite-population correction. If about 550 is a hard cap, use provisional equal quotas of 184, 183 and 183 respectively: under the same assumptions and ignoring finite-population corrections, this gives only about ±7.2–7.3 points per band, potentially wider after weighting. That version must be labelled directional and not described as meeting ±5 points. These calculations cover sampling error only, not self-selection or nonresponse bias, so the precision-versus-budget trade-off requires explicit sign-off before launch.