Benchmarks / SurveyBench / What Ledgerline's accountancy practices want built next / GLM 5.3 FlashX Legal
GLM 5.3 FlashX Legal: What Ledgerline's accountancy practices want built next
The decision: which one improvement to build next half-year: AI categorisation of bank transactions, more reliable bank feeds (fewer broken connections), a client portal for requesting documents, or bulk Making Tax Digital submissions. The brief and the data →
- Verdict
- ✓ Sound research
- Research score
- 85 of 100
- Analysis rating
- 997
- Head to head, this task
- won 10 of 14
Scorecard
Numbers 11 of 11 right
| Question | Its answer | True | |
|---|---|---|---|
| What percentage of all respondents ranked AI categorisation of bank transactions first? | ✓ | 37.8% | 37.8% |
| Among respondents on a paid plan, what percentage ranked more reliable bank feeds first? | ✓ | 48.3% | 48.3% |
| What is the mean rank of more reliable bank feeds across all respondents (1 = highest priority, 4 = lowest), to two decimal places? | ✓ | 1.99 | 1.99 |
| What percentage of all respondents ranked AI categorisation of bank transactions last (4th)? | ✓ | 42.2% | 42.2% |
| What percentage of all respondents said yes, their practice would pay £8 per user per month for AI categorisation? | ✓ | 27.7% | 27.7% |
| What is the 95% margin of error, in percentage points, on the unweighted share of all respondents who ranked AI categorisation first? | ✓ | 4.1 points | 4.06 points |
| Is the difference in the share ranking AI categorisation first between practices of 2-9 people and practices of 10-19 people statistically significant at the 95% level? (1 for yes, 0 for no) | ✓ | No | No |
| How many of the open-text answers mention bank feeds breaking, disconnecting or arriving late? | ✓ | 74 | 74 |
| How many of the open-text answers from respondents on a paid plan mention chasing clients for documents? | ✓ | 10 | 10 |
| Ledgerline has 1,230 paid practices. If paid-plan respondents are representative of them, about how many paid practices would rank bulk Making Tax Digital submissions first? | ✓ | 325 | 324 |
| What percentage of the 8,200 practice accounts invited completed the survey? | ✓ | 6.7% | 6.7% |
Traps in the data 7 of 7 handled
- ✓AI categorisation has the most first places (38%, against 35% for bank feeds) but also the most last places (42%): it is polarising. More reliable bank feeds have the best mean rank (1.99, against 2.57 for AI) and the fewest last places (7%).
- ✓Free-plan users are 46% of respondents but bring in no revenue; paid practices (15% of accounts, all of the revenue) put bank feeds first (48%) and bulk MTD (26%) ahead of AI (10%). The AI first-choice lead comes from free users (70% of whom rank it first). Weighting to the account mix (85% free) would make the AI lead larger, not smaller; what matters for revenue is paying practices.
- ✓28% said their practice would pay £8 per user per month for AI categorisation (free plan 40%, paid 17%): a stated intention about a hypothetical add-on.
- ✓Every one of the 12 practices with 20 or more people ranked the client portal first.
- ✓69% of completes came from an in-app banner seen only by users who logged in; 52% of banner respondents use Ledgerline on 5 or more days a week against 19% of email respondents and 18% of all accounts. The sample over-represents heavy users and the response rate is 6.7%.
- ✓Practices of 2-9 people rank AI first more often than practices of 10-19 (20% against 14%, bases 172 and 106), but the difference is not significant at 95%.
- ✓The chief technology officer expects the survey to confirm that AI categorisation is what practices want most.
The analysis
Build more reliable bank feeds as the one improvement for the next half-year. That is the answer to the decision. The CTO should tell the board the opposite of what was hoped: the survey does not back AI categorisation as the next build. AI's 37.8% first-place share is statistically indistinguishable from bank feeds (34.9%), is produced by free-plan sole traders (70.2% of that column), and collapses among paying customers (10.1% first, 65.5% last). Bank feeds have the best mean rank overall (1.99), are first among paid plans (48.3%), first among 2-9 and 10-19 person practices, least likely to be ranked last (7.5%), and are the most common unprompted complaint (74 of 186 open-texts). They are also a top-two priority for both free and paid segments, which no other option is. The finance director should not take an £8/user/month revenue number from this survey: only 16.9% of paid-plan respondents would pay (about 50 of 296; about 208 of 1,230 paid practices), and the question was hypothetical. The head of sales should not re-prioritise the client portal: the 20+ result is 12 people, and 10-19 person practices rank the portal last among the four (6.6% first). If feed reliability is already funded as maintenance rather than as a feature, bulk MTD is the strongest paid-plan alternative (26.4% first, mean rank 2.23, roughly 325 of 1,230 paid practices). Do not present unweighted headline tables as describing the customer base without the over-sample caveats. Confirm with engineering how much of the feed problem is in-product versus bank-side before scoping the work.
Findings
-
Do not tell the board the survey backs building AI categorisation next. Its first-place lead over bank feeds is inside the margin of error, and it is the last priority among the paying customer base.
AI ranked first by 37.8% of 548 respondents versus 34.9% for bank feeds (gap 2.9 points; 95% MoE on the AI share is ±4.1 points). Mean rank across all respondents favours bank feeds (1.99) over AI (2.57), portal (2.77) and bulk MTD (2.67). On paid plans AI is first for only 10.1% and last for 65.5% (mean rank 3.35, n=296); bank feeds are first for 48.3% of paid plans (mean rank 1.76).
-
The market is split by segment, and bank feeds are the only option that is a top-two priority for both free and paid practices. AI is a free-plan/solo-practitioner preference; bulk MTD is a paid-plan preference.
Free plan mean ranks (n=252): AI 1.66, feeds 2.25, portal 2.90, MTD 3.19. Paid plan mean ranks (n=296): feeds 1.76, MTD 2.23, portal 2.66, AI 3.35. Just-me practices (47.1% of the sample, n=258) put AI first at 60.9%; 2-9 (n=172) and 10-19 (n=106) put bank feeds first (43.0% and 49.1%).
-
The finance director should not book £8/user/month AI revenue from this survey. Stated willingness is weak where it matters most, and the question was hypothetical with no trade-off against money or other features.
27.7% of all 548 said yes, but only 16.9% of paid-plan respondents (about 50 of 296) and 49.6% said no overall. Applied to 1,230 paid practices that is roughly 208 practices, before seats, before hypothetical-bias and before the 25.7% of paid who are 'not sure'. Free-plan respondents said yes more often (40.5%, n=252) but they already pay nothing, so that intention is the least reliable.
-
The unweighted tables must not be quoted to the board as if they describe the 8,200-account base. The sample over-collects paying and daily users, which flatters paid-plan views in the headline totals.
Sample: 54.0% paid (n=296) versus 15% of the 8,200 accounts; 41.2% use Ledgerline 5+ days (n=548) versus 18% of accounts in August 2026. 69% of completes came from the in-app banner, where 52% were 5+ day users, versus 19% of the smaller email arm. Response rate was 6.7% (548 of 8,200).
-
The head of sales does not have evidence that the client portal is the priority for winning larger firms. The 100% first-place result among 20+ practices rests on 12 people; mid-size practices want bank feeds.
20+ people: portal first 100% but base is 12 (0.0% for AI, feeds and MTD in that column). 10-19 people (n=106): bank feeds first 49.1%, bulk MTD 30.2%, AI 14.2%, portal 6.6%. Among paid plans the portal is third on mean rank (2.66) behind feeds (1.76) and bulk MTD (2.23).
-
Bulk MTD submissions are a real, secondary pain among paying and mid-size practices and are the strongest alternative if feed work is already funded as reliability engineering.
Paid-plan respondents rank MTD first at 26.4% (n=296); 10-19 person practices at 30.2% (n=106). Paid mean rank 2.23 versus portal 2.66. About 325 of 1,230 paid practices would rank it first if the paid sample is representative (1,230 × 26.4% = 324.7).
-
Unprompted complaints converge on feed failure, not on the absence of AI. Manual coding is the second theme and is often the consequence of a broken or late feed, which supports fixing feeds first even for users who say they want AI.
Of 186 open-text answers, 74 mention feeds breaking, disconnecting or arriving late; 57 mention manual coding/categorisation; 26 slow app performance; 25 one-by-one MTD; 15 chasing documents; 25 say nothing slows them down. Several coding complaints are explicitly caused by feed outages (e.g. L004, L075, L126, L235, L378, L410).
-
A 6.1-point gap in AI-first share between 2-9 and 10-19 person practices is not statistically significant, so it should not be used to claim that mid-size firms are warming to AI.
AI first: 20.3% of 2-9 (n=172) versus 14.2% of 10-19 (n=106); two-proportion z ≈ 1.3, p > 0.05.
Caveats
- The survey is not representative of the 8,200 accounts. It over-represents paid plans (54% of respondents vs 15% of accounts) and heavy users (41.2% on 5+ days vs 18% of accounts). Headline unweighted totals therefore overstate the views of paying, daily users. Weighting to the population would raise AI's first-place share (free users are more pro-AI) but lower average willingness to pay, because free users already pay nothing.
- Only 6.7% of invited accounts completed (548/8,200), and 69% of those came from an in-app banner that heavy users were more likely to click (52% vs 19% on 5+ days). Non-response is likely related to engagement and satisfaction.
- The 20+ person column has a base of 12. The 100% who ranked the client portal first is a signal, not evidence. Do not put it in a board paper as a basis for reordering the roadmap.
- Q3 is a yes/no hypothetical at a fixed £8. It is prone to overstatement, did not force a trade-off against other features or prices, and does not measure seats per practice, so it cannot produce a credible revenue forecast.
- Open-text is 186 of 548 completes (34%). People with a gripe are over-represented relative to the silent majority; 25 people still said nothing slows them down.
- Forced ranking does not capture indifference or whether any option is worth building. A last-place rank for AI among paid users does not mean they would refuse a well-priced add-on; it means it was not their priority for the next half-year of product investment.
- 'More reliable bank feeds' may be partly outside Ledgerline's control (open-text names Santander, Tide, Barclays, Monzo, Starling and 90-day reauthorisation). Engineering must confirm how much is fixable in-product (detection, re-auth, deduplication) versus third-party.
- Fieldwork was 7-20 September 2026, near a VAT quarter point. That may have primed MTD and feed complaints. Q6 matched account records for every respondent, which is a strength; only the first response per practice was kept, which is also a strength.
Open-text themes it coded
Bank feeds breaking, disconnecting or arriving late 74Manual coding of bank transactions 57MTD/VAT submissions one client at a time 25Chasing clients for documents 15Slow app performance 26No complaints 25
The survey it planned
3 screening questions and 8 questions, as the model wrote them.
-
S1
Which best describes the practice or organisation you work in?
One answer- A UK accountancy practice
- A UK bookkeeping practice
- A UK practice that offers both accountancy and bookkeeping
- An accountancy or bookkeeping practice based outside the UK
- I do not work at an accountancy or bookkeeping practice
- I work at Ledgerline
Continues if A UK accountancy practice; A UK bookkeeping practice; A UK practice that offers both accountancy and bookkeeping
-
S2
Does the practice you work at currently use Ledgerline?
One answer- Yes
- No
Continues if Yes
-
S3
Which best describes your involvement in software decisions at your practice? This includes decisions such as whether to stay on the free plan, upgrade, add users, or change software.
One answer- I decide, or I am one of the main decision-makers
- I have some influence over those decisions
- I have little or no influence over those decisions
Continues if I decide, or I am one of the main decision-makers; I have some influence over those decisions
-
Q1
Including yourself, how many people work in your practice? Count owners, employees, and anyone who regularly does client work for the practice.
One answer- Just me (sole practitioner)
- 2 to 9 people
- 10 or more people
-
Q2
In a typical week, how often is Ledgerline used in your practice (by you or anyone else)?
One answer- Every day
- Most days
- A few times a week
- About once a week
- Less than once a week
-
Q3
Which of these does your practice use in Ledgerline? Select all that apply.
Any that apply- Client records
- Deadlines and key dates
- Bank feeds
- Making Tax Digital submissions
- None of these
-
Q4
Thinking about the last three months, which of these have slowed your practice down when using Ledgerline? Select all that apply.
Any that apply- Bank feeds disconnecting, failing to sync, or needing to be reconnected
- Manually categorising bank transactions, or correcting categories
- Chasing clients for documents, records, or information
- Submitting Making Tax Digital returns one client or one return at a time
- Keeping client records up to date
- Managing deadlines
- Finding features or learning how to do things in Ledgerline
- Speed, crashes, or other reliability problems (apart from bank feeds)
- Attention check: please select this option
- Something else
- Nothing has particularly slowed us down
-
Q5
Ledgerline can build only one major improvement in the next six months. Which one would most help your practice? Select one.
One answer- AI categorisation of bank transactions
- More reliable bank feeds (fewer broken connections)
- A client portal for requesting documents
- Bulk Making Tax Digital submissions
-
Q6
Put the same four improvements in order of value to your practice, where 1 is the most valuable and 4 is the least valuable.
Ranking- AI categorisation of bank transactions
- More reliable bank feeds (fewer broken connections)
- A client portal for requesting documents
- Bulk Making Tax Digital submissions
-
Q7
If Ledgerline offered AI categorisation of bank transactions as an optional paid add-on, which statement best describes your practice? Select one.
One answer- We would pay extra for it as an add-on
- We would only use it if it were included in our current plan at no extra cost
- We would not be interested in AI categorisation
- Not sure
-
Q8
Is there anything else that slows your practice down in Ledgerline, or anything else you would most like us to improve? Write as much or as little as you like. This question is optional.
Open text
Sample plan
Invite the admin of every one of the 8,200 practice accounts by email, and show an in-app banner to every user who logs in during fieldwork. Link each response to the practice account ID and keep one response per practice: if several people respond, keep the admin's response if they finished, otherwise the first person who passed the screener. Before fieldwork, pull plan (free/paid), last login, and practice size if held in CRM, to set quotas and target reminders; do not ask plan in the survey because it is already known from the account. Soft-launch the first 40 completes to check attention-fail rate, speeders, and comprehension, then go live. Fieldwork: 10 working days. Reminder to all non-responders on day 4; a second reminder on day 7 targeted at paid-plan admins and any practice-size band that is behind quota. Target about 550 completes. Use monitoring targets rather than hard stops: at least 100 completes from practices with 10 or more staff, at least 200 from 2-9 staff, the remainder sole practitioners; and at least 150 from paid plans (oversampling the 15% paid base). Weight overall figures back to the known 85/15 free/paid mix (and to the CRM size mix if available). The in-app banner will skew toward active users; the admin email will include less active practices, which is the range needed for O3. Frame the invitation as a vote on the next major build; a small prize draw is optional if response is slow. PRECISION CONFLICT TO FLAG TO PRODUCT: plus or minus 5 percentage points at 95% confidence in each of the three practice-size bands needs about 385 completes per band (1.96^2 x 0.25 / 0.05^2), or roughly 1,160 completes in total. 550 completes cannot meet that requirement. At 550 with equal bands (about 183 each) the margin of error is about plus or minus 7.2 points; a 10+ band of n=100 is about plus or minus 9.8 points. If product must have plus or minus 5 points per size band, either raise the target to at least 1,160 completes or accept wider intervals, especially for 10+ staff. I would not launch on the current brief without product choosing which of those two it wants.