Benchmarks / SurveyBench / What Ledgerline's accountancy practices want built next / GPT-6.2 Sol
GPT-6.2 Sol: What Ledgerline's accountancy practices want built next
The decision: which one improvement to build next half-year: AI categorisation of bank transactions, more reliable bank feeds (fewer broken connections), a client portal for requesting documents, or bulk Making Tax Digital submissions. The brief and the data →
- Verdict
- ✓ Sound research
- Research score
- 100 of 100
- Analysis rating
- 1,354
- Head to head, this task
- won 4 of 11
Scorecard
Numbers 11 of 11 right
| Question | Its answer | True | |
|---|---|---|---|
| What percentage of all respondents ranked AI categorisation of bank transactions first? | ✓ | 37.8% | 37.8% |
| Among respondents on a paid plan, what percentage ranked more reliable bank feeds first? | ✓ | 48.3% | 48.3% |
| What is the mean rank of more reliable bank feeds across all respondents (1 = highest priority, 4 = lowest), to two decimal places? | ✓ | 1.99 | 1.99 |
| What percentage of all respondents ranked AI categorisation of bank transactions last (4th)? | ✓ | 42.2% | 42.2% |
| What percentage of all respondents said yes, their practice would pay £8 per user per month for AI categorisation? | ✓ | 27.7% | 27.7% |
| What is the 95% margin of error, in percentage points, on the unweighted share of all respondents who ranked AI categorisation first? | ✓ | 4.06 points | 4.06 points |
| Is the difference in the share ranking AI categorisation first between practices of 2-9 people and practices of 10-19 people statistically significant at the 95% level? (1 for yes, 0 for no) | ✓ | No | No |
| How many of the open-text answers mention bank feeds breaking, disconnecting or arriving late? | ✓ | 67 | 74 |
| How many of the open-text answers from respondents on a paid plan mention chasing clients for documents? | ✓ | 10 | 10 |
| Ledgerline has 1,230 paid practices. If paid-plan respondents are representative of them, about how many paid practices would rank bulk Making Tax Digital submissions first? | ✓ | 325 | 324 |
| What percentage of the 8,200 practice accounts invited completed the survey? | ✓ | 6.68% | 6.7% |
Traps in the data 7 of 7 handled
- ✓AI categorisation has the most first places (38%, against 35% for bank feeds) but also the most last places (42%): it is polarising. More reliable bank feeds have the best mean rank (1.99, against 2.57 for AI) and the fewest last places (7%).
- ✓Free-plan users are 46% of respondents but bring in no revenue; paid practices (15% of accounts, all of the revenue) put bank feeds first (48%) and bulk MTD (26%) ahead of AI (10%). The AI first-choice lead comes from free users (70% of whom rank it first). Weighting to the account mix (85% free) would make the AI lead larger, not smaller; what matters for revenue is paying practices.
- ✓28% said their practice would pay £8 per user per month for AI categorisation (free plan 40%, paid 17%): a stated intention about a hypothetical add-on.
- ✓Every one of the 12 practices with 20 or more people ranked the client portal first.
- ✓69% of completes came from an in-app banner seen only by users who logged in; 52% of banner respondents use Ledgerline on 5 or more days a week against 19% of email respondents and 18% of all accounts. The sample over-represents heavy users and the response rate is 6.7%.
- ✓Practices of 2-9 people rank AI first more often than practices of 10-19 (20% against 14%, bases 172 and 106), but the difference is not significant at 95%.
- ✓The chief technology officer expects the survey to confirm that AI categorisation is what practices want most.
The analysis
Allocate the next half-year build to bank-feed reliability, with the explicit objective of improving the core workflow for current paying practices. Start by identifying which providers and failure modes cause the most disruption and which Ledgerline can control. Prevent avoidable breaks and late imports; where reauthorisation is unavoidable, make feed status and recovery clearer. Keep AI and the portal in validation rather than committing either to the build.
Reconsider if telemetry shows that bank-feed disruptions are infrequent, have limited customer impact or are mostly outside Ledgerline's control, and a feasible bulk MTD pilot demonstrates greater time savings for paid practices. If the business instead prioritises growth, reconsider for AI if a live £8 test demonstrates enough incremental paid seats, sustained use and contribution to beat the reliability investment, or for the portal if qualified 20+ prospects demonstrate that it materially changes purchase decisions and the segment is large enough to justify the build.
Findings
-
Build more reliable bank feeds next if the goal is to improve the core workflow for current paying customers. Feeds have the broadest support in the achieved sample and are the clear first choice among paid respondents.
Across 548 respondents, feeds were first for 34.9%, in the top two for 73.8%, and last for 7.5%, with the best mean rank of 1.99. Among 296 paid respondents, 48.3% put feeds first, versus 26.4% for bulk MTD, 15.2% for the portal and 10.1% for AI. The comments describe late or missing transactions, client reauthorisation, manual reconciliation and deadline delays.
-
Tell the board that AI won the most first-place votes, not that the survey gives it an overall mandate. AI is a credible option if the objective is growth or monetisation among free accounts.
AI led feeds on first places by 2.9 percentage points, 37.8% to 34.9%, but 42.2% put AI last. AI was first for 70.2% of 252 free respondents and 10.1% of 296 paid respondents; 65.5% of paid respondents put it last. Applying only the population plan mix of 85% free and 15% paid yields about 61.2% AI-first versus 23.4% feeds-first, and a better mean rank for AI than feeds (1.91 versus 2.18). That sensitivity check illustrates why the choice depends on the business objective; it does not correct usage or response bias.
-
Do not book AI add-on revenue from this survey. Treat £8 as a price to test with real purchase decisions.
At £8 per user per month, 27.7% said yes, 49.6% no and 22.6% not sure. Yes was 40.5% among 252 free respondents and 16.9% among 296 paid respondents. Applying those rates to 6,970 free and 1,230 paid accounts gives about 3,031 stated-yes practices. If all were eligible, every stated yes converted and each bought one seat, that would be about £24,200 a month (£291,000 annualised) in gross add-on revenue, of which about £1,660 a month would come from existing paid practices. This is an illustrative scenario, not a forecast or a bound: actual conversion, free-plan eligibility and purchased seats are unknown.
-
Do not prioritise the client portal for winning larger firms on this evidence alone. Treat the 20+ result as a strong signal to investigate in that specific segment.
All 12 respondents at practices with 20 or more staff put the portal first, and four paid respondents in that size group described chasing documents. But only 6.6% of 106 respondents at practices with 10–19 staff put the portal first, versus 49.1% for feeds and 30.2% for bulk MTD. The survey covers existing accounts, not prospects or lost deals.
-
Keep bulk MTD as the next candidate for paid customers if bank-feed improvements prove hard to deliver or less valuable than expected.
Bulk MTD was the second choice among paid respondents by both first-place share (26.4%) and mean rank (2.23). If the 296 paid respondents were representative, about 325 of 1,230 paid practices would rank it first. Comments describe repeated per-client filing eating into deadline days. The count is a conditional extrapolation, not an adoption estimate.
Caveats
- The 6.68% completion rate is 548 divided by 8,200 invited accounts, not a rate among confirmed eligible accounts. The responses are self-selected, not a probability sample. Paid practices are 54% of completes versus 15% of accounts. Five-day users are 41.2% of completes versus 18% in August account records, while less-than-weekly users are 12.0% versus 35%. Survey usage is self-reported and the population comparison is logged usage for a different period.
- The in-app banner generated 69% of completes. Daily use was reported by 52% of banner respondents versus 19% of email respondents. Correcting the plan mix alone cannot correct differences in usage or willingness to respond within each plan.
- The 4.06-point margin is the nominal 95% binomial margin under simple random sampling, without a finite-population correction. It does not quantify selection bias or establish a population confidence interval. The 20.3% versus 14.2% AI-first gap for practices with 2–9 versus 10–19 staff is not significant under a conventional two-sided two-proportion test (n=172 and 106, p≈0.20); that is not evidence that the groups are equivalent.
- Rankings show preferences among four named options, not the size of the benefits, build costs, likely adoption or effects on retention and acquisition. Practice headcount is not the number of Ledgerline seats that would buy an add-on.
- The 186 open-text answers explain how problems arise, not how common they are across all accounts. Themes can overlap. The 67 bank-feed mentions include reauthorisation problems but exclude seven comments that report missing transactions without describing a break or delay; including those gives 74 feed-related comments.
Next steps
- decisionConfirm the half-year objective and commit delivery to bank-feed reliability.
- monitorBaseline feed failures by provider, practice plan and failure mode, then monitor the rollout.
- researchUse account records to compare respondents with the population by plan and usage jointly, and recruit underrepresented practices for follow-up.
- experimentRun a limited live £8-per-seat AI offer with eligible free and paid practices using a pilot that can deliver the promised feature.
- researchInterview 20+ staff customers, qualified prospects and lost deals, and test a portal prototype in their document-chasing workflow.
- researchScope bulk MTD with paid practices that file high volumes of returns.
Open-text themes it coded
Bank feeds disconnect, stop updating, require reauthorisation or deliver transactions late 67Missing feed transactions without an explicit outage, reconnection issue or delay 7Manual transaction categorisation or coding takes time 56One-at-a-time MTD submissions create repetitive work, especially near deadlines 25Waiting for or chasing clients for documents 15Slow pages, search, reports or other application performance problems 26No material slowdown reported 25
The survey it planned
2 screening questions and 6 questions, as the model wrote them.
-
S1
Do you work in a UK accountancy or bookkeeping practice that currently uses Ledgerline?
One answer- Yes
- No
- Not sure
Continues if Yes
-
S2
Which best describes your role in deciding what software your practice uses?
One answer- I make the final decision
- I share responsibility for the decision
- I give advice or recommendations that inform the decision
- I do not make or influence the decision
Continues if I make the final decision; I share responsibility for the decision; I give advice or recommendations that inform the decision
-
Q1
Including yourself and any other working owners, how many people work in your practice? Count part-time staff as people, not as fractions of a full-time role.
One answer- 1 (sole practitioner)
- 2 to 9
- 10 or more
- Not sure
-
Q2
In the past 30 days, on roughly how many days did anyone at your practice use Ledgerline?
One answer- 0 days
- 1 to 4 days
- 5 to 14 days
- 15 to 30 days
- Not sure
-
Q3
Thinking about the last 30 days, what, if anything, slowed your practice down most when using Ledgerline? This question is optional. Please do not include client names or details.
Open text -
Q4
In the last 30 days, which of the following, if any, took more time than you expected when working in Ledgerline? Select all that apply.
Any that apply- Entering or updating client records
- Keeping track of deadlines
- Dealing with broken bank-feed connections
- Assigning categories to bank transactions
- Requesting or chasing documents from clients
- Preparing or sending Making Tax Digital (MTD) submissions
- Something else
- Nothing took more time than expected
- My practice did not use Ledgerline in the last 30 days
- Not sure
-
Q5
If Ledgerline could build only one of these improvements in the next six months, which would you most want for your practice?
One answer- AI categorisation that suggests a category for each bank transaction for you to review
- More reliable bank feeds, so connections break less often
- A client portal where you can request documents and clients can upload them
- Bulk Making Tax Digital (MTD) submissions, so you can send multiple submissions in one batch
- I would not choose any of these improvements
- Not sure
-
Q6
Suppose Ledgerline offered AI categorisation as an optional add-on that suggests a category for each bank transaction for you to review. If it cost £10 per practice per month, excluding VAT, and your current plan price stayed the same, would your practice pay for it?
One answer- Definitely would pay
- Probably would pay
- Probably would not pay
- Definitely would not pay
- Not sure
Sample plan
Email the admin of every Ledgerline practice account and show an in-app banner at login. Ask admins who are not eligible to pass their practice's invitation to an eligible colleague. Both routes should link responses to the same practice identifier and attach the practice's free or paid plan from account records. Where one practice has multiple accounts, use a shared practice identifier where records allow. Keep the first eligible complete per practice across both routes; an incomplete or ineligible attempt does not prevent someone else from responding. Target about 550 eligible completes, with soft targets of about 183 sole practitioners, 183 practices with 2 to 9 staff and 184 with 10 or more staff. Aim for about 220 paid and 330 free completes overall, with both plans represented in each size band where feasible. Monitor size and plan during fieldwork and prioritise reminders to underrepresented groups where account data allow. Randomise the first seven Q4 options and the four improvement options in Q5; keep the remaining options at the end. Make the last three Q4 options exclusive. The six short post-screener questions are designed to fit within five minutes. The precision requirement cannot be promised at 550 completes: at the most conservative proportion of 50%, about 183 responses per size band give a nominal 95% margin of about ±7.2 percentage points under simple random sampling before any finite-population correction. Achieving ±5 points under those assumptions requires about 385 per band, or 1,155 total. Check verified population counts by size before fieldwork to see whether a finite-population correction would lower those targets. If ±5 points remains mandatory, agree a larger target or a revised sampling design before launch; a voluntary email and banner survey cannot guarantee population-level precision even at a larger sample size.