Benchmarks / SurveyBench / What Ledgerline's accountancy practices want built next / GPT-6.2 Astra
GPT-6.2 Astra: What Ledgerline's accountancy practices want built next
The decision: which one improvement to build next half-year: AI categorisation of bank transactions, more reliable bank feeds (fewer broken connections), a client portal for requesting documents, or bulk Making Tax Digital submissions. The brief and the data →
- Verdict
- ✓ Sound research
- Research score
- 100 of 100
- Analysis rating
- 1,467
- Head to head, this task
- won 8 of 10
Scorecard
Numbers 11 of 11 right
| Question | Its answer | True | |
|---|---|---|---|
| What percentage of all respondents ranked AI categorisation of bank transactions first? | ✓ | 37.8% | 37.8% |
| Among respondents on a paid plan, what percentage ranked more reliable bank feeds first? | ✓ | 48.3% | 48.3% |
| What is the mean rank of more reliable bank feeds across all respondents (1 = highest priority, 4 = lowest), to two decimal places? | ✓ | 1.99 | 1.99 |
| What percentage of all respondents ranked AI categorisation of bank transactions last (4th)? | ✓ | 42.2% | 42.2% |
| What percentage of all respondents said yes, their practice would pay £8 per user per month for AI categorisation? | ✓ | 27.7% | 27.7% |
| What is the 95% margin of error, in percentage points, on the unweighted share of all respondents who ranked AI categorisation first? | ✓ | 4.06 points | 4.06 points |
| Is the difference in the share ranking AI categorisation first between practices of 2-9 people and practices of 10-19 people statistically significant at the 95% level? (1 for yes, 0 for no) | ✓ | No | No |
| How many of the open-text answers mention bank feeds breaking, disconnecting or arriving late? | ✓ | 74 | 74 |
| How many of the open-text answers from respondents on a paid plan mention chasing clients for documents? | ✓ | 10 | 10 |
| Ledgerline has 1,230 paid practices. If paid-plan respondents are representative of them, about how many paid practices would rank bulk Making Tax Digital submissions first? | ✓ | 325 | 324 |
| What percentage of the 8,200 practice accounts invited completed the survey? | ✓ | 6.68% | 6.7% |
Traps in the data 7 of 7 handled
- ✓AI categorisation has the most first places (38%, against 35% for bank feeds) but also the most last places (42%): it is polarising. More reliable bank feeds have the best mean rank (1.99, against 2.57 for AI) and the fewest last places (7%).
- ✓Free-plan users are 46% of respondents but bring in no revenue; paid practices (15% of accounts, all of the revenue) put bank feeds first (48%) and bulk MTD (26%) ahead of AI (10%). The AI first-choice lead comes from free users (70% of whom rank it first). Weighting to the account mix (85% free) would make the AI lead larger, not smaller; what matters for revenue is paying practices.
- ✓28% said their practice would pay £8 per user per month for AI categorisation (free plan 40%, paid 17%): a stated intention about a hypothetical add-on.
- ✓Every one of the 12 practices with 20 or more people ranked the client portal first.
- ✓69% of completes came from an in-app banner seen only by users who logged in; 52% of banner respondents use Ledgerline on 5 or more days a week against 19% of email respondents and 18% of all accounts. The sample over-represents heavy users and the response rate is 6.7%.
- ✓Practices of 2-9 people rank AI first more often than practices of 10-19 (20% against 14%, bases 172 and 106), but the difference is not significant at 95%.
- ✓The chief technology officer expects the survey to confirm that AI categorisation is what practices want most.
The analysis
Build more reliable bank feeds next half-year, with reducing disruption for existing paying practices as the explicit objective. It is their clear first choice and rarely the last choice in either plan. Scope the work around timely, complete transactions and recovery when client action is unavoidable. Do not present this as an all-account vote for feeds, a proven retention lift or a rejection of AI: if the board prioritises free-account reach instead, the plan-adjusted preference points to AI.
Reconsider if bank-level diagnostics show little disruption Ledgerline can actually prevent or shorten. Switch to AI if a realistically priced pilot demonstrates sustained purchases, enough paid seats and enough net value to outweigh the attainable feed improvement. Prioritise the portal for acquisition only if qualified larger prospects repeatedly treat it as a purchase condition and commit to deals; reconsider bulk MTD if observed peak-period filing time and delivery cost make it a better investment for paying practices.
Findings
-
CTO: No, not on first-place votes alone. AI has the most first choices, but feeds have the broadest support among respondents. Choosing between them depends on whose problem the board intends to solve.
Of 548 respondents, 37.8% ranked AI first versus 34.9% for feeds, a 2.9-point lead. But more ranked AI last (42.2%) than first. Feeds were in the top two for 73.8%, versus 47.3% for AI, and last for only 7.5%. Feeds had the best mean rank (1.99), ahead of AI (2.57), bulk MTD (2.67) and the portal (2.77).
-
For protecting the experience of existing paying practices, choose feeds. Bulk MTD is a credible alternative for deadline-heavy practices, but it is not paid respondents' leading priority.
Of 296 paid respondents, 48.3% ranked feeds first and only 5.1% last. Bulk MTD was first for 26.4% (about 325 of 1,230 paid practices if representative); the portal was first for 15.2% and AI for 10.1%. Fully 65.5% placed AI last. All current revenue comes from paid practices, although the survey did not measure revenue at risk.
-
For serving the largest number of accounts rather than today's revenue base, AI has the stronger case. The unweighted result understates the preferences of free accounts, not paid accounts.
AI was first for 70.2% of 252 free respondents versus 10.1% of 296 paid respondents. Applying the actual 85:15 free-to-paid mix to those rates gives about 61.2% AI-first versus 23.4% feeds-first; even mean rank then favours AI (about 1.91 versus 2.18). This is a plan-mix sensitivity check, not a correction for differences in who responds within each plan.
-
Finance: The survey does not support an add-on revenue forecast. There is stated interest at £8, especially on the free plan, but neither actual purchases nor purchased seats were observed.
Of 548 practices, 27.7% said yes, 49.6% no and 22.6% were unsure. Yes was 40.5% of 252 free respondents and 16.9% of 296 paid respondents. For scale only: if paid respondents were representative, every paid-plan yes converted and each buyer bought one seat, 1,230 × 0.169 × £8 would be about £1,663 per month from existing paid accounts. This is a conditional illustration, not expected revenue or a bound.
-
Sales: Test the portal with larger prospects; do not yet prioritise it to win larger firms. The unanimous result is worth following up, but it is from existing customers and does not extend to the next size band.
All 12 respondents from practices of 20 or more ranked the portal first; four independently described chasing documents. In contrast, only 6.6% of the 106 respondents with 10–19 staff put the portal first, versus 49.1% for feeds and 30.2% for bulk MTD. No prospective or lost buyer was surveyed.
-
Fix the loss of usable bank data, not just the connection status. Distinguish the coding work caused by an outage from the coding work that remains even when feeds run perfectly.
Of 186 Q1 answers, 74 described feed failures, missing or late data, or reconnecting. L051 reports a silent stoppage followed by duplicates of manually entered transactions; L506 double-checks against statements because payments are missing. By contrast, L176 and L277 explicitly say their feeds are fine but hand-coding takes too long. AI would not recover missing transactions, and feed repairs would not remove routine coding.
Caveats
- 548 of 8,200 invited accounts completed (6.68%); 548 of 631 starters completed (86.8%). One UK software decision-maker or influencer per responding practice avoids duplicate practice votes but does not make this a random sample.
- Paid accounts are 54% of respondents versus 15% of accounts. Frequent users are also overrepresented: 41.2% reported use on 5+ days versus 18% in August records; 12% reported less-than-weekly use versus 35% in records. The banner supplied 69% of completes; 52% of banner respondents versus 19% of email respondents reported 5+ days. Plan weighting alone cannot resolve engagement or nonresponse bias.
- The ±4.06-point margin is the conventional unweighted simple-random-sample approximation, 1.96 × sqrt(0.378 × 0.622 / 548), expressed in percentage points. It does not cover selection bias and is not a test of the lead over feeds: the same people ranked both options.
- AI-first was 20.3% in 2–9-person practices (35/172) versus 14.2% in 10–19-person practices (15/106). The roughly 6-point gap is not significant in a two-sided two-proportion test at 5%; that does not establish equal preferences. Size, plan and usage may be related.
- Rank is relative to four supplied choices, not an estimate of hours saved, retention, acquisition or return on build cost. Only 186 of 548 supplied open text; themes overlap and count people, not incidents. Unprompted performance complaints show that not every bottleneck was on the ballot.
- Q3 measured hypothetical practice-level intent at one price. Headcount is not billable seats. Free users' answers also need testing under the actual offer: buying an add-on on the free plan and having to upgrade first are different decisions.
Next steps
- decisionApprove feeds as the single build and establish a baseline by bank and practice plan before setting the release target.
- researchRecruit directly from account records across plan and activity bands, including infrequent users. Observe reconciliation and deadline-week filing in paid practices.
- experimentTest AI with a limited £8-per-seat paid pilot, with clear free-plan eligibility, rather than extrapolating survey yeses.
- researchTest a portal prototype with qualified 20+ staff prospects and review recent won and lost deals.
Open-text themes it coded
Unreliable bank feeds: disconnected or expired links, reauthorisation, missing or late transactions 74Time spent manually coding or categorising transactions 56Repeated one-client-at-a-time MTD submissions 25Waiting for or chasing client documents 15Slow application performance: loading, search, reports or screen updates 27No slowdown or complaint reported 25
The survey it planned
2 screening questions and 6 questions, as the model wrote them.
-
S1
Does your practice provide accountancy or bookkeeping services in the UK and currently use Ledgerline?
One answer- Yes
- No
- Not sure
Continues if Yes
-
S2
What part do you play in choosing software for your practice?
One answer- I make the final decision, alone or with others
- I help assess options or make recommendations, but do not make the final decision
- I am not involved in choosing software
Routing Shown only if S1 is: Yes
Continues if I make the final decision, alone or with others; I help assess options or make recommendations, but do not make the final decision
-
Q1
Thinking about the past 30 days, what, if anything, slowed your practice down most when using Ledgerline? A few words are enough. If nothing comes to mind, you can leave this blank.
Open text -
Q2
In the past 30 days, which of these tasks, if any, took longer than you would have liked when your practice used Ledgerline? Select all that apply.
Any that apply- Keeping client records up to date
- Keeping track of deadlines
- Getting bank-feed connections working again
- Assigning categories to bank transactions
- Getting documents from clients
- Sending Making Tax Digital (MTD) submissions
- Another task
- None of these tasks took longer than I would have liked
- My practice did not use Ledgerline in the past 30 days
- Not sure
-
Q3
If Ledgerline could make just one improvement in the next six months, which would you choose for your practice?
One answer- AI categorisation: suggest categories for bank transactions for your practice to review
- Bank-feed reliability: reduce how often bank-feed connections break
- Client portal: request documents from clients and receive them in one place
- Bulk MTD submissions: send MTD submissions for several clients at once
- None of these
- Not sure
-
Q4
Suppose AI suggestions for bank transaction categories were available as an optional add-on to your current plan. What is the most, in whole pounds per Ledgerline user per month, excluding VAT, that your practice would be willing to pay on top of its current price? Enter 0 if your practice would not pay extra. Leave blank if you cannot estimate.
Number -
Q5
How many people work in your practice, including you, other owners or partners, and part-time staff? Count people whether or not they use Ledgerline.
One answer- 1 person (sole practitioner)
- 2–9 people
- 10 or more people
-
Q6
In the past four weeks, on about how many days did anyone in your practice use Ledgerline? Count a day once even if several people used it.
One answer- 0 days
- 1–3 days
- 4–11 days
- 12 or more days
- Not sure
Sample plan
Email every practice admin and show a banner on login across all 8,200 accounts. Ask for one software decision-maker or adviser per practice; allow the admin to pass the practice-specific invitation to that person. Use the same practice identifier across channels, close access after one eligible completion and suppress further reminders. Resolve known duplicate practice accounts before counting responses. Link plan at invitation from account records rather than asking respondents. Target about 550 eligible practices, aiming for roughly 183 per size band and, across bands, about 200 paid and 350 free to support plan comparisons. Use soft quotas: monitor size by plan, reserve places for underrepresented cells and target reminders where profile data permit. Pilot timing on desktop and mobile; keep the questionnaire under five minutes. Randomise the substantive task options in Q2 and the four improvements in Q3; keep residual answers last and make the last three Q2 answers exclusive. Do not mention the CTO's preference or ticket trends in recruitment. The requested precision cannot be promised: under simple random sampling, absent a material finite-population correction, a proportion near 50% needs about 385 completes per size band (1,155 total) for ±5 percentage points at 95%; 183 gives about ±7.2 points before weighting. Check actual size-band population counts for finite-population corrections. If ±5 is mandatory, renegotiate the target and use a stratified probability design with nonresponse follow-up; voluntary banner/email responses do not support a guaranteed design-based margin of error.