Sign inTest your prompt

Benchmarks / SurveyBench / What Ledgerline's accountancy practices want built next / GPT-6 Luna

Measured by Spring Prompt

GPT-6 Luna: What Ledgerline's accountancy practices want built next

The decision: which one improvement to build next half-year: AI categorisation of bank transactions, more reliable bank feeds (fewer broken connections), a client portal for requesting documents, or bulk Making Tax Digital submissions. The brief and the data →

Verdict
✓ Sound research
Research score
100 of 100
Analysis rating
1,102
Head to head, this task
won 7 of 11

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of all respondents ranked AI categorisation of bank transactions first?✓37.8%37.8%
Among respondents on a paid plan, what percentage ranked more reliable bank feeds first?✓48.3%48.3%
What is the mean rank of more reliable bank feeds across all respondents (1 = highest priority, 4 = lowest), to two decimal places?✓1.991.99
What percentage of all respondents ranked AI categorisation of bank transactions last (4th)?✓42.2%42.2%
What percentage of all respondents said yes, their practice would pay £8 per user per month for AI categorisation?✓27.7%27.7%
What is the 95% margin of error, in percentage points, on the unweighted share of all respondents who ranked AI categorisation first?✓4.1 points4.06 points
Is the difference in the share ranking AI categorisation first between practices of 2-9 people and practices of 10-19 people statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
How many of the open-text answers mention bank feeds breaking, disconnecting or arriving late?✓7374
How many of the open-text answers from respondents on a paid plan mention chasing clients for documents?✓1010
Ledgerline has 1,230 paid practices. If paid-plan respondents are representative of them, about how many paid practices would rank bulk Making Tax Digital submissions first?✓325324
What percentage of the 8,200 practice accounts invited completed the survey?✓6.7%6.7%

Traps in the data 7 of 7 handled

  • ✓AI categorisation has the most first places (38%, against 35% for bank feeds) but also the most last places (42%): it is polarising. More reliable bank feeds have the best mean rank (1.99, against 2.57 for AI) and the fewest last places (7%).
  • ✓Free-plan users are 46% of respondents but bring in no revenue; paid practices (15% of accounts, all of the revenue) put bank feeds first (48%) and bulk MTD (26%) ahead of AI (10%). The AI first-choice lead comes from free users (70% of whom rank it first). Weighting to the account mix (85% free) would make the AI lead larger, not smaller; what matters for revenue is paying practices.
  • ✓28% said their practice would pay £8 per user per month for AI categorisation (free plan 40%, paid 17%): a stated intention about a hypothetical add-on.
  • ✓Every one of the 12 practices with 20 or more people ranked the client portal first.
  • ✓69% of completes came from an in-app banner seen only by users who logged in; 52% of banner respondents use Ledgerline on 5 or more days a week against 19% of email respondents and 18% of all accounts. The sample over-represents heavy users and the response rate is 6.7%.
  • ✓Practices of 2-9 people rank AI first more often than practices of 10-19 (20% against 14%, bases 172 and 106), but the difference is not significant at 95%.
  • ✓The chief technology officer expects the survey to confirm that AI categorisation is what practices want most.

The analysis

Recommendation

Prioritise improving bank-feed reliability for the next half-year, with a short technical discovery phase to identify which failure modes Ledgerline can materially reduce. This is the most defensible choice for protecting the paid customer base: paid respondents favour it, it is consistently ranked highly, and feed problems recur in open text. Do not choose AI solely because it has the largest overall first-place share: that result is concentrated among free-plan and solo respondents, while paid respondents largely rank AI last and show limited stated willingness to pay. Treat bulk MTD as the strongest follow-on candidate for paid practices, and validate portal demand with larger firms before committing to it.

What would change it

I would reconsider AI if a representative paid-customer test showed meaningful paid-seat conversion at £8 and a measured reduction in transaction-processing time that outweighs the opportunity cost of reliability work. I would reconsider the portal if a larger, properly sampled set of larger practices demonstrated that it materially affects selection or retention. I would reconsider bulk MTD if usage data showed high eligible filing volumes and a pilot demonstrated substantially greater time savings or willingness to pay than the feed-reliability option.

Findings

  1. Build more reliable bank feeds next, subject to confirming the main failure modes are addressable. AI has the largest single first-place share overall, but the rankings do not show a broad or consistent preference for it; bank-feed reliability is the stronger choice among paid practices, which generate all Ledgerline revenue.

    Medium confidenceBase 548

    Overall, AI ranked first for 37.8% versus 34.9% for reliable feeds, but reliable feeds had a better mean rank (1.99 versus 2.57) and was ranked last by only 7.5% versus 42.2% for AI. Among paid-plan respondents, 48.3% ranked reliable feeds first and its mean rank was 1.76. Bases: 548 overall; 296 paid.

  2. The CTO can say AI received the most first-place votes, but should not present that as evidence it is the best next build. Preference is sharply divided by plan, and most paid respondents put AI last.

    Medium confidenceBase 548

    AI ranked first for 70.2% of free-plan respondents but 10.1% of paid-plan respondents; it ranked last for 65.5% of paid respondents. Free-plan respondents were 252 and paid-plan respondents were 296. Overall, AI's first-place share was 37.8%, only 2.9 percentage points above reliable feeds.

  3. The survey cannot support a reliable add-on revenue forecast for AI at £8 per user per month. A simple, conditional scenario is about £1,664 per month if 16.9% of the 1,230 paid practices bought exactly one seat; that is not a forecast, because seat counts, actual conversion, and representativeness are unknown.

    Low confidenceBase 296

    16.9% of paid-plan respondents said yes to paying £8 per user per month (base 296); 27.7% said yes overall (base 548). Applying the paid-plan share to 1,230 practices gives about 208 practices; multiplied by £8 for one seat each gives about £1,664 monthly. Practice size is not licensed-seat count, and stated intent is not purchase behaviour.

  4. Do not prioritise the client portal for larger-firm acquisition on this evidence alone. The 20-or-more group's result is striking but rests on just 12 respondents, while portal interest was modest among the larger 10–19 group.

    Low confidenceBase 12

    All 12 respondents from practices with 20 or more people ranked the portal first; among 10–19-person practices, 6.6% ranked it first (base 106). The 20-or-more group is too small to establish a dependable larger-firm preference or predict sales impact.

  5. Bulk MTD submissions are a credible alternative, especially for paid practices with deadline-volume pain, but the survey does not put them ahead of feed reliability for the next build. The population extrapolation is conditional on paid respondents being representative.

    Medium confidenceBase 296

    26.4% of paid respondents ranked bulk MTD first (base 296), implying about 325 of 1,230 paid practices if representative. Open-text responses also describe one-by-one filing as a quarter-end bottleneck. This is stated preference, not validated demand or measured time saved.

  6. Feed reliability is a recurring operational problem in the open text, while document chasing is present but gives less support for a portal as the next build.

    Medium confidenceBase 186

    73 of the 186 supplied open-text answers mention feeds breaking, disconnecting, or arriving late. Ten open-text answers from paid respondents mention chasing clients for documents. These are coded mentions, not population prevalence estimates.

Caveats

  • The 95% margin of error of 4.1 percentage points for AI's first-place share is the conventional simple-random-sample calculation. Because this was a voluntary in-app/email survey rather than a probability sample, that margin does not account for selection bias and should not be treated as a population-level confidence interval.
  • Only 548 of 8,200 invited practices completed the survey (6.7%). Respondents were disproportionately paid-plan practices (54% versus 15% of accounts) and frequent users (41.2% used Ledgerline on 5 or more days a week versus 18% of accounts). In-app respondents were especially frequent users. Results are unweighted.
  • Q6 matching account records validates respondents' plan classification, not whether respondents represent all Ledgerline practices. Survey respondents were decision-makers or influencers, and the results may not generalise to non-UK accounts or less-engaged practices.
  • The size comparison between 2–9 and 10–19 people is not statistically significant at the 95% level (20.3% versus 14.2%; bases 172 and 106). The 20-or-more group has only 12 respondents, so its 100% portal result is highly uncertain.
  • The AI willingness-to-pay question measures stated intent at a proposed price, not conversion, retention, or incremental revenue. Revenue also depends on paid seats per practice, eligibility, take-up, and whether free-plan practices would convert.
  • Open-text themes are qualitative coding of 186 answers. Responses can express multiple themes, and mention counts should not be interpreted as representative prevalence or as proof that a proposed feature will solve the reported problem.

Next steps

  • decisionMake bank-feed reliability the provisional next-half-year priority and commission a short technical audit of disconnects, delays, missing transactions, and bank/provider causes.
  • researchRun a representative paid-customer study, stratified by practice size and usage, comparing feed reliability, bulk MTD, and AI; include observed workflow volumes and willingness to pay.
  • experimentPilot improvements to the most common addressable feed failure modes with a group of paid practices.
  • experimentTest AI categorisation as a separate pricing and workflow experiment rather than committing it as the next build.
  • monitorMonitor bulk MTD usage and validate demand with high-volume filing practices; conduct targeted discovery with larger practices about document collection and portal requirements.

Open-text themes it coded

Bank feeds breaking, disconnecting, delayed, or missing transactions 73Manual categorisation or transaction-entry work 59One-at-a-time Making Tax Digital submissions 25Chasing clients for documents, records, or paperwork 15Slow loading, lag, or performance problems 26No issue reported or explicitly positive about Ledgerline 25

The survey it planned

3 screening questions and 8 questions, as the model wrote them.

  1. S1

    Is your organisation an accountancy or bookkeeping practice based in the UK?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    Has anyone at your practice used Ledgerline in the past 3 months?

    One answer
    • Yes
    • No
    • Not sure

    Routing Shown only if S1 is: Yes

    Continues if Yes

  3. S3

    Do you make or influence decisions about which software your practice uses?

    One answer
    • Yes
    • No

    Routing Shown only if S2 is: Yes

    Continues if Yes

  4. Q1

    In the past 3 months, which parts of using Ledgerline have slowed your practice down? Select all that apply. If none have slowed you down, select “None of these”.

    Any that apply
    • Keeping bank-feed connections working or reconnecting them
    • Categorising bank transactions
    • Requesting or collecting documents from clients
    • Preparing or submitting Making Tax Digital (MTD) information
    • Keeping client records up to date
    • Tracking deadlines
    • Finding information or navigating Ledgerline
    • Entering the same information more than once
    • Something else
    • None of these
  5. Q2

    Which one of these causes the most delay for your practice when using Ledgerline?

    One answer
    • Keeping bank-feed connections working or reconnecting them
    • Categorising bank transactions
    • Requesting or collecting documents from clients
    • Preparing or submitting Making Tax Digital (MTD) information
    • Keeping client records up to date
    • Tracking deadlines
    • Finding information or navigating Ledgerline
    • Entering the same information more than once
    • Something else
    • None of these
  6. Q3

    Rank these possible Ledgerline improvements from the one that would be most valuable to your practice to the one that would be least valuable. Please rank all four.

    Ranking
    • AI categorisation of bank transactions
    • More reliable bank feeds (fewer broken connections)
    • A client portal for requesting documents
    • Bulk Making Tax Digital (MTD) submissions
  7. Q4

    Which Ledgerline plan does your practice currently use?

    One answer
    • Free plan
    • Paid plan
    • Not sure
  8. Q5

    How many people work in your practice, including you, providing accountancy or bookkeeping services?

    One answer
    • 1 (sole practitioner)
    • 2–9
    • 10 or more
    • Not sure
  9. Q6

    In a typical month, how often does someone at your practice use Ledgerline?

    One answer
    • Every day or nearly every day
    • Several days a week
    • About once a week
    • 2–3 times a month
    • About once a month or less
    • Not sure
  10. Q7

    Suppose Ledgerline offered AI categorisation as an optional add-on that suggests categories for bank transactions for your practice to review. What is the most your practice would be willing to pay for it per user per month? Enter an amount in pounds; enter 0 if your practice would not pay for it.

    Number
  11. Q8

    What is the main reason you ranked your top improvement first?

    Open text
Sample plan

Invite the admin of every one of the 8,200 Ledgerline practice accounts by email and show an in-app banner to users when they log in. Aim for about 550 eligible completes. Accept one response per practice, using the practice account ID to deduplicate responses across the banner and email and to prevent multiple responses from the same practice. Set equal practice-size targets of approximately 183 completes per size group (sole practitioners, 2–9 people, and 10 or more) so comparisons are possible; aim for at least 150 paid-plan and 400 free-plan completes overall, monitoring the overlap with size targets. Weight results back to the known plan mix (85% free, 15% paid); weight by practice size too if reliable population proportions are available. The requested precision cannot be achieved with about 550 completes: even with 183 completes in each size group, the worst-case 95% margin of error is about ±7.2 percentage points per group, before weighting or nonresponse effects. To achieve ±5 percentage points at 95% confidence for each size group requires about 385 completes in each group, or at least 1,155 total, before allowing for weighting or nonresponse. If ±5-point precision is mandatory, increase the target accordingly; otherwise report the achieved sample sizes and uncertainty and do not claim that precision. Results are from a voluntary response survey and may also be affected by nonresponse bias.