Benchmarks / SurveyBench / What Ledgerline's accountancy practices want built next / GPT-6.1 Terra

Measured by Spring Prompt

GPT-6.1 Terra: What Ledgerline's accountancy practices want built next

The decision: which one improvement to build next half-year: AI categorisation of bank transactions, more reliable bank feeds (fewer broken connections), a client portal for requesting documents, or bulk Making Tax Digital submissions. The brief and the data →

Verdict
✗ Not sound
Research score
96 of 100
Analysis rating
1,273
Head to head, this task
won 5 of 10

Why it is not sound

The survey

  • A research objective the survey could not answer

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of all respondents ranked AI categorisation of bank transactions first?✓37.8%37.8%
Among respondents on a paid plan, what percentage ranked more reliable bank feeds first?✓48.3%48.3%
What is the mean rank of more reliable bank feeds across all respondents (1 = highest priority, 4 = lowest), to two decimal places?✓1.991.99
What percentage of all respondents ranked AI categorisation of bank transactions last (4th)?✓42.2%42.2%
What percentage of all respondents said yes, their practice would pay £8 per user per month for AI categorisation?✓27.7%27.7%
What is the 95% margin of error, in percentage points, on the unweighted share of all respondents who ranked AI categorisation first?✓4.06 points4.06 points
Is the difference in the share ranking AI categorisation first between practices of 2-9 people and practices of 10-19 people statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
How many of the open-text answers mention bank feeds breaking, disconnecting or arriving late?✓6474
How many of the open-text answers from respondents on a paid plan mention chasing clients for documents?✓1010
Ledgerline has 1,230 paid practices. If paid-plan respondents are representative of them, about how many paid practices would rank bulk Making Tax Digital submissions first?✓325324
What percentage of the 8,200 practice accounts invited completed the survey?✓6.68%6.7%

Traps in the data 7 of 7 handled

  • ✓AI categorisation has the most first places (38%, against 35% for bank feeds) but also the most last places (42%): it is polarising. More reliable bank feeds have the best mean rank (1.99, against 2.57 for AI) and the fewest last places (7%).
  • ✓Free-plan users are 46% of respondents but bring in no revenue; paid practices (15% of accounts, all of the revenue) put bank feeds first (48%) and bulk MTD (26%) ahead of AI (10%). The AI first-choice lead comes from free users (70% of whom rank it first). Weighting to the account mix (85% free) would make the AI lead larger, not smaller; what matters for revenue is paying practices.
  • ✓28% said their practice would pay £8 per user per month for AI categorisation (free plan 40%, paid 17%): a stated intention about a hypothetical add-on.
  • ✓Every one of the 12 practices with 20 or more people ranked the client portal first.
  • ✓69% of completes came from an in-app banner seen only by users who logged in; 52% of banner respondents use Ledgerline on 5 or more days a week against 19% of email respondents and 18% of all accounts. The sample over-represents heavy users and the response rate is 6.7%.
  • ✓Practices of 2-9 people rank AI first more often than practices of 10-19 (20% against 14%, bases 172 and 106), but the difference is not significant at 95%.
  • ✓The chief technology officer expects the survey to confirm that AI categorisation is what practices want most.

The analysis

Recommendation

Prioritise more reliable bank feeds for the next half-year, subject to a short technical feasibility assessment. Be explicit that this favours broad support among surveyed paying practices and current revenue protection over the strong AI preference among surveyed free practices. Retain bulk MTD as the principal paid-practice alternative; test AI commercial demand and the larger-firm portal opportunity without treating either as proven.

What would change it

Change the priority if representative customer and operational evidence shows feed failures are rare, largely outside Ledgerline's control, or too costly to improve materially; or if validated retention, acquisition or paid-conversion value for bulk MTD, AI or the portal exceeds the achievable benefit of feed work. Strong evidence of substantial near-term revenue from larger prospects specifically requiring a portal would also change the trade-off.

Findings

  1. Build more reliable bank feeds next if the objective is to protect and improve the experience of existing paying practices. This is a conditional recommendation, not a demonstrated retention or financial impact.

    Medium confidenceBase 296

    Among 296 paid-plan respondents, 48.3% ranked feeds first and 5.1% last; their mean rank was 1.76. Across all 548 respondents, 73.8% ranked feeds first or second and the mean rank was 1.99. All current subscription revenue comes from 1,230 paid practices.

  2. The CTO should not tell the board that the survey establishes AI as the best next build. AI led on first-place votes in this sample but polarised respondents, especially paying practices.

    High confidenceBase 548

    AI was ranked first by 37.8% of all 548 respondents, versus 34.9% for feeds; 42.2% ranked AI last. Among 296 paid respondents, 10.1% ranked AI first and 65.5% last. Among 252 free respondents, 70.2% ranked AI first. Applying the known 85% free/15% paid account mix to these plan-specific shares gives an illustrative 61.2% AI-first versus 23.4% feeds-first, assuming respondents represent their plan groups.

  3. The finance director cannot calculate credible AI add-on revenue from this survey. Treat stated willingness to pay as a hypothesis, not a sales forecast.

    High confidenceBase 548

    Of 548 respondents, 27.7% said yes, 49.6% no and 22.6% were unsure. Yes shares were 40.5% among 252 free-plan respondents and 16.9% among 296 paid-plan respondents. £8 per paying user would be £96 per user over 12 paid months, but the survey does not establish purchase conversion or the number of add-on seats.

  4. The head of sales has a lead to investigate, not enough evidence to prioritise the portal for winning larger firms.

    Medium confidenceBase 118

    All 12 respondents from practices with 20 or more people ranked the portal first, but only 6.6% of the 106 respondents from 10–19-person practices did so. The survey covers existing users, not prospective firms, and does not establish the size or revenue of the larger-firm opportunity.

  5. Bulk MTD is a credible next candidate for paying practices, particularly those handling many returns; it should remain in the roadmap comparison.

    Medium confidenceBase 296

    Of 296 paid respondents, 26.4% ranked bulk MTD first and its mean rank was 2.23. Applying 26.4% to 1,230 paid practices gives approximately 325 practices, conditional on paid respondents being representative.

  6. Do not infer a reliable difference in AI-first preference between the two middle-sized practice groups from this sample.

    Medium confidenceBase 278

    AI was ranked first by 20.3% of 172 respondents from 2–9-person practices and 14.2% of 106 from 10–19-person practices. The 6.1-percentage-point difference is not significant at the conventional 95% level in an approximate two-proportion test.

  7. Investigate the causes of feed failures and client re-authorisation friction before promising a particular reliability improvement.

    Medium confidenceBase 186

    64 of the 186 open answers explicitly describe feed disconnection, re-authorisation or delays. Responses describe silent failures, client chasing and delayed work; some also describe manual coding after an outage.

Caveats

  • The survey is voluntary and unweighted. The 6.68% completion figure uses 548 completes divided by 8,200 invited practice accounts; nonresponse and self-selection may affect every population inference.
  • Paid accounts are 54.0% of completes but 15% of accounts. Respondents reporting use on at least five days a week are 41.2% of completes, versus 18% of accounts in August records. Survey-reported typical use and recorded August use are not identical measures.
  • The 4.06-point margin of error is a conventional simple-random-sample calculation, not a bound on the bias of this voluntary sample. The significance test likewise does not correct for selection bias.
  • Plan weighting is illustrative: it corrects the known plan mix only, not differences within plan groups in usage, practice size or likelihood of responding.
  • The feed count uses a conservative coding rule: explicit disconnection, re-authorisation or late feed data, including implied lapses. Missing transactions alone and ambiguous references to bank data are excluded. Open-text counts are not population prevalence estimates.
  • Practice headcount is not the number of licensed users or add-on seats. Stated intent to pay is not a purchase, and neither churn reduction nor acquisition impact was measured.
  • Only 12 respondents were from practices of 20 or more people. The survey does not measure prospects, the population distribution of practice size, implementation cost, or the revenue of individual accounts.

Next steps

  • decisionTime-box a feasibility assessment and choose the feed-reliability scope before committing the half-year plan.
  • monitorMeasure feed disconnects, transaction delays, missing data and time to recovery by bank, plan and usage; separate required re-authorisations from preventable failures.
  • experimentTest the AI add-on with a priced offer or commitment experiment across free and paid practices, recording intended seat counts and actual conversion.
  • researchInterview and test a portal proposition with larger existing practices and qualified prospective firms; examine sales and account data alongside it.
  • researchValidate priorities with a targeted sample covering underrepresented low-use and free accounts as well as paid practices, and assess bulk MTD workflow volume and cost.

Open-text themes it coded

Bank feed disconnection, re-authorisation or delayed feed data 64Paid-plan respondents chasing clients for paperwork or documents 10

The survey it planned

2 screening questions and 8 questions, as the model wrote them.

  1. S1

    Do you work in a UK accountancy or bookkeeping practice that currently uses Ledgerline?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    Do you make or influence decisions about the software your practice uses?

    One answer
    • Yes
    • No

    Continues if Yes

  3. Q1

    How many people work in your practice, including you and both full-time and part-time staff?

    One answer
    • 1 — just me
    • 2–9
    • 10 or more
    • Don't know
  4. Q2

    Which Ledgerline plan does your practice currently use?

    One answer
    • Free plan
    • Paid plan
    • Don't know
  5. Q3

    On roughly how many days in the past 30 days did you personally use Ledgerline?

    One answer
    • None
    • 1–4 days
    • 5–9 days
    • 10–19 days
    • 20–30 days
  6. Q4

    Which of the following have slowed your practice down when using Ledgerline? Select all that apply.

    Any that apply
    • Categorising bank transactions
    • Fixing or reconnecting bank feeds
    • Requesting or collecting documents from clients
    • Preparing or submitting Making Tax Digital returns
    • Finding or updating client records
    • Tracking deadlines
    • Slow performance or technical errors
    • Learning how to use Ledgerline or finding help
    • Something else
    • Nothing has slowed us down
    • Don't know
  7. Q5

    Which one of these improvements would be most valuable to your practice?

    One answer
    • AI categorisation of bank transactions
    • More reliable bank feeds, with fewer broken connections
    • A client portal for requesting documents
    • Bulk Making Tax Digital submissions
    • None of these
    • Don't know
  8. Q6

    Which of these improvements, if any, would be useful to your practice? Select all that apply.

    Any that apply
    • AI categorisation of bank transactions
    • More reliable bank feeds, with fewer broken connections
    • A client portal for requesting documents
    • Bulk Making Tax Digital submissions
    • None of these
    • Don't know
  9. Q7

    Imagine AI categorisation of bank transactions were available as an optional add-on to your current Ledgerline plan for £5 per user per month. Would your practice pay for it at that price?

    One answer
    • Yes
    • No
    • Not sure
  10. Q8

    What is the most your practice would pay for AI categorisation as an optional add-on to your current Ledgerline plan? Enter an amount in whole pounds per user per month. Enter 0 if your practice would not pay. You may skip this question if you are unsure.

    Number
Sample plan

Invite the admin of every practice account by email and show an in-app banner to logged-in users. Ask for one eligible software decision-maker or influencer per practice. Use account-linked survey access across both channels, without exposing account identifiers in the answers; allow an incomplete response to be resumed and retain only the first eligible completed response per practice. Aim for about 550 completes, with soft targets of approximately 183 per practice-size group and at least 200 paid-plan completes, subject to eligible populations and response. Monitor quotas and send reminders to underrepresented groups of account admins; do not present these targets as guaranteed. Make 'Nothing has slowed us down' and 'Don't know' exclusive in Q4, and 'None of these' and 'Don't know' exclusive in Q6. At 95% confidence, a simple random sample needs approximately 385 completes per size group for ±5 percentage points at a 50% estimate before finite-population correction. With 183 per group, the corresponding margin is approximately ±7.2 points before correction. The 550-complete target therefore cannot generally guarantee the requested precision. Once the eligible practice count N in each size group is known, calculate the finite-population target as approximately 384N/(N+383); increase the total target if necessary and feasible. Self-selection and nonresponse mean even these nominal margins do not guarantee accuracy.