Benchmarks / SurveyBench / What Ledgerline's accountancy practices want built next / Particle-1

Measured by Spring Prompt

Particle-1: What Ledgerline's accountancy practices want built next

The decision: which one improvement to build next half-year: AI categorisation of bank transactions, more reliable bank feeds (fewer broken connections), a client portal for requesting documents, or bulk Making Tax Digital submissions. The brief and the data →

Verdict
✗ Not sound
Research score
92 of 100
Analysis rating
1,193
Head to head, this task
won 3 of 13

Why it is not sound

The survey

  • A research objective the survey could not answer

The analysis

  • Reported a material finding the data does not support

Findings the judges found unsupported

  • The headline first-place result for AI categorisation is not a valid board mandate because the sample is heavily skewed toward free-plan users, who are over-represented versus the account base and who overwhelmingly prefer AI.The plan-mix interpretation is reversed: free-plan practices are under-represented, not over-represented (46% of respondents versus 85% of accounts). Weighting to the account mix would increase AI's first-choice lead, not reduce it. The recommendation can be justified by paying practices' priorities, but not by claiming that over-represented free users inflated AI support.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of all respondents ranked AI categorisation of bank transactions first?✓37.8%37.8%
Among respondents on a paid plan, what percentage ranked more reliable bank feeds first?✓48.3%48.3%
What is the mean rank of more reliable bank feeds across all respondents (1 = highest priority, 4 = lowest), to two decimal places?✓1.991.99
What percentage of all respondents ranked AI categorisation of bank transactions last (4th)?✓42.2%42.2%
What percentage of all respondents said yes, their practice would pay £8 per user per month for AI categorisation?✓27.7%27.7%
What is the 95% margin of error, in percentage points, on the unweighted share of all respondents who ranked AI categorisation first?✓4.1 points4.06 points
Is the difference in the share ranking AI categorisation first between practices of 2-9 people and practices of 10-19 people statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
How many of the open-text answers mention bank feeds breaking, disconnecting or arriving late?✓7474
How many of the open-text answers from respondents on a paid plan mention chasing clients for documents?✓1010
Ledgerline has 1,230 paid practices. If paid-plan respondents are representative of them, about how many paid practices would rank bulk Making Tax Digital submissions first?✓325324
What percentage of the 8,200 practice accounts invited completed the survey?✓6.7%6.7%

Traps in the data 7 of 7 handled

  • ✓AI categorisation has the most first places (38%, against 35% for bank feeds) but also the most last places (42%): it is polarising. More reliable bank feeds have the best mean rank (1.99, against 2.57 for AI) and the fewest last places (7%).
  • ✓Free-plan users are 46% of respondents but bring in no revenue; paid practices (15% of accounts, all of the revenue) put bank feeds first (48%) and bulk MTD (26%) ahead of AI (10%). The AI first-choice lead comes from free users (70% of whom rank it first). Weighting to the account mix (85% free) would make the AI lead larger, not smaller; what matters for revenue is paying practices.
  • ✓28% said their practice would pay £8 per user per month for AI categorisation (free plan 40%, paid 17%): a stated intention about a hypothetical add-on.
  • ✓Every one of the 12 practices with 20 or more people ranked the client portal first.
  • ✓69% of completes came from an in-app banner seen only by users who logged in; 52% of banner respondents use Ledgerline on 5 or more days a week against 19% of email respondents and 18% of all accounts. The sample over-represents heavy users and the response rate is 6.7%.
  • ✓Practices of 2-9 people rank AI first more often than practices of 10-19 (20% against 14%, bases 172 and 106), but the difference is not significant at 95%.
  • ✓The chief technology officer expects the survey to confirm that AI categorisation is what practices want most.

The analysis

Recommendation

Build more reliable bank feeds next half-year. It is the best average priority across all respondents (mean rank 1.99), the clear paid-plan priority (48.3% first, mean 1.76), and the most repeated open-text pain (74 of 186 answers). Do not tell the board that the survey backs AI categorisation next: its headline first-place share is inflated by over-represented free-plan users and it is polarising (37.8% first but 42.2% last). Treat AI categorisation as a possible later paid add-on only after a representative paid-plan survey and pricing test; do not size £8 revenue from this survey. For larger firms, bank feeds and bulk MTD are stronger near-term priorities than the client portal; the 20+ portal result is too small to rely on.

What would change it

A representative, account-weighted paid-plan survey showing bank-feed pain is lower than measured and AI add-on willingness is materially higher, or a validated pricing test proving paid conversion at £8 per user per month, would switch the recommendation toward AI categorisation. Evidence that bank-feed reliability cannot be materially improved within two quarters would also change the sequencing. A larger, adequately powered large-practice sample confirming client-portal demand could raise the portal's priority.

Findings

  1. The headline first-place result for AI categorisation is not a valid board mandate because the sample is heavily skewed toward free-plan users, who are over-represented versus the account base and who overwhelmingly prefer AI.

    High confidenceBase 548

    Free plan is 252/548 completes (46.0%) but only 85% of the 8,200 accounts; paid plans are 296/548 (54.0%) but only 15% of accounts. Free respondents rank AI first at 70.2%, paid respondents at 10.1%.

  2. Paid-plan respondents, who generate all revenue, clearly prioritise more reliable bank feeds over AI categorisation.

    High confidenceBase 296

    Paid-plan base 296: bank feeds ranked first 48.3% and mean rank 1.76; AI categorisation ranked first 10.1%, last 65.5%, mean rank 3.35. Bulk MTD is second (26.4% first, mean 2.23), client portal third (15.2% first, mean 2.66).

  3. AI categorisation is polarising rather than a consensus priority: it attracts the most first-place votes but also the most last-place votes, while bank feeds have the best average priority.

    High confidenceBase 548

    All respondents base 548: AI first 37.8%, last 42.2%, mean rank 2.57. Bank feeds first 34.9%, last 7.5%, mean rank 1.99.

  4. Free-plan enthusiasm for AI does not translate into a strong paid add-on signal; paid willingness to pay £8 per user per month is weak.

    High confidenceBase 548

    All respondents: 27.7% would pay, 49.6% would not, 22.6% unsure. Free plan: 40.5% yes. Paid plan: 16.9% yes, 57.4% no, 25.7% unsure.

  5. The finance director cannot derive a credible add-on revenue estimate from this survey because exact paid-user counts are missing and Q3 measures stated intent, not purchase.

    High confidenceBase 296

    Paid-plan yes rate is 16.9% of 296 paid respondents. Applied naively to 1,230 paid practices, that is about 208 practices saying yes, but revenue = 208 × users × £8 × 12 and the survey has no exact per-practice user count or paid-plan size cross-tab.

  6. The evidence does not support making the client portal the priority for winning larger firms; the apparent 20+ practice unanimity rests on only 12 respondents.

    Low confidenceBase 118

    Among 20+ practices, 12/12 rank client portal first, but base is only 12. Among 10-19 practices (base 106), portal is first for 6.6%, versus bank feeds 49.1% and bulk MTD 30.2%. Among 2-9 practices (base 172), portal is first for 14.0%, versus bank feeds 43.0%.

  7. Open-text responses corroborate bank-feed reliability and manual transaction coding as the most frequently cited operational drags, with bulk MTD and document chasing also material.

    Medium confidenceBase 186

    Of 186 open-text answers, 74 mention bank feeds breaking, disconnecting, arriving late or missing; 57 mention manual coding/categorisation; 25 mention one-by-one MTD filing; 15 mention chasing clients for documents; 26 mention slowness/performance.

  8. The survey has a low response rate and engaged-user skew, so unweighted totals should not be projected directly to the 8,200 account base.

    High confidenceBase 548

    548 completes from 8,200 invited accounts = 6.7%. In-app banner produced 376/548 completes (69%); 52% of banner respondents use Ledgerline 5+ days/week versus 19% of email respondents.

Caveats

  • The survey is self-selected and achieved only 6.7% of invited accounts; it is not a probability sample of the 8,200 practices.
  • Plan mix is badly skewed versus the account base: free plan is 46.0% of completes but 85% of accounts; paid plans are 54.0% of completes but only 15% of accounts. Unweighted totals overstate free-plan preferences.
  • In-app banner respondents are more engaged/frequent users (52% use Ledgerline 5+ days/week versus 19% of email respondents), so the sample over-represents active users.
  • Q3 measures stated purchase intent, not actual purchase, and the survey lacks exact per-practice user counts; a reliable £8 per user per month revenue estimate cannot be produced from these data.
  • The 20+ practice subgroup is only 12 respondents, and 10-19 is 106; subgroup comparisons involving the largest practices are fragile.
  • Open-text answers are optional and self-selected (186 of 548 completes), so mention counts are indicative, not representative.
  • Q2 was randomised but results are unweighted; mean ranks are useful but not a significance-tested preference model.
  • The decision options are not mutually exclusive in operational terms: bank-feed reliability, manual coding and bulk MTD interact, so a single build choice may not capture all value.

Next steps

  • decisionPrioritise bank-feed reliability for the next half-year and define the build scope around connection resilience, re-authorisation, and late/missing transaction detection.
  • researchRun a representative, account-weighted survey or account-level analysis stratified by plan and exact user count, with Q3 framed as a priced choice.
  • experimentIf AI categorisation is still considered, test it as a limited paid add-on beta with real conversion/pricing, not as a full next-half-year build.
  • researchResearch larger practices separately with adequate sample size before making client portal a priority.
  • monitorMonitor open-text and support-ticket themes for bank-feed failures, manual coding and bulk MTD; use them as leading indicators alongside survey data.

Open-text themes it coded

Bank feed reliability: disconnections, re-authorisation, late or missing transactions 74Manual coding or categorisation of bank transactions 57Bulk MTD submissions / one-by-one VAT filing 25Chasing clients for documents, records, statements or paperwork 15Performance and slowness 26No significant blockers or no issues reported 25

The survey it planned

4 screening questions and 7 questions, as the model wrote them.

  1. S1

    Do you work in a UK accountancy or bookkeeping practice that currently uses Ledgerline?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    Do you make or influence decisions about the software your practice uses?

    One answer
    • Yes
    • No

    Continues if Yes

  3. S3

    How many people work in your practice, including partners/directors and all staff?

    One answer
    • Sole practitioner
    • 2–9 people
    • 10 or more people

    Continues if Sole practitioner; 2–9 people; 10 or more people

  4. S4

    Which Ledgerline plan does your practice currently use?

    One answer
    • Free plan
    • Paid plan
    • Not sure

    Continues if Free plan; Paid plan; Not sure

  5. Q1

    In a typical week, how often do you personally log in to Ledgerline?

    One answer
    • Daily
    • Several times a week
    • About once a week
    • A few times a month
    • Monthly or less
  6. Q2

    Which of these, if any, slow your practice down when using Ledgerline? Select all that apply.

    Any that apply
    • Categorising or reconciling bank transactions
    • Broken bank-feed connections or reconnecting bank feeds
    • Chasing clients for documents or information
    • Preparing or submitting Making Tax Digital (MTD) returns
    • Tracking client records or deadlines
    • None of these
  7. Q3

    What is the single biggest thing that slows your practice down when using Ledgerline?

    Open text
    Judges: assumptive

    Assumes that something slows the practice down. It is shown even to respondents who selected 'None of these' in Q2, and neither the screeners nor other answers establish that they experience a slowdown.

    It is shown to everyone with no display logic, so it assumes something slows the practice down, even for respondents who chose 'None of these' at Q2.

  8. Q4

    Please rank these four possible Ledgerline improvements from 1 (would help your practice most) to 4 (would help your practice least).

    Ranking
    • AI categorisation of bank transactions (automatically suggesting categories for bank transactions)
    • More reliable bank feeds (fewer broken connections)
    • A client portal for requesting documents
    • Bulk Making Tax Digital (MTD) submissions
  9. Q5

    If Ledgerline could build only one of these improvements in the next six months, which would you choose?

    One answer
    • AI categorisation of bank transactions (automatically suggesting categories for bank transactions)
    • More reliable bank feeds (fewer broken connections)
    • A client portal for requesting documents
    • Bulk Making Tax Digital (MTD) submissions
  10. Q6

    How often does your practice experience a bank-feed connection breaking or needing to be reconnected in Ledgerline?

    Scale

    Scale 1-5: Never to Very often

    Judges: ambiguous

    No reference period is specified for connection failures. Respondents could answer about recent experience or their entire time using Ledgerline, making the frequency ratings difficult to compare.

    There is no reference period, and the labels 'Never' to 'Very often' are vague frequency terms. Respondents will interpret how often differently.

  11. Q7

    If Ledgerline offered AI categorisation of bank transactions as an add-on, what would your practice be willing to pay per user per month?

    One answer
    • I would not pay extra
    • £1–£5 per user per month
    • £6–£10 per user per month
    • £11–£15 per user per month
    • £16–£20 per user per month
    • More than £20 per user per month
    • Don’t know
Sample plan

Invite all 8,200 Ledgerline practice accounts: an email invitation to the admin of every account, plus an in-app banner shown to logged-in users. Both routes carry a unique practice ID from the Ledgerline account so that only one response per practice is retained; duplicate submissions from the same practice are removed before analysis. Target 550 completes, with quotas to support comparisons: sole practitioners 180, 2–9 staff 190, 10+ staff 180; paid-plan minimum 200 completes, with the remainder from free-plan practices. Because ±5 percentage points at 95% confidence for each of three practice-size groups requires about 384 completes per group (≈1,152 total before finite-population correction), 550 completes will not meet that precision for every size group unless the population in a group is small. Fieldwork will therefore initially aim for 550 completes but top up under-filled size strata to the finite-population-adjusted requirement, up to about 1,150 completes, before making ±5pp claims. If the 550 cap is absolute, report practice-size margins of about ±7pp and treat size-group results as directional. Weight results to the known account population by plan and practice size.