Benchmarks / SurveyBench / What Ledgerline's accountancy practices want built next / Mercury 3

Measured by Spring Prompt

Mercury 3: What Ledgerline's accountancy practices want built next

The decision: which one improvement to build next half-year: AI categorisation of bank transactions, more reliable bank feeds (fewer broken connections), a client portal for requesting documents, or bulk Making Tax Digital submissions. The brief and the data →

Verdict
✗ Not sound
Research score
92 of 100
Analysis rating
752
Head to head, this task
won 14 of 15

Why it is not sound

The analysis

  • Missed a trap: AI categorisation has the most first places (38%, against 35% for bank feeds) but also the most last places (42%): it is polarising. More reliable bank feeds have the best mean rank (1.99, against 2.57 for AI) and the fewest last places (7%).
  • Missed a trap: The chief technology officer expects the survey to confirm that AI categorisation is what practices want most.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of all respondents ranked AI categorisation of bank transactions first?✓37.8%37.8%
Among respondents on a paid plan, what percentage ranked more reliable bank feeds first?✓48.3%48.3%
What is the mean rank of more reliable bank feeds across all respondents (1 = highest priority, 4 = lowest), to two decimal places?✓1.991.99
What percentage of all respondents ranked AI categorisation of bank transactions last (4th)?✓42.2%42.2%
What percentage of all respondents said yes, their practice would pay £8 per user per month for AI categorisation?✓27.7%27.7%
What is the 95% margin of error, in percentage points, on the unweighted share of all respondents who ranked AI categorisation first?✓4.06 points4.06 points
Is the difference in the share ranking AI categorisation first between practices of 2-9 people and practices of 10-19 people statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
How many of the open-text answers mention bank feeds breaking, disconnecting or arriving late?✓7474
How many of the open-text answers from respondents on a paid plan mention chasing clients for documents?✓1010
Ledgerline has 1,230 paid practices. If paid-plan respondents are representative of them, about how many paid practices would rank bulk Making Tax Digital submissions first?✓325324
What percentage of the 8,200 practice accounts invited completed the survey?✓6.68%6.7%

Traps in the data 5 of 7 handled

  • ✗AI categorisation has the most first places (38%, against 35% for bank feeds) but also the most last places (42%): it is polarising. More reliable bank feeds have the best mean rank (1.99, against 2.57 for AI) and the fewest last places (7%).
  • ✓Free-plan users are 46% of respondents but bring in no revenue; paid practices (15% of accounts, all of the revenue) put bank feeds first (48%) and bulk MTD (26%) ahead of AI (10%). The AI first-choice lead comes from free users (70% of whom rank it first). Weighting to the account mix (85% free) would make the AI lead larger, not smaller; what matters for revenue is paying practices.
  • ✓28% said their practice would pay £8 per user per month for AI categorisation (free plan 40%, paid 17%): a stated intention about a hypothetical add-on.
  • ✓Every one of the 12 practices with 20 or more people ranked the client portal first.
  • ✓69% of completes came from an in-app banner seen only by users who logged in; 52% of banner respondents use Ledgerline on 5 or more days a week against 19% of email respondents and 18% of all accounts. The sample over-represents heavy users and the response rate is 6.7%.
  • ✓Practices of 2-9 people rank AI first more often than practices of 10-19 (20% against 14%, bases 172 and 106), but the difference is not significant at 95%.
  • ✗The chief technology officer expects the survey to confirm that AI categorisation is what practices want most.

The analysis

Recommendation

Build more reliable bank feeds next. The paid customer base—which supplies all current revenue—ranks this first (48.3%), gives it the best mean rank (1.76), and open-text answers show feed failures are the most common operational drag. Put bulk MTD submissions next on the roadmap, especially for mid-size practices. Do not commit the next half-year to AI categorisation or a client portal without further validation; AI is primarily a free/solo-plan priority and its paid willingness-to-pay is low, while the client portal case rests on only 12 very large practices.

What would change it

A switch to AI categorisation would be justified if a paid-segment pilot showed at least 30% of paid users adopting the add-on at £8/user/month and measurably cutting manual coding time, or if the company decides the priority is converting free users, in which case the weighted population data favour AI. A switch to the client portal would be justified if a properly powered study of 20+ practices (roughly 100+ respondents) and sales win/loss data showed it is a deciding factor for target firms.

Findings

  1. Paid-plan customers put bank feed reliability first; AI categorisation is their lowest-ranked option.

    High confidenceBase 296

    Among 296 paid-plan respondents, 48.3% ranked more reliable bank feeds first versus 10.1% for AI; mean ranks were 1.76 for bank feeds and 3.35 for AI. AI was ranked last by 65.5% of paid respondents.

  2. The overall AI first-place plurality is driven by free and solo practices, not by the current revenue base.

    High confidenceBase 252

    Free-plan respondents ranked AI first 70.2% (mean 1.66); 'just me' practices ranked it first 60.9%. Free plans are 85% of accounts but generate none of the revenue.

  3. Bulk MTD is the clearest second priority in paid and mid-sized practices.

    High confidenceBase 296

    26.4% of paid respondents ranked it first (mean 2.23); among 10-19-person practices 30.2% ranked it first.

  4. Client portal support is weak overall and the 20+ segment is too small to drive a build decision.

    Medium confidenceBase 548

    Only 11.7% of 548 respondents and 15.2% of paid respondents ranked the portal first. All 12 respondents from 20+ practices ranked it first, but only 6.6% of 106 respondents from 10-19 practices did.

  5. AI add-on willingness to pay is modest among paid practices; revenue cannot be fully estimated without seats-per-practice data.

    Medium confidenceBase 296

    16.9% of paid respondents said yes, implying about 208 of 1,230 paid practices if representative; 40.5% of free respondents said yes, implying a possible free-to-paid conversion route. Seats per paid practice were not collected.

  6. Open-text answers corroborate bank feeds as the most frequent operational friction.

    Medium confidenceBase 186

    Of 186 open-text answers, about 74 mention bank feed failures/delays, 59 manual coding/categorisation, 25 one-by-one MTD filing, and 15 chasing clients for documents.

Caveats

  • Survey completions are unweighted and over-represent paid plans: 54% of completes are paid, but paid plans are only 15% of the 8,200 account population. Consequently, all overall percentages are sample-mix figures, not population estimates.
  • Response rate is 6.68% and recruitment was self-selected via in-app banner and email; banner respondents were much more likely to be daily users (52% vs 19% for email and 18% population), so heavy users are over-represented.
  • Base sizes for very large practices are tiny (12 in the 20+ group); any conclusions about the largest firms are indicative only.
  • Q3 measures stated willingness to pay, not actual purchasing; stated intent tends to overstate true uptake.
  • Revenue cannot be calculated precisely because the survey did not collect the number of paid seats/users per practice.
  • Open-text classifications are analyst-coded from 186 answers and should be treated as directional themes, not exact measurement.

Next steps

  • decisionBuild more reliable bank feeds (fewer broken connections) as the primary product investment.
  • experimentPilot AI categorisation with a small group of paid practices and measure actual take-up at £8/user/month.
  • researchCollect seat counts and monthly revenue per paid practice, then model AI add-on MRR.
  • researchRecruit a larger, targeted sample of practices with 20+ people to test the client portal proposition.
  • decisionPlan bulk MTD submissions as the next item after bank feeds and scope it for paid and 10-19-person practices.

Open-text themes it coded

Bank feeds – breaking, disconnecting, late or missing transactions 74Manual categorisation/coding of bank transactions 59Bulk MTD / individual VAT return filing 25Chasing clients for documents/records 15Performance – slow loading, search, lag 26No significant slowdowns / positive experience 25

The survey it planned

4 screening questions and 6 questions, as the model wrote them.

  1. S1

    I confirm that I am happy to take part in this survey and for my answers to be used for research.

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    Do you work for an accountancy or bookkeeping practice based in the UK?

    One answer
    • Yes
    • No

    Continues if Yes

  3. S3

    Does your practice currently use Ledgerline?

    One answer
    • Yes
    • No

    Continues if Yes

  4. S4

    Are you involved in making or influencing decisions about the software your practice uses?

    One answer
    • Yes
    • No

    Continues if Yes

  5. Q1

    What is the single biggest thing that slows your practice down when using Ledgerline today? Please be as specific as you can.

    Open text
  6. Q2

    Including yourself, how many staff does your practice have?

    One answer
    • Just me (sole practitioner)
    • 2 to 9 staff
    • 10 or more staff
  7. Q3

    Which Ledgerline plan does your practice currently use?

    One answer
    • Free plan
    • Paid plan
    • Not sure
  8. Q4

    In a typical week, approximately how many hours in total do the people in your practice spend using Ledgerline?

    One answer
    • Less than 1 hour
    • 1 to 5 hours
    • 6 to 15 hours
    • More than 15 hours
  9. Q5

    If Ledgerline builds only one improvement in the next six months, which would be most valuable to your practice? Please rank the options from 1 (most valuable) to 4 (least valuable).

    Ranking
    • AI categorisation of bank transactions (automatically coding bank transactions)
    • More reliable bank feeds (fewer broken connections)
    • Client portal for requesting documents (online document requests from clients)
    • Bulk Making Tax Digital (MTD) submissions (submit multiple returns at once)
  10. Q6

    If Ledgerline offered AI categorisation of bank transactions as a paid add-on, would your practice pay for it?

    One answer
    • Yes
    • No
    • Not sure
Sample plan

Invite the admin contact of every Ledgerline practice account (8,200 practices) by email and show the in-app banner to all logged-in users. Aim for 550 completes. Use soft quotas of approximately 183 sole practitioners, 184 practices with 2-9 staff and 183 practices with 10+ staff; also oversample paid-plan practices to at least 220 completes (distributed across the size bands) so free/paid comparisons are robust. Use account-level plan data to assign quotas where possible, otherwise use the self-reported plan question. Randomise the order of the Q5 ranking options. Deduplicate by practice account ID and keep the first complete response per practice. Weight the final sample to the population distribution of practice size and plan if achieved quotas differ. Important limitation: the requested +/-5 percentage-point precision at 95% confidence for each practice-size band requires about 384 completes per band, so a total of 550 completes cannot meet that target for all three bands; if that precision is mandatory, the total sample would need to be about 1,200 completes.