Benchmarks / SurveyBench / What Ledgerline's accountancy practices want built next / Muse Wonder

Measured by Spring Prompt

Muse Wonder: What Ledgerline's accountancy practices want built next

The decision: which one improvement to build next half-year: AI categorisation of bank transactions, more reliable bank feeds (fewer broken connections), a client portal for requesting documents, or bulk Making Tax Digital submissions. The brief and the data →

Verdict
✗ Not sound
Research score
86 of 100
Analysis rating
1,460
Head to head, this task
won 5 of 14

Why it is not sound

The survey

  • A research objective the survey could not answer
  • Ignored the precision the brief asked for, which the sample cannot deliver
  • Answer bands that overlap or leave gaps

The analysis

  • Reported a material finding the data does not support

Findings the judges found unsupported

  • Do not tell the board the survey backs AI categorisation next: its headline lead is an artefact of who responded and it is polarising, not broadly preferredCalling AI's first-choice lead an artefact of who responded misstates the direction of the known plan skew. Free accounts favour AI but are under-represented; weighting to the population's plan mix would increase AI's first-choice lead. Its polarisation and weaker ranking among paid respondents are supported.
  • The unweighted total cannot be projected to the 8,200-account base because the sample badly over-represents paid and heavy usersThe associated caveat says reweighting by plan/use would likely move totals further from AI. Plan weighting would instead increase AI's lead, and no priority-by-usage data are supplied to establish the effect of usage weighting.
  • The head of sales' larger-practice portal claim rests on 12 respondents and is contradicted by paid and open-text ranking evidenceThe larger-practice portal result is not contradicted by preferences among all paid practices or practices with 10–19 people. Those are different groups. Likewise, fewer document mentions across paid open-text answers do not contradict portal demand within the 20-or-more segment. The small-base warning is justified.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of all respondents ranked AI categorisation of bank transactions first?✓37.8%37.8%
Among respondents on a paid plan, what percentage ranked more reliable bank feeds first?✓48.3%48.3%
What is the mean rank of more reliable bank feeds across all respondents (1 = highest priority, 4 = lowest), to two decimal places?✓1.991.99
What percentage of all respondents ranked AI categorisation of bank transactions last (4th)?✓42.2%42.2%
What percentage of all respondents said yes, their practice would pay £8 per user per month for AI categorisation?✓27.7%27.7%
What is the 95% margin of error, in percentage points, on the unweighted share of all respondents who ranked AI categorisation first?✓4.06 points4.06 points
Is the difference in the share ranking AI categorisation first between practices of 2-9 people and practices of 10-19 people statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
How many of the open-text answers mention bank feeds breaking, disconnecting or arriving late?✓7474
How many of the open-text answers from respondents on a paid plan mention chasing clients for documents?✓1010
Ledgerline has 1,230 paid practices. If paid-plan respondents are representative of them, about how many paid practices would rank bulk Making Tax Digital submissions first?✓325324
What percentage of the 8,200 practice accounts invited completed the survey?✓6.68%6.7%

Traps in the data 7 of 7 handled

  • ✓AI categorisation has the most first places (38%, against 35% for bank feeds) but also the most last places (42%): it is polarising. More reliable bank feeds have the best mean rank (1.99, against 2.57 for AI) and the fewest last places (7%).
  • ✓Free-plan users are 46% of respondents but bring in no revenue; paid practices (15% of accounts, all of the revenue) put bank feeds first (48%) and bulk MTD (26%) ahead of AI (10%). The AI first-choice lead comes from free users (70% of whom rank it first). Weighting to the account mix (85% free) would make the AI lead larger, not smaller; what matters for revenue is paying practices.
  • ✓28% said their practice would pay £8 per user per month for AI categorisation (free plan 40%, paid 17%): a stated intention about a hypothetical add-on.
  • ✓Every one of the 12 practices with 20 or more people ranked the client portal first.
  • ✓69% of completes came from an in-app banner seen only by users who logged in; 52% of banner respondents use Ledgerline on 5 or more days a week against 19% of email respondents and 18% of all accounts. The sample over-represents heavy users and the response rate is 6.7%.
  • ✓Practices of 2-9 people rank AI first more often than practices of 10-19 (20% against 14%, bases 172 and 106), but the difference is not significant at 95%.
  • ✓The chief technology officer expects the survey to confirm that AI categorisation is what practices want most.

The analysis

Recommendation

Build more reliable bank feeds next half-year. It is the only option with broad, non-polarising support among revenue-generating paid practices (first 48.3%, mean 1.76, last 5.1%), corroborated by 74 spontaneous open-text complaints, whereas AI is rejected last by 65.5% of paid users. Keep bulk MTD as the sequenced second candidate for paid/mid-sized practices. Do not position AI as board-backed by this survey and do not make the client portal the priority on the basis of n=12 current 20+ respondents.

What would change it

Evidence that would change this: representative, weighted/account-based data showing paid practices (especially by users/revenue at risk) rank another option above feed reliability; operational telemetry showing feed failure is rare or outside Ledgerline's control and not linked to churn/time loss; a real-payment AI pilot (not stated intent) with paid practices achieving high take-up at GBP8/user and accuracy/time savings without increasing corrections; or direct research with prospective and current 20+ firms, with adequate n, showing the portal wins deals or prevents churn worth more than feed fixes.

Findings

  1. Do not tell the board the survey backs AI categorisation next: its headline lead is an artefact of who responded and it is polarising, not broadly preferred

    High confidenceBase 548

    AI is first for 37.8% (207/548, 95% margin +/-4.1pp) but last for 42.2%; its mean rank 2.57 is worse than bank feeds 1.99. Bank feeds are first 34.9%, second 38.9% and last only 7.5%

  2. Among the practices that produce all revenue, reliable bank feeds are the clear priority and AI is the clear reject

    High confidenceBase 296

    Paid plan (n=296): bank feeds first 48.3%, mean rank 1.76; AI first 10.1%, last 65.5%, mean rank 3.35. Bulk MTD is second among paid: first 26.4%, mean 2.23

  3. The unweighted total cannot be projected to the 8,200-account base because the sample badly over-represents paid and heavy users

    High confidenceBase 548

    Sample is 54.0% paid vs 15% in the population and 46.0% free vs 85%; sample 41.2% use it 5+ days/week vs 18% in August account records. Banner completes were 52% 5+ day users vs 19% for email. Only 548/8,200 = 6.68% completed

  4. AI enthusiasm is concentrated in free, solo practices; that is not evidence of willingness to pay among current payers

    High confidenceBase 548

    Free (n=252): AI first 70.2%, mean 1.66. Paid: AI first 10.1%. 'Just me' (n=258): AI first 60.9%; 2-9: 20.3%; 10-19: 14.2%; 20+: 0.0%. The 2-9 vs 10-19 gap (20.3% vs 14.2%, n=172 vs 106) is not significant at 95%

  5. No defensible add-on revenue forecast can be made from Q3 at GBP8/user/month

    High confidenceBase 548

    Only 27.7% overall say Yes, 49.6% No, 22.6% Not sure. Yes is 40.5% among free users who currently pay nothing but 16.9% among paid users (n=296), 57.4% of whom say No. The survey has no number of chargeable users per practice, no actual purchase, no price test and hypothetical intent overstates payment; even 'about 208 of 1,230 paid practices' (16.9%) cannot be converted to per-user revenue from these data

  6. The head of sales' larger-practice portal claim rests on 12 respondents and is contradicted by paid and open-text ranking evidence

    Medium confidenceBase 12

    All 12 practices of 20+ ranked the portal first (100%), but n=12 gives extreme uncertainty and they are 2.2% of the sample. Among all paid users the portal is first for only 15.2% and last for 21.6% (mean 2.66); among 10-19 it is first for 6.6%. Only 10 paid open-text answers mention chasing documents, versus pervasive bank-feed complaints

  7. Open text corroborates bank-feed unreliability as a live operational pain, often creating the manual-coding burden AI would only partly treat

    Medium confidenceBase 186

    74 of 186 open-text answers mention feeds breaking, disconnecting, expiring, missing transactions or arriving late; many describe re-authorisation chasing, duplicate/missing transactions and then hand-coding backlogs (e.g. L012, L051, L105, L159, L354, L473)

  8. Bulk MTD is a credible second candidate for paid practices, especially mid-sized ones, but does not beat bank feeds

    Medium confidenceBase 296

    Paid: bulk MTD first 26.4%, mean 2.23 vs bank feeds 48.3%/1.76; 10-19: bulk MTD first 30.2%. If paid respondents were representative, about 325 of 1,230 paid practices would rank it first. Repeated open-text accounts cite one-by-one filing at quarter/month ends

  9. A separate performance problem appears in open text that was not one of the four ranked options, so the forced ranking cannot show it is unimportant

    Low confidenceBase 186

    Numerous respondents spontaneously cite slow loading, search, reports and client records (e.g. L032, L093, L279, L309, L329), and many say nothing slows them down. Q1 was answered by only 186/548 and was not coded in the fielded tables, so theme counts are indicative, not population estimates

Caveats

  • Unweighted, self-selected in-app/email sample: 6.68% of accounts completed, paid practices are 54% of completes vs 15% of the population, and heavy users are heavily over-represented. Reweighting by plan/use would likely move totals further from AI and cannot fix unknown non-response bias within segments.
  • Decision-makers/influencers only (screened), UK only, first response per practice only; results do not represent all staff or prospective larger firms who are not current Ledgerline accounts.
  • The 20+ segment is n=12: 100% portal-first could change by 8.3pp per respondent and must not drive strategy.
  • Ranking is forced choice among four options only; it measures relative priority, not absolute demand, severity, willingness to switch, or ROI. Performance problems emerging in Q1 were not ranked.
  • Q3 is a single hypothetical price with no commitment, no user-count base and 'Not sure' at 22.6%; it cannot yield a revenue forecast. Free users' Yes answers are not current revenue evidence.
  • Q1 was optional/open (186 answers), mentions overlap, and counts depend on wording; they show salience among commenters, not prevalence in all 548 or 8,200 practices.
  • Account-record population splits by practice size were not provided, so size-segment representativeness cannot be weighted or checked; Q6 matched records, which validates plan classification not representativeness.

Next steps

  • decisionLock bank-feed reliability as the next-half-year build and define it from failure modes in Q1: disconnect/expiry detection, proactive alerts, late/missing/duplicate transactions and re-authorisation flow
  • researchRe-analyse the existing dataset weighted to the 8,200 population by plan and August usage, and report paid results by practice size and estimated users/revenue before board sign-off
  • monitorPull product/telemetry and support data on feed disconnections, expiry, latency, missing/duplicate transactions by bank, segment and client count
  • experimentRun an AI willingness-to-pay validation only as a secondary track: recruit paid practices, disclose accuracy limits, test GBP8/user with a clickable checkout or paid pilot and measure actual conversion and correction time
  • researchInterview and survey a separately recruited sample of current and prospective 20+ practices about portal needs, current document-chasing cost and deal criteria
  • researchScope bulk MTD as the follow-on paid-segment improvement and quantify filing volumes and deadline peaks from account data

Open-text themes it coded

Bank feeds breaking, disconnecting, expiring, missing transactions or arriving late 74Manual categorisation/coding of bank transactions is slow 54Making Tax Digital submissions one client at a time / no bulk submission 25Chasing clients for documents, records, statements or paperwork 15Performance: slow loading, search, reports or pages 26Nothing slows us down / no complaints 25

The survey it planned

4 screening questions and 8 questions, as the model wrote them.

  1. S1

    Is your practice based in the UK and does it provide accountancy or bookkeeping services to clients?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    Does your practice currently use Ledgerline?

    One answer
    • Yes
    • No

    Continues if Yes

  3. S3

    Which of these best describes your role in decisions about the software your practice uses?

    One answer
    • I make the final decision
    • I influence the decision
    • I have no influence on the decision

    Continues if I make the final decision; I influence the decision

  4. S4

    Has someone else from your practice already completed this survey, as far as you know?

    One answer
    • No
    • Yes
    • I'm not sure

    Continues if No; I'm not sure

  5. Q1

    Which Ledgerline plan is your practice currently on?

    One answer
    • Free plan
    • Paid plan
    • Don't know
  6. Q2

    How many people, including you, work in your practice?

    One answer
    • Just me (sole practitioner)
    • 2-9 people
    • 10 or more people
  7. Q3

    In a typical week, on how many days do you personally use Ledgerline?

    One answer
    • 5 or more days
    • 3-4 days
    • 1-2 days
    • Less than 1 day a week
  8. Q4

    When using Ledgerline today, which of the following slow your practice down? Please select all that apply.

    Any that apply
    • Broken bank-feed connections, or having to reconnect bank feeds
    • Manually categorising bank transactions
    • Chasing clients for documents or records
    • Submitting Making Tax Digital returns one client at a time
    • Entering or correcting client records
    • Keeping track of deadlines
    • Something else
    • None of these slow us down
  9. Q5

    Thinking about the one thing that slows your practice down most when using Ledgerline today, please describe it in your own words. What happens, and how does it affect your work?

    Open text
    Judges: assumptive

    Assumes something slows the practice down. It is shown even to respondents who select 'None of these slow us down' in Q4, and no screener establishes that they experience a slowdown.

    Q5 has no display logic. Respondents who chose 'None of these slow us down' at Q4 are still asked to describe the one thing that slows them down most, which assumes something does.

  10. Q6

    Ledgerline can build one major improvement in the next half-year. Please rank the four improvements below from 1 to 4, where 1 is the one you would most like Ledgerline to build and 4 is the one you would least like Ledgerline to build. Please use each number once.

    Ranking
    • AI categorisation of bank transactions (automatically suggests a category for each bank transaction for you to review)
    • Bulk Making Tax Digital submissions (submit Making Tax Digital returns for multiple clients in one batch)
    • Client portal for requesting documents (a place to request, receive and track documents from clients)
    • More reliable bank feeds (fewer broken bank-feed connections and less need to reconnect them)
  11. Q7

    If AI categorisation of bank transactions were available only as an optional add-on, charged per user per month on top of your current plan, would your practice pay for it?

    One answer
    • Yes, we would pay an additional charge
    • Maybe - it would depend on the price and how well it worked
    • No - we would only use it if it were included in the price of our current plan
    • No - we would not use it at all
    • Don't know
  12. Q8

    What is the most your practice would be willing to pay per user, per month, for AI categorisation of bank transactions as an add-on? Please enter an amount in pounds (for example, 5). Enter 0 if you would not pay anything.

    Number

    Routing Shown only if Q7 is: Yes, we would pay an additional charge; Maybe - it would depend on the price and how well it worked

Sample plan

Invite the admin of all 8,200 Ledgerline practice accounts by email using a unique link tied to the practice account, and show an in-app banner to Ledgerline users on login that requires the practice account to be identified, so responses can be de-duplicated by practice account and only the first complete response per practice is kept. Respondents must pass S1-S4. The stated target of about 550 completes is not sufficient for the required precision: for a proportion near 50%, +/-5 percentage points at 95% confidence requires 1.96^2 x 0.25 / 0.05^2 = 385 completes per practice-size group. Therefore set a minimum total of 1,155 completes, with minimum quotas of 385 sole practitioners (Q2: Just me), 385 practices with 2-9 people, and 385 practices with 10 or more people; aim for about 1,200 completes to allow for quality-control exclusions. Monitor plan mix during fieldwork and, because paid practices are only 15% (1,230) of accounts, prioritise reminders to paid-plan admins and to any under-represented size group, closing over-filled size quotas first while keeping the survey open for under-represented groups. Report overall results weighted back to the 8,200-account population by plan and practice size, and report each size subgroup unweighted against its 385 minimum.