Sign inTest your prompt

Benchmarks / SurveyBench / What Ledgerline's accountancy practices want built next / Gemini 3.8 Flash

Measured by Spring Prompt

Gemini 3.8 Flash: What Ledgerline's accountancy practices want built next

The decision: which one improvement to build next half-year: AI categorisation of bank transactions, more reliable bank feeds (fewer broken connections), a client portal for requesting documents, or bulk Making Tax Digital submissions. The brief and the data →

Verdict
✗ Not sound
Research score
86 of 100
Analysis rating
898
Head to head, this task
won 3 of 12

Why it is not sound

The survey

  • A double-barrelled question: Q4

The analysis

  • Reported a material finding the data does not support

Findings the judges found unsupported

  • AI categorisation is a trap for the board: its high aggregate first-place vote is driven entirely by non-paying solo users, while paying customers actively reject it.The separate plan and size tables do not establish that AI's aggregate lead is driven entirely by non-paying solo users. Ranking AI last establishes low relative priority, not active rejection or alienation if it is built.
  • Add-on revenue projections of £8/user/month for AI categorisation are economically unviable and based on stated-intent illusions.The survey cannot establish economic unviability, negligible MRR, a 70–80% stated-to-actual drop-off, or alienation of paying customers. There are no behavioural conversion results, development costs or user-seat counts supporting those conclusions. Being on the free plan also does not establish an absence of willingness-to-pay history.
  • Bulk MTD submissions represent an overlooked, high-impact retention driver for mid-sized commercial practices.Bulk MTD's priority among paying respondents is supported, but its effect on retention and its claimed high impact were not measured. Approximately 325 is an arithmetic extrapolation, not an established count of paying practices with that preference in this self-selected sample.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of all respondents ranked AI categorisation of bank transactions first?✓37.8%37.8%
Among respondents on a paid plan, what percentage ranked more reliable bank feeds first?✓48.3%48.3%
What is the mean rank of more reliable bank feeds across all respondents (1 = highest priority, 4 = lowest), to two decimal places?✓1.991.99
What percentage of all respondents ranked AI categorisation of bank transactions last (4th)?✓42.2%42.2%
What percentage of all respondents said yes, their practice would pay £8 per user per month for AI categorisation?✓27.7%27.7%
What is the 95% margin of error, in percentage points, on the unweighted share of all respondents who ranked AI categorisation first?✓4.1 points4.06 points
Is the difference in the share ranking AI categorisation first between practices of 2-9 people and practices of 10-19 people statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
How many of the open-text answers mention bank feeds breaking, disconnecting or arriving late?✓7374
How many of the open-text answers from respondents on a paid plan mention chasing clients for documents?✓1010
Ledgerline has 1,230 paid practices. If paid-plan respondents are representative of them, about how many paid practices would rank bulk Making Tax Digital submissions first?✓325324
What percentage of the 8,200 practice accounts invited completed the survey?✓6.7%6.7%

Traps in the data 7 of 7 handled

  • ✓AI categorisation has the most first places (38%, against 35% for bank feeds) but also the most last places (42%): it is polarising. More reliable bank feeds have the best mean rank (1.99, against 2.57 for AI) and the fewest last places (7%).
  • ✓Free-plan users are 46% of respondents but bring in no revenue; paid practices (15% of accounts, all of the revenue) put bank feeds first (48%) and bulk MTD (26%) ahead of AI (10%). The AI first-choice lead comes from free users (70% of whom rank it first). Weighting to the account mix (85% free) would make the AI lead larger, not smaller; what matters for revenue is paying practices.
  • ✓28% said their practice would pay £8 per user per month for AI categorisation (free plan 40%, paid 17%): a stated intention about a hypothetical add-on.
  • ✓Every one of the 12 practices with 20 or more people ranked the client portal first.
  • ✓69% of completes came from an in-app banner seen only by users who logged in; 52% of banner respondents use Ledgerline on 5 or more days a week against 19% of email respondents and 18% of all accounts. The sample over-represents heavy users and the response rate is 6.7%.
  • ✓Practices of 2-9 people rank AI first more often than practices of 10-19 (20% against 14%, bases 172 and 106), but the difference is not significant at 95%.
  • ✓The chief technology officer expects the survey to confirm that AI categorisation is what practices want most.

The analysis

Recommendation

Build more reliable bank feeds (fewer broken connections) next half-year. It is the #1 priority for paying customers (48.3% first-place, 1.76 mean rank), protects 100% of current revenue from churn, and fixes the root cause of downstream manual work. Do not build AI categorisation next: it alienates 65.5% of paying customers who ranked it last.

What would change it

Evidence that bank feed breakages are primarily external bank API/Open Banking constraints that Ledgerline engineering cannot materially resolve, OR evidence that churn among 20+ seat firms due to missing client portal capabilities exceeds the net retention risk of 2-19 seat firms suffering from feed drops.

Findings

  1. AI categorisation is a trap for the board: its high aggregate first-place vote is driven entirely by non-paying solo users, while paying customers actively reject it.

    High confidenceBase 548

    AI categorisation was ranked 1st by 70.2% of free-plan practices and 60.9% of solo practices, but by only 10.1% of paying practices. Crucially, 65.5% of paying practices ranked AI categorisation dead last (4th), giving it an abysmal mean rank of 3.35 among paying firms.

  2. Bank feed reliability is the undisputed top priority for revenue-generating customers and the overall sample consensus.

    High confidenceBase 548

    Bank feeds achieved the best overall mean rank (1.99) across all respondents and ranked #1 for 48.3% of paid practices, 43.0% of 2-9 person firms, and 49.1% of 10-19 person firms. Only 5.1% of paying practices ranked it last, and 73 open-text answers cited feed drops, delays, or re-auth friction as their primary bottleneck.

  3. Add-on revenue projections of £8/user/month for AI categorisation are economically unviable and based on stated-intent illusions.

    High confidenceBase 548

    Only 16.9% of paying practices stated they would pay £8/user/month, while 57.4% said 'No' and 25.7% 'Not sure'. While 40.5% of free users said 'Yes', free users have zero willingness-to-pay history; assuming industry-standard stated-vs-actual discount rates (typically 70-80% drop-off), add-on uptake would yield negligible MRR while alienating the paying core.

  4. Bulk MTD submissions represent an overlooked, high-impact retention driver for mid-sized commercial practices.

    High confidenceBase 296

    Bulk MTD was ranked #1 by 26.4% of paid practices (representing ~325 paying firms) and 30.2% of 10-19 person practices (ranked 2nd among paid firms with a 2.23 mean rank). In open-text feedback, quarter-end VAT filing bottlenecks were repeatedly highlighted as severe time sinks.

  5. Large practices (20+ staff) show unanimous demand for a client document portal, but the segment is currently too small to justify an immediate roadmap pivot.

    Medium confidenceBase 12

    100% of practices with 20+ staff (12/12) ranked the client portal #1, reinforced by 10 open-text complaints from paid practices regarding chasing client paperwork. However, across the core commercial engine (2-19 staff), portal interest ranked lowest (6.6% to 14.0% first-place votes).

Caveats

  • Sample composition over-indexes on paid plans (54% in survey vs 15% in true population) and daily active users (41.2% in survey vs 18% in August telemetry), caused by recruiting primarily via in-app banners.
  • Unweighted aggregate rankings create a severe Simpson's paradox: aggregating free solo users with multi-seat paying practices masks total divergence in priorities.
  • The 20+ staff practice segment base is critically small (n=12, 2.2% of sample), making their 100% preference for a client portal indicative rather than definitive.
  • Stated willingness-to-pay for AI categorisation (Q3) suffers from hypothetical bias; without card-on-file commitment or price sensitivity testing, actual uptake will be a fraction of stated numbers.

Next steps

  • decisionCommit next half-year core roadmap to Bank Feed Reliability (proactive status dashboard, automated client re-authorisation nudges, and multi-aggregator fallback).
  • researchRun discovery and pilot for Bulk Making Tax Digital (MTD) submissions as the fast-follow improvement.
  • researchConduct targeted qualitative and willingness-to-pay interviews with 20+ staff accounting firms regarding client portal requirements.
  • experimentDe-risk AI categorisation by testing lightweight heuristics/rules-based categorisation within the free tier before considering expensive AI development.

Open-text themes it coded

Bank feed unreliability, disconnections, 90-day re-auth friction, and delayed data 73Manual bank transaction categorisation overhead 56Lack of bulk Making Tax Digital (MTD) filing causing deadline bottlenecks 25Chasing clients for missing paperwork, statements, and receipts 15Application performance, slow loading times, and sluggish search 26High satisfaction with no major blockers 25

The survey it planned

2 screening questions and 8 questions, as the model wrote them.

  1. S1

    Does your organisation operate as an accountancy or bookkeeping practice based in the UK?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    What is your role regarding decisions about the software and technology tools your practice uses?

    One answer
    • I make the final decisions
    • I have significant influence or provide direct input into decisions
    • I use the software but have no influence on purchasing decisions
    • I am not involved in software decisions

    Continues if I make the final decisions; I have significant influence or provide direct input into decisions

  3. Q1

    How many total staff (including partners, qualified accountants, bookkeepers, and administrative staff) work at your practice?

    One answer
    • Sole practitioner (1 person)
    • 2 to 9 staff
    • 10 or more staff
  4. Q2

    Which Ledgerline plan does your practice currently use?

    One answer
    • Free plan (single user)
    • Paid plan (per-user monthly subscription)
    • Not sure
  5. Q3

    On average, how frequently does someone in your practice log in to Ledgerline?

    One answer
    • Multiple times a day
    • Once a day
    • 2 to 4 days a week
    • About once a week
    • A few times a month or less
  6. Q4

    When your practice uses Ledgerline today, which of the following areas cause the most friction or slow you down? (Select up to 2)

    Any that apply
    • Reconnecting broken bank feeds and fixing missed/duplicate transactions
    • Manually categorising and reconciling high volumes of bank transactions
    • Chasing clients for missing receipts, invoices, and bank statements
    • Filing Making Tax Digital (MTD) returns one client at a time
    • Tracking statutory deadlines and filing schedules across clients
    • Managing client contact records and engagement letters
    • None of these cause significant friction
    Judges: double-barrelled

    Several options combine separate tasks under one choice: reconnecting feeds and fixing missed or duplicate transactions; categorising and reconciling; contact records and engagement letters. A selection cannot be traced to a single source of friction, which matters most for separating categorisation from reconciliation, since only categorisation maps to the AI improvement.

    Several options combine distinct tasks (e.g., 'Reconnecting broken bank feeds and fixing missed/duplicate transactions', 'categorising and reconciling'), so it is impossible to tell which part causes the friction.

  7. Q5

    In a sentence or two, what is the single most time-consuming task you or your team spend time on within Ledgerline each week?

    Open text
    Judges: assumptive

    Assumes the practice performs tasks within Ledgerline every week, although Q3 explicitly allows practices using it only a few times a month or less. No display logic excludes those practices.

    It asks what they spend time on 'each week', assuming weekly usage, but Q3 allows respondents to say they log in 'A few times a month or less'.

  8. Q6

    Ledgerline can build ONE major improvement over the next six months. Please rank the following four potential improvements in order of value to your practice (1 = Most valuable, 4 = Least valuable).

    Ranking
    • More reliable bank feeds (fewer broken connections and reduced sync errors)
    • AI categorisation of bank transactions (automatic rule suggestions and smart transaction matching)
    • Client portal for requesting documents (secure area where clients upload requested receipts and records)
    • Bulk Making Tax Digital (MTD) submissions (submitting VAT and tax returns for multiple clients in one batch)
  9. Q7

    If Ledgerline could only release ONE of these four improvements, which single one would you choose as your top priority?

    One answer
    • More reliable bank feeds (fewer broken connections)
    • AI categorisation of bank transactions
    • Client portal for requesting documents
    • Bulk Making Tax Digital (MTD) submissions
  10. Q8

    If Ledgerline introduced AI-assisted categorisation of bank transactions, would your practice pay an additional monthly add-on fee for it?

    One answer
    • Yes, we would pay an additional monthly add-on fee
    • We might pay, depending on the price
    • No, we would only use it if included in our existing plan at no extra cost
    • No, we would not use AI categorisation regardless of price
Sample plan

Target: N=550 verified UK accountancy/bookkeeping practice decision-makers (1 response per practice deduplicated via unique practice account ID/admin email). Recruitment via an in-app banner for active users and direct email invitations to practice admins across all 8,200 accounts. Stratification & Quota Note: Product requested ±5% margin of error at 95% confidence for each practice size (sole practitioner, 2-9 staff, 10+ staff). Achieving ±5% strictly within each of the 3 subgroups would require ~384 completes per group (N=1,152 total). Within the 550-complete constraint, we establish equal-sized quotas of n≈183 per practice size tier (sole practitioners: 183; 2-9 staff: 183; 10+ staff: 184). This yields ±4.1% margin of error overall and ±7.1% within each size tier at 95% confidence. In addition, paid plan accounts (1,230 practices generating 100% of revenue) will be oversampled with a minimum quota of n=200 completes to ensure high commercial validity for roadmapping decisions.