Benchmarks / SurveyBench / What Ledgerline's accountancy practices want built next / Gemini 4 Carbon

Measured by Spring Prompt

Gemini 4 Carbon: What Ledgerline's accountancy practices want built next

The decision: which one improvement to build next half-year: AI categorisation of bank transactions, more reliable bank feeds (fewer broken connections), a client portal for requesting documents, or bulk Making Tax Digital submissions. The brief and the data →

Verdict
✗ Not sound
Research score
96 of 100
Analysis rating
1,076
Head to head, this task
won 2 of 10

Why it is not sound

The analysis

  • Reported a material finding the data does not support

Findings the judges found unsupported

  • Answer to the CTO: Do not tell the board the survey backs building AI categorisation next. Its first-place plurality is a statistical mirage caused by split-voting among multi-person paid firms, sample over-representation of heavy daily users, and sharp polarisation: AI categorisation is actually ranked dead last (4th) by more respondents than any other option, and by two-thirds of paying practices.The claimed causes of AI's plurality—split-voting among multi-person paid firms and heavy-user over-representation—are not established by the supplied tables. The population-weighting claim also reverses the calculation: weighting mean ranks to 85% free and 15% paid gives AI approximately 1.91 and bank feeds 2.18, so AI ranks better on that account-weighted measure. Revenue-based prioritisation is a different calculation.
  • More reliable bank feeds is the clear consensus priority across both paying customers and the broader user base, and it relieves a major root cause of manual coding complaints.Bank feeds lead across the unweighted respondent sample and among paying respondents, not demonstrably across the broader account population. The supplied plan-weighted means favour AI. The open-text count is also incorrect: 74 answers mention bank-feed problems, not 69.
  • Answer to the Finance Director: Do not bank on £8/user/month add-on revenue from AI categorisation based on the raw 27.7% 'Yes' figure. Paying firms largely reject it, free solo users face a base-plan paywall hurdle, and stated willingness-to-pay overstates actual purchase.The revenue scenarios turn hypothetical willingness to pay into adoption and revenue projections using unsupported assumptions about 4–6 paid seats and a supposedly typical 25%–50% conversion of stated intent. Neither conversion calibration nor seat counts were supplied. A free plan's one-user limit also does not establish that every free practice is a solo practice, and a required base-plan upgrade was not established.
  • Bulk Making Tax Digital (MTD) submissions is the clear second-highest priority for revenue-generating practices, creating acute quarter-end bottlenecks as client counts scale.The open-text answers contain 25 mentions of one-by-one MTD submissions, not 27.
  • Unprompted open-text answers reveal an unmeasured fifth problem—application slowness and page/report load latency—that rivals bulk MTD and exceeds client document chasing.The open-text counts are incorrect: application slowness appears in 26 answers, MTD in 25, and document chasing in 15, including 10 paid practices—not 27, 27, and 14 including 9 paid. The analysis also gives inconsistent counts elsewhere.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of all respondents ranked AI categorisation of bank transactions first?✓37.8%37.8%
Among respondents on a paid plan, what percentage ranked more reliable bank feeds first?✓48.3%48.3%
What is the mean rank of more reliable bank feeds across all respondents (1 = highest priority, 4 = lowest), to two decimal places?✓1.991.99
What percentage of all respondents ranked AI categorisation of bank transactions last (4th)?✓42.2%42.2%
What percentage of all respondents said yes, their practice would pay £8 per user per month for AI categorisation?✓27.7%27.7%
What is the 95% margin of error, in percentage points, on the unweighted share of all respondents who ranked AI categorisation first?✓3.9 points4.06 points
Is the difference in the share ranking AI categorisation first between practices of 2-9 people and practices of 10-19 people statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
How many of the open-text answers mention bank feeds breaking, disconnecting or arriving late?✓6974
How many of the open-text answers from respondents on a paid plan mention chasing clients for documents?✓910
Ledgerline has 1,230 paid practices. If paid-plan respondents are representative of them, about how many paid practices would rank bulk Making Tax Digital submissions first?✓325324
What percentage of the 8,200 practice accounts invited completed the survey?✓6.7%6.7%

Traps in the data 7 of 7 handled

  • ✓AI categorisation has the most first places (38%, against 35% for bank feeds) but also the most last places (42%): it is polarising. More reliable bank feeds have the best mean rank (1.99, against 2.57 for AI) and the fewest last places (7%).
  • ✓Free-plan users are 46% of respondents but bring in no revenue; paid practices (15% of accounts, all of the revenue) put bank feeds first (48%) and bulk MTD (26%) ahead of AI (10%). The AI first-choice lead comes from free users (70% of whom rank it first). Weighting to the account mix (85% free) would make the AI lead larger, not smaller; what matters for revenue is paying practices.
  • ✓28% said their practice would pay £8 per user per month for AI categorisation (free plan 40%, paid 17%): a stated intention about a hypothetical add-on.
  • ✓Every one of the 12 practices with 20 or more people ranked the client portal first.
  • ✓69% of completes came from an in-app banner seen only by users who logged in; 52% of banner respondents use Ledgerline on 5 or more days a week against 19% of email respondents and 18% of all accounts. The sample over-represents heavy users and the response rate is 6.7%.
  • ✓Practices of 2-9 people rank AI first more often than practices of 10-19 (20% against 14%, bases 172 and 106), but the difference is not significant at 95%.
  • ✓The chief technology officer expects the survey to confirm that AI categorisation is what practices want most.

The analysis

Recommendation

Build 'More reliable bank feeds (fewer broken connections)' as the committed improvement next half-year—scoping it specifically around feed stability, proactive disconnection alerts, frictionless one-click client re-authorisation chasers, and missing/duplicate transaction safeguards—while tackling quick-win database/query optimisations for slow page and report loading alongside it, and queuing Bulk MTD Submissions as the top candidate for the following cycle.

What would change it

Two pieces of evidence would shift the build decision: (1) An engineering spike showing that >80% of bank feed failures and delays stem from unfixable upstream bank API limits and mandatory Open Banking 90-day rules where proactive alerting/re-auth UX and aggregator changes cannot materially reduce downtime—in which case pivot the half-year slot to Bulk MTD Submissions (the clear #2 priority for paid firms at 26.4% first-place / 2.23 mean rank); or (2) A live pricing/upgrade smoke test proving that solo free-plan users actually convert to paid plans + £8/mo AI categorisation at a net new ARR rate that outweighs retention risk across the 1,230 paid practices.

Findings

  1. Answer to the CTO: Do not tell the board the survey backs building AI categorisation next. Its first-place plurality is a statistical mirage caused by split-voting among multi-person paid firms, sample over-representation of heavy daily users, and sharp polarisation: AI categorisation is actually ranked dead last (4th) by more respondents than any other option, and by two-thirds of paying practices.

    High confidenceBase 548

    Unweighted, AI categorisation wins 37.8% of first-place votes (n=548) vs 34.9% for reliable bank feeds, 15.7% for bulk MTD, and 11.7% for client portal. However, 42.2% of all respondents and 65.5% of paid practices (n=296) rank AI categorisation 4th (lowest priority), giving it a worse overall mean rank (2.57) than reliable bank feeds (1.99; paid mean 1.76). Support comes overwhelmingly from solo free-tier users (70.2% of free plan rank it 1st, n=252; 60.9% of solo practices, n=258) vs just 10.1% of paid practices (n=296). Even reweighting by plan tier alone (85% free, 15% paid) gives reliable bank feeds a better population mean rank (2.18) than bank feeds' nearest rivals when accounting for paid revenue weight—where 100% of current revenue sits.

  2. More reliable bank feeds is the clear consensus priority across both paying customers and the broader user base, and it relieves a major root cause of manual coding complaints.

    High confidenceBase 548

    Among paid practices (n=296, representing 100% of revenue), 48.3% rank reliable bank feeds 1st and only 5.1% rank it last (mean rank 1.76). Across all respondents (n=548), 73.7% rank reliable bank feeds 1st or 2nd, and only 7.5% rank it last (mean rank 1.99). In open-text responses (n=186), broken, disconnected, missing, or delayed bank feeds are the #1 complaint (69 mentions, 37.1%), with 22 respondents explicitly stating that manual coding surges because feeds drop out, expire every 90 days, or arrive late.

  3. Answer to the Finance Director: Do not bank on £8/user/month add-on revenue from AI categorisation based on the raw 27.7% 'Yes' figure. Paying firms largely reject it, free solo users face a base-plan paywall hurdle, and stated willingness-to-pay overstates actual purchase.

    Medium confidenceBase 548

    Only 16.9% of paid practices (50 of 296) say 'Yes' to paying £8/user/month (57.4% say 'No'), and willingness to pay drops as firm size (and seat count) grows: 36.8% for 'Just me' (n=258), 21.5% for 2–9 people (n=172), 17.0% for 10–19 people (n=106), and 16.7% for 20+ people (n=12). Even if paid respondents matched all 1,230 paid practices at stated face value (~208 practices) and averaged ~4–6 paid seats, unadjusted stated intent among paid accounts implies roughly £80k–£120k/year—and realistic B2B conversion of stated 'Yes' (typically 25%–50%) cuts that to ~£20k–£60k/year from the paid base. While 40.5% of free practices (n=252, all 1-user capped) say 'Yes'—theoretically ~2,820 of 6,970 free accounts (£271k/year if sold standalone at £8/mo)—the survey did not test whether free users must first upgrade to a paid base plan, nor adjust for heavy-user sample skew (45.2% of free respondents use Ledgerline 5+ days/week vs 18% of all accounts).

  4. Answer to the Head of Sales: The survey cannot bear the weight of prioritising the client portal to win larger firms. The 100% figure for 20+ person firms rests on just 12 existing customers, while 10–19 person firms strongly prefer reliable bank feeds and bulk MTD.

    Low confidenceBase 118

    All 12 respondents in practices of 20+ people ranked the client portal 1st (100.0%, 95% Wilson CI ~76%–100%) and 4 of those 20+ practices mentioned chasing client paperwork in Q1 (L215, L509, L540, L544). However, n=12 is a tiny base of existing accounts (zero non-customer prospects were surveyed), and adjacent mid-sized practices of 10–19 people (n=106) ranked the client portal dead last among first-place choices at just 6.6% (27.4% ranked it 4th), heavily favouring reliable bank feeds (49.1% 1st) and bulk MTD submissions (30.2% 1st).

  5. Bulk Making Tax Digital (MTD) submissions is the clear second-highest priority for revenue-generating practices, creating acute quarter-end bottlenecks as client counts scale.

    High confidenceBase 296

    26.4% of paid practices rank bulk MTD 1st (mean rank 2.23, second only to bank feeds at 1.76; only 7.8% rank it last), rising from 5.8% in solo practices to 22.7% in 2–9 person practices and 30.2% in 10–19 person practices. Extrapolated directly to 1,230 paid practices, ~325 paid firms rank it #1. Open-text responses contain 27 mentions of tedious one-by-one MTD filing eating entire days around the 7th of the month and VAT quarter-ends.

  6. Unprompted open-text answers reveal an unmeasured fifth problem—application slowness and page/report load latency—that rivals bulk MTD and exceeds client document chasing.

    Medium confidenceBase 186

    27 of 186 Q1 respondents (14.5%, spanning free and paid plans across solo to 10–19 person firms) complained of slow page loads, sluggish search, lagging deadline views, and report timeouts—compared with 27 mentions of individual MTD filing and just 14 mentions (9 paid) of chasing client documents. Because app speed was omitted from Q2, its priority relative to feature work is unquantified.

Caveats

  • Severe unweighted sample skew on plan tier and usage frequency: Paid plans are 54.0% of respondents vs 15.0% of the 8,200 account population, and 5+ day/week users are 41.2% of respondents vs 18.0% of the population (driven by 69% of completes coming via the in-app banner). Unweighted totals represent neither revenue (100% paid) nor all accounts (85% free, 35% <1 day/week).
  • Stated willingness to pay (Q3) is uncalibrated and structurally ambiguous for free users: Hypothetical single-price questions ('Would you pay £8/user/month?') routinely overstate real adoption by 2x–4x, we lack exact paid-seat counts per respondent account to turn practice-level 'Yes' into exact user-level MRR, and Q3 did not specify whether free-plan practices (capped at 1 user, £0/month) could buy a standalone £8 add-on or would also have to upgrade to a paid base plan.
  • Tiny sample and survivorship bias for large firms: Only 12 practices of 20+ people completed the survey, and all respondents are existing Ledgerline users—giving zero direct evidence on why non-customer mid-tier/large firms choose competitors in sales deals.
  • Forced ranking (Q2) measures relative preference among four pre-selected options, not absolute engineering ROI or feasibility: Some bank feed issues (e.g., Open Banking 90-day re-authorisation rules or upstream bank API outages at Barclays/Monzo/Santander/Tide) may only be partially solvable via UX/alerting/aggregator changes rather than 100% eliminable, and app performance/speed (26 open-text mentions) was excluded from Q2.
  • Non-response bias: 6.7% overall completion rate (548 of 8,200 practices); the 35% of practices logging in less than once a week are heavily under-represented (12.0% of sample), masking potential churn or activation blockers among dormant accounts.

Next steps

  • decisionCommit the H1 roadmap slot to Bank Feed Reliability, starting with a 2-week technical root-cause audit of feed drops, missing lines, and late syncs by bank/aggregator vs UX gaps (silent disconnects and 90-day expiry friction).
  • experimentCompute exact paid-seat-weighted priority and revenue models by joining the 296 paid survey responses (or Q6/size segments) to internal billing records of seat counts per practice, and run a live painted-door / pre-order pricing test on AI categorisation (testing standalone £8 add-on vs bundling into paid plans to drive free-to-paid conversion).
  • researchAudit CRM win/loss logs and run 12–15 qualitative discovery interviews with non-customer prospects and lost deals of 10–50+ staff.
  • monitorInstrument application telemetry (P95 load times on client list, search, deadline views, and reports, especially around 9am and month/quarter-end peaks) and ship targeted query/indexing fixes outside the main feature slot.

Open-text themes it coded

Broken, disconnected, missing, or delayed bank feeds 74Manual categorisation and coding of bank transactions 59Lack of bulk Making Tax Digital (MTD) / VAT submissions 25Slow application performance, page loads, search, and reports 26Chasing clients for missing documents, statements, and receipts 15Nothing slows the practice down / satisfied with Ledgerline 25

The survey it planned

3 screening questions and 8 questions, as the model wrote them.

  1. S1

    Is your accountancy or bookkeeping practice based in the UK?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    Does your practice currently use Ledgerline?

    One answer
    • Yes
    • No

    Continues if Yes

  3. S3

    Which of the following best describes your involvement in choosing or paying for software for your practice?

    One answer
    • I make the final decision on software for the practice
    • I share or influence decisions on software for the practice
    • I have no involvement in software decisions

    Continues if I make the final decision on software for the practice; I share or influence decisions on software for the practice

  4. Q1

    Including yourself, how many people work in your practice (full-time or part-time)?

    One answer
    • 1 (sole practitioner, no other staff)
    • 2 to 9 people
    • 10 or more people
  5. Q2

    Which Ledgerline plan is your practice currently on?

    One answer
    • Free plan (1 user)
    • Paid plan (charged per user per month)
    • Not sure
  6. Q3

    In a typical working week, how often do you or members of your practice use Ledgerline?

    One answer
    • Every working day
    • 2 to 4 days a week
    • About once a week
    • Less than once a week
  7. Q4

    Over the past 6 months, which of the following have slowed your practice down when using Ledgerline? Select all that apply.

    Any that apply
    • Broken bank-feed connections or having to reconnect bank feeds
    • Categorising bank transactions manually
    • Chasing clients for documents or records outside Ledgerline
    • Submitting Making Tax Digital (MTD) returns one client at a time
    • Updating client records or tracking deadlines
    • Slow loading speeds or general technical errors
    • Something else in Ledgerline
    • None of these — Ledgerline has not slowed us down
  8. Q5

    In a sentence or two, what is the single biggest thing that slows your practice down in Ledgerline today, and how does it affect your work?

    Open text

    Routing Shown only if Q4 is: Broken bank-feed connections or having to reconnect bank feeds; Categorising bank transactions manually; Chasing clients for documents or records outside Ledgerline; Submitting Making Tax Digital (MTD) returns one client at a time; Updating client records or tracking deadlines; Slow loading speeds or general technical errors; Something else in Ledgerline

  9. Q6

    We have room to build one major improvement to Ledgerline over the next six months. Please rank these four options from 1 (most valuable to your practice) to 4 (least valuable to your practice).

    Ranking
    • AI categorisation of bank transactions (automatically suggests categories for bank lines)
    • More reliable bank feeds (fewer broken connections and reconnections)
    • A client portal for requesting and receiving documents from clients
    • Bulk Making Tax Digital (MTD) submissions across multiple clients at once
  10. Q7

    Suppose AI categorisation of bank transactions were offered as an optional paid add-on (charged on top of your current free or paid plan), rather than included at no extra cost. What would your practice most likely do?

    One answer
    • We would very likely pay extra for it as an add-on
    • We might consider paying extra, depending on the price
    • We would only use it if it were included in our plan at no extra cost
    • We would not use it, even at no extra cost
  11. Q8

    What is the highest extra fee per user per month (excluding VAT) your practice would realistically pay for an AI bank-transaction categorisation add-on?

    One answer
    • Up to £5 per user per month
    • £6 to £10 per user per month
    • £11 to £20 per user per month
    • More than £20 per user per month
    • On reflection, we would not pay an extra monthly fee

    Routing Shown only if Q7 is: We would very likely pay extra for it as an add-on; We might consider paying extra, depending on the price

Sample plan

INVITATIONS & DEDUPLICATION: Invite all 8,200 UK practice accounts via (1) a personalised email to the account admin and (2) an in-app login banner shown to logged-in users. Embed a hashed practice_id token in both channels so the survey platform blocks duplicate entries once one qualified decision-maker from a practice completes the survey (if simultaneous entries occur, retain the account admin's complete). Randomise option display order in Q4 (first six items, keeping 'None' exclusive at bottom) and Q6 (all four improvements) to prevent order bias and counter stakeholder confirmation bias. TARGET COMPLETES, QUOTAS & PRECISION FEASIBILITY NOTE: A strict +/-5 percentage point margin of error at 95% confidence requires ~385 completes per subgroup in large populations (~785–1,155 total even after finite-population correction), which cannot be met across all three practice-size bands simultaneously within a ~550-complete cap. Within ~550 completes, enforce stratified quotas by practice size (Q1) and plan (Q2/account billing flag): target ~200 sole practitioners, ~220 practices with 2–9 people, and ~130 practices with 10+ people (assuming a small finite pool of 10+ staff practices within the 1,230 paid accounts), while ensuring at least 260 completes come from Paid plans (which represent 15% of accounts but 100% of revenue) and ~290 from Free plans. At n=550, expect +/-6.8 pp for sole practitioners, ~+/-6.0 pp for 2–9 staff (with finite-population correction), and ~+/-5.0–6.0 pp for 10+ staff (with finite-population correction); if strict +/-5.0 pp in every size band is mandatory, raise the total cap to ~800–900 completes and send targeted reminder emails to 2–9 and 10+ paid accounts.