Benchmarks / SurveyBench / What Ledgerline's accountancy practices want built next / DeepSeek V4.1 Flash 1017

Measured by Spring Prompt

DeepSeek V4.1 Flash 1017: What Ledgerline's accountancy practices want built next

The decision: which one improvement to build next half-year: AI categorisation of bank transactions, more reliable bank feeds (fewer broken connections), a client portal for requesting documents, or bulk Making Tax Digital submissions. The brief and the data →

Verdict
✗ Not sound
Research score
89 of 100
Analysis rating
1,221
Head to head, this task
won 9 of 14

Why it is not sound

The survey

  • A research objective the survey could not answer
  • Answer bands that overlap or leave gaps

The analysis

  • Reported a material finding the data does not support

Findings the judges found unsupported

  • Bulk MTD submission is the clear second priority for paying practices and the top operational ask among mid-sized firms; it is close to irrelevant to sole practitioners on the free plan.Bulk MTD is second among paying practices, but calling it the top operational ask among mid-sized firms contradicts their first-choice results: bank feeds lead MTD by 49.1% to 30.2% among 10–19-person practices. The tables also do not provide the joint free-plan/sole-practitioner breakdown used in the claim.
  • The client portal is not the priority for larger firms. The 100% figure for 20+ practices rests on 12 respondents, and among 10-19 person practices the portal is the least-wanted of the four options.The small 20+ base warrants caution, not the conclusion that larger firms do not prioritise a portal. All 12 selecting it first is an observed result, not a rounding artefact. Among 10–19-person practices the portal has the fewest first choices, but that does not establish the lowest overall ranking. The portal is fourth, not third, by overall mean rank.
  • The survey cannot size add-on revenue for AI categorisation. Users per account were not collected, so any figure is an assumption dressed as a forecast; the honest range runs from tens of thousands to low hundreds of thousands of pounds a year, before cannibalisation and churn.The £40,000–£200,000 annual range is an illustrative calculation assuming stated yes responses become purchases and assuming seat counts. It is not an evidence-based revenue range or bound. Acknowledging those assumptions does not establish that actual revenue lies within it.
  • The open text confirms the ranking: broken, disconnected and late bank feeds are the single most-mentioned problem, and feed failure is itself the main generator of the manual coding that free-plan users complain about.The responses establish that feed failures and manual coding often coexist, sometimes with an explicit causal connection. They do not establish feed failure as the main generator of free users' manual-coding complaints; many free users describe coding as an independent problem.
  • Chasing clients for documents is a genuine, recurring pain - mainly for larger paid practices - but it is not what respondents vote for when forced to rank, which argues against a full portal as the next build.Four, not three, 20+ practices supplied document-chasing comments: L215, L509, L540 and L544. The evidence does not establish that document chasing is mainly a larger-practice problem. The portal is fourth overall by mean rank, not third.
  • App slowness is a substantial issue that the four options do not include, so the ballot has forced a choice set that omits a live complaint.There are 26 supplied comments mentioning app slowness, not 25.
  • The sample does not represent the account base and must be weighted before any board statement is made about 'the survey' as a whole.Weighting is not mandatory before reporting clearly labelled respondent results or paid/free comparisons, and weighting to the account mix cannot by itself make this opt-in sample representative. The appropriate weighting depends on whether the decision concerns account-wide preferences or paying customers.
  • The gap between 2-9 and 10-19 person practices on ranking AI first is not statistically significant; the apparent decline with size is noise.A non-significant difference does not establish that the apparent decline is noise; it means the evidence does not reliably distinguish it from sampling variation. Conventional margins of error also do not capture this opt-in survey's selection bias.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of all respondents ranked AI categorisation of bank transactions first?✓37.8%37.8%
Among respondents on a paid plan, what percentage ranked more reliable bank feeds first?✓48.3%48.3%
What is the mean rank of more reliable bank feeds across all respondents (1 = highest priority, 4 = lowest), to two decimal places?✓1.991.99
What percentage of all respondents ranked AI categorisation of bank transactions last (4th)?✓42.2%42.2%
What percentage of all respondents said yes, their practice would pay £8 per user per month for AI categorisation?✓27.7%27.7%
What is the 95% margin of error, in percentage points, on the unweighted share of all respondents who ranked AI categorisation first?✓4.1 points4.06 points
Is the difference in the share ranking AI categorisation first between practices of 2-9 people and practices of 10-19 people statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
How many of the open-text answers mention bank feeds breaking, disconnecting or arriving late?✓7474
How many of the open-text answers from respondents on a paid plan mention chasing clients for documents?✓1010
Ledgerline has 1,230 paid practices. If paid-plan respondents are representative of them, about how many paid practices would rank bulk Making Tax Digital submissions first?✓325324
What percentage of the 8,200 practice accounts invited completed the survey?✓6.7%6.7%

Traps in the data 7 of 7 handled

  • ✓AI categorisation has the most first places (38%, against 35% for bank feeds) but also the most last places (42%): it is polarising. More reliable bank feeds have the best mean rank (1.99, against 2.57 for AI) and the fewest last places (7%).
  • ✓Free-plan users are 46% of respondents but bring in no revenue; paid practices (15% of accounts, all of the revenue) put bank feeds first (48%) and bulk MTD (26%) ahead of AI (10%). The AI first-choice lead comes from free users (70% of whom rank it first). Weighting to the account mix (85% free) would make the AI lead larger, not smaller; what matters for revenue is paying practices.
  • ✓28% said their practice would pay £8 per user per month for AI categorisation (free plan 40%, paid 17%): a stated intention about a hypothetical add-on.
  • ✓Every one of the 12 practices with 20 or more people ranked the client portal first.
  • ✓69% of completes came from an in-app banner seen only by users who logged in; 52% of banner respondents use Ledgerline on 5 or more days a week against 19% of email respondents and 18% of all accounts. The sample over-represents heavy users and the response rate is 6.7%.
  • ✓Practices of 2-9 people rank AI first more often than practices of 10-19 (20% against 14%, bases 172 and 106), but the difference is not significant at 95%.
  • ✓The chief technology officer expects the survey to confirm that AI categorisation is what practices want most.

The analysis

Recommendation

Build more reliable bank feeds next half-year. It is the only option that is a consensus priority rather than a divisive one: mean rank 1.99 overall and 1.76 among paying practices, first choice for 48.3% of paid plans and last for only 7.5% of anyone, and it sits upstream of the manual-coding pain that dominates the free-plan free text (74 of 186 answers describe feeds breaking or arriving late, and the same people describe re-keying the backlog afterwards). Tell the board plainly that the raw first-place count for AI categorisation (37.8%) does not back building it: that lead is produced almost entirely by free-plan respondents (70.2% rank AI first, versus 10.1% of paid plans), while paying practices rank AI last (65.5%) and only 16.9% would pay £8 for it. Do not present an AI add-on revenue number to the finance director from this survey - 1,230 paid practices x 16.9% is about 208 practices, and without seat counts the £8/user/month revenue could be anywhere from roughly £40k to £200k a year on assumptions alone. On the head of sales' question, do not prioritise the client portal on the strength of the 20+ person column: that is 12 people, and among 10-19 person practices the portal is the least-wanted of the four (6.6% first, versus 49.1% for bank feeds and 30.2% for bulk MTD). Bulk MTD submission should be queued as the next substantial build after feeds (26.4% first among paid plans, 30.2% among 10-19 person practices, mean rank 2.23 paid), and 325 of the 1,230 paid practices would put it first. The document-chasing problem is real and worth solving cheaply - automated reminders and a document request list rather than a full portal - and the slowness theme needs a roadmap answer even though it was not on the ballot.

What would change it

Evidence that would overturn this: (a) a properly weighted read of the paid base, or a re-run of Q2 with the paid-plan population alone, showing bank feeds is not first once free-plan respondents are removed; (b) revealed rather than stated demand for AI - for example a paid beta where more than 30-40% of invited paying practices adopt an £8/user/month add-on, or a conjoint/pricing test showing willingness to pay well above the 16.9% stated here; (c) telemetry showing feed disconnections are already rare and short (say under 2% of connections per month, resolved within 24 hours), which would make the perceived problem a communication failure rather than an engineering one; (d) 8-12 structured interviews with 10+ person paid practices showing that a client portal or bulk MTD is a deal-breaker in competitive tenders, or that feed reliability is not raised unprompted in lost-deal reviews; (e) churn or downgrade data showing paying practices leave over missing AI categorisation. If free-to-paid conversion is the commercial goal rather than retention of existing payers, AI categorisation could be reconsidered as an acquisition wedge, but that requires a separate conversion experiment, not this survey.

Findings

  1. AI categorisation's first-place lead is an artefact of free-plan respondents, who are 46% of the sample but 85% of the account base and 0% of the revenue. Among paying practices AI is the lowest priority of the four, ranked last by two-thirds.

    High confidenceBase 548

    AI ranked first: 37.8% overall, but 70.2% of free plan (n=252) vs 10.1% of paid plan (n=296). AI ranked last: 42.2% overall vs 65.5% of paid plans. Mean rank for AI: 1.66 free vs 3.35 paid (bank feeds 1.76 paid). The distribution is bimodal: 37.8% put AI 1st and 42.2% put it 4th.

  2. More reliable bank feeds is the consensus priority: it is ranked first or second by nearly three-quarters of all respondents and last by almost nobody, and it is first among paying practices by a wide margin.

    High confidenceBase 548

    Ranked 1st 34.9%, 2nd 38.9%, 3rd 18.8%, 4th 7.5% (lowest 'last' share of the four options). Mean rank 1.99 vs 2.57 AI, 2.67 bulk MTD, 2.77 portal. Paid plans: 48.3% ranked it first, mean rank 1.76. 10-19 people: 49.1% first.

  3. Bulk MTD submission is the clear second priority for paying practices and the top operational ask among mid-sized firms; it is close to irrelevant to sole practitioners on the free plan.

    High confidenceBase 296

    Bulk MTD ranked first: 26.4% of paid plans vs 3.2% of free plan; 30.2% of 10-19 person practices. Mean rank 2.23 paid vs 3.19 free. Ranked last by 42.5% of free-plan respondents but only 7.8% of paid. Open text: 25 separate answers describe filing VAT returns one client at a time (e.g. L289 'multiply by 90 and you lose a day').

  4. The client portal is not the priority for larger firms. The 100% figure for 20+ practices rests on 12 respondents, and among 10-19 person practices the portal is the least-wanted of the four options.

    High confidenceBase 118

    20 or more people column base is 12 people (2.2% of sample): 100% portal, 0% on all other options - a rounding artefact of a dozen people, not a finding. In the 10-19 band (n=106): portal first 6.6% vs bank feeds 49.1% and bulk MTD 30.2%; portal mean rank 2.77 overall and 2.66 among paid plans, i.e. third of four in both.

  5. Stated willingness to pay £8/user/month for AI categorisation is weak where it matters: fewer than one in five paying practices say yes, and over half say no outright.

    Medium confidenceBase 548

    Paid plans: 16.9% yes, 57.4% no, 25.7% not sure (n=296). 10-19 people: 17.0% yes, 56.6% no. 20+ people: 16.7% yes, 66.7% no. The 27.7% headline yes is inflated by free-plan respondents (40.5% yes) who currently pay nothing. Overall 49.6% say no and 22.6% are unsure.

  6. The survey cannot size add-on revenue for AI categorisation. Users per account were not collected, so any figure is an assumption dressed as a forecast; the honest range runs from tens of thousands to low hundreds of thousands of pounds a year, before cannibalisation and churn.

    Low confidenceBase 296

    1,230 paid practices x 16.9% yes = ~208 practices. At an assumed 4 users per paid practice x £8 = ~£6,700/month (~£80,000/year); at 2 users ~£40,000, at 10 users ~£200,000. Q3 is a hypothetical at a stated price with no purchase commitment, and 25.7% of paid respondents are unsure. Q5 gives size bands only (just me 47.1%, 2-9 31.4%, 10-19 19.3%, 20+ 2.2%), not seat counts.

  7. The open text confirms the ranking: broken, disconnected and late bank feeds are the single most-mentioned problem, and feed failure is itself the main generator of the manual coding that free-plan users complain about.

    High confidenceBase 186

    74 of 186 open answers mention feeds breaking, dropping, needing re-authorisation or arriving days late; 54 mention manual coding/categorisation. Many answers pair the two ('feed breaks... then I code from PDF statements', L495; 'feed was down... manually categorising a few hundred lines', L126; 'transactions go missing and I have to download CSVs', L318). Only 10 paid-plan answers are about chasing documents, 25 about one-at-a-time MTD filing.

  8. Chasing clients for documents is a genuine, recurring pain - mainly for larger paid practices - but it is not what respondents vote for when forced to rank, which argues against a full portal as the next build.

    Medium confidenceBase 186

    15 open answers describe chasing paperwork (10 of them from paying practices, including the only three 20+ practices that gave free text: L215, L509, L540, L544). Yet the portal ranks 3rd of 4 overall (mean 2.77) and 3rd among paid plans (2.66), with 26.6% ranking it last.

  9. App slowness is a substantial issue that the four options do not include, so the ballot has forced a choice set that omits a live complaint.

    Medium confidenceBase 186

    25 open answers raise page load, search, report or dashboard speed (e.g. L279 'fine when we had 50 clients, but since growing to about 300 everything has got noticeably slower'; L309 'opening an individual client record takes 8 to 10 seconds'). 25 of 186 open answers (13%) say nothing slows them down at all.

  10. The sample does not represent the account base and must be weighted before any board statement is made about 'the survey' as a whole.

    High confidenceBase 548

    Q6 shows 46.0% free / 54.0% paid in the sample; the account base is 85% free / 15% paid - paying practices are over-sampled roughly 3.6x. Heavy users are over-represented: 41.2% of respondents use Ledgerline 5+ days a week vs 18% of accounts, driven by the in-app banner (52% of banner respondents use it 5+ days vs 19% of email respondents) which supplied 376 of 548 completes. Response rate 548/8,200 = 6.7%, with self-selection into a topic they care about.

  11. The gap between 2-9 and 10-19 person practices on ranking AI first is not statistically significant; the apparent decline with size is noise.

    High confidenceBase 278

    2-9 people: 20.3% of 172 = 35 respondents; 10-19 people: 14.2% of 106 = 15 respondents. Two-proportion z = 1.31, well below 1.96. Subgroup margins of error are roughly +/-6pp (n=296), +/-8pp (n=172) and +/-9.5pp (n=106); the overall margin on 37.8% is +/-4.1pp.

Caveats

  • The sample is not representative of the account base: 46% free / 54% paid in the data versus 85% free / 15% paid in the records, so paying practices are over-sampled ~3.6x while free practices (who generate no revenue but most of the account count) are under-sampled. Any 'total' column figure must be reweighted before it is presented as the voice of the customer base.
  • Heavy users are over-represented: 41.2% of respondents use Ledgerline 5+ days a week against 18% of accounts. The in-app banner (376 of 548 completes) drew 52% daily users versus 19% from the email invite, so engaged, logged-in users are disproportionately heard; practices who rarely log in (35% of accounts use it less than weekly) are largely missing.
  • Only 6.7% of the 8,200 invited accounts completed, and participation was self-selected around a topic respondents care about. Non-response bias is likely to inflate the salience of whichever problem motivated people to answer.
  • Small subgroup cells are not usable for decisions: the 20+ person column has 12 respondents (2.2%), so its 100%/0% pattern is noise; the 10-19 column (n=106) carries roughly a +/-9.5pp margin and the 2-9 column (n=172) roughly +/-8pp, against +/-4.1pp overall.
  • Q3 measures stated intent at a hypothetical price with no purchase commitment; 22.6% are unsure and the free-plan 'yes' respondents (40.5%) are not currently paying customers at all. Stated-intent inflation is normally large, so treat 16.9% of paid practices as an upper bound.
  • Seat counts per practice were not collected, so add-on revenue cannot be computed. Any £ figure requires an assumption about users per account.
  • The ballot omitted app performance/slowness, which 25 open answers raise unprompted; the ranking therefore forces a choice between four options that may not include the real bottleneck for larger practices.
  • Feed breakage and manual coding are entangled: respondents describe coding as worse precisely because feeds fail and duplicate or drop transactions, so these are not fully independent priorities.
  • One response per practice was kept, so these are practice-level views, generally from the person who handles the work rather than necessarily the budget holder, despite the S1 screen for involvement.

Next steps

  • decisionCommit the next half-year to bank feed reliability (fewer broken connections), scoped with reconnect self-service and clear in-app status of dead connections
  • researchBuild a defensible revenue model for the AI add-on using actual seat counts per paid account, and correct the £8 figure against a real price test rather than Q3 intent
  • experimentRun a paid beta of AI categorisation with a sample of paying practices at £8/user/month and measure actual adoption and 90-day retention of the add-on
  • researchInterview 8-12 decision-makers at 10+ person paid practices specifically on bulk MTD and document chasing, and on what wins and loses competitive tenders
  • experimentShip low-cost document request and automated reminder functionality (chase lists, escalation emails) rather than a full client portal, and measure the effect
  • researchInvestigate and publish a plan for app performance (client list, search, reports, deadline views), which the ballot omitted
  • monitorRebuild the survey instrument and sampling for the next wave: weight to the 85/15 plan mix, stratify the email invite to reach low-usage accounts, collect seat counts, and add performance to the ballot

Open-text themes it coded

Bank feeds breaking, disconnecting, expiring or arriving late 74Manual coding/categorisation of bank transactions is the main time sink 54Making Tax Digital returns must be filed one client at a time 25Chasing clients for documents, records and paperwork 15Slow performance, page loads, search and reports 25Nothing slows us down / fully satisfied 25

The survey it planned

3 screening questions and 8 questions, as the model wrote them.

  1. S1

    Do you work in a UK accountancy or bookkeeping practice that uses Ledgerline?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    Which best describes your role in decisions about the software your practice uses?

    One answer
    • I make the final decision
    • I influence the decision
    • I use the software but do not influence software decisions
    • I do not use Ledgerline

    Continues if I make the final decision; I influence the decision

  3. S3

    Please enter your Ledgerline practice account ID (you can find it under Settings > Account).

    Open text

    Continues if Valid Ledgerline practice account ID

  4. Q1

    In the last month, which of these tasks have caused your practice the most difficulty or taken the most time in Ledgerline? Select up to three.

    Any that apply
    • Bank feeds disconnecting or needing reconnection
    • Categorising or coding bank transactions
    • Chasing clients for documents or information
    • Preparing or submitting Making Tax Digital returns
    • Keeping client records and deadlines up to date
    • Reconciling accounts or reviewing reports
    • Other
  5. Q2

    How often do you experience bank-feed connections that are broken or need reconnecting?

    Scale

    Scale 1-5: Never to Daily or more often

  6. Q3

    Please rank these four possible Ledgerline improvements from 1 (highest priority) to 4 (lowest priority) for your practice over the next six months.

    Ranking
    • AI categorisation of bank transactions (automatically suggests categories for bank transactions)
    • More reliable bank feeds (fewer broken bank connections)
    • Client portal for requesting documents (a secure place for clients to upload documents you request)
    • Bulk Making Tax Digital submissions (submit multiple MTD returns in one go)
  7. Q4

    If Ledgerline could build only one of these improvements in the next six months, which would you choose?

    One answer
    • AI categorisation of bank transactions
    • More reliable bank feeds
    • Client portal for requesting documents
    • Bulk Making Tax Digital submissions
  8. Q5

    If Ledgerline offered AI categorisation of bank transactions as a paid add-on that automatically suggested categories for most transactions, what is the most your practice would be willing to pay per user per month?

    One answer
    • £0 — not willing to pay
    • £1–£5
    • £6–£10
    • £11–£15
    • £16–£20
    • More than £20
    • Not sure
  9. Q6

    What single change to Ledgerline would save your practice the most time?

    Open text
  10. Q7

    How often do you personally log in to Ledgerline?

    One answer
    • Daily
    • 3–4 days a week
    • 1–2 days a week
    • 2–3 times a month
    • Once a month or less
  11. Q8

    Including you, how many people work in your practice?

    One answer
    • 1 (sole practitioner)
    • 2–9
    • 10 or more
Sample plan

Invite all 8,200 practice accounts: email the admin of every account with a unique link pre-populated with the Ledgerline practice account ID, plus in-app banners to logged-in users. Screen for decision-maker/influencer role and valid account ID. De-duplicate by account ID and keep one complete per practice. Target 1,200 completes (400 in each practice-size quota: sole practitioner, 2–9 staff, 10+ staff) because 550 completes cannot meet ±5 percentage points at 95% confidence for all three size groups; 550 gives roughly ±7–8 points per group. If the 550 cap is fixed, use 183 per size as a fallback and label size estimates as not meeting the ±5-point precision requirement. Invite all paid accounts to support plan comparisons, aim for at least 250 paid completes, send two reminders, and weight to the account population by plan and size if response differs.