Benchmarks / SurveyBench / What Ledgerline's accountancy practices want built next / Claude Fable 5.2

Measured by Spring Prompt

Claude Fable 5.2: What Ledgerline's accountancy practices want built next

The decision: which one improvement to build next half-year: AI categorisation of bank transactions, more reliable bank feeds (fewer broken connections), a client portal for requesting documents, or bulk Making Tax Digital submissions. The brief and the data →

Verdict: ✗ Not sound
Research score: 94 of 100
Analysis rating: 1,683
Head to head, this task: won 9 of 10

Why it is not sound

The survey

Answer bands that overlap or leave gaps

Scorecard

Numbers 11 of 11 right

Question		Its answer	True
What percentage of all respondents ranked AI categorisation of bank transactions first?	✓	37.8%	37.8%
Among respondents on a paid plan, what percentage ranked more reliable bank feeds first?	✓	48.3%	48.3%
What is the mean rank of more reliable bank feeds across all respondents (1 = highest priority, 4 = lowest), to two decimal places?	✓	1.99	1.99
What percentage of all respondents ranked AI categorisation of bank transactions last (4th)?	✓	42.2%	42.2%
What percentage of all respondents said yes, their practice would pay £8 per user per month for AI categorisation?	✓	27.7%	27.7%
What is the 95% margin of error, in percentage points, on the unweighted share of all respondents who ranked AI categorisation first?	✓	4.1 points	4.06 points
Is the difference in the share ranking AI categorisation first between practices of 2-9 people and practices of 10-19 people statistically significant at the 95% level? (1 for yes, 0 for no)	✓	No	No
How many of the open-text answers mention bank feeds breaking, disconnecting or arriving late?	✓	74	74
How many of the open-text answers from respondents on a paid plan mention chasing clients for documents?	✓	10	10
Ledgerline has 1,230 paid practices. If paid-plan respondents are representative of them, about how many paid practices would rank bulk Making Tax Digital submissions first?	✓	325	324
What percentage of the 8,200 practice accounts invited completed the survey?	✓	6.7%	6.7%

Traps in the data 7 of 7 handled

✓AI categorisation has the most first places (38%, against 35% for bank feeds) but also the most last places (42%): it is polarising. More reliable bank feeds have the best mean rank (1.99, against 2.57 for AI) and the fewest last places (7%).
✓Free-plan users are 46% of respondents but bring in no revenue; paid practices (15% of accounts, all of the revenue) put bank feeds first (48%) and bulk MTD (26%) ahead of AI (10%). The AI first-choice lead comes from free users (70% of whom rank it first). Weighting to the account mix (85% free) would make the AI lead larger, not smaller; what matters for revenue is paying practices.
✓28% said their practice would pay £8 per user per month for AI categorisation (free plan 40%, paid 17%): a stated intention about a hypothetical add-on.
✓Every one of the 12 practices with 20 or more people ranked the client portal first.
✓69% of completes came from an in-app banner seen only by users who logged in; 52% of banner respondents use Ledgerline on 5 or more days a week against 19% of email respondents and 18% of all accounts. The sample over-represents heavy users and the response rate is 6.7%.
✓Practices of 2-9 people rank AI first more often than practices of 10-19 (20% against 14%, bases 172 and 106), but the difference is not significant at 95%.
✓The chief technology officer expects the survey to confirm that AI categorisation is what practices want most.

The analysis

Recommendation

Build more reliable bank feeds next half-year. It is the first choice of the practices that generate all the revenue (48.3% first, mean rank 1.76, 5.1% last), the second choice of free practices, the best-ranked option overall (1.99), almost nobody's last choice, and the most common unprompted complaint (74 of 186). Scope it around what respondents describe: visible alerts when a connection drops, an easier client re-authorisation flow, and fewer late, missing or duplicated transactions. Before locking scope, confirm from feed logs how much of the failure is within Ledgerline's control. Do not present AI categorisation to the board as survey-backed: its support is concentrated in free solo accounts, paying practices rank it last, and its lead in first-place votes is within the margin of error. If the company wants AI as a free-to-paid conversion play, that is a strategic choice to validate with a live price test, not this survey. Hold bulk MTD as next in line (26.4% of paid first, rising with size). Treat the client portal as a discovery item for 20+ firms, not a build commitment.

What would change it

Feed-log analysis showing most disconnections are caused by banks or mandatory re-consent and Ledgerline can do little about them (alerts, re-authorisation flow, de-duplication); bulk MTD would then become the lead candidate for paying practices. 2) An explicit strategic goal of converting free accounts to paid, together with a live price test showing real take-up of the AI add-on close to the stated 40% among free users; that would make the case for AI. 3) A re-cut by seats or revenue, or churn and support-ticket data, showing a different option dominates among the highest-revenue accounts, for example the portal among 20+ firms holding a large share of seats or MTD among the 10-19 band. 4) Win/loss evidence that larger prospects are being lost specifically for lack of a client portal, alongside a decision that moving up-market is the half-year goal. 5) A weighted re-analysis including low-usage paid accounts that overturns the paid-plan ranking; unlikely given the size of the gap, but untested.

Findings

CTO: no, do not tell the board the survey backs AI categorisation. Its first-place lead is within sampling error of bank feeds, and it is also the option most often ranked last. Bank feeds is the consensus choice.

High confidenceBase 548

AI was ranked first by 37.8% (±4.1 points) and bank feeds by 34.9%; the 2.9-point gap is not statistically significant (z about 0.8). AI was ranked last by 42.2%, bank feeds by 7.5%. Bank feeds was in the top two for 73.8%, AI for 47.3%. Mean rank: bank feeds 1.99, AI 2.57, bulk MTD 2.67, portal 2.77.
AI's first-place votes come almost entirely from free-plan practices, which bring in no revenue. Paying practices put it at the bottom. The 'Total' column blends two opposed groups and should not be quoted as customer demand.

High confidenceBase 548

AI ranked first: free plan 70.2% (about 177 of 252), paid 10.1% (about 30 of 296), so roughly 85% of AI's first-place votes are from free accounts. Among paid, 65.5% ranked AI last and its mean rank was 3.35, the worst of the four. The sample is 54% paid against 15% of accounts. Re-weighted to the account mix AI would be about 61% first; restricted to revenue-bearing accounts it is 10%.
Paying practices clearly want more reliable bank feeds first, and bulk MTD second. Bank feeds is also the free plan's second choice, so it is the one option that serves both groups.

High confidenceBase 296

Paid plan (base 296, about 24% of all 1,230 paid practices; plan verified against account records): bank feeds first 48.3% (±5.7 points), bulk MTD 26.4% (±5.0), portal 15.2%, AI 10.1%. Mean ranks 1.76, 2.23, 2.66, 3.35. Only 5.1% ranked feeds last. Free plan: feeds mean rank 2.25 (second), 19.0% first, 10.3% last. Open text from paid respondents (96 answers): feeds 51, MTD 22, manual coding 16, slow performance 11, chasing documents 10, nothing 8.
Broken, late or missing bank feeds are the most common unprompted complaint. Much of the manual-coding pain, especially among paying practices, is a knock-on effect of feed failures, so fixing feeds also relieves part of the problem AI is meant to solve.

Medium confidenceBase 186

74 of 186 open-text answers (40%) mention feeds, against 56 for manual coding, 26 for slowness, 25 for one-at-a-time MTD and 15 for chasing documents. 25 of the 56 coding mentions tie the coding to a feed outage or backlog. Among paid respondents that is 13 of 16 coding mentions; only 3 paid answers raise coding on its own. Recurring specifics: silent disconnections, clients having to re-authorise, transactions arriving days late, missing or duplicated lines. Open text is from a self-selected third of respondents, so it indicates rather than measures.
Finance director: the survey cannot give a usable revenue figure for the AI add-on. Stated willingness to pay is low among the practices that actually pay, and the free-plan 'yes' answers come from heavy users of a product they currently get for nothing.

Low confidenceBase 548

Would pay £8 per user per month: 27.7% overall, paid 16.9% (about 50 of 296, ±4.3 points), free 40.5% (about 102 of 252). Among paid, 57.4% said no and 25.7% not sure. Taken at face value, 16.9% of 1,230 is about 208 paid practices (seat counts not collected) and 40.5% of 6,970 free accounts is about 2,800 single seats, roughly £270k a year. That is an unrealistic ceiling: hypothetical intent, a single price point, and 45% of free respondents use the product 5 or more days a week against 18% of all accounts. Do not put it in a forecast.
Head of sales: the largest practices in the sample do all want the client portal, but that rests on 12 respondents. It does not extend to 10-19-person practices, and it says nothing about prospects. Treat it as a lead for targeted discovery, not a basis for the half-year priority.

Low confidenceBase 12

20 or more people (base 12): 12 of 12 ranked the portal first, none last, and 9 of 12 ranked AI last. With 12 respondents the plausible range is still roughly 74-100%. All four open-text answers from this group are about chasing clients for documents. But 10-19 people (base 106): portal first 6.6%, last 27.4%. Paid overall: 15.2% first. The survey covered existing accounts only and did not ask about buying or switching.
Bulk MTD is the clear runner-up for paying practices and matters more as practices grow. It is the natural candidate for the following half, or the fallback if feed reliability proves largely outside Ledgerline's control.

Medium confidenceBase 296

Bulk MTD ranked first: paid 26.4%, free 3.2%; Just me 5.8%, 2-9 people 22.7%, 10-19 people 30.2%. Only 7.8% of paid ranked it last. If paid respondents are representative, that is about 325 of 1,230 paid practices, roughly 265-385. 25 open-text answers describe one-at-a-time filing, 22 of them from paid respondents, often citing 40-140 VAT clients due on the same day.
General slowness is an unprompted pain point that is not among the four options. It appears as often as MTD in the open text and seems to worsen as client lists grow.

Low confidenceBase 186

26 of 186 open-text answers (14%; 11 paid, 15 free) describe slow page loads, search, reports or the deadlines and client screens. Several link it to scale (growing from 50 to about 300 clients; 8-10 seconds to open a client record with 200 clients). It was not offered in Q2, so its priority against the four options is unmeasured.
The meaningful split on AI is solo practices against everyone else, not a gradient by size. The gap between 2-9 and 10-19 people is not statistically significant.

Medium confidenceBase 536

AI ranked first: Just me 60.9% (base 258), 2-9 people 20.3% (base 172), 10-19 people 14.2% (base 106). For 2-9 against 10-19 the difference is 6.1 points, z about 1.3, not significant at 95%. Size and plan are confounded and no plan-by-size table was produced.

Caveats

The sample does not mirror the customer base. Paid practices are 54% of respondents but 15% of accounts (about 24% of paid practices responded, against about 3.6% of free). 'Total' figures answer neither 'what do customers want' nor 'what does revenue want'; read the plan columns separately.
Respondents skew heavily to frequent users: 41% use Ledgerline 5 or more days a week against 18% of accounts, and 12% use it less than weekly against 35%. 69% of completes came from the in-app banner, which only reaches people who log in. Light and lapsing users are barely represented.
The overall response rate is 6.7% (548 of 8,200) and the tables are unweighted. The margins of error quoted cover sampling error only, not this self-selection.
Q3 is hypothetical stated intent at a single price, with no seat counts collected. The survey did not establish whether free-plan practices could or would buy an add-on without upgrading. Such questions typically overstate real purchase.
The 20-or-more-people group is 12 respondents. Its 100% and 0% figures show direction only, and the number and revenue of such practices in the base was not provided.
Plan and practice size are confounded (free skews solo) and no plan-by-size, channel or usage cuts of Q2 were produced, so size effects cannot be separated from plan effects.
Q2 is a forced ranking of four pre-chosen options. It shows relative order, not strength of need, and could not surface performance, which the open text raises unprompted.
Q1 asked what slows practices down 'when using Ledgerline'. That may under-report problems that happen outside the product, such as chasing clients for documents. Only 186 of 548 answered, and theme counts rest on my coding of free text.
One response per practice (the first received) stands for the whole practice. In larger practices that person's view may not be the firm's.
The survey covers existing accounts only. It cannot speak to what would win new or larger firms, and it did not measure churn risk or whether any option would change renewal, upgrade or seat growth.
Some feed failures may sit with the banks or the open-banking re-consent rules rather than Ledgerline (two respondents say they cannot tell which). The survey shows the pain, not how much of it is fixable.

Next steps

decisionCommit the next half-year to bank-feed reliability: disconnection alerts, a simpler client re-authorisation flow, and fewer late, missing or duplicated transactions.
researchBefore scope is locked, analyse feed logs and support tickets to break failures down by cause (consent expiry, bank-side outage, Ledgerline fault) and by bank; respondents name Barclays, Monzo, Santander, Starling and Tide.
researchRe-run the Q2 and Q3 tables by channel (banner or email), usage frequency and plan by size, and weight paid respondents to the paid account profile, including by seats or revenue.
experimentRun an in-product fake-door or pre-order test for AI categorisation at £8 per user per month, shown separately to free and paid accounts including light users, with seat counts drawn from account records.
researchPull the count, seats and revenue of 20+ practices from account records, review win/loss notes for larger prospects, and interview 8-10 larger firms (customers and lost deals) about document collection and what decides their purchase.
monitorInstrument and track page-load, search and report times by account size (number of clients), and track performance-related tickets.
monitorKeep bulk MTD as the lead candidate for the following half: track submissions per practice on peak days from product data and do light scoping in parallel.

Open-text themes it coded

Bank feeds breaking, disconnecting, needing re-authorisation, or transactions arriving late or missing 74Manual coding or categorisation of bank transactions 56MTD/VAT returns have to be submitted one client at a time (no bulk filing) 25Chasing clients for documents and records 15Slow performance (page loads, search, reports, lag) - not among the four options 26Nothing slows us down / satisfied 25

The survey it planned

3 screening questions and 8 questions, as the model wrote them.

S1
Which of these best describes the organisation you work for?
One answer
- An accountancy practice based in the UK
- A bookkeeping practice based in the UK
- A practice based in the UK that offers both accountancy and bookkeeping
- A practice based outside the UK
- Something else (for example, the finance team of a business)
Continues if An accountancy practice based in the UK; A bookkeeping practice based in the UK; A practice based in the UK that offers both accountancy and bookkeeping
S2
When did your practice last use Ledgerline for client work?
One answer
- In the past 7 days
- More than a week ago, but in the past month
- More than a month ago, but in the past 3 months
- More than 3 months ago
- We have never used Ledgerline for client work
Continues if In the past 7 days; More than a week ago, but in the past month; More than a month ago, but in the past 3 months
S3
What part do you play when your practice chooses or changes the software it uses?
One answer
- I make the final decision
- I make the decision jointly with others
- I give input or recommendations, but others decide
- I am not involved in these decisions
Continues if I make the final decision; I make the decision jointly with others; I give input or recommendations, but others decide
Q1
Thinking about the past 3 months, what one thing has slowed your practice down most when using Ledgerline? A sentence is plenty. If nothing has slowed you down, please write 'nothing'.
Open text
Q2
In the past 3 months, which of these, if any, have slowed your practice down when using Ledgerline? Please select all that apply.
Any that apply
- Bank feeds disconnecting or having to be reconnected
- Categorising bank transactions by hand
- Getting documents or information from clients
- Submitting Making Tax Digital (MTD) returns one client at a time
- Keeping track of deadlines
- Setting up or updating client records
- Ledgerline being slow or unavailable
- Finding the feature or information I need in Ledgerline
- Something else not listed here
- Nothing has slowed us down in the past 3 months
Q3
Ledgerline is deciding which one of these four improvements to build next. Please rank them from 1 (most useful to your practice) to 4 (least useful to your practice).
Ranking
- More reliable bank feeds – fewer connections that break and have to be reconnected
- Suggested categories for bank transactions – AI proposes a category for each transaction, for you to check
- Client portal – request documents from clients and have them upload them in one place
- Bulk Making Tax Digital (MTD) submissions – submit for several clients in one go
Q4
Now think about the improvement you ranked 1. How much difference would it make to your practice's day-to-day work?
Scale

Scale 1-5: No difference at all to A very big difference
Q5
Suppose suggested categories for bank transactions (AI proposes a category for each transaction, for you to check) were offered as an optional extra that costs more than your current plan. What is the most your practice would pay for it, per user per month?
One answer
- We would not use it, even at no extra cost
- We would use it only at no extra cost
- Up to £3 per user per month
- More than £3, up to £6 per user per month
- More than £6, up to £10 per user per month
- More than £10 per user per month
- Not sure
Q6
Including yourself, how many people work in your practice? Please count partners, employees and regular contractors, whether full-time or part-time.
One answer
- 1 (just me)
- 2 to 4
- 5 to 9
- 10 to 19
- 20 or more
Q7
About what share of your practice's clients do you manage in Ledgerline?
One answer
- Fewer than a quarter
- A quarter to a half
- More than half, but not all
- All or nearly all
- Not sure
Q8
Which Ledgerline plan is your practice on?
One answer
- The free plan
- A paid plan
- Not sure

Sample plan

WHO IS INVITED: All 8,200 practice accounts (a census, not a sample). Each account admin gets an email with a unique link, and all users see an in-app banner at login. Both routes carry the practice account ID. The invitation states that answers will be linked to the account. The banner is switched off for a practice once it has a complete.

ONE RESPONSE PER PRACTICE: If several people from one practice complete, keep the one with the most senior software-decision role (S3), then the earliest.

FIELDWORK: 2 to 3 weeks with two reminders, avoiding the weeks before the 31 January self-assessment deadline and MTD quarterly deadlines, when workload would distort both response and reported pain points. Soft-launch to about 50 practices first to check timing (target 4 minutes) and the spread of answers across the Q5 price bands.

PRESENTATION: Option order is randomised per respondent in Q2 (keeping 'Something else' and 'Nothing' last) and in Q3, so no improvement benefits from position. Q1 (open) comes before any list so the first answer is unprompted. The AI price question (Q5) comes after the ranking so it cannot prime it. There is no attention-check question, because the 8 slots are needed for content. Quality is controlled by removing speeders (under about 60 seconds), nonsense open text and duplicate practices.

COMPLETES AND PRECISION: The stated precision cannot be met with 550 completes.

Requirement: ±5 points at 95% confidence within each of three size groups needs about 385 completes per group, roughly 1,150 in total. That is about a 14% response rate. A group with only a few hundred practices needs fewer, but a very high share of them.
What 550 gives: 550 completes is a 6.7% response rate. It gives about ±4 points overall. It gives about ±7 points per size group if the groups were equal (about 183 each). In practice the 10-or-more group is likely to be the smallest and may reach only ±10 points or worse.
Recommendation: Do not cap at 550. Keep the survey open and accept every qualifying complete. Either extend fieldwork and reminders to aim for about 1,150, or agree with Product before launch that size-group results will be read at about ±7 to 8 points, with the 10-or-more group indicative only.

MINIMUM TARGETS (soft, monitored daily, chased with targeted reminders rather than closing cells):

At least 180 completes in each size group (1; 2 to 9; 10 or more), or as many as the 10-or-more population allows.
At least 200 completes from the 1,230 paid practices (about ±6 to 7 points). Paid practices bring in all the revenue, and a proportional sample would give only about 80 of them.

WEIGHTING AND BIAS CHECKS: Compare respondents with all accounts on plan, seats, activity level and bank-feed ticket history, using account records. Weight to the account population by plan (85% free, 15% paid) and activity. Report paid practices separately as well. The banner over-reaches frequent users, and practices with bank-feed problems may be keener to respond; both are checked against records and flagged in reporting.