Benchmarks / SurveyBench / Why Tallyroom trials don't convert / Kimi K3.1 Pro

Measured by Spring Prompt

Kimi K3.1 Pro: Why Tallyroom trials don't convert

The decision: where to put next quarter's product and sales effort to lift trial-to-paid conversion: a new accounting integration, assisted setup, a cheaper plan, or a shorter trial. The brief and the data →

Verdict
✗ Not sound
Research score
30 of 100
Analysis rating
608
Head to head, this task
won 0 of 6

Why it is not sound

The analysis

  • Produced no usable analysis

Scorecard

The analysis

This model produced no usable analysis for this task.

The survey it planned

3 screening questions and 10 questions, as the model wrote them.

  1. S1

    Thinking about the team that tried Tallyroom, which best describes your role at the time of the trial?

    One answer
    • I was the main Tallyroom administrator
    • I shared Tallyroom administrator responsibilities
    • I used Tallyroom but was not an administrator
    • I did not use or administer Tallyroom for that team

    Continues if I was the main Tallyroom administrator; I shared Tallyroom administrator responsibilities

  2. S2

    Which statement best describes that team's Tallyroom trial?

    One answer
    • It has ended, and we did not start a paid Tallyroom subscription
    • It is still running
    • It ended, and we started a paid Tallyroom subscription during or after the trial
    • I am not aware that this team tried Tallyroom
    • Don't know

    Continues if It has ended, and we did not start a paid Tallyroom subscription

  3. S3

    At the time of the trial, was that team part of a business or organisation based in the UK?

    One answer
    • Yes
    • No
    • Don't know

    Continues if Yes

  4. Q1

    Approximately how many employees did the whole business or organisation have when it tried Tallyroom? Include working owners and employees at all locations, not just Tallyroom users.

    One answer
    • 1-9 employees
    • 10-49 employees
    • 50-249 employees
    • 250 or more employees
    • Don't know
  5. Q2

    What did your team do during the Tallyroom trial? Select all that apply.

    Any that apply
    • Added users or changed settings
    • Photographed or submitted at least one receipt
    • Approved at least one expense as a manager
    • Connected our accounting software to Tallyroom
    • Sent at least one approved expense to our accounting software
    • Used Tallyroom for real business expenses, rather than only testing
    • Asked Tallyroom for help or used its setup guidance
    • None of these activities
    • Don't know
  6. Q3

    What was the main reason your team did not start paying for Tallyroom? Please select the one reason that mattered most.

    One answer
    • We never got it set up or tested properly
    • It was too difficult or time-consuming to set up or use
    • Staff or managers did not want to use it
    • It did not fit our expense or approval process
    • An integration we needed was missing or inadequate
    • A feature we needed was missing or inadequate
    • Receipt capture or another part of the product was unreliable or inaccurate
    • It was too expensive or not worth £6 per user per month
    • We chose or kept another product or way of managing expenses
    • We had too few expenses to need it
    • We intended to pay but could not get internal approval or complete the purchase
    • The trial ended before we had enough time to evaluate it
    • We made no active decision; we forgot about Tallyroom or the trial simply ended
    • Our business circumstances changed
    • Another reason not listed above
    • Don't know
  7. Q4

    Which accounting or bookkeeping software did your business use for its main accounts when it tried Tallyroom?

    One answer
    • Xero
    • QuickBooks Online
    • QuickBooks Desktop
    • Sage Accounting (cloud)
    • Sage 50 (desktop)
    • FreeAgent
    • KashFlow
    • Zoho Books
    • Another accounting or bookkeeping package
    • None; we used spreadsheets, paper or manual records
    • Don't know
  8. Q5

    Which, if any, of these contributed to your team's decision not to pay? Select all that apply.

    Any that apply
    • An integration with our accounting or bookkeeping software was missing
    • The existing Xero or QuickBooks integration did not work as we needed
    • An integration with our bank or company-card provider was missing or inadequate
    • An integration with payroll, HR or another business system was missing or inadequate
    • Receipt capture was not accurate or fast enough
    • Approval workflows or user permissions did not meet our needs
    • VAT, mileage or expense-categorisation features did not meet our needs
    • Reporting or exports did not meet our needs
    • Another feature or improvement was missing or inadequate
    • None of these contributed to our decision
    • Don't know
  9. Q6

    What was the most important missing or inadequate item you selected in the previous question? For an integration, name the software and what you needed it to do; for a feature, describe the capability you needed.

    Open text

    Routing Shown only if Q5 is: An integration with our accounting or bookkeeping software was missing; The existing Xero or QuickBooks integration did not work as we needed; An integration with our bank or company-card provider was missing or inadequate; An integration with payroll, HR or another business system was missing or inadequate; Receipt capture was not accurate or fast enough; Approval workflows or user permissions did not meet our needs; VAT, mileage or expense-categorisation features did not meet our needs; Reporting or exports did not meet our needs; Another feature or improvement was missing or inadequate

  10. Q7

    Assuming Tallyroom had otherwise met your team's needs, which statement is closest to what your team would have done?

    One answer
    • We would have paid £6 per user per month
    • We would not have paid £6, but might have paid for a cheaper plan, depending on its price and features
    • We would not have paid for Tallyroom at any price
    • Not applicable; our need for an expenses app had gone away
    • Don't know
  11. Q8

    Suppose Tallyroom had offered a free 30-minute one-to-one video call in the first week of your trial, to help configure expenses and approvals and connect your accounting software. What difference would this have made to your decision?

    One answer
    • We would probably have started paying
    • It would have helped, but would not have been enough on its own
    • It would have made no difference; setup was not a problem for us
    • It would have made no difference; another issue stopped us paying
    • We would not have wanted a setup call
    • Don't know
  12. Q9

    During the 30-day trial, when did your team last actively use Tallyroom? This includes adding receipts, approving expenses, changing settings or trying to connect software. Please give your best estimate.

    One answer
    • We never actively used it after signing up
    • Days 1-3
    • Days 4-7
    • Days 8-14
    • Days 15-21
    • Days 22-30
    • Don't remember
  13. Q10

    Thinking about your team's situation at the time, what would most likely have happened if the trial had lasted 14 days instead of 30?

    One answer
    • We would have started paying by day 14
    • We would have set up, tested or decided sooner, but we still would not have paid
    • We would not have done anything sooner, and the final decision would have been the same
    • We would have needed more time, so we would have been less likely to pay
    • We would have needed more time, but could still have paid if an extension had been available
    • Don't know
Sample plan

Census-invite the eligible product-database frame: one invited admin per UK team whose 30-day trial ended in the previous 90 days without conversion to paid. Remove test/internal accounts, invalid emails, duplicates and teams now paying; if one admin has several eligible teams, randomly select one team for that admin to answer about. Use a unique single-use email link, offer the £10 voucher for an eligible complete regardless of answers, and send up to two reminders to non-respondents. Set quotas from Q1; responses from businesses with 250+ employees or unknown size should be reported separately and not fill the three required band quotas. Weight overall results to the eligible frame's size-band distribution if disproportionate quotas are used, and examine non-response using available frame variables. There is a conflict between about 450 completes and the required precision. Using the conservative worst-case calculation for a proportion, without a finite-population correction, each band needs 1.96^2 × 0.25 / 0.06^2 = 266.8, rounded up to 267 completes. Therefore, if ±6 percentage points at 95% confidence in every band is non-negotiable, target at least 801 completes across the three bands: 267 in 1-9, 267 in 10-49 and 267 in 50-249. Check actual frame counts by band before promising feasibility and recalculate only if a defensible finite-population correction applies. If a band cannot reach its target, pool successive survey waves while still surveying each admin within 90 days of lapse, or report the shortfall; do not backfill from larger bands. If the total is immovably capped at 450, use 150 completes per band and obtain explicit sign-off that the worst-case band margin is approximately ±8.0 percentage points, not ±6. These margins describe sampling variability only and do not eliminate email non-response bias.