# Methodology — budgeting-app review mining

Companion to the study published at
https://www.cashjot.com/blog/why-budgeting-apps-fail. Collected and coded
2026-07-19. Published here: `coded.csv` (one row per review: id, app, rating,
date, category, one-line gist) and `analyze.py` (the stats script). We do not
republish the raw review texts, which belong to their authors and to Apple's
platform; the review ids in `coded.csv` let anyone look up the originals.

## Source

Apple's public App Store customer-reviews feed:
`https://itunes.apple.com/{storefront}/rss/customerreviews/page={1-10}/id={appId}/sortby=mostrecent/json`

The feed returns up to the ~500 most recent reviews per app per storefront (10 pages
of 50). Storefronts pulled: US, GB, CA, AU. Reviews
deduplicated by review id across storefronts.

## Apps (9), chosen to cover the three classes fairly

| App | App Store id | Class |
|---|---|---|
| YNAB | 1010865877 | zero-based |
| EveryDollar | 942571931 | zero-based |
| Goodbudget | 471112395 | zero-based (envelope) |
| Rocket Money | 1130616675 | bank-linked |
| Copilot Money | 1447330651 | bank-linked |
| Monarch Money | 1459319842 | bank-linked |
| PocketGuard | 949414211 | bank-linked |
| Money Manager (Realbyte) | 560481810 | manual |
| Spendee | 635861140 | manual (with optional bank sync) |

Including manual trackers (our own product class) was deliberate: mining only
competitors' weaknesses would not be honest data.

## Sample

- Pulled: 7,777 unique reviews (all ratings).
- Rating ≤ 2 stars: 2,680.
- Coded corpus: **1,586** (1,178 one-star, 408 two-star) after capping each app at
  its 200 most recent low-star reviews. The cap keeps any single app (or any single
  review-bombing wave) from dominating the corpus.
- Date range varies by app because the feed is recency-limited and volume differs:
  Rocket Money's 200 span ~7 weeks (May–Jul 2026); Goodbudget's span back to 2018.
  Per-app ranges are printed by `analyze.py`.

## Coding

Every review in the corpus was read and assigned exactly one primary-complaint
category (`coded.csv`: review id, app, rating, date, category, one-line gist).
Coding was done by us with AI assistance, one pass over the full corpus, no keyword
pre-filtering (which would bias the distribution). Categories:

| Code | Meaning |
|---|---|
| sync | Bank feed broken, institution unsupported, re-auth loops, import failures |
| bugs | Crashes, data loss, wrong math, broken logins, duplicates |
| redesign | An update made it worse (UI regressions, removed features, forced changes) |
| price | Cost, paywalls, ads, undisclosed pricing, free-tier removal |
| missing | Feature absent, product doesn't fit the need, too confusing to use |
| support | Billing gone wrong (charged after cancel), unreachable support, trust/privacy |
| upkeep | Fell behind, method too much work, backlog, reconciliation fatigue |
| rant | Venting or unclear; no codeable complaint |

Judgment calls, applied consistently:
- A named bank that *fails to connect* codes as `sync`; a bank or region that is
  *not supported at all* usually codes as `missing` when the reviewer frames it as
  a coverage gap.
- Duplicated transactions code as `bugs` unless the reviewer ties them to a feed
  reconnect, then `sync`.
- Ads and upsell pressure code as `price`.
- Complaints that mix causes were coded by the lead complaint (title first).

## Known limitations (say these in the piece)

1. **Low-star reviews are the unhappy tail**, not a sample of all users. Ratings for
   these apps average 4.5–4.9 stars.
2. **Quiet abandonment is invisible.** People who fall behind and drift away rarely
   review; people hit by a charge or a data wipe review the same day. The 1% upkeep
   share is a floor, not an estimate of how many people fall behind.
3. **Recency skew.** "Most recent" windows over-represent current controversies:
   YNAB's June 2026 UI backlash, Monarch's April 2026 pricing-tier backlash,
   Spendee's 2021 v5 data-loss wave. The per-app cap contains but does not remove
   this.
4. **English storefronts only** (US, GB, CA, AU).
5. **One coder, one pass.** Category boundaries have judgment in them; the gist
   column in `coded.csv` lets anyone audit any row.
6. We make CashJot, a manual expense tracker. That is a conflict of interest;
   it is why the raw data, codes, and this method are published.