I wanted a personal score tracker for beatmania IIDX — a dynamic table of every song and chart, sortable and filterable, with a way to log my own scores. The app part was the easy part: an Astro frontend, a Supabase Postgres backend, a handful of pages. What actually ate the time was the thing I hadn’t planned for at all: getting a trustworthy song/chart catalog into the database in the first place, and then trusting it once it was there.

The data problem before the app problem

IIDX has been running for over 25 years, across more than 30 arcade versions, with songs that get added, removed, ported between play styles, and occasionally resurrected years later as paid DLC packs. There’s no official public API for any of this. If you want a structured dataset of every song, every chart’s difficulty level, and which version it debuted in, you’re on your own.

The obvious place to look is Textage, the community’s most complete chart database. I ruled it out almost immediately. Its data is served as intentionally obfuscated JS tables with no public API, the site’s author has a documented history of pushing back on tools that reverse-engineer and redistribute that data, and I wanted this project to be a public, forkable repo. Building a scraper for it wouldn’t just be a personal convenience — it’d be shipping a general-purpose extraction tool for someone else’s compiled database to anyone who forked the project. That’s a different thing entirely, and not one I was willing to build.

What I used instead was iidx-db, an MIT-licensed dataset whose data is extracted directly from IIDX Infinitas’ memory via a tool called Reflux — datamined from a copy of the game the extractor owns, not scraped from a third party’s site. Cleaner provenance, and a license that actually says “you can use this.” It got me 1,740 songs and 10,698 charts to start with — a real, working catalog.

Chasing the last few hundred songs

The gap was the DLC. iidx-db’s snapshot didn’t include song packs the original extractor hadn’t purchased, and IIDX sells a lot of packs. My first instinct was to fix this the “proper” way: build my own extraction. I cloned Reflux, patched its source to output the extra fields I needed, retargeted the build to a modern .NET SDK, and got a working custom build running against my own copy of Infinitas.

It didn’t work — but not for a code reason. My Infinitas installation was on a newer game build than any offset calibration Reflux had published, even from its own upstream releases. The tool’s interactive offset-calibration fallback found the right memory addresses for play data but never located a working pointer to the song list itself, and just hung forever waiting for data that was never going to arrive. I could have kept guessing at raw memory offsets, but that’s exactly the kind of thing where being wrong is silent and expensive — corrupt data that looks plausible is worse than no data. I shelved it.

The unglamorous fix: doing the research myself

What actually closed the gap was less interesting from an engineering standpoint and more interesting from a “how do you validate data” standpoint: I built my own spreadsheet, going song by song through Infinitas’ catalog, cross-referencing Textage, RemyWiki, and in-game data as a personal reference — the same way you’d use a strategy guide, not the same way you’d scrape one. Slower than an API. Also the only approach that was actually available to me without cutting a corner I wasn’t willing to cut.

Once I’d reviewed the sheet myself, I needed a way to get it into the database without creating a mess.

Validating two “correct” datasets against each other

Before treating my own sheet as gospel, I ran it against the existing iidx-db-backed catalog to see where they disagreed. Out of about 3,500 rows, roughly 2,760 matched cleanly. 597 didn’t.

That’s a big number, and my first assumption was that it meant one dataset was right and the other was wrong — probably mine, since it was hand-compiled and hand-compiled data has typos. So I spot-checked a couple of the disagreements directly against Textage to find out which side was actually correct.

The first one confirmed my assumption: the automated dataset was right, my sheet had a transcription slip (I’d apparently copied a Leggendaria chart’s level into the Another chart’s cell instead).

The second one flipped it: the automated dataset was wrong, and my sheet was right.

That was the useful result. It wasn’t one dataset being sloppy — both had independent errors, and there was no shortcut that let me trust either one wholesale. The only honest thing to do with 597 disagreements was hand them back to myself as a reviewable list and go through them individually, rather than pick a “winner” dataset and blindly overwrite the other. Cross-referencing two independently-sourced-but-imperfect datasets caught mistakes that neither source alone would have surfaced.

Underneath that, there was a second, quieter validation problem: matching the same song across two datasets that didn’t format titles identically. “A MINSTREL ～ver. short-scape～” and “A MINSTREL ～ ver. short-scape ～” are the same song with one extra space. Matching titles exactly missed 57 real matches this way, and treating them as case/whitespace/tilde-variant-insensitive fixed nearly all of it, at the cost of two unrelated existing songs (“SHOOTING STAR” and “Shooting Star”) accidentally colliding under the loose match. Fuzzy matching a large dataset always trades false negatives for false positives — you don’t get to avoid both, only pick which one you’d rather manually clean up.

The bug that only showed up once the data got big enough

After syncing the reviewed sheet in, the catalog grew from 1,740 songs to 1,927, and from 10,698 charts to 11,950. And a brand-new song — one I’d specifically just added — silently didn’t show up on the live site. No error, no console warning, just… missing.

The cause was Supabase capping a single .select() query at 1,000 rows by default. Every query in the app had been written back when the catalog was under that limit, so it worked fine for months — right up until the chart count crossed 1,000 and started silently truncating results everywhere: the main table, the score-entry search, the CSV import matcher. All three needed the same fix — paging through results explicitly instead of trusting a single request to return everything:

const pageSize = 1000;
let data = [];
for (let from = 0; ; from += pageSize) {
  const { data: page, error } = await supabase
    .from('charts')
    .select('...')
    .range(from, from + pageSize - 1);
  data = data.concat(page ?? []);
  if (!page || page.length < pageSize) break;
}

This is the kind of bug that’s easy to miss entirely, because it doesn’t throw. It just quietly returns less than you asked for, and everything downstream looks correct until you go looking for the one thing that happens to be past row 1,000.

Where the data stands right now

The database in this release is a v1 dataset — the reviewed sheet merged with the automated iidx-db catalog, disagreements resolved by hand wherever I could confirm the right answer against Textage. It covers the large majority of songs and charts correctly, but I’m not calling it authoritative yet. There are almost certainly a few charts still wrong somewhere in there — a level I mis-transcribed, or a fuzzy-matched title that landed on the wrong song. I’ll keep correcting these as they surface, and I’m treating the catalog as something that improves release over release rather than a one-time import — the goal is to get it to a point where I’d call it definitively trustworthy, not to call it finished today.

What I’d take away from this

The schema design, the auth policies, the UI — all of that came together in a normal amount of time. The actual bulk of the effort went into questions that had nothing to do with code: where is this data allowed to come from, how do I know when two “correct” sources disagree, and how do I know my own dataset is complete once it’s big enough that I can’t just eyeball it?

If there’s one habit worth carrying into any project with a real dataset behind it, it’s this: don’t just validate data once at import time and call it done. Re-validate when the shape of the data changes — a new source, a bulk edit, a dataset that crosses some size threshold you didn’t think mattered. That’s usually exactly when a silent assumption you didn’t know you were making finally breaks.