Data feeds fail quietly

Most software tells you when it breaks. A page returns an error, a test goes red, someone opens a ticket. A data feed to a partner does none of that.

I work on the feeds that send Bandsintown concert listings to apps like Spotify, YouTube, Shazam and Amazon Music. Every few hours we build a file for each partner and deliver it. The partner reads the file and shows the concerts to people who listen to those artists.

When something goes wrong in that chain, the usual result is not an outage. It is a smaller number. A few hundred events drop out. A field arrives empty. A ticket link points to the wrong place. The partner app still works. It just shows less, and nobody who uses it knows what they are missing.

Nobody files a bug for a concert they never saw

That is the hard part. The people who would notice the problem are the fans, and the fans cannot see it. If a show does not appear on an artist page, there is no gap where it should be. There is only a shorter list.

So the job is to notice for them.

Count what you send

The first check is the simplest one: how many events went out, compared with last time? A feed that usually carries a steady number and suddenly carries far fewer is almost always broken, even if every row in it is valid.

I keep this count for each partner, every run. A drop past a set limit raises an alert before the file leaves.

Check the shape, not only the rows

Row checks catch a bad date or a missing venue. They do not catch a column that is now empty for every row because something upstream changed its name. For that you need checks on the whole file: what share of rows have a ticket link, how many distinct cities appear, whether the time zones look right.

These checks are cheap, and they find the problems that row checks miss.

Watch before you block

When I add a new check, it does not block anything at first. It runs in observe mode, logs what it would have stopped, and I read those logs for a while. Some checks are too strict. Some catch real problems that nobody knew about.

Only when a check has proved itself does it start to block. And when it blocks, the feed sends nothing for that run rather than something wrong. A partner can wait two hours for a correct file. A wrong file can sit on their side for much longer.

The takeaway

If your system sends data to someone else, the failures you need to worry about most are the ones that look like success. Count what you send, check the shape of it, and earn the right to block before you do.