Broadcast pre-filtering — prune before you pay to move

The partner catalog touches a handful of fare zones; route_stops covers the whole country. Compute the zones that are actually present, broadcast that tiny set, and discard reference rows that provably can't match — before the shuffle pays to move them.

Step 1 · The broadcast: a tiny set, shipped everywhere

stopsPrepared
zone_key "0042" ×41
zone_key "0043" ×17
zone_key "0117" ×6

the partner's 64 stops

→.select("zone_key").distinct()

004200430117

presentZones · 3 rows, ~a few hundred bytes

→broadcast(…)

executor 1

executor 2

…

executor 200

every executor holds the full set — no shuffle needed to check membership

Step 2 · The gate: route_stops, one cell per fare zone

224

zones in the feed

1.83M

rows in the feed

—

rows that survive

Each cell is one fare zone, sized here as ~8k route_stops rows on average. Blue cells are zones in presentZones — the only rows the join could ever match.

Step 3 · What the shuffle actually moves

without pre-filter

with pre-filter

Semantically this join is a no-op — an inner join against the keys we're about to match on anyway removes nothing the outer join wouldn't have dropped. In practice it's the difference between shuffling a national feed and shuffling three zones' worth of it. Spark can do this automatically (dynamic partition pruning) only when the big side is directory-partitioned on the join key; ours isn't, so the code does it by hand.