Nidan AlamResume
PreviousNextClose

AIUB World Cup 2026 Data Pipeline

Data Pipeline, Jul 2026. Role: Solo developer.

Code on GitHub

Tooling for the university's World Cup 2026 football tournament: a re-runnable scraper, a path-to-final projection and a local Flask dashboard with a team simulator. It was built in the week before kickoff on 28 July 2026.

Highlights

  • Scrapes the site's own HTML fragment endpoint: no headless browser, 0.5 s between requests, 3 retries
  • Parsers handle both the empty pre-tournament state and played matches
  • Deterministic list of possible opponents for every team and knockout round, Round of 32 to Final
  • Monte Carlo simulator: 1,000 trials by default (up to 10,000) with a bias dial from uniform to standings-based strength
  • Refresh re-runs scraper and projection as a single background job without blocking the page
  • 101 pytest tests across 29 modules

Main challenge

The tournament site only shows rendered pages, and before kickoff every score, table and bracket slot is empty. The data had to be captured cleanly in both states. Knockout slots are labels such as group winners or "Winner of match N", so finding a team's possible opponents means resolving those labels through the bracket.

How it was solved

The scraper reads the site's own HTML fragment endpoint and team pages with Beautiful Soup, with delays and retries, and writes JSON plus timestamped snapshots. A resolver walks the bracket labels with memoised reachability to build opponent sets per round. The simulator applies what-if picks or samples whole tournaments, weighting teams by points, goal difference and goals scored, blended with uniform odds by a user-set bias. Simulator state lives in a file-locked JSON store, separate from the scraped data.

Outcomes

  • Seven JSON datasets per run: teams, rosters, fixtures, standings, scorers, bracket and a run manifest
  • Opponent sets per knockout round that shrink as real results come in
  • Simulator with three modes, saved and named scenarios, and a JSON API
  • 101 pytest tests covering scraping, projection, simulation and the dashboard

Built with

  • Python
  • Web Scraping
  • Beautiful Soup
  • Requests
  • Flask
  • pytest
  • Monte Carlo Simulation
  • JavaScript