Why this project has a frontend

A typical data engineering portfolio is a README, an architecture diagram, and maybe a notebook with a few charts. The pipeline underneath might be genuinely good, tested, idempotent, careful about cost, and none of that comes across, because there is nothing to look at. You read the code or you take the author's word for it. I have clicked through a lot of these projects and remembered almost none of them.

That is the gap I wanted to close. compete ingests competitor web pages, runs the changed ones through an LLM, scores what it finds, and files the results into a warehouse. That is real work. If the only way to see it is to clone the repo and run the pipeline locally, then almost nobody ever will.

I gave compete a proper product surface: an overview, a page per competitor, a filterable feed of detected changes, and weekly reports you can download as PDF. It shares one design system with the marketing site and this page, warm paper and ink surfaces, a single lime accent, one sans typeface and one serif for editorial moments. Every data view has explicit loading, empty, and error states, which is most of what makes a dashboard feel finished instead of half-built.

The design exists to make the engineering legible. Good pipeline work deserves somewhere to show up, and a dry repo isn't it.

A polished shell over nothing is easy to spot and worth nothing, so I wanted a reader to be able to confirm the system actually runs. The dashboard reads real dbt marts through a FastAPI service. The whole pipeline runs end to end from one CLI. The numbers on screen come from an actual DuckDB file, not a JSON someone hand-wrote to look plausible.

The docs page walks through every stage, collection, extraction, the change-detection gate, the warehouse, the API, and it is honest about where the system falls short. If you want to check my work, the seams are all exposed on purpose.

Here is the part that surprises people. The version of compete you are looking at has no backend. There is no FastAPI server running, no database, nothing to wake up.

Before each deploy, scripts/export_static_data.py queries the warehouse once and writes the results to plain JSON under /api-static: competitors, signals, changes, per-competitor pricing and hiring, the reports. The frontend detects static mode and reads those files directly instead of calling an API. Create, edit, and delete still work; they are applied to a localStorage overlay in your browser, so the settings screen stays interactive with nothing on the other end to write to.

Two reasons this is the right call for a portfolio piece. It costs nothing to host, static files on a CDN, no server bill, no database to keep alive. And it never embarrasses me with a cold-start API returning an empty dashboard to the one person who opens the link on a Sunday. The tradeoff is that the data is a snapshot. A weekly GitHub Actions run regenerates the JSON and commits it, so the site stays current without ever running a server in production.

If you just want to look, the deployed site is the whole thing. Open the dashboard from the nav and click around.

If you want live data flowing through the real pipeline, it is a short setup from a clean clone:

uv sync --extra dev --extra dbt --extra api   # Python env + deps
uv run compete init-db                        # create the DuckDB schema
uv run python scripts/seed_demo.py --build    # demo data + dbt build
uv run compete-api                            # FastAPI on :8000

Then start the dashboard against the live API with NEXT_PUBLIC_STATIC_DATA=0 npm run dev inside web/. Left unset, dev reads the same static snapshot as production, which is the faster way to just click around.

One honest caveat: the seeded dataset is curated. The collectors are real and run against live sites, but the demo data is planted so that every feature has something to show without waiting a month for real history to accumulate. Everything you click is wired to the actual system. The contents are just primed. If something looks broken, it is a bug rather than a mockup, because the wiring underneath is all real.