Retired · September 2026
Early Flags
Early Flags was a market-signal research agent: a pipeline that scanned public developer and research communities for problems people were starting to complain about, and tried to flag emerging markets before they became obvious. It is no longer running. This page is what remains.
What it did
Every five hours it ingested new posts from Hacker News (stories, comments and Ask HN), Reddit, Stack Exchange, arXiv, GitHub (new repos and issue volume) and Product Hunt, plus adoption curves from npm and PyPI downloads, GitHub star velocity, Wikipedia pageviews and YouTube. Once a day it:
- Extracted the concrete problem each post described, using a cheap model to discard the noise and a stronger one to structure the rest.
- Clustered similar problems together with embeddings, merging and splitting clusters as they evolved over days.
- Ranked the clusters on seven factors: velocity, how early the sources are historically, pain intensity, inverse press coverage, whether people were already building workarounds, breadth across independent sources, and whether download or pageview curves agreed.
- Wrote a digest of the top ten, with evidence links, and captured a human verdict on each (real signal, noise, skip) as append-only history so the ranking could later be scored against what actually grew.
How it was built
Next.js on Vercel, Postgres on Supabase, extraction through Anthropic's batch API, GitHub Actions as the scheduler. Every run was logged immutably with its pinned config version, and a fixed holdout of sources was withheld so blind spots could be measured rather than guessed. The design called for a backtest harness and then a monitor agent that would critique the digest and propose its own tuning; the pipeline was retired before those stages.
What was learned
- Searching is easy; ranking is the whole product. Volume becomes unusable within a week without a defensible answer to “ahead of the crowd, or already obvious?”
- Nearly every real bug lived at a boundary that only a live run crosses: source APIs that silently cap results, rate limits by request count, a platform that kills a function at five minutes, a transaction poisoned by one bad row.
- Token cost scales with the noisiest source. Half the volume was Hacker News comments, and most of the spend went to deciding they were not signal.
Interested?
If you want to talk about the approach, the data, or picking it back up, get in touch.