20 years of VC research searchable in seconds

The Problem

The firm had been collecting content for years: VC blogs, founder interviews, fundraising guides, board deck templates, market analyses, hiring playbooks, operational teardowns. Nearly 900,000 URLs spanning two decades of startup and investment knowledge. The problem wasn't that they didn't have the information. It was that they couldn't find it. No search. No filtering. No way to ask 'what did top VCs say about pricing strategy in 2019?' and get a real answer with sources. Some links were paywalled. Some were dead. Many were personal bios or about pages with no real content. And the firm was explicit about what they didn't want: 'Don't build us a chatbot that makes things up. We want to search real content, ranked by relevance, with citations we can verify.'

Our Approach

1. 01

   Collecting and Cleaning 20 Years of Content

   900,000 URLs from every corner of the internet: modern blogs, decade-old archives, paywalled sites, dead links. We built a system that pulls each page, converts it to clean text, and filters out the noise: navigation menus, ads, footers, bio pages, anything that isn't real content. The result is a clean library of actual articles, guides, and analyses.

   • +Handles modern sites and 20-year-old archives
   • +Detects and skips paywalled or dead content
   • +Strips navigation, ads, and boilerplate automatically
   • +Classifies content by type (article, guide, interview, memo, etc.)
   • +Started with 100k-page pilot before full scale
2. 02

   Making Every Piece of Content Findable

   Raw content isn't searchable. It needs structure. We run each document through AI to extract consistent metadata: who wrote it, when, what it's about, what type of content it is, and a short summary. This means you can filter by author, date range, content type, or topic, not just search keywords. A blog post from 2008 with no title tag gets the same treatment as a well-structured 2024 article.

   • +Extracts author, date, topic, and content type from every piece
   • +Generates consistent titles and summaries even for poorly structured pages
   • +Tags content by subject matter (fundraising, hiring, operations, etc.)
   • +Enables filtering by date range, author, and content type
   • +Only indexes content that's actually useful, with no wasted processing
3. 03

   Two Ways to Search: Keywords and Meaning

   Some searches are exact: 'Reid Hoffman board presentation.' Others are conceptual: 'advice on managing a down round.' The system handles both. It combines traditional keyword search with AI-powered semantic search, then merges and re-ranks the results. You get the precision of Google with the understanding of someone who's read every article in the archive.

   • +Keyword search for exact names, titles, and phrases
   • +Semantic search that understands what you mean, not just what you typed
   • +Results are merged and re-ranked for relevance
   • +Every result links back to the original source
   • +Optional AI summary across top results, only when you want it

Results & Impact

What used to require hours of manual searching, or just not happen at all, now takes seconds. The investment team can search two decades of VC knowledge by topic, author, date, or concept and get verified sources back instantly. The 100k-page pilot proved that most firms think they want AI-generated answers, but what they actually need is reliable search with citations. By pruning aggressively upfront, we avoided processing hundreds of thousands of irrelevant pages and cut costs by 60%. The system is built to scale to the full 900k+ corpus and can ingest new content types (newsletters, PDFs, LinkedIn posts) without any architecture changes.

Under the hood

• +Combines keyword search and semantic search, handling both exact lookups and conceptual queries
• +AI cleans and structures every document at ingestion, making downstream search dramatically better
• +Every result includes a direct link to the original source, with no hallucinated citations
• +AI summaries are optional rather than default, so the firm controls when they want interpretation vs. raw sources
• +Started with a 100k-page pilot that proved the approach and saved months of wasted processing
• +Architecture handles blogs, newsletters, PDFs, and LinkedIn posts without changes