On the same clinical research engagement mentioned in a couple of my other posts, we built a pipeline to extract insights from a large corpus of medical literature. The goal was not search or retrieval. The goal was discovery: finding patterns and relationships across hundreds of papers that no single researcher would have time to read systematically.
This post describes what we built, what failed, and what the pipeline actually produced.
Medical literature is dense, highly structured, and domain-specific. A general-purpose embedding model trained on web text produces poor embeddings for clinical language. "Response rate" in an oncology paper and "response rate" in a customer service context embed similarly under a general model, even though they are entirely different concepts.
We tested three embedding approaches:
text-embedding-ada-002: general-purpose, fast, cheapFor our corpus, PubMedBERT consistently produced tighter clusters on semantically related content. We validated this by embedding 50 known-related paper pairs and checking that they ranked in the top percentiles by cosine similarity. PubMedBERT had the best signal-to-noise ratio on this test.
A paper is not a single unit of meaning. The methods section, results section, and discussion carry different information and should be embedded separately. We split each paper into:
Each chunk was embedded independently and stored with its metadata (paper ID, section, authors, year, journal). This allowed us to search within a section type, not just across the full corpus, which dramatically improved precision for specific queries.
Chunk size was a source of ongoing tuning. Chunks that are too large lose specificity. Chunks that are too small lose context. For methods sections, 512 tokens worked well. For results sections with dense tables, we went smaller (256 tokens) because each sentence in a results section often carries independent information.
With embeddings computed, we used HDBSCAN (hierarchical density-based clustering) to find groups of semantically related chunks across the corpus. HDBSCAN was preferable to k-means because:
Each cluster was then summarized with an LLM prompt: given the 10 most central chunks in a cluster, produce a one-paragraph summary of the shared theme. This gave researchers a navigable map of the corpus organized by topic, without requiring them to read every paper.
The most useful output was not the clusters themselves. It was the relationships between clusters. Papers on a specific biomarker clustered together, but those clusters had measurable distances from clusters on related therapeutic targets. Clusters that appeared close in embedding space but were methodologically distinct (different patient populations, different endpoints) were the most interesting to researchers: they represented apparent contradictions in the literature that warranted deeper review.
We visualized this with UMAP dimensionality reduction, projecting the embeddings to 2D for interactive exploration. The UMAP plot let researchers navigate the corpus spatially, click on clusters, read the LLM-generated summaries, and drill into individual papers.
The biggest one: a 110M-parameter PubMedBERT beat OpenAI's much larger
text-embedding-ada-002 on domain-specific clustering. Domain alignment mattered more
than scale here, which isn't the default assumption most people bring to embedding-model choice.
The LLM calls themselves worked best kept small and late in the pipeline: the top chunks of a cluster, a specific summarization task, rather than thrown at full papers upfront. Feeding the model whole papers is expensive and, worse, produces weaker output, because attention gets spread too thin across everything at once instead of focused on what matters.
Clustering itself never became a source of conclusions on its own. It stayed an exploration tool. The pipeline surfaced groupings; whether those groupings meant anything scientifically was always a call for the researchers, not the system. And there was no way to shortcut evaluation early on: with no ground-truth labels for "correct" clusters, the first several iterations were judged purely by domain experts eyeballing whether a cluster made sense. Quantitative benchmarks only became possible once enough of that qualitative feedback had accumulated.