At Tecla, working on a clinical research project for Incyte, we built a pipeline to extract insights from a large corpus of medical literature. The goal was not search or retrieval. The goal was discovery: finding patterns and relationships across hundreds of papers that no single researcher would have time to read systematically.
This post describes what we built, what failed, and what the pipeline actually produced.
Medical literature is dense, highly structured, and domain-specific. A general-purpose embedding model trained on web text produces poor embeddings for clinical language. "Response rate" in an oncology paper and "response rate" in a customer service context embed similarly under a general model, even though they are entirely different concepts.
We tested three embedding approaches:
text-embedding-ada-002: general-purpose, fast, cheapFor our corpus, PubMedBERT consistently produced tighter clusters on semantically related content. We validated this by embedding 50 known-related paper pairs and checking that they ranked in the top percentiles by cosine similarity. PubMedBERT had the best signal-to-noise ratio on this test.
A paper is not a single unit of meaning. The methods section, results section, and discussion carry different information and should be embedded separately. We split each paper into:
Each chunk was embedded independently and stored with its metadata (paper ID, section, authors, year, journal). This allowed us to search within a section type, not just across the full corpus, which dramatically improved precision for specific queries.
Chunk size was a source of ongoing tuning. Chunks that are too large lose specificity. Chunks that are too small lose context. For methods sections, 512 tokens worked well. For results sections with dense tables, we went smaller (256 tokens) because each sentence in a results section often carries independent information.
With embeddings computed, we used HDBSCAN (hierarchical density-based clustering) to find groups of semantically related chunks across the corpus. HDBSCAN was preferable to k-means because:
Each cluster was then summarized with an LLM prompt: given the 10 most central chunks in a cluster, produce a one-paragraph summary of the shared theme. This gave researchers a navigable map of the corpus organized by topic, without requiring them to read every paper.
The most useful output was not the clusters themselves. It was the relationships between clusters. Papers on a specific biomarker clustered together, but those clusters had measurable distances from clusters on related therapeutic targets. Clusters that appeared close in embedding space but were methodologically distinct (different patient populations, different endpoints) were the most interesting to researchers: they represented apparent contradictions in the literature that warranted deeper review.
We visualized this with UMAP dimensionality reduction, projecting the embeddings to 2D for interactive exploration. The UMAP plot let researchers navigate the corpus spatially, click on clusters, read the LLM-generated summaries, and drill into individual papers.
PubMedBERT (110M parameters) outperformed text-embedding-ada-002 on domain-specific
clustering despite being a much smaller model. Domain alignment beats scale for specialized corpora.
Every expensive LLM call in this pipeline was on a small, focused input: the top chunks of a cluster, a specific summary task, a concrete question. Calling the LLM on full papers upfront is expensive and produces lower-quality output because the model cannot attend equally to everything.
Clusters require human interpretation. The pipeline produced interesting groupings; researchers decided whether those groupings were scientifically meaningful. Building the system as a tool for human experts, rather than a system that produces conclusions, was the right framing.
We had no ground-truth labels for "correct" clusters. The initial evaluation was entirely qualitative: domain experts reviewed the clusters and flagged whether they made sense. Only after several iterations did we have enough human feedback to build quantitative benchmarks.
LLM pipelines over specialized corpora work best when domain-specific embeddings handle the semantic structure and the LLM is reserved for generation and summarization tasks where its language capabilities are uniquely valuable. For medical literature, this division of labor produced a system that clinical researchers found genuinely useful for navigating a corpus they could not have read manually.
If you are building something similar, feel free to reach out.