May 2026 – Aug 2026 · San Jose, CA

IBM

Software Engineer Intern · watsonx Orchestrate

Built the evaluation pipeline that picked the retrieval backend for IBM's agentic AI platform, plus internal tools for triage and release quality.

hallucinations after the team adopted the top backend: −35%; hallucinations after the team adopted the top backend
citation precision: 94%; citation precision
GitHub issues synced into a filterable view: 65k+; GitHub issues synced into a filterable view

How the RAG evaluation worked

Start with a question from the hand-labeled golden dataset. Every question has a ground-truth answer.

Query · golden set #1,284

How long do customers have to request a refund on an annual plan?

Ground truth (hand-labeled)

Annual plans can be fully refunded within 30 days of purchase. After that, refunds are prorated.

↓ sent to all three backends

Milvus

Vector search

Retrieved

Answer

NLI vs ground truth

–

Embedding similarity

–

Graph

Knowledge graph

Retrieved

Answer

NLI vs ground truth

–

Embedding similarity

–

Hybrid

Combined retrieval

Retrieved

Answer

NLI vs ground truth

–

Embedding similarity

–

Illustrative recreation with synthetic data. No company code, data, or internal tools.

The problem

The team had three candidate retrieval backends for its RAG features and no reliable way to tell which one produced answers that were actually grounded in the source documents. Separately, customer issues were spread across GitHub in a form that was slow to triage.

What I built

A faithfulness evaluation pipeline

I hand-labeled a 3,000-query golden dataset and scored every generated answer sentence by sentence, using NLI entailment to check whether each claim was supported by the retrieved passages and embedding similarity to check that citations pointed at the right ones. Running all three backends through it gave the team a clear winner.

An issue triage view

Asynchronous services sync 65,000+ GitHub issues into IBM DB2, so the team can filter and group them by customer and status instead of paging through GitHub's UI.

Faster release gates

I parallelized the 1,200+ test Playwright and pytest suite that gates every release, and built a telemetry dashboard to track client quality metrics across releases.

Stack

Python
NLI models
Embeddings
Django
Next.js
IBM DB2
Playwright
pytest-xdist

The illustrations on this page are recreations built for this site with synthetic data. They don't contain IBM's code, data, or internal tools.

Next case studyWaste Management