Oak Ridge National Laboratory · SEARCH WITH GENAI
How might we rebuild enterprise search around generative AI, and prove it worked?
Staff at Oak Ridge National Laboratory couldn't find what they needed in internal search; 70% couldn't name one thing it did well. I led UX research for a redesign that added generative AI summaries, from foundational research through AI trust testing and usability testing to launch. Then I set up the measurement that proved the result: a 51% improvement in UX score over the pre-launch benchmark.
BY THE NUMBERS
- 51%improvement in UX score
- 79%improvement in usability score
- 81,000Searches performed in the first year
- $1MEstimated annual cost avoidance (peer institutions require 4-6 dedicated FTEs)
- DATE
- Summer / Fall 2024
- ORGANIZATION
- Oak Ridge National Laboratory
- ROLE
- ResearchStrategyAI trust & transparencyDesignContent designContinuous measurement
- DELIVERABLES
- Multi-phase research plan, benchmark report, design prototypes, user testing results decks, new enterprise search experience, continual intercept measurement program & post-launch reporting
Research Process
- 01
Find out what's broken
A short org-wide survey showed 70% of staff couldn't name anything search did well, and 90% described a bad experience. I coded 343 open-text responses and split results by employee type. The top complaints were irrelevant, outdated, and inaccessible results.
- 02
Test whether users trust AI search summaries
In moderated, task-based sessions, I scored success and trust as people used AI summaries and filters. Search success depended entirely on whether the AI summary was clear and helpful, so summary quality became the design priority.
- 03
Refine and ship
Rapid usability tests found confusion around card labels, the AI summary toggle, and breadcrumbs. These recommendations shipped in the org-wide release:
- Say what is and isn't being searched
- Let people filter by content source
- Make AI summaries easier to read, with clear citations
- Relabel "Suggested Links" as "Top Match," the wording users chose in testing
- 04
Measure after launch
An intercept survey on the live product showed usability scored high while usefulness lagged, and the open text explained why: most complaints were about getting no results at all, an indexing problem rather than an interface one. I recommended an indexing audit, better synonym and acronym handling, and AI summaries that match the results below them.
Staff now know what they're searching, how AI is working on their behalf, and how to recover from a failed query
This work directly shaped the redesign of ORNL’s internal search and the integration of generative AI in a high-trust setting.
Staff now understand what they're searching, how AI is being used on their behalf, and how to recover from a query that fails. A year of measurement shows the experience substantially outperforming what it replaced, at a fraction of the staffing a traditionally curated portal would require.
Designing the research strategy, the trust-building approach, the AI evaluation framework, and the post-launch measurement program together demonstrate that I can take a mission-driven organization through AI adoption in a complex environment and then prove whether it worked.