VA CIPHER · CIPHER

How might we enable a partner to run their own research?

ORNL was brought in as an external research partner to evaluate a Department of Veterans Affairs data-discovery platform. I could have delivered findings and left. Instead, I treated the engagement as an apprenticeship: I moderated the first session as a worked example, then sat in on every session the VA ran, debriefed them after each one, and closed the engagement with a written set of standards for running their own studies. I also ran the full analysis, and built the analysis walkthrough into the report itself so they could attempt it next time. The deliverable wasn't findings - it was enabling a team to run research without me.

BY THE NUMBERS

  • 4rounds of findings and design recommendations
  • 4things handed off: worked example, session instrument, scoring standard, analysis walkthrough
DATE
Winter 2026
ORGANIZATION
Oak Ridge National Laboratory with the U.S. Department of Veteran Affairs
ROLE
ResearchStrategyMethod designCapability transferCoaching & enablementMixed-methods analysis
DELIVERABLES
Repeatable usability testing protocol, instrumented session-capture instrument, success and ease scoring rubric, thematic analysis framework, tiered reporting template, moderation and data-integrity standards, session-by-session moderator coaching, four rounds of findings and design recommendations

How was the engagement developed?

  1. 01

    Designing for Repeatability

    CIPHER is a searchable catalog of health-data assets used by analysts, epidemiologists, and informatics leaders — a high-stakes, high-jargon environment where the wrong search result costs a researcher weeks. Rather than scope a one-off study, I designed the protocol to be repeatable: one form that carries the script, the task rubric, live notes, and the recording link into a single record per participant.

  2. 02

    Building Measurement into the Method

    I paired task-level outcome scoring (complete / partial / failure) with the Single Ease Question, asked immediately after each task so the rating stays valid. That combination let me run head-to-head prototype comparisons and report a decision rather than a debate. In the most recent round, two search-and-filter concepts diverged sharply on the primary task: one variant reached full task success and a near-ceiling ease score, the other failed for most participants. A design question the team had been circling became a single unambiguous chart.

  3. 03

    Demonstrating, then Observing, then Coaching

    I moderated the first session myself as a worked example, then I attended every session the VA ran, watched them work, and debriefed with them afterward while the session was fresh — what to do differently, and why the methodology required it. The engagement closed with written standards for their future studies: when to rescue a struggling participant, why never to characterize someone's performance before asking them to rate it, why the ease question has to come immediately after the task, why focus groups can't substitute for 1:1 sessions.

  4. 04

    Making the Analysis Teachable

    I ran the full analysis but wrote the report so it wasn't a black box. The findings sections double as a walkthrough of how the analysis was built: sequence the numbers first so they set the story, layer the qualitative themes in to explain the numbers, then reduce it to an insight-and-recommendation table. I also showed how to structure the same readout three ways for three tiers of stakeholder attention. That's the hardest part of research to hand off, and it's the part that determines whether their next study produces a decision.

What I handed off:

  • 1 / A worked example

    One session moderated by me, then observation and post-session coaching on every session they ran

  • 2 / A session instrument

    Script, task rubric, notes, and recording capture in one record

  • 3 / A scoring standard

    Task success categories plus ease-question timing rules, producing a defensible number instead of an impression

  • 4 / An analysis walkthrough

    The reports explain how the analysis was built, not just what it found

Main Takeaways

Slide titled 'Testing highlighted confusing language, an unmet need for global search, and opportunities to simplify the search and Resource Catalog experiences' with three findings and recommendations: users expect a single, simple global search; jargon and unclear terminology are creating friction; Resource Catalog works well, but content could be refined.
Main takeaways from my analysis
Steps for making sense of user research results.
Teaching the VA about a way to think about results

Four rounds in, prototype decisions at CIPHER get settled with evidence rather than opinion, and the partner team has moderated real sessions with coaching rather than watching a consultant work.

Contracting into an organization and leaving behind a practice they can run themselves is the highest-leverage version of research consulting. It demonstrates that I can operate as an external expert in a regulated federal environment, translate rigorous methodology into something a non-researcher can execute correctly, and build capability that outlasts my involvement.