Human Preference Data
Pairwise choices, rankings, and explanations tied to the outputs a person evaluated. Collections can be specified for preference analysis, reward modeling, and post-training.
Data & Evals
Human preference signals and task-level evidence for model development.
Our work spans output evaluation, agent behavior, and purpose-built data collection. A project can begin with an existing task family or a new collection specification.
Model & Checkpoint Evaluations
Comparative studies of new releases and candidate checkpoints. Define the task distribution, reference models, audience, and review protocol around the question being tested.
Agent Trajectories
Task context, tool interactions, intermediate artifacts, and final outputs. Study execution, recovery, and the relationship between an agent’s process and its result.
Multimodal Evaluation
Human judgments of websites, images, video, documents, and other artifacts. Evaluate visual quality, instruction following, coherence, and usefulness in the context of the task.
Multi-Turn Tasks
Interactions that develop through feedback and revision. Examine context retention, corrections, and the differences between an initial output and the result a person continues working with.
Custom Evals & Benchmarks
Task sets, source material, rubrics, and review criteria for a particular capability or domain. Designed for repeatable comparison across models and checkpoints.
Tool-Use & RL Environments
Custom tasks around APIs, MCP tools, browsers, and software workflows. Environment design and verifiers can be developed as part of a scoped research engagement.
Domain-Specific Collections
Data collection with defined domain, participant, and source requirements. Collection workflows, review, and delivery formats are developed with the research team.
Failure Analysis
Analysis of losing comparisons and unsuccessful attempts. Identify recurring failures in quality, instruction following, tool use, and task completion to guide subsequent evaluations or collection.
Data access
Samples, collection requirements, and custom data agreements.