The engineer will own the evaluation codebase and pipelines that inform training decisions and releases.

Evaluation infrastructure

• Build and maintain the evaluation codebase and pipelines for Apertus models, from checkpoints during training to released models
• Make evaluations run quickly and at scale: parallel execution on Alps, efficient use of inference backends, caching, and result tracking
• Reduce mismatch between evaluations during training and during serving: consistent tokenisation, chat templates, prompting, and sampling across evaluation harnesses and inference engines
• Debug evaluation failures, regressions, and inconsistencies across backends

Benchmark coverage

• Integrate and run the evaluations the project cares about. The design of new evaluations is owned by collaborating researchers and engineers; this role makes them run reliably and at scale
• Cover image and audio evaluations alongside text within the same pipeline
• Integrate new benchmarks as the field evolves, working with our academic collaborators to onboard the benchmarks they create, and validate that metrics and harness implementations are trustworthy

Comparative and third-party evaluation

• Evaluate third-party services and other open and closed models against the same benchmark suite, producing directly comparable and reproducible results
• Provide evaluation results, reports, and dashboards that support training decisions (data mixtures, ablations) and release decisions
• Work closely with the engineers focused on safety, deployment, and community needs, and integrate the evaluations they create into the shared pipeline