Assignment Description:
We are looking for a Senior Traffic Analyst to join our dynamic team.
What you will work on:
- Develop and implement evaluation frameworks, scoring rubrics, and multidimensional assessment models to measure the quality of AI-generated and AI-translated content.
- Create, maintain, and refine datasets, scoring protocols, and language-specific guidelines to ensure accuracy, fluency, and cultural relevance.
- Collaborate with ML engineers and language experts to calibrate evaluators, analyze model errors, and improve assessment logic.
- Conduct prompt testing, model calibration, and scoring validation for Generative AI evaluation processes.
- Identify quality gaps, propose improvements, and contribute to continuous feedback loops that enhance the linguistic performance of AI platforms.
- Support the training of assessment models and contribute to documentation, process development, and adoption of best practices.
What you bring:
- Experience in language quality evaluation, applied linguistics, computational linguistics, or similar language-focused research environments.
- Hands-on experience with LLM evaluation, machine translation assessment, or annotation processes in multilingual contexts.
- Strong understanding of linguistic and translation assessment frameworks such as MQM, MetricX, or COMET, with experience creating scoring rubrics.
- Familiarity with Generative AI evaluation methods, including prompt testing, model calibration, and scoring validation.
- Experience collaborating with ML teams on data pipelines, annotation workflows, or fine-tuning of linguistic evaluators.
- Deep linguistic and cultural knowledge across multiple markets, with the ability to define and measure content quality for different contexts.
- Advantageous experience with programmatic QA using Python, YAML, gRPC, or rule-based validation.
- Advantageous knowledge of inter-rater reliability measurement methods such as Krippendorff’s alpha or Cohen’s kappa, and experience comparing human and AI-generated assessments
- Fluent in English and Swedish.
If this sounds interesting, please send your CV to cv@veritaz.se
