Domain Contributor - Evaluation Specialist

Overview

Responsibilities

• Design nuanced, domain-specific tasks that probe AI model limits and surface reasoning flaws.
• Author questions and scenarios demanding deep subject knowledge, beyond general understanding.
• Assess AI-generated outputs, validating accuracy against authoritative, primary sources in your area of expertise.
• Document clear rationales for model shortcomings, specifying missed reasoning, source errors, or nuances.
• Apply detailed task guidelines and evaluation rubrics consistently to each submission.
• Submit completed deliverables independently on a rolling basis, meeting project requirements for written and verbal clarity.

Requirements

• Required qualifications: not separately stated in the listing; the qualifications below are labelled preferred.

Preferred

• Proven hands-on experience with AI model training, annotation, RLHF, or evaluation.
• Deep expertise in a core domain: history, archival or library science, law (paralegal, legal academic, compliance), finance (equity research, PE/VC, forensic accounting), journalism, or technology/industry history.
• Advanced research skills with a track record of verifying complex claims against authoritative sources.
• Exceptional written and verbal communication skills to articulate findings and feedback.
• Demonstrated ability to precisely follow detailed instructions and evaluation frameworks.
• Background in adversarial or "stomp the model" task design is advantageous.
• Graduate-level research, experience at a Fortune 100 company, or professional fluency in a non-English language a plus.

Engagement and pay

• Role type: contractor, about 15 hours per week, as stated on the source.
• Pay: $30-$60/hour, as listed on the source.
• Compensation is output-based: experts are paid per task that meets the project specifications; time required varies by expert, and minimum weekly submission requirements apply.
• The source states roles are typically filled within 48 hours and selected experts are expected to start their first tasks within 24-48 hours of completing onboarding.

Eligibility

• Country eligibility not stated. Check with the provider before applying.