Research Evaluation Specialist (PhD or Masters / Researcher / Professor)

Overview

Responsibilities

• Create original, high-difficulty question-and-answer pairs at the cutting edge of your discipline to rigorously assess AI model capabilities.
• Source, verify, and thoroughly document answers using primary sources and authoritative references, including clear citations and reasoning.
• Design questions demanding advanced reasoning, methodological nuance, or synthesis across multiple sources, ensuring resistance to oversimplified or shortcut solutions.
• Test questions against AI systems, identify items that are not sufficiently challenging, and iteratively increase complexity while maintaining accuracy.
• Write each question and answer with high precision and clarity, eliminating ambiguity and supporting defensible responses.
• Integrate reviewer feedback and consistently uphold all project guidelines and quality standards throughout your deliverables.

Requirements

• Required qualifications: not separately stated in the listing; the qualifications below are labelled preferred.
• The source states prior experience in AI training, data labeling, or model evaluation is required.

Preferred

• Masters or PhD completed, active candidacy, or equivalent research experience as a specialist, researcher, assistant professor or professor in your domain.
• Demonstrated track record of scholarly research, in-depth subject matter expertise, and familiarity with primary literature.
• Exceptional analytical thinking, close attention to detail, and strong written communication skills in English.
• Experience sourcing and triangulating information from authoritative references.
• Proven self-direction and reliability in delivering expert-level work independently within remote engagements.
• Skill in composing challenging, original, and methodologically sound questions.

Engagement and pay

• Role type: contractor, about 15 hours per week, as stated on the source.
• Pay: $30-$60/hour, as listed on the source.
• Compensation is output-based: experts are paid per task that meets the project specifications; time required varies by expert, and minimum weekly submission requirements apply.
• The source states roles are typically filled within 48 hours and selected experts are expected to start their first tasks within 24-48 hours of completing onboarding.

Eligibility

• The source states the location as Remote, worldwide.