JobFetch

Senior Software Engineer - AI Evaluation / Coding Agents

Pay
$100–$150/hour
Where
Remote
Type
Freelance · Senior
Eligibility
Country eligibility not stated. Check with the provider before applying.
Listing
Last verified Nov 8, 2026

The application asks you to state your expected hourly rate.

Apply opens the provider’s application page. You apply there yourself: we never submit applications, and opening the page doesn’t mean you’ve applied. Nobody can promise you’ll be hired.

Some Apply links are referral links. If you apply through them, the provider may pay us a commission. Referral disclosure

Explore weekly job alerts for your skills

Overview

Evaluate and improve AI coding models by reviewing generated code.

Work with coding agents across real-world repositories and assess the quality of their work, the way you would review another engineer's output: whether it understood the task, chose the right approach, and produced correct, robust, maintainable code, with a precise explanation of where it succeeded or failed.

Responsibilities

  • Design, refine, and iterate on rubrics and evaluation criteria for preference data.
  • Review and label data to a high quality bar, catching subtle issues others might overlook.
  • Build and maintain pipelines and infrastructure supporting data generation, collection, and evaluation workflows.
  • Synthesize findings from data work into clear write-ups, updates, and recommendations for the team.
  • Collaborate closely with researchers and engineers to translate qualitative judgment into scalable processes.

Requirements

  • 5+ years of hands-on software engineering experience.
  • Strong proficiency in Python, TypeScript/JavaScript, Go, or another major production language.
  • Experience working in substantial real-world codebases.
  • Strong code-review skills and technical judgment.
  • Ability to clearly explain why an implementation is correct, incorrect, or could be improved.
  • Strong written communication.
  • Experience using modern LLMs or AI coding tools.

Preferred

  • Experience with LLM evaluation, coding agents, RLHF, preference data, rubric design, or post-training is a plus, but not required.

Engagement and pay

  • Compensation: $100-$150 per hour; the application asks you to provide a specific hourly rate expectation.
  • Availability: 20-40 hours per week, with at least 6 hours of Pacific Time overlap per day.
  • Independent contractor engagement of approximately 3 months, starting as soon as possible.
  • Evaluation process: an AI interview of about 25 minutes, a practical code/AI evaluation exercise of about 30 minutes focused on reviewing and evaluating AI-generated code (not competitive programming or algorithm puzzles), and a hiring manager interview of about 20 minutes.

Eligibility

  • The source states the location as Remote, North America (only).
  • Individual countries are not listed in the source.

Skills

  • python
  • typescript
  • javascript
  • go
  • java
  • github
  • ci/cd
  • open source
  • llm
  • code reviews
  • code analysis