AI Quality Analyst (Personalization) - Hindi

Overview

Responsibilities

• Design and execute multi-turn conversational prompts (typically 1-5 turns) that require the AI to use the analyst's personal information and experiences.
• Evaluate model responses against the intent of the starting prompt, checking whether personalization was appropriately applied.
• Analyze responses for grounding issues, ensuring claims about the user are supported by evidence rather than flawed inferences or hallucinations.
• Assess integration quality so personal data is woven naturally into responses without robotic overnarrating.
• Rigorously evaluate and stack-rank two model responses side by side to determine which is more helpful, easy to use, and enjoyable.
• Write clear, defensible rationales for comparisons, explicitly referencing where issues or positive aspects occurred in the conversation.
• Extract and verify debug information from the model to confirm chat summaries and data sources were properly used.
• Maintain strict data hygiene by deleting evaluation conversations so they do not pollute future chat history.

Requirements

• Hindi proficiency: ability to read and write in Hindi with a high degree of competence, as Hindi is the focus language for this project.
• Willingness to use your primary personal Google account (not a testing account) and enable personal data sources for a genuine assessment.
• Full-time availability in your local time zone; the project staffs a global 24-hour operations team.
• Demonstrated ability to evaluate nuanced and ambiguous AI responses, specifically assessing personalization quality.
• Experience designing creative, multi-turn starting prompts based on personal context to thoroughly test model capabilities.
• Understanding of personalization concepts, including the ability to identify incorrect personalization, poor inferences, and forced connections.
• Meticulous attention to detail when reviewing side-by-side model responses and spotting subtle differences in naturalness and overnarrating.
• Superior ability to write clear, concise, and structured rationales for model rankings, explicitly referencing specific turn numbers.
• Ability to provide constructive feedback and detailed annotations; excellent communication and collaboration skills.
• Self-motivated and able to work independently in a remote setting; desktop/laptop setup with a good internet connection.

Preferred

• BS/BA degree or equivalent experience in a relevant field such as policy, law, ethics, linguistics, journalism, computer science, or a related analytical field.
• Experience in data annotation, AI quality evaluation, content moderation, or a related role is strongly preferred.

Engagement and pay

• Pay: $15 per hour, as listed on the source. Contractor role with an engagement length of 3 months. Commitment: at least 4 hours per day and up to 40 hours per week, with 4 hours of overlap with PST.

Eligibility

• The source requires Hindi proficiency, use of a primary personal Google account, and full-time availability in the applicant's local time zone.
• Country eligibility is not stated in the listing. Remote alone does not guarantee eligibility.