AI Model Assessment Specialist
As an AI Model Assessment Specialist, you will play a pivotal role in evaluating and measuring generative AI systems that drive user engagement and satisfaction. This is a unique opportunity to collaborate with a multidisciplinary team to create innovative AI applications that address real-world challenges.
* Evaluate and assess generative AI models using human-in-the-loop assessments and automated evaluation methods.
* Collaborate with product managers and engineers to design and measure metrics for user engagement, chatbot effectiveness, and data quality.
* Perform data cleansing, transformation, and quality analysis on product data to optimize system behavior.
* Design monitoring systems to track LLM behavior in production and capture key metrics around information retrieval, hallucinations, and latency.
* Run A/B tests to optimize chatbot interactions and user engagement.
* Develop and maintain interactive dashboards for real-time visualization of performance metrics and insights.
* Bachelor's degree in Computer Science, Data Science, Machine Learning, Statistics, or a related field.
* Proficiency in Python for data handling and analysis.
* Advanced NoSQL and SQL skills for querying large datasets.
* Strong problem-solving abilities with a focus on experimental design and data analysis.
* Excellent verbal and written communication skills with the ability to explain complex technical concepts to non-technical stakeholders.
* A flexible work environment that fosters collaboration, community, and team building.
* A home office stipend and flexible work allowance to help cover the costs of working from home.
* Paid time off to recharge and pursue personal interests.
* Parental leave to support new parents and extended maternity leave.
This is an exciting opportunity to contribute to the development of cutting-edge AI applications that drive meaningful impact.