Job Description:
We are hiring a Data Scientist to our GenAI team in Portugal.
As a Data Scientist specializing in Generative AI, you will play a crucial role in evaluating, measuring, and assessing Agentic systems that have as main goal improving the wellbeing of all Wellhub users.
Metric Design & Analysis: Collaborate with product managers and engineering teams to define and measure metrics for user engagement, satisfaction, and chatbot effectiveness;
Data Preparation: Perform data cleansing, transformation, and quality analysis on product data;
Evaluation and Testing: Assist and design robust evaluation metrics to assess agentic AI performance, combining automated evaluation (LLM-as-judge), adversarial testing, human-in-the-loop evaluations, and custom behavioral metrics;
Prompt Engineering: Craft reusable templates and clear instructions to measure system behavior, optimizing for challenging edge cases;
LLM Observability: Design monitoring systems that track LLM behavior in production, capturing key metrics around information retrieval, hallucinations, and latency;
A/B testing: Design, run, and analyze A/B tests to optimize chatbot interactions and user engagement;
Dashboard Development: Create and maintain interactive dashboards using BI tools (e.g., Tableau, Power BI, Looker, Metabase) for real-time visualization of performance metrics and insights.
Main Requirements
Bachelor's degree in Computer Science, Data Science, Machine Learning, Statistics, or a related field;
Proficiency in Python for data handling and analysis;
Advanced NoSQL and SQL skills for querying large datasets;
Strong problem-solving abilities, with a focus on experimental design and data analysis;
Ability to work collaboratively in a team environment and communicate effectively across different departments;
Excellent verbal and written communication skills, with the ability to explain complex technical concepts to non-technical stakeholders;
Fluent in English.