We are seeking an exceptionally skilled and visionary AWS Senior / Expert AI Engineer to spearhead our advanced DevOps, MLOps, and LLMOps initiatives. This is a senior technical role where you will define, establish best practices, and lead the implementation of robust, scalable, and secure operational pipelines for our software, machine learning models, and large language model applications across multiple, high-impact project teams.
What you’ll be doing:
- Cloud Infrastructure & Orchestration: Design, implement, and manage scalable and secure AWS-based infrastructure for AI/ML workloads, utilizing services like AWS Step Functions, EventBridge, AWS.
- Managed Workflows for Apache Airflow (MWAA), and AWS Lambda for workflow orchestration.
- Data Processing & ETL: Develop, optimize, and maintain robust big data ETL and analytics pipelines using PySpark with Python and/or Spark with Scala on AWS Glue, Amazon EMR, and Amazon EKS.
- Data Storage & Management: Implement efficient data storage solutions primarily on Amazon S3, ensuring data accessibility, security, and integrity for AI/ML applications.
- Data Querying & Analysis: Utilize AWS Athena for ad-hoc querying and analysis of large datasets stored in S3, supporting data exploration and model development.
- Hadoop Ecosystem Integration: Leverage expertise in the Hadoop ecosystem (Hive, Impala, Sqoop, HDFS, Oozie) for managing and processing large-scale datasets.
- Programming & Data Transformation: Apply strong Python programming skills, including extensive experience with Pandas DataFrame transformations, for data manipulation and analysis.
- Deployment & MLOps: Implement and maintain Continuous Integration (CI) and Continuous Delivery (CD) pipelines using Jenkins, and manage infrastructure as code (IaC) with Terraform for automated deployment of AI/ML solutions.
- Performance Optimization: Continuously monitor, evaluate, and optimize the performance, cost-efficiency, and reliability of deployed AI/ML infrastructure and data pipelines.
- Collaboration: Work closely with data scientists, product managers, and business stakeholders to translate requirements into scalable technical solutions.
- Code Quality: Write clean, well-documented, and testable code, adhering to best practices in software development, MLOps, and cloud engineering.
- Mentorship (Senior/Expert): Mentor junior engineers, share knowledge, and contribute to the overall growth and technical excellence of the team.