Sr. Software Engineer, Machine Learning Infrastructure for Recommendations
- Los Gatos, California
- Product Engineering
Today, we think of the Netflix service less as a monolithic product and more as hundreds of millions of different products, serving uniquely personalized experiences to each of our member profiles. One of the teams powering this effort is the Personalization Infrastructure (PI) team, which builds scalable Machine Learning infrastructure for accelerating innovation for Netflix recommendations, content promotion, and search algorithms.
We are looking for a senior engineer with experience in scaling machine learning frameworks and platforms for large scale production use cases. In this role, you will be innovating on the core ML platform that powers Netflix recommendations. This platform is built on top of two Scala/Java platform libraries - Boson and AlgoCommons, widely adopted by the ML pipelines used to power recommendations and promotion algorithms. Boson is built on top of Spark, using the core dataframe abstraction for data preparation and feature encodings . Boson also provides a unified training API for different trainers (TensorFlow, XGBoost, etc.) as well as the metric and model evaluation layers, post-training. AlgoCommons provides the core building blocks and primitives for machine learning, feature encoding, and a common model format that allows trained models be used for inferencing/scoring in online and pre-compute environments.
Additionally, this role will position you as a subject matter expert in Apache Spark and leverage your Spark background to help other researchers and algorithm engineers scale and tune their Spark jobs. This involves understanding how Spark execution plans are laid out when dealing with large data sets and optimizing them to achieve compute efficiency within acceptable latency limits.
To be successful in this role, you must have a passion for performance tuning and discovering the low-level details of how subsystems work. You should also possess outstanding software development skills, ability to learn fast, and work cross-functionally with several teams. You will need to exhibit strong leadership and communication skills to successfully execute on the engineering and release priorities. We value work-life balance and believe that some of the best work happens in dynamic low-process environments.
This is a great opportunity to gain intimate knowledge of Netflix member personalization, while working for a unique and pioneering company that is redefining how video content is consumed globally.
- Strong bias towards action and great curiosity
- Experience building large-scale applications on Apache Spark with a good understanding of Spark internals
- Production experience with functional languages like Scala
- 4+ years of relevant work experience post BS/MS in Computer Science, Electrical Engineering or related field
- Excellent communication and people engagement skills
- Experience working with large-scale distributed systems like Mesos, Zookeeper, and HDFS
- Experience building systems and solutions for and with ML Engineers/Researchers
- Experience in Personalization/Recommendation domain
- Experience with Cloud Computing platforms like Amazon AWS
To learn more, here are some talks and blog posts from the PI Team:
You can learn more about Netflix’s unique culture of Freedom and Responsibility, that presents an opportunity to work with some of the best and the brightest, allowing you to make a difference to the business in a meaningful way.