Senior Distributed Systems Engineer - Big Data Infrastructure
- Los Gatos, California
- Cloud and Platform Engineering
At Netflix, the big data platform is the core foundation in driving all of our product decisions that directly impact our customer experience. The platform is leveraged across the company to build data models for recommendations, analyze customer streaming experience, content take-rate, plus more. It is critical to the success of our business as we heavily rely on analyzing data to revolutionize internet TV!
We are the big data services team that builds an ecosystem of micro services and near real-time data ingestion pipelines that exposes the big data platform as a robust and highly available service to the rest of Netflix. Check out Genie to see one of the micro services we have open sourced in the Netflix OSS.
We are constantly evolving our services and infrastructure. We are also at the beginning of designing new micro services to make our ecosystem more comprehensive for increasing data and metadata needs. We are looking for an experienced senior distributed systems engineer to help architect, build, and scale the big data services layer.
To learn more, here is our recent talk at re:Invent conference (slides or video) that describes what we do in the big data space!
Specifically, you will help:
- Architect and build new microservices that help manages our rapidly growing 60+ PB data hub on AWS S3.
- Architect and build federated metadata services (like Metacat) to manage metadata and lineage across all data sources we support (e.g., Kafka, Cassandra, RDS, Redshift.)
- Build and evolve our federated job management (RESTful) service, Genie, to manage job executions in various open source data processing engines like Presto, Spark, Hive, Pig, Sqoop, etc.
- Improve efficiency, reliability and scalability of our data ingestion pipelines. Currently, one of our distributed data pipelines ingests hundreds of billions of events from Kafka into S3 with a 5 minute SLA.
- Lead cross-functional initiatives and collaborate with engineers across organizations.
- Help contribute to our services and tools on Netflix OSS. Share our experiences and collaborate with the open source communities.
- You have 6+ years of experience in building large scale distributed applications.
- You are proficient in design and development of RESTful web services.
- You are an expert in Java. Python expertise is a plus.
- You have a BS/MS/PhD in Computer Science or related field.
- Most importantly, you are thrilled to help solve our big data challenges while revolutionizing internet TV!