Senior Software Engineering Manager – Stream Processing Infrastructure
Posted: December 17, 2016
Reference ID: 1698515587
Application developers are moving from old school batch processing systems to real time stream processing frameworks. Stream processing is completely revolutionizing the big data space. At LinkedIn, events pertaining to application and system monitoring, member behavior tracking, inter-application communication, etc. , are all ingested into our pub-sub messaging system (Kafka). A staggering 1. trillion events are published into Kafka per day with peaks of 4. million messages/sec per cluster. The stream processing team is responsible for developing Apache Samza and its surrounding ecosystem at LinkedIn for processing this deluge of events in real time. If you are passionate in the real time data/stream processing domain then this team is at the center of the action. The technologies built by this team are not only used inside LinkedIn but will also be impacting the broader open source ecosystem.
The ideal candidate will help take LinkedIn's real time stream processing infrastructure to the next stage of its evolution:
Manage the performance and career development of a team of junior and senior engineers, and own significant parts of this infrastructure which will require design, architecture, and coding. Service: The Samza team is responsible for running a YARN based 24/7 production service that enables application developers to rapidly innovate, experiment, build and run Samza based stream processing applications.
Framework: The Samza team is also responsible for building an embedded version of the stream processing framework which is distributed as a client library and used by various event processing applications at LinkedIn.
Develop a long term technology roadmap: As senior manager of the Samza team, you will be responsible for working closely with the broader application developer community inside LinkedIn, other data infrastructure teams to come up with a technology roadmap for real time event processing. You would also be responsible for influencing teams working on complementary/related technologies in data infrastructure at LinkedIn.
Community: We have a deep culture of open sourcing the software the we build. As the leader of the stream processing team, you would also be responsible for nurturing the relationship with the open source community.
Experience hiring, mentoring, coaching and developing top engineering talent. project management skills. Experience building and running 24/7 operations of infrastructure systems/distributed large-scale systems. Strong management skills for planning and executing complex multi-team projects Knowledge of multi-threading, concurrency, and parallel processing technologies. Experience with industry, open-source projects and/or academic research in large-data, parallel and distributed systems. Published work in academic conferences or industry circles. Experience in any other data processing technologies like Map-Reduce, Storm, Spark, Cosmos, Google Dataflow, Azure Stream Analytics etc. , CEP frameworks and/or pub-sub technologies like Kafka, AWS Kinesis, Google Pub-Sub, Azure EventHubs etc. is highly desired