Discover Financial Services is seeking a bright, driven, and talented individual to join a team of passionate and innovative technologists. As a Master Data Engineer, you will use your experience and hands on approach to contribute to a team that delivers the latest streaming & next generation data technologies.
In this role, you will provide engineering leadership to create and enhance data solutions enabling seamless integration and flow of data across our data ecosystem. Additionally you will provide senior level technical consulting to peer data engineers during design and development for highly complex and critical data projects. Some of these projects will include designing and developing data ingestion and processing/transformation frameworks leveraging open source tools such as NiFi, Sqoop, Hive, Java, Pig, Python, Spark, etc.
The Master Data Engineer will work on real time processing solutions using tools such as Apex, Flink, Storm and Kafka. You will deploy application code and analytical models using CI/CD tools and techniques and provide support for deployed data applications and analytical models.
* Develop data driven solutions utilizing current and next generation technologies to meet evolving business needs
* Ability to quickly identify an opportunity and recommend possible technical solutions
* Utilize multiple development languages/tools such as Python, SPARK, HBase, Hive, Microsoft R, Java to build prototypes and evaluate results for effectiveness and feasibility
* Operationalize open source data-analytic tools for enterprise use
* Develop real-time data ingestion and stream-analytic solutions leveraging technologies such as Kafka, Apache Spark, NIFI, Python, HBase and Hadoop
* Custom Data pipeline development (Cloud and locally hosted)Work heavily within the Hadoop ecosystem and migrate data from Teradata to Hadoop
* Provide support for deployed data applications and analytical models by being a trusted advisor to Data Scientists and other data consumers by identifying data problems and guiding issue resolution with partner Data Engineers and source data providers
* Provide subject matter expertise in the analysis, preparation of specifications and plans for the development of data processes
* Ensure proper data governance policies are followed by implementing or validating Data Lineage, Quality checks, classification, etc.
* Promote a risk-aware culture, ensure efficient and effective risk and compliance management practices by adhering to required standards and processes.
* Bachelor's Degree required
* 8 + years' work experience
* 5+ years' experience working with Hadoop
* Experience using Apache Spark with Scala
* Experience developing ETL on Hadoop
#LI-KB1 We are an Equal Opportunity Employer and do not discriminate against any employee or applicant for employment because of race, color, sex, age, national origin, religion, sexual orientation, gender identity, status as a veteran, and basis of disability or any other federal, state or local protected class.
A little about us:
Discover is one of the most recognized brands in U.S. financial services. We’re a direct banking and payment services company built on a legacy of innovation and customer service.