Manager, Data Engineer - Scala

  • Company: Capital One
  • Location: New York, New York
  • Posted: January 12, 2017
  • Reference ID: R17367
114 5th Ave (22114), United States of America, New York, New York

Manager, Data Engineer - Scala

We are building an amazing data playground. A highly scalable distributed platform that takes all of our data (petabytes of it), and makes it available to analysts and systems so that we can continue doing what we’ve always done – use data to make banking better.

As a Manager, Senior Data engineer on the Data Intelligence team, you will contribute to building a fast data and machine learning platform scalable to solve diverse business problems.  We envision, create, deploy, and maintain full stack technology solutions powered by streaming big data, state of the art machine learning, micro-service architecture, and intuitive visualizations in the cloud.

At Capital One, we have seas of big data and rivers of fast data. To manage this, we are working with a number of cutting edge machine learning technologies, and are actively developing more. We are highly technical with strong backgrounds in what we do.  Our use cases range from cyber threat prevention to predicting environment outages to enable our always on 24/7 services.  We have the highest executive support and direct impact on our customer experience and bottom line.

On any given day you will:

  • Transform complex analytical models into scalable, production-ready solutions

  • Develop applications from ground up using a modern technology stack such as Scala, Spark, Postgres and NoSQL

  • Lead and develop sustainable data driven solutions with current new gen data technologies to meet the needs of our organization and business customers

  • Master new technologies rapidly as needed to progress varied initiatives

  • Provide technical guidance to team members

  • Manage, develop and lead solution engineering on top of Cloud platforms like AWS
  • Automate the provisioning of environments using tools like Ansible, and the deployment of those environments using containers like Docker
  • Design and develop workflows to automate the deployment of applications and infrastructure environments 
  • While you don’t have to scale buildings, you will implement scalable systems solutions
  • Identify technical obstacles early and work closely with team to find creative solutions to build prototypes & develop deployment strategies, procedures
  • Investigate the impact of new technologies on the platform, Capital One users, customers & recommend solutions

Ideal Qualifications:

  • Successful candidates will possess strong, demonstrable skills in: big data tools, web services, cloud services, systems automation and cutting-edge technologies such as Cassandra, Docker, Ansible, Spark, Storm, Hadoop, Kafka etc., as well as possess experience working alongside architecture & development teams
  • Our solutions need to be scalable and robust, as well as simple and easily digestible for our end-users. In that context, you will be expected to research and develop cutting-edge technologies to accomplish your goals, should you choose to accept the challenge
  • Strong verbal and written communication skills are required due to the dynamic nature of discussions with customers, vendors, and other engineering and product teams
  • You know what continuous integration and deployment means, and believe automation is the path to happiness!


What we have:

- A startup mindset with the backing of a top 10 bank

- Monthly Innovation-Days dedicated to test driving cutting edge technologies - Flexible work schedules

- Convenient office locations

- Generous salary and merit-based pay incentives

- Your choice of equipment (MacBook/PC, iPhone/Android Device)

Basic Qualifications:

- Bachelor’s Degree or military experience

- 3+ years in coding in Scala, Python or Java

- At least 3 years experience with big data technologies (Cassandra, Accumulo, HBase, Spark, Hadoop, HDFS, AVRO, MongoDB, Zookeeper or similar)

Preferred Qualifications:

- Master's Degree

- 3+ years with Agile engineering practices

- 3+ years in-depth experience with the Hadoop stack (MapReduce, Pig, Hive, Hbase)

- 3+ years experience with NoSQL implementation (Mongo, Cassandra, etc. a plus)

- 3+ years experience with Relational Database Systems and SQL

- 3+ years experience with Linux including basic commands and shell scripting

- Familiarity with data science tools and concepts

- Experience in industry, i.e. code committer, published, white paper, etc.

Share this Job