Title: Sr Data Engineer / Sr Big Data Consultant
Location: Irving, TX
Duration: 1 Year
Data engineer will have the unique combination of business acumen needed to interface directly with key stakeholders to understand the problem along with the skills and vision to translate the need into a world-class technical solution using the latest technologies
As a senior data engineer, you’ll be handling the design and construction of large data structures, ensuring all clinical data is ready for analysis, and also research new uses for data acquisition. The position requires constant innovation surrounding data mining practices, algorithms, and how best to leverage new techno
Data engineering is about building the underlying infrastructure while working with data architects and data scientist. We want to see candidates with mechanical tendencies who know how to wrangle data.
This person will be a hands-on role who is responsible for building data engineering solutions for NMG Enterprise using cloud based data platform. They will provide day-to-day technical leadership and active oversight for technical design, development and support for data engineering workloads. In this role, you need to be equally skilled with the whiteboard and the keyboard.
Senior Data Engineer Responsibilities
- Architect, design, develop and engineering end-to-end data pipelines across multiple data sources and systems of record.
- Ensure data quality, integrity, security and completeness throughout the data lifecycle.
- Develop, design data models, data structures and ETL jobs for data acquisition and manipulation purposes.
- Develop deep understanding of the data sources, implement data standards, maintain data quality and master data management.
- Manage and maintain cloud-based data and analytics platform
- Deep understanding of the cloud offerings and engage in quick proof of concepts and proof of value in prototyping data and analytics solutions and derive viability
- Ability to interact with the business stakeholders to understand requirements and translating into technology solutions
- Experience in Cloud platform AWS eco-system.
- Data Engineering/Development experience with SQL (Snowflake, Oracle, SQL Server, MySQL).
- Strong development background creating pipelines and complex data transformations and manipulations using one of the languages Python, Java, R, or Scala.
- Experience in NoSQL Databases and Big data technologies including Hadoop, MongoDB, Cassandra.
- Experience with API / RESTful data services.
- Worked on real-time data capture, processing and storing using technologies like Kafka, AWS Kinesis.
- Understanding of different data formats including Parquet, Avro, CSV, ORC etc.
- Prior experience with MPP databases and maintains large amount of data processing
- Past working experience on a fast paced and agile environment
- Perform ongoing monitoring, automation and refinement of data engineering solutions
- Experience in leading high visibility transformation projects that interacts with multiple business lines
- Experience working with an on-shore / off-shore model that consists of Data architects and Visualization Engineers
- Build and meet project timelines and manage delivery commitments with proper communication to management
- Collaborate and communicate with key business lines, technology partners, vendors and architects
- BS in Computer Science or related field
- 10+ years of experience in the data and analytics space
- Certification –preferably AWS Certified Big Data or any other cloud data platforms, big data platforms
- 6+ year’s experience developing and implementing enterprise-level data solutions utilizing Python (Scikit-lean, Scipy, Pandas, Numpy, Tensorflow) , Java, Spark, and Scala, Airflow , Hive and Python.
- 4+ years in key aspects of software engineering such as parallel data processing, data flows, REST APIs, JSON, XML, and micro service architectures.
- 4+ year of experience working on Big Data Processing Frameworks and Tools – Map Reduce, YARN, Hive, Pig, Oozie, Sqoop, and good knowledge of common big data file formats (e.g., Parquet, ORC, etc.)
- 6+ years of RDBMS concepts with Strong Data analysis and SQL experience
- 6+ years of Linux OS command line tools and bash scripting proficiency
Knowledge, Skills and Abilities:
- Flexibility to work in matrix reporting structure
- Experienced in implementing large scale event based streaming architectures
- Background in all aspects of software engineering with strong skills in parallel data processing, data flows, REST APIs, JSON, XML, and micro service architecture
- Solid Programing experience in Python - needs to be an expert in this 4/5 level
- Working knowledge of data engineering aspects within machine learning pipelines (e.g., train/test splitting, scoring process, etc.)
- Experience working in a scrum/agile environment and associated tools (Jira)
Director - Resource Development