The Data Engineer III is responsible for developing and supporting advanced reports that provide accurate and timely data for internal and external clients and senior management. The incumbent will design and grow a data infrastructure that powers our ability to make timely and data-driven decision. This role requires familiarity of data architecture and extraction & manipulation of data sets of large sizes.
Position Responsibilities:
Design and Methodology
Contribute to the architecture and implementation of next generation data and BI solutions from the ground up.
Review code to ensure performance and reliability of data extraction and processing.
Design, implement and operate large-scale, high-volume, high-performance data structures for analytics and data science.
Implement data ingestion routines both Real Time and batch using best practices in data modeling, ETL/ELT processes by leveraging big data tools.
Create data products for analytics and data scientist team members to improve their productivity.
Design, develop and build Real Time data pipelines from a variety of sources (streaming data, APIs, data warehouse, messages etc.)
Leverage the understanding of software architecture and software design patterns to write scalable, maintainable, well-designed, and future-proof software.
Manage existing pipelines and create new pipelines from a variety of sources (relational, XML, etc.)
Design and build solutions to track data quality, stabilize data pipeline, etc. to ensure reliable operations.
Technical Consulting
Gather business and functional requirements and translate these requirements into robust, scalable operable solutions with a flexible and adaptable data architecture.
Collaborate with engineers to help adopt best practices in data system creation, data integrity, test design, analysis, validation, and documentation.
Collaborate with Data Scientists to create fast and efficient algorithms that exploit our rich data sets for optimization, statistical analysis, prediction, clustering, and machine learning.
Coordination with other teams to design optimal patterns for data ingest and egress, as well as lead and coordinate data quality initiatives and troubleshooting.
Participate in sprint planning meetings and provide estimations on technical implementation.
Advise, consult, mentor and coach other data and analytic professionals on data standards and practices.
Foster a culture of sharing, re-use, design for scale stability, and operational efficiency of data and analytical solutions.
Optimize support for ad-hod analysis across various data sources.
Recommend infrastructure changes to improve storage capacity or performance.
Planning & Organizing
Perform cost-benefit and return on investment analyses for proposed systems to aid management in making implementation decisions.
Support purchasing efforts through conducting research on software and systems products to justify recommendations.
Ensure best practices are followed across architecture, codebase, and configuration.
Proactively identify and implement opportunities to automate tasks and develop reusable frameworks.
Lead the exploration and understanding of new tools and techniques and propose improvements to the data pipeline.
Continually improve ongoing reporting and analysis processes, automating or simplifying self-service modeling and production support for customers.
Identify opportunities to improve performance by improving database structure or indexing methods.
Risk Management
Optimize automated processes for performance and fault tolerance.
Review functional and technical designs to identify areas of risk and any missing requirements.
Design and implement security measures to protect data from unauthorized access or misuse.
Design backup and recovery procedures to ensure data integrity is maintained.
Administration/Support
Collaborate with Technology and Business colleagues in an agile development environment.
Review, create and maintain documentation as it relates to system configuration, mapping, processes, and service records.
Contribute to Data Governance, system documentation and sharing of data asset knowledge.
Act as a team lead and mentor for lower-level Data Engineers.
Keeps abreast of leading-edge technologies in the Data Engineering space.
Position Qualifications:
Bachelor’s Degree in computer science, engineering or in a technology related field OR equivalent through a combination of education and/or technology experience OR 12 years of technology experience
6 years of experience in Data Engineering, BI Engineering or related field in architecting and developing end-to-end scalable data applications and data pipelines
6 years of experience in extracting, transforming/manipulating, and loading data sets of various sizes using programming (eg, Python) or querying languages (SQL)
6 years of working knowledge of different databases (eg, SQL & NoSQL) and available cloud technologies to leverage (eg, AWS, Azure)
6 years of experience coding with modern programming or Scripting language (Python, Scala, Java)
6 years of experience using Informatica PowerCenter with 3 years IICS experience in extracting, transforming/manipulating, and loading data sets using programming (eg, Python) or querying languages (SQL) and relevant experience in data profiling, data quality, catalog tools
5 years of experience in data strategy and road map exercises, data warehouse design and implementation
5 years of experience developing strong collaborative relationships with key partners in data engineering, business intelligence, software development, finance, modelling, and product teams
4 years of experience working with software engineering best practices across the development lifecycle, including agile methodologies, coding standards, code reviews, source management, build processes and testing
3 years of experience working in enterprise data warehouse solutions and platforms
Licenses/Certifications:
Amazon Web Services (AWS) Certified Data Analytics
Preferred, Informatica Data Management Cloud Certification – Developer or Administration
Plano
8:00am – 5:00pm Monday – Friday
Job Description Year Up is a one-year or less, intensive job training program that provides young adults with hands-on skill...
Apply For This JobOur client is looking for an Audit Manager to join their successful team. By joining them, you will have the...
Apply For This JobJob Description Location: 7000 Target Pkwy N, Brooklyn Park, Minnesota, United States, 55445 Pay range: $129,147 to $150,800/year In addition...
Apply For This JobJob Description Job order – J(Apply online only) – Permanent Full Time Title Systems / Technical Architects Category Architecture City...
Apply For This JobJob Description We’re on the search for The Junior BI Developer is responsible for supporting both the BI Engineer and...
Apply For This JobJob Description Systems Engineer (Windows Server Technologies) – Brussels Job Purpose: We are in search of a committed and meticulous...
Apply For This Job