Apex Systems, Inc. Data Engineer in Chicago, Illinois
This job was posted by https://illinoisjoblink.illinois.gov : For more information, please see: https://illinoisjoblink.illinois.gov/ada/r/jobs/6423174 Job DescriptionnPlease note that this is a CTH position and all candidates must have a college degree.nDesign and Document the new architecture and development process to convert Database and data warehouse models into Hadoop based systems.nDevelop the Large-scale data processing pipelines to handle Peta bytes of transaction data and egress to analytical sources.nWork with Apache Spark 2.0 Utilizing the Spark SQL and Streaming components to support the intraday and real-time data processingn Use the Scala 2.12 and Java 7 Programming to build the UDFs and supporting utilities for the standardizing the data pipelinesn Develop Batch and streaming workflows with in-built Stone branch scheduler and bash scripts to automate the Data Lake systemsnImplement the Spark Best practices to efficiently process data to meet ETAs by utilizing features like partitioning, resource tuning, memory management and Check pointing featuresnResolve any open Spark and Yarn resource management issues, OOM errors, Shuffle exceptions, heap space errors, Null Pointer Exceptions and schema compatibility in SparknWork on multiple data formats like ORC, Parquet, Avro, JSON and XML etcnWork on converting the multiple SQL Server and Oracle stored procedures into Hadoop using Spark SQL, Hive, Scala and JavanExtract, Parse, Clean and ingest the incoming data feeds, data and transaction logsnEEO EmployernApex is an Equal Employment Opportunity/Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, age, sexual orientation, gender identity, national origin, disability, protected veteran status, or any other characteristic protected by law. Apex will consider qualified applicants with criminal histories in a manner consistent with the requirements of applicable law. If you have visited our website in search of information on employment opportunities or to apply for a position, and you require an accommodation in using our website for a search or application, please contact our Employee Services Department at firstname.lastname@example.org or 844-463-6178.nVEVRAA Federal ContractorWe request Priority Protected Veteran & Disabled Referrals for all of our locations within the state.We are an equal opportunity employer. We evaluate qualified applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, veteran status, or any other protected characteristic. The EEO is the Law poster is available here.