nuestras búsquedas

Data Engineer Malaga

WE ARE LOOKING FOR A DATA ENGINEER TO BE RELOCATED TO MALAGA, SPAIN. The data engineer is a key role in any of the different data squads in charge of development and maintenance of new big data data platforms and data products. His/her main mission is to develop the different data pipelines which ingest, transform and prepare data from different sources into the different layers of our datahub platform and/or data products ROLES / RESPONSIBILITIES  Develop data pipelines with Spark and Scala to ingest and transform data from different sources (API, files, Kafka topics) into HDFS, IBM Cloud Storage, Parquet or SQL/NOSQL databases  Manage big data storage solutions in the platform (HDFS, IBM Cloud Storage, Parquet)  Implementation of schemas and queries in SQL/NOSQL databases like postgres, MongoDB or Oracle  Develop and configure scheduling of data pipelines with a combination of shell scripting and autosys, oozie or AirFlow  Configure Dremio Data Virtualization to interface with Parquet or as a way to expose the data in the different data products  Configure Dataiku pipelines as a data preparation tool for loading and transformation of data from our big data platform into specific datamarts for data or business products  Provide N3 level support to end users

Requisitos Tecnicos:
IT TOOLS  Spark and Scala as legacy data pipeline development language  SparkaaS and Scala as data pipeline development platform  Management of legacy big data storage solutions (HDFS and Parquet)  Management of big data storage solutions (IBM Cloud Object Storage and Parquet)  Implementation of SQL/NO SQL database schemas and queries (MongoDB, Oracle, Postgres)  Ksh, Autosys and Oozie as legacy data pipeline scheduling solution  Ksh and Airflow as data pipelie scheduling solution  Dremio as data virtualization tool