Senior Data Engineer

  • Job Ref: 8143
  • Greater London
  • IT

We are currently on the lookout for a Senior Data Engineer, the client is a Scala house but they are happy with Python either.

This is a contract opportunity that is fully remote from across the EU and is outside IR35.

What to expect:

You will work in an agile team of data engineers and backend developers responsible for building and operating a data processing platform.

  • Near-real time ingestion pipelines handling >100M events per day, built using Kinesis, Kafka and Spark-streaming and written in Scala
  • Batch pipelines written in Spark or SQL, with scheduling and dependencies defined in Airflow
  • S3 data lake holding >60TB of data, underpinning our Redshift Spectrum warehouse
  • Modern, AWS based infrastructure using k8s and automated CI/CD with Jenkins and Terraform, making the team fully autonomous

If you feel this is of interest to you please reach out and I can provide you with a further info about the company and project.