Summary
Overview
Work history
Education
Skills
Certification
Timeline
Generic

Koushik Reddy

Watford

Summary

Data Engineer with 10 years of IT experience, including 5+ years of hands-on expertise in building scalable data solutions across AWS, Snowflake, Hadoop, and modern ELT/ETL frameworks. Strong background in migrating legacy systems to cloud data platforms, optimizing pipelines, and enabling high-quality analytics. Experienced in orchestration, CI/CD, data modelling, and infrastructure automation. Proven ability to deliver efficient and cost-optimized data pipelines in fast-paced Agile environments.

Overview

10
10
years of professional experience
1
1
Certification

Work history

Data engineer

HSBC
London
11.2023 - Current
  • Migrated internal data pipelines to Snowflake using Snowpipe & Snowflake Connector, improving automation by 40%.
  • Designed & orchestrated end-to-end ELT pipelines using Airflow, covering ingestion, transformation & curation.
  • Built DBT models with incremental loads, schema tests & documentation.
  • Developed ingestion pipelines using AWS S3, Glue, Lambda, and optimized processing with PySpark.
  • Managed cloud infrastructure using Terraform, ensuring scalable & secure deployments.
  • Implemented CI/CD using GitLab/Jenkins for automated pipeline deployment.
  • Migrated legacy Hive pipelines to cloud-native Snowflake workflows, enhancing performance by 50%.
  • Implemented monitoring & alerting using Airflow logs and notifications.
  • Designed reusable pipeline components to standardize ETL/ELT patterns.
  • Supported real-time ingestion using Kafka → Snowflake.
  • Optimized compute cost and query performance using partitioning, clustering & caching strategies.

Big Data Engineer

IBM
Hyderabad
07.2020 - 09.2023
  • Designed scalable batch & streaming pipelines using PySpark on AWS EMR.
  • Developed and orchestrated AWS Glue ETL workflows and Step Functions.
  • Built data lakes on S3 and HDFS for structured & semi-structured data.
  • Used Kafka for real-time event ingestion and processing.
  • Optimized Spark jobs using partitioning, bucketing & caching, reducing runtime significantly.
  • Implemented advanced transformations for cleansing, aggregation & enrichment.
  • Containerized Spark apps using Docker for consistent deployments.
  • Built CI/CD pipelines for data apps improving release efficiency.

Software engineer

EY GLOBAL SERVICES LIMITED
, india
10.2015 - 05.2020
  • Developed scalable Java, React, and Android applications ensuring high performance and usability.
  • Designed and consumed RESTful APIs for microservices-based systems.
  • Improved performance by debugging, tuning backend services & optimizing SQL queries.
  • Automated tasks using Python/Bash to increase development efficiency.
  • Implemented CI/CD using Git and Jenkins, improving deployment speed.
  • Participated in Agile workflows including sprint planning & code reviews.

Education

Master of Science - Computer Science

University of East London
London
07.2021

Skills

Data Engineering

Snowflake, AWS Glue, Airflow, DBT, Kafka, Snowpipe, PySpark, Hive, Impala, ETL/ELT, Terraform, GitLab, Jenkins, CI/CD, Data Modeling, Real-time Pipelines, Data Lake, API Integration, Linux/Unix

Programming

Python, SQL, PL/SQL, Scala, Bash

Cloud & Big Data

AWS (S3, Lambda, EMR, Glue, Athena, IAM, Lake Formation), Hadoop (HDFS, MapReduce, YARN), Redshift

Visualization Tools

Power BI, Tableau

Databases

Snowflake, Oracle, SQL Server, MySQL, PostgreSQL, DB2

Tools

IntelliJ, VS Code, Eclipse, Jira, Confluence, Git, Docker

Certification

  • AWS Certified Data Analytics – Specialty
  • Databricks Certified Associate Developer for Apache Spark 3.0
  • Snow pro-Core Certified

Timeline

Data engineer

HSBC
11.2023 - Current

Big Data Engineer

IBM
07.2020 - 09.2023

Software engineer

EY GLOBAL SERVICES LIMITED
10.2015 - 05.2020

Master of Science - Computer Science

University of East London

Similar Profiles

Create profile
Koushik Reddy