Kishore Chitturi
Data Engineer
Slough
Summary
- With over 8+ years of experience in IT solutions, this professional has led enterprise architecture practices and successfully managed large-scale, complex projects across various industries, including Life Sciences, Banking, Retail, Healthcare, Airline, and Automotive. They possess extensive expertise in the analysis, design, development, implementation, and testing of software applications, particularly in Big Data technologies. Their skill set includes hands-on experience with tools and platforms such as Databricks, Azure, AWS, Palantir Foundry, Spark, PySpark, Airflow, Hadoop, Hive, Sqoop, Oozie, HBase, Kafka, MongoDB, Cassandra, Redshift, Snowflake, DynamoDB, and Mode Analytics, demonstrating a deep proficiency in managing and analyzing vast datasets.
- Hands on experience on Spark framework components like Spark Core, Spark SQL, and Spark Streaming.
- Experienced with implementing latest languages like Spark, PySpark, Scala.
- Excellent understanding knowledge of Apache Spark & Hadoop YARN architecture and ecosystem.
- Strong experience in collecting and storing stream data in HDFS using Apache Kafka.
- Experience in column-family based Databases (HBase), DynamoDB.
- Good Experience in job scheduling tools like Airflow, control M, OOZIE
- Experienced with processing different file formats like Avro, Parquet, JSON and Sequence file formats.
- Understanding of Snowflake cloud technology.
- Experience with Snowflake cloud data warehouse and AWS S3 bucket for integrating data from multiple source system which include loading data into snowflake table.
- Worked on implementing a use case on Palantir Foundry using Spark to load the data to AWS S3.
- Hands on experience with various Data modeling techniques for OLTP & OLAP processes
- Experienced in CI/CD using Jenkins, GitHub, SVN
- Experience working in Agile Development process using Scrum.
- Committed to professionalism, highly organized, ability to work under strict deadline schedules with attention to details, possess excellent written and communication skills.
- Ability to work effectively in a multi-cultural environment with a team and individually as per the project requirement.
- Have good ability to understand Functional Specifications and good experience in writing Technical Documents for the required Functional Specifications.
- Good interpersonal and managerial skills. Result oriented, quick learner, hard working with a quest and zeal to learn new technologies.
Overview
8
8
years of professional experience
3
3
years of post-secondary education
6
6
Certifications
Work History
Senior Data Engineer
Nomo Fintech
10.2022 - Current
- Ensured data quality through rigorous testing, validation, and monitoring of all data assets, minimizing inaccuracies and inconsistencies.
- Reengineered existing ETL workflows to improve performance by identifying bottlenecks and optimizing code accordingly.
- Championed the adoption of agile methodologies within the team, resulting in faster delivery times and increased collaboration among team members.
- Participated in strategic planning sessions with stakeholders to assess business needs related to data engineering initiatives.
- Designed robust database architecture that supported seamless integration of new datasets and facilitated rapid analysis capabilities.
- Optimized data pipelines by implementing advanced ETL processes and streamlining data flow.
- Delivered exceptional results under tight deadlines, consistently prioritizing tasks effectively to meet project timelines without compromising quality or accuracy.
- Mentored junior team members in best practices for software development, code optimization, and troubleshooting techniques.
- Collaborated on ETL (Extract, Transform, Load) tasks, maintaining data integrity and verifying pipeline stability.
Senior Data Engineer
Version1
07.2022 - 10.2022
- Data pipeline development by using Pyspark, Databricks, Python, Redshift, AWS S3, DynamoDB, Glue, Athena, IAM, SQS, SES, Lambda, Terraform, Kubernetes, Redshift, DBT
- Experience in developing Spark applications using Spark (Python) in Databricks for data extraction, transformation, and aggregation from multiple file formats for analyzing and transforming data
- Implemented Data Modeling pipeline by using Glue ETL
- Contributing in all round development as product and doing R&D for current as well as new modules
- Reengineered existing ETL workflows to improve performance by identifying bottlenecks and optimizing code accordingly.
Data Engineer
Novartis, Accenture AI
12.2019 - 06.2022
- Responsible for architecting the project
- Involved in Requirement gathering, Technical and Architectural Documentation
- Data pipeline (ETL framework tool) development by using Pyspark, Databricks, Python, Snowflake, PostgreSQL RDS, AWS S3, Glue, EC2, SQS, SES, Lambda, API Gateway, Airflow, Jenkins
- Implemented Data ingestion pipeline from Multiple sources to F1
- As a ETL developer to define the transformation stage and underlying technologies that will perform formatting (Data Modeling)
- Airflow is used to schedule the tasks
- Implemented the generic tool for data reconcile (data validation according to the business logic)
- Optimization of Spark Data Frame code submit query and Databricks cluster configuration
- Developed Spark Scripts using Pyspark and SparkSQL for faster testing and processing of data
- Developed clinical metrics data loading from AWS s3 to Snowflake using Databricks ETL framework
- Reengineered existing ETL workflows to improve performance by identifying bottlenecks and optimizing code accordingly.
- Ensured data quality through rigorous testing, validation, and monitoring of all data assets, minimizing inaccuracies and inconsistencies.
- Assigning the activities to team and manage the team activity
- Work closely with software asset management to help support integration of hardware and software lifecycles
Data Engineer
UNITED AIRLINES, Accenture AI
06.2019 - 11.2019
- Data pipeline development by using Azure Databricks, Azure Blob Storage Azure Data Lake, Azure Storage, ADF, Pyspark, AWS S3, Glue, Athena, Palantir, Jenkins
- Implemented Data ingestion pipeline from Teradata to Foundry raw
- Involved in data analysis using spark sql using Data frame, Action and Transformation
- Data cleaning through Foundry Pyspark Transforms
- Converting the data frames into the required file format and Ontology Layer
- Developed Spark Scripts using Pyspark and SparkSQL for faster testing and processing of data
- Report creation through Foundry Contour Reports and Slate
- Responsible for performance optimization and Spark cluster management
- Pipeline scheduling and Data Lineage through Foundry Monocle
- Training the team on Palantir Foundry
- Optimized data processing by implementing efficient ETL pipelines and streamlining database design.
- Collaborated on ETL (Extract, Transform, Load) tasks, maintaining data integrity and verifying pipeline stability.
Data Engineer
FRDM - DBS, Accenture AI
11.2016 - 05.2019
- Analyzing the Oracle stored procedures based on the business documents
- Mapping the functional logic for each and every stored procedure defined
- Converting the Oracle stored procedure logic into Spark Data Frames using Java
- Data pipeline development by using Hadoop, Spark, Sqoop, Hive, Java, Scala, Kafka, MariaDB, Oracle, AWS S3, Redshift, Collibra, Jenkins
- Unit Testing the code using Junit
- Involved in Sonar and Emma code coverage for Junit
- Code quality checks using Jenkins and peer reviews in an Agile methodology
- Optimization of Spark Data Frame code and submit query
- Worked collaboratively on ETL tasks, ensuring data quality and pipeline reliability.
- Enhanced analytical capabilities through the design of comprehensive data models.
- Transformed data processing by establishing efficient ETL pipelines and optimizing database structures for better performance.
Education
Master of computer Applications -
Osmania University
Hyderabad, India 01.2001 - 01.2004
Skills
Python programming
Data pipeline design
ETL development
Advanced SQL
Data warehousing
Git version control
Performance tuning
Data modeling
NoSQL databases
Big data processing
API development
Data quality assurance
Hadoop ecosystem
Real-time analytics
Kafka streaming
Data security
Spark development
Data curating
Scala programming
Metadata management
Data integration
Continuous integration
Java development
SQL programming
SQL and databases
Data analysis
Relational databases
Database administration
RDBMS
Backup and recovery
Database design
Big data technologies
Problem-solving abilities
Teamwork and collaboration
Analytical thinking
Team building
Team collaboration
Excellent communication
Data aggregation processes
Data governance
Problem-solving aptitude
Amazon redshift
Multitasking
Problem-solving
Adaptability
Organizational skills
Decision-making
Data programming
Adaptability and flexibility
Data repositories
Interpersonal skills
Time management abilities
Professionalism
Continuous improvement
Analytical skills
Critical thinking
Large dataset management
Teamwork
Data analytics
Project management
Complex Problem-solving
Cloud computing
Database structures
Certification
AWS Certified Solutions Architect Associate
Countries Of Work Experience
- France
- Singapore
Timeline
Senior Data Engineer
Nomo Fintech
10.2022 - Current
Senior Data Engineer
Version1
07.2022 - 10.2022
Data Engineer
Novartis, Accenture AI
12.2019 - 06.2022
Data Engineer
UNITED AIRLINES, Accenture AI
06.2019 - 11.2019
Data Engineer
FRDM - DBS, Accenture AI
11.2016 - 05.2019
Master of computer Applications -
Osmania University
01.2001 - 01.2004