Mangesh Yadav

GCP Data Engineer - EDH Migration - UK Bank - Feb '23 to Present

• Developed high-quality GCP compatible code for on-prem Hadoop code using Spark in Scala, Java.
• Migrated data from on prem Hadoop to google cloud platform using GCS, Hive, Big Query using Dataproc, Cloud Composer with Airflow.
• Created dataset, groups, tables in big query and loaded data in it from hive and tagged PII columns using Google Dataplex, Data Catalog.
• Used Jenkins Pipelines, GitHub, K8S, and Docker for code deployment, job executions and Spinnaker for image deployment into higher environment.

GCP Data Engineer - TD2C Migration / Walmart / USA - Feb '21 to May '23

• Migrated data of Walmart users data from Teradata warehouse to Google Big Query warehouse using DataStage as ETL tool and Mainframe for processing data.
• Converted complex Teradata queries having ~3000+ LOC with respective to Big Query compatible format.
• Created CI/CD pipeline using composer with Apache Airflow by preparing Dag, prop, and CIA load config files.
• Helped team members to convert complex queries and improving query performance in Big Query.

GCP Data Engineer - CDDGCP / Macy’s / USA - Oct '18 to Jan '21

• Laydown pipeline to ingest database tables from onsite database to Google Cloud Storage and Big Query using Composer with Apache Airflow.
• Created source and domain for respective customer data according to its category in Infoworks.
• Ingested data from source to Base Table, Pipeline target table in hive by designing Pipeline using Infoworks.
• Prepared python DAG for cloud composer orchestration service.
• Created historical and snapshot tables in Big Query on top of cloud storage using Cloud Composer with Apache Airflow.
• Exported data to Big Query and validate it and designed workflow for same, Scheduled jobs through IBM Control-M.

Big Data Developer - CDD / Macy’s / USA - Jan '18 to Sep '18

• Objective of this project is to migrate data to defined Data lake. Client has their customers data stored in different sources (Oracle, Teradata etc.) according to different category like LTY, EDW etc.
• Performed initial and delta load according to client requirement.
• Developed Hive DDL, Sqoop, spark processing configurations.
• Developed Oozie workflow for ingesting one TB of data.

Hadoop Developer - Web Log Processing / Cisco / USA - Apr '17 to Nov '18

• Migrated Weblog processing data of cisco.com from Hadoop to snowflake warehouse on AWS. All logs will ingested in GCS for further processing using gsutil.
• Migrated data from Hadoop cluster to Snowflake warehouse on AWS cloud by creating snowflake scripts and from Snowflake to DOMO, which is reporting tool on AWS by using DOMO connector.
• Created IBM Control-M jobs to automate execution process.

AWS Developer - Data Lake / Astro / Malaysia - Apr '17 to Sep '17

• Objective of Astro project is to migrate all client data to defined Data lake. Client has their website-accessed data stored in different sources. Migrated this data to Redshift following guidelines specified by client.
• Ingested data into S3 bucket by using Sqoop jobs.
• Developed spark code for transforming raw data into clean data.
• Worked on Face Recognition using Image Recognition service.
• Used Big Data Hadoop – Hive, Sqoop, Spark, AWS Services: S3, EC2, Redshift, Data Pipeline and Tools: GIT, JIRA, winscp.

ERP Developer - ERP / BMS Solutions / Dubai - Dec '13 to Apr '17

• Requirements gathering from client and analyze them. Prepared design flow model according to requirement.
• Collaborated with teams regarding technical issues, software system design and maintenance.
• Built ERP system for client to add, remove, and update their employee's information. Employee should be able to see their in-out timings, apply their leaves, and update their daily work status through ERP.
• Developed employee leave module, employee profile module from start to end i.e. from entry to exit from organization through ERP system.