Muhammad Shahzad Munir
London,United Kingdom
Summary
- I’m a passionate Big Data Engineer with 17+ years of experience, including 10+ years in Data Engineering, designing and implementing scalable solutions using a variety of big data frameworks.
- In-depth knowledge with proven experience of Data Mesh, Data Products, Data Lakehouse, Domain Driven Data Design, Data Quality, Lambda Architecture, Kappa Architeture, Medallion Architecture, Datawarehouse (Star & Snowflake Scehma).
- I specialize in crafting high-quality, optimized Big Data code in 𝐒𝐜𝐚𝐥𝐚, 𝐏𝐲𝐭𝐡𝐨𝐧, and 𝐉𝐚𝐯𝐚 tailored to project requirements. I create clean, efficient, and maintainable code that you’ll not only appreciate but genuinely enjoy working with.
- I have a proven track record of successfully managing technical teams throughout my professional career, ensuring efficient collaboration and delivering outstanding results.
- I have consistently excelled in engaging with diverse stakeholders and clients, understanding their unique project requirements, and delivering customized, effective solutions that align perfectly with their goals.
- Hands-on experience in developeing GEN-AI products using Open AI, Snowflake Cortex, and RAG.
- Strong In-depth hands on experience in Cloud Computing (Amazon Web Services – AWS and Azure)
- I have extensive experience working on large-scale, compute-intensive, and distributed projects for organizations across the globe, including in the UK, Belgium, USA, Germany, and Pakistan. My portfolio includes collaborations with early-stage startups as well as medium and large-scale enterprises. These projects have been successfully deployed in both on-premises environments, leveraging Cloudera and HortonWorks distributions, and cloud platforms like AWS and Azure.
- Highly confident, motivate and competent individual with excellent communication, proven verbal, analytical and problem solving skills. Possess unique skill set of handling big and complex projects, with the ability to work supportively within a team environment, and communicate effectively at all levels. Experience of interacting with the valuable, local and foreign clients and resolving their technical issues.
Overview
20
20
years of professional experience
4
4
years of post-secondary education
1
1
Certification
Work history
Sr. Data Engineer / Technical Analyst
Xorlogics LTD
London
07.2023 - Current
- Served as Technical Lead for Data Products, overseeing their design and development using advanced data architectures
- Selected and implemented the most suitable tools and technologies, such as Apache Spark, Snowflake, DBT, Scala, Python, and AWS
- Conducted R&D and implemented GEN AI Chatbots and Semantic Analysis using Snowflake Cortex AI LLMs and OpenAI LLMs
- Developed RAG (Retrieval-Augmented Generation) applications with Python, fine-tuning LLMs for specific contexts or products
- Implemented comprehensive quality checks for incoming data, ensuring the integrity and reliability of ingested and processed datasets
- Optimized system performance in terms of time and cost, including developing custom load-generation tools and conducting performance tests
- Analyzed project requirements and collaborated with Enterprise Architects to finalize solutions
- Coordinated with data scientists to translate algorithms into production systems and resolve bottlenecks in existing code
- Innovated through research, prototyped advanced features, defined key metrics, and fostered team growth by sharing knowledge of emerging technologies
- Developed and maintained features in the Streaming and Batch Layer of the system
- Monitored running states, handled release management, resolved bottlenecks, and maintained operational efficiency.
- Mentored junior engineers on best practices in big-data handling, nurturing their professional growth along the way.
- Fixed many pipelines with Data quality, skewness, resources, and performance issues
Sr. Data Engineer / Application Expert
Telenet
Mechelen
07.2019 - 06.2023
- Designed and developed Data Products using cutting-edge architectures and tools such as Apache Spark, Snowflake, and DBT with Scala and Python
- Implemented both ELT and ETL pipelines tailored to use case requirements
- Ensured strong data quality to deliver reliable Data Products
- Led the migration of an on-prem Big Data platform (100+ projects, ~4 PB of data) to AWS Cloud using alternate tools and technologies.
- Played a pivotal role as a Technical Lead in the Data Engineering department
- Analyzed project requirements, collaborated with Enterprise Architects, and finalized system architectures
- Optimized performance in terms of time and cost
- Developed and maintained features for the Batch Layer within the Lambda architecture
- Ensured system reliability through quality checks, performance monitoring, and release management
- Partnered with data scientists to productionize algorithms and integrate complex human behavior models into scalable systems
- Identified bottlenecks in code and collaborated with data science teams to resolve iss
- Conducted research and prototyped advanced features, including tools and technologies for telco data
- Innovated and shared knowledge within the team, fostering continuous improvemen
- Developed custom tools for load generation and tested performance on existing clusters
- Wrote and executed MapReduce jobs for data synchronization between system components
- Defined metrics and evaluation procedures
- Educated and mentored team members on emerging technologies and best practices.
Sr. Data Engineer / Key Analyst
Xorlogics BVBA/SPRL, The Proximus Group
Brussels
02.2018 - 05.2019
- Designed and developed Data Products using cutting-edge architectures and tools such as Apache Spark,Apache Strom, Hadoop eco system, using Scala and Python
- Implemented both ELT and ETL pipelines tailored to use case requirements
- Ensured strong data quality to deliver reliable Data Products
- Integrate complex human behavior models into scalable production platforms using Lambda architecture
- Contribute to the quality and profiling of ingested data for improved delivery
- Work with data scientists to productionize algorithms and address code bottlenecks
- Verify and optimize solutions for performance and scalability
- Fully manage features in the Batch Layer of the system
- Implement quality checks on incoming data and optimize system performance for time and cost
- Develop custom tools for load generation and conduct load tests on clusters
- Write and execute MapReduce jobs for data synchronization between system components
- Monitor application performance using various tools and technologies
- Research, prototype advanced features, and define metrics for evaluation
- Innovate, educate, and share knowledge with team members
- Foster a culture of continuous improvement and collaboration
- Maintain system running states, manage datamarts, and oversee release management
- Develop and track KPIs for various use cases.
Sr. Software Engineer / Sr. Data Engineer
Xorlogics BVBA/SPRL
Brussels
12.2016 - 01.2018
- Designed and developed Products and Projects using cutting-edge architectures and tools such as Apache, Spark, Strom, Hadoop eco system using Scala and Python
- Integrate complex human behavior models into scalable production platforms using Lambda architecture
- Develop and maintain features on the Batch Layer of the system, ensuring data quality and optimized performance
- Explore and experiment with state-of-the-art tools and technologies to enhance product capabilities
- Research and prototype advanced features while innovating and educating team members
- Coordinate with data scientists to productionize algorithms and resolve bottlenecks in existing code
- Define metrics and evaluation procedures to measure system effectiveness
- Implement quality checks on incoming data, optimize system performance for time and cost, and monitor application performance
- Develop custom tools for load testing and ensure cluster monitoring and maintenance
- Provision new systems for Hadoop and Spark clusters, maintaining system running states and ensuring stability
- Write and execute MapReduce jobs for data synchronization across system components
- Oversee release management to ensure smooth deployments.
Sr. Software Engineer / Sr. Data Engineer
Xorlogics BVBA/SPRL, Sentiance N.V
Brussels
10.2015 - 12.2016
- Designed and developed Products and Projects using cutting-edge architectures and tools such as Apache Spark, Strom, Hadoop eco system using Scala and Python
- Integrate complex human behavior models into scalable production platforms using Lambda architecture
- Develop and maintain features in the Batch Layer of the system
- Ensure data quality through rigorous checks on incoming data from various sources
- Optimize system performance in terms of time and cost efficiency
- Explore and experiment with state-of-the-art tools and technologies to enhance product capabilities
- Research and prototype advanced features to improve system functionality
- Coordinate with data scientists to convert algorithms into production systems
- Identify bottlenecks in the code and work with the data science team to verify and resolve solutions
- Develop custom tools for load generation and test cluster performance
- Monitor application performance using various tools and technologies
- Maintain and monitor Hadoop and Spark clusters, ensuring system stability and provisioning new systems as needed
- Maintain the overall system running state and manage cluster performance
- Oversee release management for smooth deployment of updates
- Define metrics and evaluation procedures, while innovating and sharing knowledge with team members
- Write and execute MapReduce jobs to synchronize data across system components
Manager, Analytics - Product Development / Software Architect
Interactive Group
07.2015 - 12.2015
- Explore and experiment with state-of-the-art tools and technologies relevant to product development
- Stay updated with the latest research in the field and translate advanced algorithms into practical product features
- Conduct research, prototype advanced features, and design evaluation tasks for analytical algorithms
- Provide domain-specific support and guidance
- Advise on the product roadmap, vision, and strategies for solving business challenges
- Identify and prioritize text analytics projects aligned with company use cases
- Develop and improve analytical techniques and capabilities
- Design and implement Lambda Architecture for scalable analytics
- Prototype analytical modules using tools like R and Weka and translate them to Java for production
- Implement distributed computing using the Hadoop ecosystem
- Define metrics and evaluation procedures for analytical solutions
- Monitor and refine features based on production performance
- Manage teams and resources to complete product modules within defined timelines
- Ensure the quality of work delivered and oversee stress, load, and performance testing for large-scale applications
- Manage release processes and deliverables
- Innovate and share insights with team members, educating them on evolving technologies
- Work closely with business and technical teams to model and solve business problems as data science challenges.
Asst. Manager Data Analytics - Product Development / Software Architect
Interactive Group
06.2014 - 06.2015
- Explore and experiment with state-of-the-art tools and technologies to enhance product capabilities
- Translate advanced research-based approaches and algorithms into practical product features
- Acquire language and knowledge resources and prototype analytical modules using tools like R and Weka
- Implement distributed computing solutions using Hadoop and translate prototypes to production-ready Java modules
- Analyze and recommend platforms and technical aspects for the product and its modules
- Research and prototype advanced features, suggesting new modules that add value to the product and benefit customers
- Design evaluation tasks and metrics for analytical algorithms to ensure product effectiveness
- Communicate with clients to gather, clarify, and analyze product requirements
- Assess customer needs to align product features and functionality with business objectives
- Manage development and modifications of system modules, ensuring timely and high-quality delivery
- Define timelines for module completion and allocate resources effectively
- Oversee the quality of delivered work and manage testing processes, including stress, load, and performance testing for large-scale user bases
- Release and Quality Assurance o Supervise release management processes, ensuring smooth deployment and maintenance
- Continuously monitor and improve the system to enhance performance and scalability.
Sr. Software Engineer/ Team Lead
Interactive Group
06.2011 - 06.2014
- Translate the R and Weka modules to Java
- Implement Distributed Computing using Hadoop Eco System
- Develop the product using state of the art tools and technologies
- Analyze and suggest the platform and technical aspects for the product and product modules
- Manage the development and modifications of system modules
- Communicate with product clients to gather and clear requirements
- Requirement Gathering
- Suggest new modules for the product that help customers and increase product value
- Analyze the requirements
- Analyze the needs of customer
- Manage the team and resources to complete a product module
- Define the timeline to complete each module of the product
- Manage the quality of the work delivered
- Manage the testing (stress, load, and performance) of the product for thousands of users
- Release Management
Sr. Software Engineer/ Team Lead
NextBridge Pvt Ltd
10.2010 - 06.2011
- Analyze and suggest the platform and technical aspects for the product and product modules
- Development of product using JAVA and PHP
- Develop SDKs to manage the data transfer between the servers
- Develop the SDKs to update the data in database if the inventory is updated in the third party softwares/inventory
- Develop the main search engine and booking platform of the product
- Develop the main data flow system of the product
- Develop various modules of the system
- Manage the development and modifications of system modules
- Communicate with product clients to gather and clear requirements
- Requirement Gathering
- Suggest new modules for the product that help customers and increase product value
- Analyze the requirements
- Analyze the needs of customer
- Manage the team and resources to complete a product module
- Defining the timeline to complete each module of the product
- Manage the quality of the work delivered
- Manage the testing (stress, load, and performance) of the product for thousands of users
- Release Management
Engineer/Solu
Shopaservice Inc
04.2009 - 12.2010
- Analyze and suggest the platform and technical aspects for the product and product modules
- Develop SDKs to manage the data transfer between the servers
- Develop the SDKs to update the data in database if the inventory is updated in the third party softwares/inventory
- Develop the basic search engine and booking platform of the product
- Develop various modules of the system
- Manage a team of 10 developers, designers and testers
- Manage the development and modifications of system modules
- Communicate with product clients to gather and clear requirements
- Requirement Gathering
- Suggest new modules for the product that help customers and increase product value
- Analyze the requirements
- Analyze the needs of customer
- Manage the team and resources to complete a product module
- Define the timeline to complete each module of the product
- Manage the quality of the work delivered
- Manage the testing (stress, load, and performance) of the product for thousands of users
- Release Management
Sr. Software Developer
Cyberdesignz
04.2008 - 03.2009
- Within Cyberdesignz worked on various client projects which range from SDK development to Web portals
- Also, part of the technical resource team to provide technical support to customers in case of issues in the deployed SDKs and software systems
- Role & Responsibilities:
- Analyze and suggest the platform and technical aspects for the product and product modules
- Development of product using JAVA and PHP
- Manage the development and modifications of system modules
- Communicate with product clients to gather and clear requirements
- Requirement Gathering
- Suggest new modules for the product that help customers and increase product value
- Analyze the requirements
- Analyze the needs of customer
- Manage the team and resources to complete a product module
- Defining the timeline to complete each module of the product
- Manage the quality of the work delivered
- Manage the testing (stress, load, and performance) of the product for thousands of users
- Release Management
Sr. Software Developer
OVEX Technologies
12.2006 - 04.2008
- Within OVEX Inc worked on various internal products which range from SDK development to Intranet based Web projects
- Also, part of the technical resource team to provide technical support to customers in case of issues in the deployed SDKs and software systems
- Role & Responsibilities:
- Analyze the software requirements
- Develop the WBS (Work Breakdown Structure) of the software
- Develop and update the related documentation (SRS, DFD, ER) for each module and product
- Development of intranal product using PHP and related frameworks (Codignator, Zend)
- Design the architecture of the new modules
- Maintain the running state of the systems
- Monitor the database and perform regular backups of it
- Monitor the performance of the deployed systems
- Manage the testing (stress, load, and performance) of the product
- Release Management
Software Engineer
XpertSol Technologies
06.2005 - 12.2006
- Within XpertSol Tech worked on various Online Web Portals
- Also, part of the technical resource team to provide technical support to customers in case of issues in the deployed SDKs and software systems
- Role & Responsibilities:
- Design and develop the inhouse and outsource web and SDK projects
- Develop SDKs to sync the data between different servers
- Develop and update the related documentation (SRS, DFD, ER) for each module and product
- Development of product using PHP and VB.net
- Monitor the database and perform regular backups of it
- Provide Maintenance and support after deployment
- Manage the testing (stress, load, and performance) of the product
- Release Management.
Education
Bachelor of Engineering - Software Engineering
University of Engineering and Technology (UET Taxila)
Taxila Pakistan 08.2001 - 07.2005
Skills
- Big Data Technologies
- Scala
- Python
- Apache Hadoop
- Apache Spark
- PySpark
- Snowflake
- DBT
- Apache Parquet
- Apache Storm
- Dagster
- Apache Kafka
- Deltalake
- Apache Iceberg
- Airflow
- AWS-EMR
- AWS-EMR-EKS
- AWS-EKS
- AWS Lambda
- AWS-S3
- AWS-Athena
- AWS Glue
- AWS Dynamo DB
- AWS-Cloudwatch
- AWS-MSK
- Apache Zookeeper
- Apache Solr
- Apache YARN
- Apache HDFS
- Apache MapReduce
- Apache Hive
- Impala
- Apache Kudu
- Apache Avro
- Apache Thrift
- Hue
- Apache Presto
- Apache Kerberos
- Apache Knox
- Apache Ranger
- Hadoop KMS
- Confluent
- Redis
- Cloudera
- Redis Labs
- ORC
- Datalake
- Data Lakehouse
- Data Products
- Data Mesh
- Domain Driven Data Design
- Lambda Architecture
- Kappa Architecture
- Medallion Architecture
- Star and Snowflake Schema
- GEN AI Technologies
- Snowflake Cortex
- Open AI
- Copilot
- LLMs
- Embedding Models
- RAG
- Vector DB
- Database Management Systems
- Oracle
- MySQL
- PostgreSQL
- SQL Server
- MS Access
- DataGrip
- JAVA
- MapReduce
- PL/SQL
- Shell Scripting
- Agile
- Scrum
- UML
- OOAD
- TDD
- BDD
- Apache Maven
- SBT
- Jenkins
- Git
- PyPI
- Docker
- Kubernetes
- Cloud Computing
- Azure
- AWS
Certification
- SnowPro Core
- Generative AI Fundamentals
- DBT Fundamentals
- Databricks Lakehouse Fundamentals Academy Accreditation - Databricks Lakehouse Fundamentals
- Architecting Big Data Applications: Batch Mode Application Engineering
- Architecting Big Data Applications: Real-Time Application Engineering
- Big Data Analytics with Hadoop and Apache Spark
- Data Engineering Foundations
- GDPR Compliance: Essential Training
- CMMI Level 3
Custom
- Attended a one week workshop on PMP at Interactive Group of Companies.
- Attended a one week workshop on CMMI level 3 at Interactive Group of Companies.
- President of WIZCOM (Computing Society of University). January 2005 - September 2005.
- Participated in SAZSCOM competition as Computer Genius from University in June 2004.
- Head organizer on Annual games opening ceremony June 2004.
- Won Scholarship and Talent Award throughout the degree. 2001 - 2005.
- Attended Workshop of Visual Basic at University in 2002.
- Attended Workshop of PHP in SARF technologies in 2004.
- Attended Workshop of Oracle DBA track at University in 2004.
- Conducted a Workshop of JSP and PHP at University in 2005.
- Passed Basic Computer Diploma from CDS (College of Digital Sciences) Bank Road Saddar Cantt.
Personal Information
Nationality: Belgian
Timeline
Sr. Data Engineer / Technical Analyst
Xorlogics LTD
07.2023 - Current
Sr. Data Engineer / Application Expert
Telenet
07.2019 - 06.2023
Sr. Data Engineer / Key Analyst
Xorlogics BVBA/SPRL, The Proximus Group
02.2018 - 05.2019
Sr. Software Engineer / Sr. Data Engineer
Xorlogics BVBA/SPRL
12.2016 - 01.2018
Sr. Software Engineer / Sr. Data Engineer
Xorlogics BVBA/SPRL, Sentiance N.V
10.2015 - 12.2016
Manager, Analytics - Product Development / Software Architect
Interactive Group
07.2015 - 12.2015
Asst. Manager Data Analytics - Product Development / Software Architect
Interactive Group
06.2014 - 06.2015
Sr. Software Engineer/ Team Lead
Interactive Group
06.2011 - 06.2014
Sr. Software Engineer/ Team Lead
NextBridge Pvt Ltd
10.2010 - 06.2011
Engineer/Solu
Shopaservice Inc
04.2009 - 12.2010
Sr. Software Developer
Cyberdesignz
04.2008 - 03.2009
Sr. Software Developer
OVEX Technologies
12.2006 - 04.2008
Software Engineer
XpertSol Technologies
06.2005 - 12.2006
Bachelor of Engineering - Software Engineering
University of Engineering and Technology (UET Taxila)
08.2001 - 07.2005
Similar Profiles
Joanne LittleJoanne Little
Director of Curriculam at Newham College Sixth Form CentreDirector of Curriculam at Newham College Sixth Form Centre
Victoria MakindeVictoria Makinde
Volunteer Advocate at Rethink Mental IllnessVolunteer Advocate at Rethink Mental Illness
Natalia KhanNatalia Khan
Assistant Graphic Designer & Administrative Assistant at AvantageGoAssistant Graphic Designer & Administrative Assistant at AvantageGo
Elena GeorgievaElena Georgieva
Domestic Cleaner at Self EmployedDomestic Cleaner at Self Employed
Soniya BhadoriyaSoniya Bhadoriya
Senior Technical Data Analyst (Product Owner, Business Analyst, Data Engineer) at Commonwealth Bank of AustraliaSenior Technical Data Analyst (Product Owner, Business Analyst, Data Engineer) at Commonwealth Bank of Australia