Mallikarjun Jogannavar

Developed and optimized SQL and PySpark scripts for importing new client data products into AWS Athena and Starburst platforms, enhancing data accessibility.

Automated data workflows by implementing Apache Airflow DAGs to trigger and execute scripts on AWS Glue instances, improving operational efficiency.

Designed and created robust staging and reporting tables for new data streams, ensuring consistent and reliable data flow to business dashboards.

Proactively monitored and debugged AWS Glue jobs, including fine-tuning worker node requirements to enhance job performance and stability.

Utilized GitHub for collaborative development, ensuring version control and streamlined team coordination across data engineering initiatives.

Engineered and deployed robust data pipelines to ingest raw data from diverse sources (RDBMS, On-Prem) into ADLS, achieving a 25% reduction in data processing time.

Developed and implemented ETL processes for large raw datasets from ADLS into stage tables, integrating 50+ data quality checks via IIG Framework to ensure data integrity.

Successfully migrated over 10,000 lines of consumption/gold-layer table code and complex Hadoop SAS programs into optimized Databricks Notebooks, improving data accessibility and performance.

Designed and implemented robust Delta Lake architectures utilizing external and managed tables, significantly enhancing data lake efficiency and query performance.

Utilized Azure DevOps to facilitate seamless team collaboration, version control, and CI/CD for data engineering initiatives, improving development lifecycle efficiency.

Completed intensive training in big data architecture, PySpark, and Hive, building foundational expertise in distributed data processing and warehousing concepts.

Executed an end-to-end project to transform raw airline data using PySpark, successfully storing and processing it in Hive tables for dashboard visualization.

Applied theoretical knowledge to practical data manipulation tasks, demonstrating early proficiency in developing data solutions for analytical consumption.