Data Engineer

● Tasked with defining project scope, collecting business requirements from stakeholders, analyzing requirement documents, and providing accurate effort estimates. ● Used custom connectors in Databricks to integrate with various third-party APIs, NoSQL databases, Webpages, extending the ETL capabilities and enabling seamless data ingestion. ● Built and manage pipelines to extract data from Cosmos DB, Web Applications and SQL Server, transform, and load it into ADLS, Delta Lake using Azure Databricks, Python. ● Configured Azure Key Vault for credentials management, improving security and compliance, and used Azure Purview for unified data governance and auto data discovery. ● Developed and maintained robust CI/CD pipelines utilizing GitHub Actions and GitLab Runners, integrated with Azure services such as Azure Repos and Azure Artifacts. ● Monitored and troubleshot data pipelines using Azure Monitor, Log Analytics, and Application Insights, ensuring high availability, performance, and minimal downtime. ● Develop batch processing solutions using ADF and Databricks, worked with .JOSN, Avro and Parquet files and converted the data from either format Parsed Semi Structured JSON data. ● Ingested data in mini-batches and performs RDD transformations on those mini-batches of data by using Spark Streaming to perform streaming analytics in Databricks, Synapses. ● Utilized Azure Synapse and PolyBase for seamless data transfer, enabling efficient movement of data between different systems and achieving streamlined data integration. ● Developed interactive Power BI dashboards and reports, providing actionable insights and facilitating data-driven decision-making, and published into Power BI Services. Tech stack: Databricks, ADF, Synapses, ADLS, Python, Pyspark, SQL, DevOps, Git, Power BI, CosmosDB

Data Engineer

● Participated in delivery of exceptional customer experience by building relationships through proactive outreach as well as proposing appropriate products, services, and digital solutions to help customers succeed in providing BI, ETL/ELT, Data Lake, DWH solutions. ● Implemented enterprise-level Azure solutions such as Azure synapse analytics, Azure Data Factory, Databricks, Fabric, Logic Apps, Azure Storage Account, and Azure SQL DB. ● Worked on various SSIS data transformations such as Data Conversion, Derived Column, Script Component, Lookup, Execute SQL Task, and various containers etc. ● Used Azure Key vault as central repository for maintaining secrets and referenced the secrets in Azure Data Factory and in Databricks notebooks. ● Analyze, develop, and construct modern data solutions that allow Power BI visualizations utilizing the Azure PaaS services. Determine the impact of the new implementation on existing processes by understanding the present status of the application in production. ● Automated the manual data reconciliation process between the different sources, increasing the data accuracy and efficiency by 90% using Synapse, Databricks and ADF. ● Involved in code review and bug fixing for improving the execution performance of queries ● Automated testing which doubled code coverage and increased unit and integration tests quantity by a factor of four. Cut testing lead time from one person-month to two hours. ● Collect and analyse extensive data from multiple sources developed Power BI reports. Tech Stack: ADF, Databricks, Fabric, Synapse, Data Lake, Python, PySpark, SQL, Git, Power BI

Data Analyst

● Collaborate with subject matter experts, analysts, and managers to resolve data and production issues, ensuring process quality in BI and ETL Projects. ● Design, testing, implementation, maintenance, and control of the organization's physical, relational, and object-oriented databases across multiple platforms and environments. ● Worked extensively on building data warehouse structures, including the creation of facts, dimensions, and aggregate tables. Employed dimensional modeling techniques and utilized star and snowflake schemas to ensure optimal data organization and retrieval. ● Created complex SSIS packages using various transformations and tasks like Sequence Containers, Script, For Loop and Foreach Loop Container for data integration and loading. ● Developing Power BI, Tableau reports for large enterprise-scale clients with story-telling and multiple drill-downs that have heavy transaction data with volumes of data represented. ● Used Alteryx Designer to create analytical and ETL workflows to data transformations. ● Developed and published reports and dashboards using Power BI and wrote effective DAX formulas and expressions to derive the required metrics and KPI’s. ● Used Power Query data preparation and shaping data in editing queries like Append queries, merge queries, add conditional columns, Pivot columns, and Unpivot Columns. ● Conduct weekly status updates with business users, ensuring their satisfaction with report status, solutions and insights provided and noting changes the client wishes to be done. ● Published Power BI reports in Power BI Services, install and configured Gateways to schedule the report datasets, and assigned required permissions to users.