Building a Cloud Data Lake For a Municipal Aviation Department

Project Description

The aviation department of a major Midwest City needed to build a data lake to provide centralized analytics with fast processing time. The goal was to connect on-premises data sources, develop transformation within the cloud and report out into Power BI.

Project Goals

The City needed to build a data lake in Azure that synchronized on-premises data sources in SQL into an SQL database in Azure that could process the data into a data lake storage container, then process the needed data into an SQL pool and connect Power BI to create reports and dashboards. Because data processing on-premises in this manner would require additional server resources, leveraging data lake would allow the City to scale up or down in minutes, while providing flexibility to do reporting, big or small, within seconds.

Services Delivered

• Technology implementation
• Pilot of data lake

Project Results

With the Azure data lake up and running, the City is now able to consider scheduling restrictions of the data when running queries, since the database in Azure is synchronized every hour with no impact on the production SQL database environment that is hosted on premises. Applications like face recognition, ID scanning and others that had been intermittently functional. This POC provided the ability to not interrupt those activities, while allowing activities in the database to run at any time. Processes have been simplified, integration was made easier and user satisfaction was increased.