In this project, I aimed to solve the problem of manual report generation by automating the entire process using Apache AirFlow and MongoDB. The primary challenge was managing a growing dataset from multiple sensor streams while ensuring reports were consistently generated and delivered. Using AirFlow’s Directed Acyclic Graphs (DAGs), I created a scheduling mechanism that handles the entire reporting workflow, from data extraction to processing and emailing the final report to stakeholders.
MongoDB’s flexibility as a NoSQL database allowed me to store unstructured sensor data in a format that could be easily queried and processed. This flexibility made it possible to extend the reporting system over time, incorporating new sensor types and data formats without major structural changes. Reports generated include summaries of daily, weekly, and monthly data trends, as well as real-time alerts for data anomalies that required immediate attention.
By automating the entire reporting pipeline, this solution reduced the time spent on manual data extraction and report creation by over 50%. Moreover, the system ensured that reports were sent out in a timely manner, improving the overall responsiveness and efficiency of the monitoring process. This kind of automation is particularly useful for industries that rely on continuous data, such as environmental monitoring, manufacturing quality control, or infrastructure health tracking.