More than a decade ago I worked on a topic named Real Time Remote Monitoring of Pollution for my Doctoral Thesis. At that time Big Data technologies and advanced sensors were not available. It took sometime to come up with the monitoring system.
In this blog I would like to revisit Real Time Remote Monitoring of Environmental Parameters using simple streaming to internet using Dweet / the advanced technologies namely Hadoop (I have taken Hortonworks HDF / HDP); Rasberryi Pi 3 and Sense Hat along to demonstrate the simple flow of sensor data capture and streaming. The same scenario is also demonstrated using Microsoft Azure platform using Data Lake and Advanced Machine Learning algorithm to predicting the daily hourly temperature using Time Series analysis.
- Simple measurement of environmental parameters and their streaming using Dweet: Dweet.io is a no sign in platform to stream real time sensor data using the existing API. Default API as along with simple measurement of time bound data along with their streaming to internet can be developed using a simple code as shown in web page dweet.io. Check below site for real time updates of weather data posted at Hyderabad, India.
- Real time streaming of environmental data using HDF/HDP and Zeppelin (Hortonworks): Using Apache MiNiFi it is possible to gather sensor data from the Rasberry Pi, once collected MiNiFi data is transported using NiFi by means of site to site upload. Once uploaded the data is passed on to Hbase to perform data analytics using Zeppelin. MiNiFi is built to live on the edge for ingesting the data and using either Java or C++ flavor it is possible to collect the data from Rasberry Pi. Details on how to set up the scenario is given in Hortonworks sample projects site.
- Real Time collection using Azure Data Lake and Azure ML: Using Microsoft Azure IoT and Visual Studio, SenseHat and Rasberry PI it is possible to create excellent scenarios. One such scenario is acquiring sensor data from SenseHat for say temperature, upload data to IoT Hub, feeding to Stream Analytics, Store to an Azure SQL database and using Service Bus. This scenario can be extended to predict temperature based on machine learning model. There is literature available which used Linear Regression for selecting the appropriate features using Statsmodel / scikit learn and Neural Network for predicting the weather. There is also a reference of using directly the Microsoft IoT and Azure for weather forecast. When a default algorithm is set up as web service and deployed to IoT hub along with streaming job for input and output, it automatically predicts the measurements. Microsoft storage explorer can be used to view the data.
- IoT Leonardo based scenarios: Similar scenarios can be demoed in SAP using Big Data, SDA, SDI, HANA along with SAP EHSM Environmental Compliance to capture data as well as triggering predictive machine maintenance activities for systems generating varied environmental parameters or even triggering Exceptions and Exception reasons. Details about setting up a IoT using SAP HANA Cloud Platform is given by Craig Cmehil. In my future blog i would like to cover IoT scenario covering SAP IoT. I have written a preliminary blog on that topic here.
Compared to last time during my Doctoral days, where I have relied on multiple sensors, Microcontrollers and cluster of servers to predict meteorological information (WRF). I thought creating a real time scenarios are very easy and further advanced technologies like Big Data along with sophisticated Machine Learning technologies enable development of sophisticated scenarios. The scenarios can also be extended to assess the Pollution Scenarios by embedding with SOx, NOx, CO and other parameters monitoring. Last time I have used VB or Oracle to create the database to show the Web Pages and now using big data tools like Zeppelin or Microsoft tools I could stream the same information. Further simple tool like Dweet / Thingspeak or similar tools help in creating simple tools very rapidly. The availability of Cloud Technologies help in deploying such sensors in a matter of hours, we need to develop scenarios be it measurement of soil moisture and temperature and switching on the watering meters; or real time traffic insights or other sophisticated environmental applications.
Lastly as mentioned in the video post by Microsoft showing Scott Hanselmann, Opportunities are Immense – “Develop Without Limits !!!”.
Thanks – Jak