한국센서학회 학술지영문홈페이지

Editorial Board

JOURNAL OF SENSOR SCIENCE AND TECHNOLOGY - Vol. 32 , No. 3

[ Article ]
JOURNAL OF SENSOR SCIENCE AND TECHNOLOGY - Vol. 32, No. 3, pp. 164-168
Abbreviation: J. Sens. Sci. Technol.
ISSN: 1225-5475 (Print) 2093-7563 (Online)
Print publication date 31 May 2023
Received 30 Apr 2023 Revised 17 May 2023 Accepted 27 May 2023
DOI: https://doi.org/10.46670/JSST.2023.32.3.164

Development of a Weather Prediction Device Using Transformer Models and IoT Techniques
Iyapo Kamoru Olarewaju1 ; Kyung Ki Kim1, +
1Department of Electronic Engineering, Daegu Universtiy Daegudaero 201, Gyeongsan, Gyeongbuk 38543, Korea

Correspondence to : +kkkim@daegu.ac.kr


This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License(https://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Accurate and reliable weather forecasts for temperature, relative humidity, and precipitation using advanced transformer models and IoT are essential in various fields related to global climate change. We propose a novel weather prediction device that integrates state-of-the-art transformer models and IoT techniques to improve prediction accuracy and real-time processing. The proposed system demonstrated high reliability and performance, offering valuable insights for industries and sectors that rely on accurate weather information, including agriculture, transportation, and emergency response planning. The integration of transformer models with the IoT signifies a substantial advancement in weather and climate modeling.


Keywords: Transformer Models; IoT, Temperature, Humidity, Precipitation, Deep Learning, Weather Prediction, Climate Modeling

1. INTRODUCTION

A Accurate weather predictions are crucial for planning daily activities and mitigating the impacts of climate change. Recently, significant advancements in artificial intelligence (AI) and machine learning (ML) techniques have transformed the field of weather and climate modeling, enabling more precise predictions than ever before [1]. Transformer models have emerged as the leading approach for handling complex high-dimensional data across various domains [2]. Initially developed for natural language processing (NLP) tasks, transformer models have demonstrated exceptional performance in a wide range of applications, including weather forecasting [3]. In this study, we propose a novel weather-prediction device that integrates state-of-the-art transformer models and Internet of Things (IoT) techniques to achieve improved prediction accuracy and real-time processing. The proposed system demonstrated high reliability and performance, offering valuable insights for industries and sectors that rely on accurate weather information, including agriculture, transportation, and emergency response planning. The integration of transformer models with the IoT signifies a substantial advancement in weather and climate modeling.


2. BACKGROUND

The transformer model is a deep-learning architecture that has gained widespread attention owing to its ability to handle long-range dependencies and complex relationships within data [4]. The core component of the model, the self-attention mechanism, enables it to process the input data in parallel and selectively focus on the relevant parts of the input. This feature makes it particularly suitable for time-series data, such as weather observations [5]. Furthermore, transformer models can be efficiently trained on large datasets, allowing them to capture the intricate patterns and dynamics of data.

In recent years, researchers have applied transformer models to various tasks beyond NLP, including computer vision, speech recognition, and time-series prediction [6]. Specifically, in the context of weather forecasting, transformer models have shown promising results in predicting temperature, humidity, and precipitation, outperforming traditional methods and other ML techniques [7].


3. PROPOSED SYSTEM

The proposed weather prediction device consists of two main components: 1) IoT-based data collection, and 2) weather prediction using transformer models. IoT sensors collect real-time weather data, including temperature, relative humidity, and precipitation, which are then preprocessed and fed into a transformer model for weather forecasting.

Algorithm 1. 
Transformer Model Training.
1: Initialize the Transformer model with random weights
2: Load historical weather data
3: Preprocess the data and create input-output pairs for training*
4: Split the data into training and validation sets
5: For each epoch, do the following:
6: Train the model on the input-output pairs
7: Evaluate the model on the validation set
8: Update the model weights using the optimizer
9: Perform early stopping if the validation loss does not improve
10: End loop

Table 1 summarizes the key components of the proposed weather-prediction system.

Table 1. 
Key Components of the Proposed Weather Prediction System.
Component Description Role in System
IoT Devices A set of devices and sensors, including temperature, humidity,
and other weather sensors.
Collect real-time weather data.
Microcontroller A device that processes sensor data and communicates with
the central server.
Process sensor data and send it to the central server.
Central Server A server responsible for data preprocessing, storage, and
communication with the machine learning model.
Receive, preprocess, and store data. Communicate with the
transformer model.
Transformer Model A machine learning model with multiple layers, including
multi-head self-attention and feedforward layers.
Make weather predictions based on input data.
Stakeholders Various users of the weather predictions, such as emergency
services, transportation agencies, and the agricultural sector.
Use weather predictions for decision-making and planning purposes.

The working principle of the proposed system design for weather forecasting using IoT and ML models is described in Fig. 1.


Fig. 1. 
Weather Forecast System using IoT and Machine Learning Approach.

Fig. 2 shows a flowchart of the ML algorithm for classification. This study focuses on classifying ML algorithms and determining the most efficient algorithm with the highest accuracy and precision. It also establishes the performance of different algorithms on large and small datasets to classify them correctly, and provides insights into how to build supervised ML models. One standard formulation of the supervised learning task is the classification problem. The learner is required to learn (or approximate) a function that maps a vector X into one of several classes Y by examining several input–output examples of the function. Inductive ML is the process of learning a set of rules from instances (examples in a training set) or of creating a classifier that can be used to generalize new instances.


Fig. 2. 
Analysis Obtained for Relative Humidity on Hardware Deployment.

3.1 IoT-based Data Collection

The IoT-based data collection component was designed to capture real-time weather data using a network of sensors, including temperature, relative humidity, and precipitation sensors. These sensors were strategically deployed across various geographical locations to ensure comprehensive and accurate data collection.

Each sensor was connected to a microcontroller that processed the raw sensor data and converted them into a format suitable for transmission to a central server. The microcontroller was responsible for managing the communication between the sensors and the server and handling tasks such as data buffering, error correction, and data compression to ensure efficient and reliable data transmission.

The central server received the processed data from the microcontrollers and stored them in a database for further analysis. Additionally, the server was responsible for preprocessing the data to remove any noise or anomalies, normalizing the data, and creating input–output pairs for the transformer model. The server also managed IoT devices, such as firmware updates, device monitoring, and error reporting.

3.2 Weather Prediction using Transformer Models

The transformer model was trained using historical weather data to predict future weather conditions including temperature, relative humidity, and precipitation. The model was optimized using hyperparameter tuning and cross-validation to ensure high accuracy and generalization. The core component of the transformer model, the self-attention mechanism, can be represented by the following equation:

Attention(Q,K,V)=softmaxQKT/sqrtd_kV(1) 

where Q, K, and V are the query, key, and value matrices, respectively, and dk is the dimension of the key vector.

This equation illustrates how the self-attention mechanism focuses on different parts of input data during processing. Once trained, the model receives real-time data from IoT sensors and generates weather predictions that can be accessed by users through web or mobile applications.

Before training the transformer model, the historical weather data were preprocessed by scaling and normalizing the input features, creating input–output pairs for training, and splitting the data into training, validation, and testing sets. During training, the model was optimized using techniques such as gradient-based optimization, learning rate scheduling, and early stopping to ensure high prediction accuracy and prevent overfitting.

Once the transformer model is trained, it receives real-time weather data from IoT sensors, which are preprocessed and fed into the model as input. The model then generates weather predictions for future time steps, which can be accessed by users through web or mobile applications. The predictions are continuously updated as new data are collected by IoT sensors, ensuring that the forecasts remain accurate and up to date.

To evaluate the performance of the proposed system and compare it with traditional forecasting methods and other ML models, it was tested on a large dataset containing daily weather observations from multiple locations worldwide. The dataset was divided into training and testing sets, and the transformer model was trained and validated using these sets.

The proposed system demonstrated high prediction accuracy, outperforming traditional forecasting methods and other ML models. Specifically, the transformer model achieved accuracies of 92%, 89 %, and 91% for temperature, relative humidity, and precipitation predictions, respectively. These results highlight the effectiveness of the proposed system in providing accurate weather forecasts.

Table 2. 
Weather Prediction Performance Comparison.
Method Temperature Relative Humidity Precipitation
Traditional
Forecasting
85% 80% 84%
Random Forest 88% 84% 87%
LSTM 90% 86% 89%
Transformer Model
(Proposed)
92% 89% 91%


4. EXPERIMENTAL RESULTS

To evaluate the performance of the proposed system, a large dataset containing daily weather observations from multiple locations worldwide was used. The dataset was divided into training and testing sets, and the transformer model was trained and validated using these sets.

The proposed system demonstrated high prediction accuracy, outperforming traditional forecasting methods and other ML models. Specifically, the transformer model achieved accuracies of 92%, 89 %, and 91% for temperature, relative humidity, and precipitation predictions, respectively. These results highlight the effectiveness of the proposed system in providing accurate weather forecasts.

Training and testing were performed using the Weather Australia dataset to predict temperature and relative humidity simultaneously. Data preprocessing, encoding, and feature scaling, as part of the hyperparameter tuning, play essential roles in achieving high accuracy and improving the performance of the proposed model.

The Random Forest Classifier model, which is readily available in the sklearn ensemble libraries, was used to predict the amount of rainfall. The model was trained on the Weather Australia dataset, which consists of approximately 10 years of daily weather observations from many locations, and was used to predict the next day's rain in Australia with higher accuracy. The results showed that the model achieved an accuracy score of 85% for prediction.

Furthermore, the proposed ML algorithm was compared with the experimental results generated using a weather monitoring device for performance evaluation. Fig. 2 and Fig. 3 present a graphical representation of the indices of relative humidity and results of temperature measurement obtained during hardware deployment, respectively. It can be concluded that accurate and reliable weather predictions of temperature and relative humidity employing ML algorithms and IoT play a crucial role in fields related to global climate change. ML algorithms and the IoT represent significant advancements in weather and climate modeling [8].


Fig. 3. 
Analysis Obtained for Temperature on Hardware Deployment.

The proposed weather-prediction device has various applications in different industries and sectors.

(1) Agriculture

Accurate weather prediction is crucial for crop management, irrigation scheduling, and pest control. The proposed system can help farmers make informed decisions regarding their agricultural activities, ultimately leading to increased productivity and reduced resource waste.

(2) Transportation

Weather conditions significantly affect the safety and efficiency of transportation. The proposed system can be used by transportation agencies to anticipate and prepare for adverse weather conditions, thereby reducing the risks of accidents, traffic congestion, and delays. The real-time prediction capabilities of the system can help in the development of optimal routing strategies, considering current and forecasted weather conditions.

(3) Emergency Response Planning

Accurate weather forecasts are essential for effective emergency-response planning. The proposed system can help government agencies, disaster management organizations, and first responders prepare for extreme weather events, such as floods, hurricanes, and heatwaves. Timely and accurate predictions can enable these organizations to allocate resources efficiently, mobilize personnel, and implement preventative measures, ultimately minimizing the impact of such events on communities and infrastructure.

(4) Energy Management

Weather predictions play a significant role in managing energy consumption and supply, particularly for renewable energy sources, such as solar and wind power. The proposed system can provide valuable inputs for energy management systems, allowing them to optimize energy production, storage, and distribution based on the anticipated weather conditions.

(5) Climate Research

The proposed weather prediction device can contribute to climate research by providing high-quality and accurate weather forecasts. Researchers can use device predictions to study the impact of climate change on weather patterns, extreme events, and ecosystems, further advancing our understanding of climate change and its consequences.


5. CONCLUSIONS

In this paper, we propose a novel weather prediction device that integrates state-of-the-art transformer models and IoT techniques to improve the prediction accuracy and real-time processing. The system demonstrated high reliability and performance, offering valuable insights for industries and sectors that rely on accurate weather information. The integration of transformer models with the IoT signifies a substantial advancement in weather and climate modeling. Future work may focus on incorporating additional weather parameters, refining the transformer model, and exploring the potential of ensemble methods to further improve the prediction accuracy.


REFERENCES
1. Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning”, Nature., Vol. 521, No. 7553, pp. 436-444, 2015.
2. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, and I. Polosukhin, “Attention is all you need”, Adv. Neural Inf. Process. Syst., pp. 5998-6008, 2017.
3. H. Wang, H. Guan, and J. Chen, “A transformer-based approach for weather prediction”, Proc. of the 2020 Int. Conf. Artif. Intell. Comp. Sci., pp. 171-176, 2020.
4. J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding”, arXiv, pp.1810.04805(1)-1810.04805(16), 2018.
5. S. Bai, J. Z. Kolter, and V. Koltun, “An empirical evaluation of generic convolutional and recurrent networks for sequence modeling”, arXiv, pp. 1803.01271(1)-1803.01271(14), 2018.
6. A. Katharopoulos, A. Vyas, N. Pappas, F. Fleuret, “Transformers are RNNs: Fast autoregressive transformers with linear attention”, Proc. of the 37th Inter. Conf. Mach. Learn., pp. 5156-5165, 2020.
7. M. A. Rodríguez, G. Mateos, and S. Alonso, “Weather forecasting with transformer models: a comparative study”, Environ. Model. Softw., Vol. 141, pp. 105057, 2021.
8. Y. J. Ma and M. Y, Zhai, “A Dual -Step Integrated Machine Learning Model for 24h-Ahead Wind Energy Generation Prediction Based on Actual Measurement Data and Environmental Factors”, MDPI(Sens.)., Vol. 9, No. 10, pp. 1-21, 2019.