Current Issue

Immersive multimedia is considered to be the next generation of multimedia services because it has it significantly impacts the emotions, sense of presence, and engagement of a user [1]. The sense of presence and immersive experience in immersive multimedia are determined by sensory modalities and surround effect [2,3]. In particular, sensory modalities such as haptics, smell taste, and audio-visual content improve the quality of experience (QoE) of a user for immersive multimedia [4]. Therefore, it is important that a user perceives sufficient information through sensory modalities to experience immersive multimedia [5]. Among these sensory modalities, olfaction or the sense of smell, is an important perceptual function. Olfaction is known to enhance the quality of life and facilitate memory recall [6]. In addition, the olfactory stimuli enhance the productivity, alertness, and physical performance [7]. In the case of olfactory-enhanced multimedia, a computer-generated smell is combined with additional media content to create a richer and/or more realistic experience for a user [8]. Several studies have been conducted to investigate the impact of olfaction on the user for olfaction-enhanced multimedia. Murray, Niall, et al. showed that the type of scent affects the QoE of a user and statistically significant differences between pleasant and unpleasant scents existed [9]. Ghinea, Gheorghita, and Oluwakemi Ademoye found that the association between scent and content has a significant impact on the user-perceived experience of olfactory-enhanced multimedia [10]. Additionally, they found that olfaction significantly improves the user multimedia experience [11]. Nakamoto et al. showed that presenting smells accurately enhanced the content [12]. The studies regarding olfaction-enhanced multimedia can be divided into two parts in terms of encoding and decoding of odor (scent) metadata (odor label, strength, concentration, and harmfulness). The first part is related to research using an olfactory display which can present odors to multiple users. This research focuses on adjusting important parameters which are associated with atomization, distribution, ventilation, and synchronization of odor [13-16]. The second part focuses on coding odor metadata obtained from an electronic nose. In these studies, an electronic nose (e-nose) was used to qualify and quantify an odor [17,18]. Most of the olfaction metadata coded from an e-nose was transferred to an olfactory display through a network and the olfactory display generated an odor corresponding to the odor recipe [19,20]. Several studies have been conducted to combine olfaction with multimedia using an e-nose and olfactory display. However, their practical applications are restricted due to lack of device diversity, inability to utilize non-normative data templates, and absence of a generalized design for devices. Therefore, it is crucial to select a suitable standard that is universally accepted. ISO/IEC 23005 (MPEG-V), a representative standard related to immersive media, provides normative data templates and commands for the sensory effects of immersive multimedia [21-27]. In particular, this standard specifies information representation which can be used to control olfaction-related sensors (e-nose) and the actuators (olfactory display). This paper proposes a method to standardize odor metadata obtained from an electronic nose system which was particularly used to acquire data from different coffee aromas. Subsequently, the data were transferred to an odor display applicable for olfactory-enhanced multimedia using the MPEG-V (ISO/IEC 23005) data template. A virtual instrument and MPEG-V data templates used for e-nose system were presented in this paper.

The main controller of the e-nose system for data acquisition was developed as shown in Fig. 5. The microcontroller unit (MCU) was used to control the sensors and actuators. The four MOS-type sensors were connected to each ADC channel of the main controller. The pump was connected to OC1A (PB5) of MCU and controlled by PWM (pulse-width modulation) method. The solenoid valve was connected to the MCU through a Darlington transistor. A USB to UART cable was used to transfer the response of the sensor array to the virtual instrument.

Fig. 5. 
Main controller board of E-Nose system.

The eight normalized patterns obtained from the e-nose system or the training data, are listed in Table 4. The traditional backpropagation (BP) algorithm was used for coffee label recognition (0: Dharkan, 1: Rosabaya). The network results for the training data are listed in Table 5. The trained network accurately recognized the coffee labels of all samples. The trained networks were embedded in the virtual instrument to recognize the coffee label of the test data.

Fig. 6 shows the implemented virtual instrument graphical interface. The response of sensor array to the sample is displayed at the left-top side. The normalized pattern for the response is displayed at the left-bottom side. The recognition result, strength, chemical gas density (unit), and harmfulness are displayed at the right-top side. At the right-bottom side, the two tree views display information about the recognized odor metadata and the capability of the e-nose, which are encoded to each template of MPEG-V.

Fig. 6. 
VI (Virtual Instrument) graphical interface.

Fig. 7-8 and Table 6 show and list results of two coffee labels, which are Dharkan and Rosabaya, classification and encoded data template for the test set. The strength and harmfulness were determined by a look-up table containing their values according to the recognized coffee label. In particular, the strength values inside the look-up table were defined considering the values of intensity listed in Table 1. Since experimental results were not available to estimate the concentration, the corresponding values for Chemical Gas Density and Chemical Gas Density Unit were treated as not applicable (N/A). The odor metadata for the Dharkan test sample and capability of the e-nose system were encoded to Enose Sensor Type and Enose Sensor Capability Type and then displayed hierarchically on the tree viewers. The odor metadata comprised of the recognized coffee label (Dharkan), strength (very_strong), harmfulness (no_harzard) was listed on the corresponding tree viewer.

Fig. 7. 
VI and results for Dharkan test sample.

The representative patterns for the test samples and their recognition results are listed in Table 6. The trained network accurately classified the coffee labels of the test set.

Although sensor 4 exhibits a relatively good selectivity to Rosabaya, there are no significant differences between the response of sensors for the samples, Dharkan, and Rosabaya. Therefore, an investigation on various gas sensors to improve selectivity is additionally required.

E-Nose has been widely used to assess the quality of coffee [30]. In particular, artificial neural network (ANN) is generally used to distinguish the coffee label. In this study, the BP algorithm, which is an ANN algorithm, was used. Although E-Nose has applications in different fields, the diffusion of e-nose in real-life applications is still limited due to absence of standards related to data template, interface, calibration techniques, and reference materials. Under such circumstances, European e-nose standardization initiative (Nose II) proposed standard data formats and utilization of IEEE 1451, a set of smart transducer interface standards using open, common, network-independent communication interfaces [31]. In addition, a systematic way to develop a standard e-nose was proposed [32]. In this article, MPEG-V standard data templates associated with e-nose were provided. In addition, the e-nose system, which adopted the data templates, was presented including the virtual instrument which can recognize the coffee label and encode odor metadata to the data templates.

In this paper, MPEG-V standard data templates related to e-nose for olfactory-enhanced media, e-nose system adopted data templates, and virtual instrument used for the recognition of coffee label and encode odor metadata to the data templates, were presented.

The olfactory-enhanced media requires standard data templates of odor metadata, which can be easily combined with existing visual-audio content rather than data templates describing the detailed information (e.g., output voltage level of each gas sensor) detected through an e-nose. The detailed information would be interpreted differently in recognition processes. Therefore, it is important to provide a generic standard metadata for future content-based multimedia application. The research for the standardization of e-nose is crucial because a normative model, interface, and data format can accelerate the commercialization and ensure interoperability of the devices, and enhance the user experience.