Multi-sensory Feedback-assisted Low-cost Smart Walking Cane Forming a Wearable Assembly for Blind People

2.1 Materials and Tools

The cane (Drive Medical Deluxe Folding) is a 4-section foldable medical-grade aluminum stick. In this study, the cane provides the structural capability for folding and portability, and the lightweight aluminum minimizes user discomfort, even when sensors are added. The cane has a reinforced nylon tip and a wrist strap, which are standards for providing reliability to the user and protecting the device. The belt (AGPtek, police security tactical combat gear belt) is made of durable nylon, making it lightweight and tear-resistant. It can withstand the stress, even when a wearable system package is attached. The electronic housing was constructed using additive manufacturing. 3D printing (flash) is a technology that is being used in small-scale production of various parts, such as the prototype used in this study [28]. This method involves melting or solidifying materials and stacking the cross sections of a structure individually, which allows for individual product design and reduces manufacturing costs by using only the material necessary for the shape of the part [29]. The 3D printing process is discussed in detail in Section 2.5. The Arduino board is powered by a 9 V lithium-ion power source (commercial battery). The inertial measurement unit (IMU) (6 DOF, ITG3200/ADXL345) has a MEMS structured gyro-sensor on board with a full-scale range of ± 2000°/s and a sensitivity of 14.375 leastsignificant bit (LSB) per degree per second. The IMU board has two mounting holes for secure attachment to the wearable assembly. A low operating current consumption of 6.5 mA ensures long battery life, and a stabilization time of 50 ms ensures fast use of the smart cane. A microcontroller (Arduino Mini Pro) was connected to a range finder (Ultrasonic Range Finder-LVMaxSonar-EZ3) and actuator (Mini Vibration Motor for Arduino). Software coding was accomplished using the open-source Arduino Software (IDE). The mechanical design used 3D design software (Inventor, Autodesk Co.). The test was performed in accordance with the relevant guidelines and regulations. Informed consent was obtained from all the subjects. All the methods were performed in accordance with the safety of the users. As the device was worn over clothing and operated while wearing laboratory gloves, it did not touch the subject's skin and did not cause any changes to the body. The participants were not manipulated directly. In addition, no invasive actions were realized, such as drug or blood collection, or eating. No sensitive information was collected or recorded.

2.2 Conceptual Design

As shown in Fig. 1, the smart cane performs two functions: keeping the user on a straight path, and warning the user of obstacles. The path deviation was analyzed using an electronic gyroscope mounted on the belt. The obstacles were detected using an ultrasonic sensor mounted on the cane. A small microcontroller interprets the signals from the sensors. It warns or corrects the user by activating vibrating motors in the hand (for obstacles) or waist (for path deviation). The motor intensity can vary depending on the degree of deviation. Users who receive haptic feedback in real-time interpret the vibrations in their hands or waist and apply appropriate path adjustments.

Fig. 1.

Conceptual diagram of the human–machine system.

2.3 Electrical Design

The electrical hardware, which minimizes cost and signal processing time, consists of the following parts, which can be purchased in the general market and are highly compatible components for constructing a miniaturized form factor. In addition, with the selected components, processing speed and range of detection can be flexibly adjusted.

● Arduino: Arduino is the brain behind the surgery. It records all the information from the sensors and determines whether a straight-path deviation occurs or whether the user is approaching an obstacle. If this occurs, it sends signals to the vibrators on the belt or cane to alert the user.

● Vibrotactors: Small motors placed on the belt and cane. They vibrate to warn the users that the path must be adjusted. The vibrator has an off-center motor to create nonlinear rotational vibrations. The motor has a simple structure and is reliable; therefore, it is used in various mobile devices.

● IMU: The IMU constantly reads the user positions. When the user deviates in either direction, the accelerometer in the IMU records information. The IMU gyroscope helps to eliminate the confounding effect that the user’s natural gait adds to the equation.

● Ultrasonic Sensor: An ultrasonic sensor measures the distance to a target object by emitting and receiving ultrasonic sound waves. Thus, an ultrasonic sensor utilizes echolocation to determine whether a user is approaching an obstacle.

The electrical configuration is shown in Fig. 2. The schematic shows the functionality and connectivity between multiple electrical components. A directly connected battery powers the Arduino microprocessor built into the housing system. Using the signal obtained from the IMU, the left and right vibration motors on the waist of the user can be operated separately when the device deviates from a straight path. When the Arduino receives a signal from the ultrasonic sensor that recognizes an obstacle in front, it sends a signal to the vibration motor on the handle of the smart cane.

Fig. 2.

Electrical components and block diagram

2.4 Computational Design

Based on instantaneous stimulation, users can walk safely by compensating for walking deviations and avoiding obstacles. As shown in Fig. 3, the straight-path maintenance algorithm of the code has two main feedback categories. The user must wear the cane assembly. The user presses a button for the desired walkingassistance function to supply an electrical current. The corresponding sensor module is then activated, and the step of recognizing the user walking begins in real time. As described in this section, the assembly contains two sensor modules. The user turns the device on while walking. In the first sensor, the alarms are analyzed for walking deviation using gyroscopic sensory feedback. In the second one, obstacle and alarm analyses are performed using ultrasonic sensory feedback. The two feedback algorithms operate in real-time. Haptic alarms are activated on the belt or cane when a walking deviation or obstacle is recognized. The haptic alarms can be simultaneously provided. Users can adjust their walking direction or walk around obstacles based on the haptic warnings. If no haptic warning occurred while walking, the subjects continue walking forward.

Fig. 3.

Straight-path maintenance algorithm

The pseudocode is shown in Fig. 4. The code for the sensors typically involves reading the sensor and making decisions based on its value. The actuator operates automatically using the input commands determined by the Arduino microprocessor based on the sensor values. The cane uses a sensor module that operates as a combination of a 3-axis gyroscope and a 3-axis accelerometer. To maintain the path, the code recognizes the rotational acceleration based on the z-axis when the user rotates the waist to the left and right, similar to the yaw shown in Fig. 3. The sensor generates a digital value. The change in the value of the IMU determines whether it is a left or right rotation, based on the positive and negative phase shifts. This direction can be indicated independently of the size of the IMU value. The sensor inputs a digital signal until the z-axis rotation is restored. The ultrasonic sensor also generates a digital value and inputs it into the microprocessor. At this time, the digital value varies depending on the obstacle distance. Therefore, the user can recognize the distance of the obstacle placed in front and prepare for an evasive action.

Fig. 4.

Pseudocode for sensing and vibration.

2.5 Printing for Fabrication

The prototyping process consisted of direct preparation of parts of the shielding cases via 3D printing. The 3D printer used was an Adventurer4 (FlashForge). The SolidWorks CAD software was used to design the structures. STL files were converted into sliced layers using FlashPrint software. The filament was made of polylactic acid (PLA, 1.75 mm, FlashForge). PLA is a polymer material used for printing that offers excellent interlayer bonding, which is the ability of recently extruded materials in the print path to bond well with recently printed and cooled materials. Excellent intralayer bonding significantly improves the blending of strands and layers, resulting in a user-compatible smoother surface [29,30]. In the printing setup, the extruder temperature was 225°C, the platform temperature was 55°C, and the printing speed was 50 mm/s. The fill density was set as 30%. The supporting aid option was linear with overhang critical angle of 55° and column diameter of 1.5 mm. Linear support printing requires more filaments but provides a good output quality. Tree-type support printing uses fewer filaments but tends to produce low-quality output at higher heights.

3.1 Wearable Cane

As shown in Fig. 5 (a), the research processes involve 3D printing to create a wearable smart cane assembly with ultrasonic sensors to detect obstacles, an inertial measurement unit (IMU) with a gyroscope for path detection, and real-time haptic feedback to provide outline path warnings.

Fig. 5.

Principles of assistive warning while walking. (a) Affordable wearable smart cane with added smart feedback assistance. (b) Warning system using sensor and actuator.

Fig. 5 (b) illustrates the principle of path warning. The ultrasonic sensor has an effective range of approximately 14.7 ft at a 15° angle. The sensor emitted ultrasonic waves at 40 Hz and detected the reflected echoes. The speed of ultrasonic waves in air at room temperature is approximately 1115 ft/s; therefore, dividing by the time it takes for the ultrasonic echo to return gives the distance between the sensor and the object, which is distance = (Vultra × Ttimedelay) / 2. Vibrations are generated when an object in front is detected, and a vibration motor that provides haptic feedback is fixed just below the handle of the cane. The gyroscopic sensor detected deviations during walking. When the belt was worn, a haptic feedback vibration motor was installed on the left and right sides of the user's waist. These two vibration stimuli use information from the gyroscope to provide vibration stimuli when the user deviates from a straight walking path.

Fig. 6 shows the prototype product and its assembly using 3D printing and mechanical design. Fig. 6 (a) shows a conceptual smart cane with an ultrasonic sensor attached to it. The primary control system of the wearable belt consists of a gyroscope sensor, microprocessor, battery, and Arduino mainboard fixed with a 3D-printed box. The vibrating box was securely connected to a wearable belt within a corrugated polymeric shield. Fig. 6 (b) shows the ultrasonic sensor attached to the cane. The ultrasonic sensor has an intentionally tilted angle that allows it to point forward when the user taps the stick. When walking using the smart cane, the ultrasonic sensor is placed at a 40° angle from the cane. This angle was established through geometric analysis using computer-aided design for testers with a height of 5.5 to 5.9 ft. Fig. 6 (c) shows the system housing used to functionalize the wearable belt. A sliding door is used to achieve light weight. A single-sided sliding open/closed door is available for quick maintenance, such as battery replacement and hardware debugging. It is equipped with toggle power buttons that instruct the user to begin or reset the sensors. The toggle switch that turns the main power source on and off is on the outside. This button has a raised volume that allows the user to actuate it without visual confirmation. As shown in Figs. 6 (d) and (e), the 3D housing design provides an ultrasonic sensor that can be securely attached, and the electric cables are extended to the back of the housing and immediately protected by a polymer shield. Shielding and customized structures can protect electric cables from damage during user scenarios, such as placing and swinging the cane.

Fig. 6.

Prototyping (a) Assembly of the cane (b) Ultrasonic sensor mounted on the cane (c) 3D design for housing (d) 3D designed sensor housing (e) 3D designed electrical connection.

3.2 Indoor Testing

Tests with ten blindfolded students were carried out to determine the effectiveness of the proposed smart cane, as shown in Fig. 7 (a). Two tests were performed with the participants. In the first, to determine how well the straight-path detection functioned, the participants walked 30 ft and it was ascertained how far off the path they ended up. In the second test, the participants had to avoid four obstacles placed in their path, thus allowing to test the ultrasonic sensor and reliability of the feedback. The users were blindfolded and evaluated in a controlled indoor environment to ensure safety. Each test was performed for the analysis.

Fig. 7.

Test schematic (a) Evaluation through straight path. (b) Walking on a straight path and deviation.

Fig. 7 (b) shows a schematic of the results. In the experiment involving walking on a straight path, the test quantitatively evaluated the deviation from the invisible line. Additionally, the idea examined the case of going off a straight path and going on a different path by utilizing an open space on the left. In the obstacle detection test, two cylindrical trashcans were set up on the left and two on the right, so that the user could encounter obstacles while walking. The obstacles were positioned at different distances from the path but could be directed towards it by the user swiping the cane left and right. There was a sidewall behind the obstacle, but this was an unintended structure; therefore, it was not included in the number of times an obstacle was encountered. If part of the user's body touched an obstacle, it was determined that the obstacle was not recognized.

3.3 Evaluation

In Fig. 8 (a), the analysis reveals the number of deviations within the 30 ft walking plan. Among the participants, 70% made deviations of less than 4 ft, two within 1 ft, two within 2 ft, two within 3 ft, and one within 4 ft. The results of this experiment showed that deviations within 4 ft meant that the participants were still in the testing space, and the feedback system allowed users to refine the straight path. The participants precisely corrected their paths using motor alarms and performed high-quality straight walking. In contrast, two participants deviated by 7 ft, and one wandered 10 ft away from the target path. In all attempts, the motor alarms worked adequately. The reasons for the errors in path correction are as follows: First, when the user's body is rotated around the z-axis to correct the path, an irregular path correction occurs if the body rotation angle per alarm is too large. If the body moves rapidly, the tolerance of each rotation increases, and the path-correction process creates irregular zigzag movements that gradually deviate from the path. Therefore, the motor warning operates at a constant current value independent of the value of the gyroscope. Hence, if the same rotation amount of approximately +/- 5° is applied, the exact path can be found as a result of the converging zigzag movement. Second, errors can occur when there is a lack of sense of vibration. The participant with a deviation of 10 ft had difficulty determining whether the vibration generated by the motor was to the left or right. The vibration of the motor was transmitted to the waist of the user through the assembled belt. In this process, the sense of the person's waist, thickness and structure of the clothing, and accessories can create an effect that further disperses or interferes with the vibration. The users can continuously scan by changing the vibration intensity. However, the sensitivity of each individual and variables such as clothing and rotation amount result in the need for proper training of the user.

Fig. 8.

Result and analysis (a) Number of deviations within 30 ft during walking. (b) Number of obstacles avoided by users. (c) Position estimation using links and joints.

Fig. 8 (b) shows the results of obstacle avoidance during walking. Two participants successfully recognized and avoided all four obstacles. Five subjects avoided three obstacles, and three avoided two obstacles. None of the users avoided one or zero obstacles. Because the alarm generated by the ultrasonic sensor changes the intensity of the motor according to the distance, the user can converge to the exact direction of the obstacle through continuous scanning of the cane and alarm intensity. Because the ultrasonic sensor has a frequency of 40 Hz and a repeat echo acquisition capability of 25 ms, forward monitoring precision can be improved through rapid scanning by increasing the arm swing speed or reducing the angle without hardware limitations. Fig. 8 (c) illustrates obstacle recognition using Cartesian coordinates [25]. The sensor system of the smart cane provides the distance between the ultrasonic sensor module and obstacle; however, Cartesian data processing using the structure of the arm can provide the user with precise obstacle location information. The idea proposed in this study is to fix the pitch and actively search for objects placed in the x-y plane by performing yaw rotation. The cane's yaw rotation angle (θ4), which determines the direction of the ultrasonic sensor, is controlled by the user's arm links and joints. The shoulders, elbows, and wrists are constrained to rotate. The angles of each joint are acquired through user scenes, and these angles can provide information about the yaw, pitch, and roll. The yaw and pitch angles made at the shoulder are θ1 and Ψ1, and those made at the elbow are θ2 and Ψ2, respectively. The wrist fixes the pitch rotation and creates θ3. The three axes of the Cartesian coordinates are x, y, and z, where the x-axis position of the object is Px, y-axis position is Py, and z-axis position is Pz. The links are L1, L2, and L3 from the shoulder, and the cane is L4. The following formulas calculate each Cartesian equation:

$$ \begin{array}{l}\mathrm{P}\mathrm{x}=\mathrm{L}1\mathrm{ }\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{\Psi }1\mathrm{ }\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{\theta }1+\mathrm{L}2\mathrm{ }\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{\Psi }2\mathrm{ }\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{\theta }2+\left(\mathrm{L}3+\mathrm{L}4\right)\mathrm{ }\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{\theta }3\\ \mathrm{P}\mathrm{y}=\mathrm{L}1\mathrm{ }\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{\Psi }1\mathrm{ }\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{\theta }1+\mathrm{L}2\mathrm{ }\mathrm{c}\mathrm{o}\mathrm{s}\mathrm{\Psi }2\mathrm{ }\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{\theta }2+\left(\mathrm{L}3+\mathrm{L}4\right)\mathrm{ }\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{\theta }3\\ \mathrm{P}\mathrm{z}=\mathrm{L}1\mathrm{ }\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{\Psi }1+\mathrm{L}2\mathrm{ }\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{\Psi }2+\left(\mathrm{L}3+\mathrm{L}4\right)\mathrm{ }\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{\Psi }3\\ \mathrm{\theta }4=\mathrm{\theta }1+\mathrm{\theta }2+\mathrm{\theta }3\end{array}$$

Because the field of view of the ultrasonic sensor in the x-y plane is approximately 15°, the Py range for recognizing obstacles in front of the ultrasonic sensor is 1.32 ft at 5 ft, 2.64 ft at 10 ft, and 3.87 ft at 14.7 ft. In the test, the effective field of view created by swinging the cane left and right by approximately 70° to 90° is approximately 85° to 105°. This angle is a valid range for detecting dangerous objects in front of the user.

As presented in Table 1, after completing the test, we asked the users whether they wanted more or less vibration. Sensibility ergonomics is an engineering approach that realizes images or emotions as desired in specific product designs. Vibration intensity and repeatability can be adjusted through sensibility ergonomics based on the deviation and obstacle avoidance abilities. Approximately 60% of the participants responded that they were satisfied with the vibration alarm. The responses of participant 4 were significant. He obtained a negative result of a 7 ft deviation and avoided two obstacles, and therefore responded that he wanted more vibrations. However, there was no correlation between the experimental and survey results for most of the participants. Participant 3 had a significant deviation but responded that the vibration was effective. In contrast, participants 7 and 8 successfully achieved a deviation of 1 ft and avoided four obstacles, but responded that the vibration should be improved further. In addition, participants 7 and 9 also succeeded in deviating and evading by 2 and 3 ft, and 3 and 4 ft, respectively, but expressed random claims that they wanted more vibration or that it was effective. Therefore, if this smart cane is to be distributed for personal use in the future, it should be able to customize the vibration and sensor sensitivity. In addition, we asked the participants whether dual-motor vibration was appropriate for providing a deviation alarm. The concept of dual motors was considered effective by 90% of the participants. Participant 3, who had the most significant deviation of 10 ft, stated that dual vibration was ineffective. Participant 3 was the only one who deviated significantly from the path and lost sight of the target. In contrast, the other nine participants had deviations of less than 7 ft and could steer according to the vibration. Therefore, the alarm is effective as long as the left and right vibrations can be distinguished. However, if the user cannot determine the vibrations, directional guidance does not work. For the variables that make it difficult to distinguish the vibrations, please refer to Fig. 8 (a).

Table 1.

Individual test results and opinions.

3.4 Outdoor Simulation

To reflect the willingness for future real-world testing and ensure the practical relevance of the study, preliminary graphical simulations were performed to test actual streets with people present. Fig. 9 (a) shows a 60 ft-wide crosswalk captured from an online map (Street View, Google). Based on the indoor test results, if the user wants to walk straight on the crosswalk, the prediction reveals minimum, average, and maximum deviations of 2, 8, and 20 ft, respectively. Approximately 70% of the users have deviations within approximately 2–8 ft, as indicated by the yellow and orange lines. Considering the repeated guidelines for pedestrians and stop lines for cars, this appears safe for pedestrians. However, approximately 30% of users may deviate from the pedestrian guidance lines. Although guidance based on single data using a gyroscope provides these predictions, walking can be improved if ultrasonic sensors work together. By recognizing cars stopped at the stop lines drawn on the left and right sides of the pedestrian as obstacles, the system suggests improved directionality to the pedestrian. Thus, errors are reduced.

Fig. 9.

Outdoor simulation (a) Skyview simulation using outdoor crosswalks. (b) Street view of the prediction.

3.5 Future Improvements

In this section, we review the limitations of this study and the improvements that can be made by addressing its limitations. First, a feasibility test through prototyping provided valuable insights into the effectiveness of the smart cane, and additional testing enhanced the robustness and reliability of the results. However, because only ten blindfolded subjects participated in the experiment, it is difficult to support the claim that the proposed solution is statistically valid and generalizable. Each blinded participant can repeat the experiment multiple times. This allows for a more comprehensive evaluation of the cane performance by calculating the mean, standard deviation, and least-squares error metrics. Establishing a statistical analysis based on statistical metrics (e.g., standard deviation and confidence interval) can help readers better understand the results. Second, the scope of obstacle recognition should be expanded from a single x-y plane to threedimensional recognition. The space in which a person walks should be analyzed using a three-axis coordinate system because obstacles and hazards exist in three dimensions. For example, a hanging panel can threaten the user’s upper body or a staircase in front can cause the user to fall. Multiple ultrasonic sensors can be utilized to enhance the performance of smart canes, and highperformance sensors, such as the time-of-flight (TOF) technology used in camera technology, can be used to improve resolution, fast detection, and 3D recognition.