Sensor Data-Driven Modular Network Model for Jump Motion in One-Legged Robot

2.1 Analysis of Human Jumping Motion and Phase Definition

The classification of “preparation, acceleration, flight, buffering, and recovery” [17] serves as a foundational reference for understanding the characteristics of human jumping and for learning humanoid robot motions. Based on this, various studies pertaining to robot jumping have classified motion into three phases, i.e., preparation, flight, and landing [18,19], or defined it as a two-phase process comprising flight and landing by merging the preparation and flight phases [20,21].

In this study, we categorized the robot’s jumping motion into three phases, i.e., push-off, flight, and landing, based on the points of system state transitions, as shown in Fig. 2. The push-off phase focuses on generating propulsion by maximizing ground reaction forces; the flight phase prioritizes center-of-mass (CoM) stabilization and posture maintenance, and the landing phase centers on shock absorption and balance restoration.

One-legged robots have an extremely narrow support base, which renders posture and balance maintenance challenging. Under these conditions, the robot can learn precise jumping motions while maintaining a stable balance by distinguishing the phases of the robot based on the system state transitions and by clearly defining the operational requirements and postures for each phase.

2.2 Design of Reward Functions

The one-legged robot used for RL in this study comprises six joints, and its action is defined as the torque of each joint at each time step.

In this study, we utilized robot state data obtained from a simulation environment based on information measured using sensors in an actual robot system. The state vector is defined as follows:

Here, h denotes the height of the robot top, which can be measured using motion-capture equipment with attached markers in an actual robot system. Meanwhile, α, β, and γ represent roll, pitch, and yaw, respectively, which allow the robot’s posture and velocity to be measured using an IMU sensor. Additionally, the angle and angular velocity of each joint can be measured using absolute encoders attached to the joints.

As illustrated in Figs. 2 and 3, the motion of the one-legged robot is divided into three phases, i.e., push-off, flight, and landing, based on the system state changes, with each phase having distinct target height and posture requirements. The reward function is decomposed into multiple components to systematically reflect the varying target motions that the robot must achieve throughout this process. Additionally, the reward function designed to achieve the target height and postural stability required in each phase of the jumping sequence is expressed as follows:

Fig. 3.

Design of reward function for implementing robot’s jumping motion at each phase.

Here, r_v represents the velocity reward, r_h the height reward, r_p the posture reward, r_c the center of mass (CoM) reward, and r_a the action constraint reward. The velocity reward contributes to maintaining propulsion, the posture reward ensures rotational stability, and the CoM reward ensures landing stability. The mathematical formulations for each reward are listed in Table 1. The velocity reward is defined as the squared error relative to the target velocity, whereas the posture reward is designed to minimize deviations in the robot’s upper-body inclination (roll and pitch). The CoM reward is designed to minimize the distance deviation from the vertical axis.

Table 1.

Definition of reward function for jumping motion

By adjusting the target values for r_h and r_p (height and posture) within a unified reward-function structure, the specific characteristics of each phase are effectively reflected. This intuitive, systematic design enables efficient learning of the propulsion, postural stability, and balance recovery required in each phase. In Table 1, P_d,h, R_d,h, and F_d,h correspond to the robot’s target height, body length, and foot height for each phase, respectively, as illustrated in Fig. 3; P_h, R_h, and F_h denote the robot’s real-time height, body length, and foot height, respectively; and p_CoM and p_foot represent the positions of the robot’s CoM and foot, respectively.

2.3 Transition Model for Phase-wise Motion Switching

In the preceding section, the jumping motion of the robot was classified into three phases, and distinct reward functions were employed in each phase to enable independent learning. However, during motion generation, if the posture of the robot at the moment of phase transition deviates significantly from the initial posture used during training, then the transition may not be smooth, thus resulting in postural instability. Such discrepancies can result in propulsion loss, balance collapse, or landing failure, thereby degrading the overall stability of the jumping motion.

Hence, we propose a skill-transition model that mitigates state discrepancies during transition intervals and ensures smooth policy switching. The architecture of the model is illustrated in Fig. 4. Each phase-specific policy network is trained independently and, upon completion, is sequentially connected by a transition manager. The transition model encodes the action vector from the policy of the previous phase and the state vector of the current phase to generate an action correction term, $$ a_t^{Trans }$$, for the transition interval. The skill-transition model receives two primary inputs, which are expressed as follows:

Fig. 4.

Schematic diagram of overall system utilizing skill-transition model

Here, $$ a_t^{\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f} }$$ denotes the difference in actions ($$ a_t^1-a_t^2$$) output by the policies of the two transitioning phases, $$ s_t^{\mathrm{s}\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{c}\mathrm{h} }$$ represents the robot’s state information at the moment of transition, $$ a_t^{\mathrm{s}\mathrm{e}\mathrm{l}\mathrm{e}\mathrm{c}\mathrm{t} }$$ and indicates the action output generated by the policy of the selected phase.

Meanwhile, $$ o_t^{\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s} }$$ and $$ o_t^{\mathrm{R}\mathrm{o}\mathrm{b}\mathrm{o}\mathrm{t} }$$ are processed through separately designed dedicated encoder models, i.e., the transition-s mand robot-state models, to extract the features. Subsequently, these features are concatenated and input into the action model. Based on the combined features, the action model derives the optimal corrective action, $$ a_t^{\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s} }$$, required during the transition interval. Finally, the corrective action $$ a_t^{\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s} }$$ is combined with the action $$ a_t^{\mathrm{s}\mathrm{e}\mathrm{l}\mathrm{e}\mathrm{c}\mathrm{t} }$$ from the selected phase model to determine the final control action of the robot.

2.4 Training Strategy for Skill-Transition Model

This section introduces a training strategy that utilizes the skill-transition model. The skill-transition model is trained to correct actions for a specified duration precisely when the phase-specific learned policies (π₁, π₂, π₃) transition. As illustrated in Fig. 4, the transition manager, which selects the subsequent model based on the robot’s current state, s_t, is designed as follows:

Here, P_h represents the target height of the robot used during the training of each phase. The next phase is determined based on the current phase and the robot’s real-time height information. Meanwhile, the robot’s action is determined by the policy for the corresponding phase, as shown in Eq. (7).

At the moment of phase transition, the skill-transition model is activated for a specified duration to correct the action and simultaneously obtain the training data for that interval. Data acquisition is repeated across Phases 1–3, and the data from each phase are used to train the skill-transition model. In this study, the activation times were experimentally set to 0.2, 0.5, and 1.0 s for the transitions from Phase 3 to Phase 1, Phase 1 to Phase 2, and Phase 2 to Phase 3, respectively. These durations represent the minimum time required to correct actions stably and allow the policy to adapt to each phase. This phase-transition training was performed in parallel across 100 environments, and by sampling at 0.01 s intervals, we obtained 2,000, 5,000, and 10,000 training data points. Utilizing the obtained data significantly improved the generalization performance and stability at action transitions and across various robot postures.

Here, R_Phase denotes the phase-specific reward function and R_Imitation represents the error between the output of the phase model and that of the skill-transition model. By introducing a reward (R_Imitation) that mimics the action of the transitioned phase, the skill-transition model is trained to stably assist the output of the phase model without generating excessive control signals.

3.1 Simulation Environment

Simulations were conducted to evaluate the performance of the phase-wise modular learning method and the skill-transition model proposed in this study. The proximal policy optimization (PPO) algorithm [22] was employed for the robot's policy training; it is a representative policy optimization method known for providing stable, efficient learning in reinforcement learning. RaiSim [23] was used as the simulation environment, as it enables precise modeling of robot dynamics, supports parallel configuration of environments, and allows control of diverse physical conditions. To verify the performance of the phase-wise learning and the Skill Transition Model, 100 parallel simulations were executed. This approach enabled the effective collection of phase-wise training data and transition-interval data, facilitating a systematic evaluation of the model's generalization performance and stability.

3.2 Results of Jumping Motion via Modular Learning

The jumping motion of the one-legged robot implemented using the proposed phase-wise modular learning method is illustrated in Fig. 5. Fig. 6 presents the training results for each phase-specific model, which show that the reward values converged rapidly. Additionally, the skill-transition model trained based on these phase models exhibited stable learning. The maximum reward for all models was normalized to 1. However, the reward for the skill-transition model was lower than that for the phase models because of the relatively short transition interval, which resulted in a smaller accumulated reward. The motions learned by each phase model were combined sequentially to constitute a single complete jumping motion.

Fig. 5.

Simulation results of robot’s jumping motion (top: without skill-transition model; bottom: with skill-transition model)

Fig. 6.

Training results for each model

The upper section of Fig. 5 shows the results obtained only with transitions between phase models, i.e., without the skill-transition model, whereas the lower section shows the results with the skill-transition model applied. In the absence of the skill-transition model, while the basic jumping motion was achieved, the robot’s posture exhibited discontinuous changes during transition intervals, resulting in instability. Conversely, when the skill-transition model was applied, this unstable transition was mitigated by reducing the postural instability observed during the flight phase. These results indicate that the skill-transition model effectively maintains a stable posture throughout the jumping motion by smoothly correcting action discrepancies at the moment of phase transition.

To verify these results more intuitively, Fig. 7 presents the robot’s top-coordinate trajectory in three dimensions. Without the skill-transition model, the robot was able to perform the jumping motion; however, the landing point deviated significantly from the target position. An analysis of the actual landing coordinates revealed an average error of x = -0.92 m and y = -0.49 m relative to the target point. By contrast, when the skill-transition model was applied, instability during the transition process was reduced significantly, thus resulting in a landing position that closely aligned with the target point (average error x = -0.1 m, y = -0.29 m). These results indicate that the skill-transition model effectively corrects the discontinuous motion changes that occur during phase transitions, thereby aiding the robot in performing more stable and accurate jumping motions.

Fig. 7.

Three-dimension results of robot’s jumping motion

3.3 Analysis of Jumping Motion Under Disturbances

To verify the robustness of the skill-transition model, experiments were conducted by artificially introducing disturbances to the robot’s initial posture and velocity errors, as listed in Table 2. These experiments aimed to evaluate the ability of the model to maintain a stable motion when unexpected environmental changes or uncertainties occur in the initial conditions.

Table 2.

Initial joint, posture, and velocity disturbance values

Fig. 8 shows the results obtained without applying disturbances. In the absence of disturbances, both cases, i.e., with and without the skill-transition model, successfully demonstrated jumping motions close to the target height of 2.5 m and maintained a generally stable posture. This indicates that basic phase-wise learning alone is sufficient to acquire the primary patterns of the jumping motion.

Fig. 8.

Analysis results of robot’s jumping motion without disturbance

However, when disturbances were introduced to the initial posture and velocity under the same conditions, the robot without the skill-transition model exhibited severe postural instability during the phase-transition intervals, as shown in Fig. 9. Specifically, the discontinuity in the transition interval caused the robot to lose its balance and fall, thus preventing the simulation from proceeding normally. These results imply that, in the presence of disturbances, the lack of smooth phase transitions causes the stability of the overall motion to degrade rapidly.

Fig. 9.

Analysis results of robot’s jumping motion under disturbance

By contrast, when the skill-transition model was applied, the postural instability during the transition intervals reduced significantly, even under disturbances. This shows that the skill-transition model effectively corrects motion discontinuities across phases and enhances the resilience against disturbances, thus ultimately enabling a more stable and consistent performance.

3.4 Analysis of Skill-Transition Model Outputs

We analyzed the manner by which the skill-transition model corrected the motion of the robot during each phase-transition interval. Fig. 10 shows the correction amounts applied by the skill-transition model to the phase-specific policy outputs for each joint. These results confirmed that the skill-transition model selectively affected different joints during each transition interval, thereby ensuring motion continuity and stability.

Fig. 10.

Joint-wise correction results of skill-transition model

First, during the transition from Phase 3 to Phase 1, the correction occurred primarily on the third joint. This suggests that the model fine-tuning the movement of this joint and its surrounding joints to maintain balance while lowering the body as the robot transitions from landing back to the jump-preparation posture.

Next, during the transition from Phase 1 to Phase 2, as the robot pushes off the ground to jump, the major joints responsible for upper-body stabilization and propulsion were primarily corrected. This correction aids efficient propulsion transmission by maintaining the directionality and balance of the jump.

Conversely, during the transition from Phase 2 to Phase 3, as the robot entered the landing phase, the most significant correction was applied to the fifth joint (knee). This represents an adjustment motion designed to absorb the impact upon landing and rapidly stabilize posture, thus demonstrating that the skill-transition model effectively performs posture control against external impacts.

These findings suggest that the skill-transition model extends beyond merely connecting outputs across phases. In fact, it enhances the continuity and stability of the entire jumping motion by performing adaptive corrections at each joint based on the motion characteristics of each transition interval.