企业官网建设流程全解析

我们一般理解为static 3DGS 是背景，轨迹回放时，障碍物是无法交互的。但是这两篇论文仍然进行了RL强化学习。

RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learning
ParkingWorld: End-to-End Autonomous Parking Reinforcement Learning from Corrective Experience in 3DG

我选择RAD的奖励模型进行分析：

3.4 奖励建模
奖励是训练信号的来源，决定了强化学习（RL）的优化方向。奖励函数的设计旨在通过惩罚不安全行为并鼓励与专家轨迹保持一致来引导自车的行为。它由四个奖励组成部分构成：(1) 与动态障碍物的碰撞，(2) 与静态障碍物的碰撞，(3) 相对于专家轨迹的位置偏差，以及 (4) 相对于专家轨迹的航向偏差：
R = { r d c , r s c , r p d , r h d } . ( 4 ) R = \{r_{dc}, r_{sc}, r_{pd}, r_{hd}\}. \quad (4)R={rdc​,rsc​,rpd​,rhd​}.(4)
如图 4 所示，这些奖励组成部分在特定条件下被触发。在 3DGS 环境中，如果自车的边界框与动态障碍物的标注边界框重叠，则检测到动态碰撞，并触发负奖励r d c r_{dc}rdc​。同样，当自车的边界框与静态障碍物的高斯分布重叠时，识别为静态碰撞，导致负奖励r s c r_{sc}rsc​。位置偏差被测量为自车当前位置与专家轨迹上最近点之间的欧几里得距离。超过预设阈值d m a x d_{max}dmax​的偏差将产生负奖励r p d r_{pd}rpd​。航向偏差计算为自车当前航向角ψ t \psi_tψt​与专家轨迹匹配航向角ψ e x p e r t \psi_{expert}ψexpert​之间的角度差。超过阈值ψ m a x \psi_{max}ψmax​的偏差会导致负奖励r h d r_{hd}rhd​。
这些事件中的任何一项，包括动态碰撞、静态碰撞、过度的位置偏差或过度的航向偏差，都会立即导致回合终止。因为在发生此类事件后，3DGS 环境通常会产生噪声传感器数据，这对强化学习训练不利。

With the reward function, it is entirely possible to perform RL training in a:

Static 3DGS Scene + Trajectory Replay + RL Agent

framework.

However, it is important to understandwhat kind of RL problem you are actually solving.

What your environment really is

Your environment dynamics are:

Dynamic vehicles: fixed replay Pedestrians: fixed replay Traffic: fixed replay Ego: controlled by RL

So:

s t + 1 = f ( s t , a t ) s_{t+1}=f(s_t,a_t)st+1​=f(st​,at​)

still exists.

The ego vehicle’s future state depends on its actions.

The difference is that:

other agents \text{other agents}other agents

do not react to the ego.

They follow prerecorded trajectories.

This is a valid MDP

Many papers call this:

• Open-loop traffic replay
• Log replay simulation
• Reactive ego / non-reactive world

The RL agent can still learn:

Steer Brake Accelerate

because:

action ↓ ego trajectory changes ↓ reward changes

For example:

Replay vehicle: --------------------> Ego: accelerate

Result:

collision

Reward:

rdc < 0

The policy receives a training signal.

Why your reward design works

Your reward contains:

rdc Dynamic collision rsc Static collision rpd Position deviation rhd Heading deviation

These define a constrained corridor around the expert trajectory.

Effectively the RL agent learns:

Stay safe Stay near expert Maintain correct heading

This is very similar to:

• imitation RL
• offline RL fine-tuning
• autonomous-driving RL benchmarks

What the RL agent can learn

It can learn:

✓ smoother steering

✓ throttle control

✓ brake timing

✓ lane keeping

✓ trajectory tracking

✓ collision avoidance against replayed traffic

What it cannot learn(3DGS+回放训练RL存在局限性：反物理常识)

Suppose:

Replay car: turns left

Your ego decides:

stop completely

In reality:

other vehicles react pedestrians react traffic evolves differently

But in replay:

all agents continue exactly as recorded

Therefore the RL agent never sees:

counterfactual futures

Example

Dataset:

Car A: x=10 x=11 x=12 x=13

Recorded assuming:

human driver accelerates

During RL:

ego brakes hard

In the real world:

Car A might also brake

In replay:

Car A still: 10→11→12→13

This is physically inconsistent.

Yet RL can still train.

Why many autonomous-driving papers accept this

Because the objective is often:

Learn a driving policy

not

Build a perfect simulator

Replay-based environments provide:

• real sensor data
• realistic traffic
• easy reward computation
• stable training

without needing:

• behavior prediction
• traffic simulation
• world models

The limitation

The strongest limitation is:

No agent interaction

Your environment is approximately:

World(t)

instead of:

World(t, EgoAction)

A true world model learns:

W o r l d t + 1 = F ( W o r l d t , E g o A c t i o n t ) World_{t+1}=F(World_t,EgoAction_t)Worldt+1​=F(Worldt​,EgoActiont​)

which enables:

• negotiation
• yielding
• merging
• interactive driving

In fact, your quoted text already reveals this

Notice:

“这些事件中的任何一项，包括动态碰撞、静态碰撞、过度的位置偏差或过度的航向偏差，都会立即导致回合终止。因为在发生此类事件后，3DGS 环境通常会产生噪声传感器数据，这对强化学习训练不利。”

This is a strong clue that the environment is fundamentally arendering/replay system, not a generative world model.

When the ego leaves the distribution of the recorded trajectory:

large deviation collision off-road

the rendered views become unreliable.

Therefore the episode terminates.

World-model simulators usually try to continue generating plausible futures after such deviations.

Conclusion

With the reward function you showed,RL training in a Static 3DGS + Trajectory Replay environment is completely feasible and is already an active research direction.

What you are learning is essentially:

Policy Learning inside a replay-based simulator

rather than:

Policy Learning inside an interactive world model

The main trade-off is:

So the answer isyes, RL can absolutely be trained successfully in your proposed setup, and a world model is not strictly necessary.The world model becomes valuable when you want other agents and the environment to react realistically to the ego vehicle’s actions rather than simply replaying recorded trajectories.