Michael Meidinger, M.Sc.

At LIS, we leverage the Duckietown hardware and software ecosystem to experiment with our reinforcement learning (RL) agents, known as learning classifier tables (LCTs), as part of the Duckiebot control system. More information on Duckietown can be found here.

In previous work, both the default PID controllers for steering and speed control were replaced independently by LCT RL agents. The modified versions of the Duckiebot control systems can keep up, respectively, with the default versions in terms of driving performance while using just slightly more computational resources. With some additional changes to the image processing pipeline, we also improved the accuracy of state measurements, e.g., the distance to the center of the lane. However, the two separate agents have not been combined so far, and have not been exposed to more complex driving scenarios.

This thesis aims to have the Duckiebots' driving be controlled entirely by RL. The student can achieve this by either merging our two RL agents into one or via a multi-agent approach. As their selected actions affect each other, they cannot be treated as entirely independent. For example, a more substantial heading angle correction is necessary to avoid leaving the lane when accelerating in curves.

The first step in this thesis will be to analyze the existing agents and investigate different concepts for their combination. When implementing the selected approach(es), new states and actions will likely extend the ruleset(s), which will demand more complex reward and Q-value update functions. Ideally, code optimizations will decrease the minimum possible RL cycle period. The student will compare the new system configuration to those with the existing separate agents and the baseline PID controller version regarding driving performance and computational resource utilization. A potential extension is the integration of more complex scenarios, such as intersections, pedestrian crossings, or traffic lights.

If the current LCT agents do not yield satisfactory results, the student could also explore other agent approaches, such as deep RL. As such methods are generally more resource-hungry, they might only be feasible by offloading parts of the control system to the Jetson Nano's GPU.

Prerequisites

At LIS, we leverage the Duckietown hardware and software ecosystem to experiment with our reinforcement learning (RL) agents, known as learning classifier tables (LCTs), as part of the Duckiebot control system. More information on Duckietown can be found here.

In previous work, we developed a tool called DuckieVisualizer to monitor our Duckiebots, evaluate their driving performance, and visualize and interact with the actively learning RL agents.

This student assistant position will involve extending the tool and its respective interfaces on the robot side by further features, e.g., more complex learning algorithms or driving statistics. The underlying camera processing program should also be ported from Matlab to a faster programming language to enable real-time robot tracking. Furthermore, more robust Duckiebot identification mechanisms should be considered.

Besides these extensions to the DuckieVisualizer, the student will also do some general system maintenance tasks. This may include the hardware of the Duckiebots and their software stack, for example, merging different sub-projects and looking into quality-of-life improvements to the building process using Docker. Another task will be to help newly starting students set up their development environment and to assist them in their first steps. Finally, the student can get involved in expanding our track and adding new components, e.g., intersections or duckie pedestrian crossings.

Prerequisites

At LIS, we leverage the Duckietown hardware and software ecosystem to experiment with our reinforcement learning (RL) agents, known as learning classifier tables (LCTs), as part of the Duckiebot control system. More information on Duckietown can be found here.

We use a Duckiebot's Time-of-Flight (ToF) sensor to measure the distance to objects in front of the robot. This allows it to stop before crashing into obstacles. The distance measurement is also used in our platooning mechanism. When another Duckiebot is detected via its rear dot pattern, the robot can adjust its speed to follow the other Duckiebot at a given distance.

Unfortunately, the measurement region of the integrated ToF sensor is very narrow. It only detects objects reliably in a cone of about 5 degrees in front of the robot. Objects outside this cone, either too far to the side or too high/low, cannot reflect the emitted laser beam to the sensor's collector, leading to crashes. The distance measurement is also fairly noisy, with measurement accuracy decreasing for further distances, angular offsets from the sensor, and uneven reflection surfaces. This means that the distance to the other Duckiebot is often not measured correctly in the platooning mode, causing the robot to react with unexpected maneuvers and to lose track of the leading robot.

In this student assistant project, the student will investigate how to resolve these issues. After analyzing the current setup, different sensors and their position on the robot's front should be considered. A suitable driver and some hardware adaptations will be required to add a new sensor to the Duckiebot system. Finally, they will integrate the improved distance measurement setup in our Python/ROS-based autonomous driving pipeline, evaluate it in terms of measurement region and accuracy, and compare the new setup to the baseline.

These modifications should allow us to avoid crashes more reliably and enhance our platooning mode, which will be helpful for further development, especially when moving to more difficult-to-navigate environments, e.g., tracks with intersections and sharp turns.

Prerequisites