Real-time Control In ROS And ROS 2 - Robot Operating System

Real-time control in ROS and ROS 2.0Jackie Kayjackie@osrfoundation.orgAdolfo Rodriguez Tsouroukdissianadolfo.rodriguez@pal-robotics.com

Table of ContentsA motivating exampleReal-time computingRequirements and best practicesROS 2 designComparison with ROS 1 and ros controlDemo and results2

A motivating example3

A motivating example4

A motivating example– Blocks can be composed by other blocks– Some blocks are subject to real-time constraints5

A motivating example– Blocks can be composed by other blocks– Some blocks are subject to real-time constraints6

A motivating example– Blocks can be composed by other blocks– Some blocks are subject to real-time constraints– System topology can change at runtime7

A motivating example– Blocks can be composed by other blocks– Some blocks are subject to real-time constraints– System topology can change at runtime8

A motivating example– Blocks can be composed by other blocks– Some blocks are subject to real-time constraints– System topology can change at runtime9

A motivating example– Blocks can be composed by other blocks– Some blocks are subject to real-time constraints– System topology can change at runtime10

Table of ContentsA motivating exampleReal-time computingRequirements and best practicesROS 2 designComparison with ROS 1 and ros controlDemo and results11

Real-time computing– It's about determinism, not performance– Correct computation delivered at the correct time– Failure to respond is as bad as a wrong response12

Real-time computing13

Real-time computing14

Real-time computingUsefulness of results after missing a deadline?15

Real-time computingHard real-time systems– Missing a deadline is considered a system failureOverruns may lead to loss of life or financial damage– Safety- or mission-critical systemsExamples: reactor, aircraft and spacecraft control16

Real-time computingSoft real-time systems– Missing a deadline has a cost, but is not catastrophicResult becomes less useful after deadline– Often related to Quality of ServiceExamples: audio / video streaming and playback17

Real-time computingFirm real-time systems– Missing a deadline has a cost, but is not catastrophicResult becomes useless after deadline– Cost might be interpreted as loss of revenueExamples: Financial forecasting, robot assembly lines18

Real-time computingWhy do we care?– Event responsee.g. parts inspection– Closed-loop controle.g. manipulator control– Added benefit: Reliability, extended uptimeDowntime is unacceptable or too expensiveThe above is prevalent in robotics software19

Goal of ROS 2Real-time compatibility, from day one20

Table of ContentsA motivating exampleReal-time computingRequirements and best practicesROS 2 designComparison with ROS 1 and ros controlDemo and results21

Requirements and best practicesUse an OS able to deliver the required determinism– Linux variantsOSreal-timemax latency (μs)Linuxno104RT PREEMPTsoft101-102Xenomaihard101– Proprietary: e.g. QNX, VxWorksPOSIX compliant, certified to IEC 61508 SIL3 et.al.22

Requirements and best practicesPrioritize real-time threads– Use a real-time scheduling policy23

Requirements and best practicesPrioritize real-time threads– Use a real-time scheduling policy24

Requirements and best practicesAvoid sources of non-determinism in real-time code– Memory allocation and management ( malloc, new )Pre-allocate resources in the non real-time pathReal-time safe O(1) allocators exist– Blocking synchronization primitives (e.g. mutex)Real-time safe alternatives exist (e.g. lock-free)– Printing, logging ( printf, cout )Real-time safe alternatives exist25

Requirements and best practicesAvoid sources of non-determinism in real-time code– Network access, especially TCP/IPRTnet stack, real-time friendly protocols like RTPS– Non real-time device driversReal-time drivers exist for some devices– Accessing the hard disk– Page faultsLock address space (mlockall), pre-fault stack26

Table of ContentsA motivating exampleReal-time computingRequirements and best practicesROS 2 designComparison with ROS 1 and ros controlDemo and results27

ROS2 design - architecture PROSROS 2 Middleware APIOpenspliceFreeRTPSetc.28

ROS2 design - real-time architectureusercode.cppros control ROS 2 Middleware APITCPROSReal-time Operating SystemOpenspliceFreeRTPSetc.Real-time Operating System29

ROS2 design – Modularity– ROS2 allows customization for real-time use-cases Memory management Synchronization Schedulingare orthogonal to each other, and to node topology30

ROS 2 - current implementationExecutorinitializationpreallocate memory.spinrmw wait(timeout){pass conditions to waitsetwait (in DDS)wake-up if timed-out}do work if it came incleanupdeallocate memory.non real-timereal-timeloop until interruptednon real-time31

ROS2 design – Node lifecycle– Standard node lifecycle state machine Opt-in feature Node lifecycle can be managed without knowledgeof internals (black box)– Best practice from existing frameworks microblx OpenRTM Orocos RTT ros control32

ROS2 design – Node lifecyclecredit: Geoffrey Biggs et.al.WIP, design subject to change33

ROS2 design – Node lifecyclecredit: Geoffrey Biggs et.al.WIP, design subject to change34

ROS2 design – Node lifecyclecredit: Geoffrey Biggs et.al.WIP, design subject to change35

ROS2 design – Node lifecycleBenefits of managed lifecycle– Clear separation of real-time code path– Greater control of ROS network Help ensure correct launch sequence Online node restart / replace– Better monitoring and supervision Standard lifecycle standard tooling36

ROS2 design – Node composition37

ROS2 design – Node composition– Composite node is a black box with well-defined API– Lifecycle can be stepped in sync for all internal nodes– Resources can be shared for internal nodes38

ROS2 design – Communications– Inter-processDDS can deliver soft real-time commsCustomizable QoS, can be tuned for real-time use-case– Intra-processEfficient (zero-copy) shared pointer transport– Same-threadNo need for synchronization primitives. Simple, fast39

ROS 2 – alpha release– Real-time safety is configurable– Can configure custom allocation policy thatpreallocates resources– Requires hard limit on number of pubs, subs, services– Requires messages to be statically sized40

ROS2 – progress overviewIn progress– Component lifecycle– Composable components– Complete intra-process pipelineFuture work– Pre-allocate dynamic messages– CI for verifying real-time constraints– Lock-free multi-threaded executor41

Table of ContentsA motivating exampleReal-time computingRequirements and best practicesROS 2 designComparison with ROS 1 ros controlDemo and results42

Comparison with ROS1 ros control–Real-time safe communications–Lifecycle management–Composability43

Comparison with ROS1 ros control–Real-time safe communications–Lifecycle management–Composability44

Comparison with ROS1 ros controlROS1 ros control:ROS2 equivalent: drop non-standard lifecycle / interfaces gentler learning curve smaller codebase easier to maintain45

Table of ContentsA motivating exampleReal-time computingRequirements and best practicesROS 2 designComparison with ROS 1 and ros controlDemo and results46

ROS 2 Real-time Benchmarking: Setupreal-time processmotor commandcontrollersimulatorsensor feedbackuser commandteleopnon real-time processprofiling resultsloggernon real-time process47

ROS 2 Real-time Benchmarking: SetupConfiguration﹣ RT PREEMPT kernel﹣ Round robin scheduler (SCHED RR), thread priority: 98﹣ malloc hook: control malloc calls﹣ getrusage: count pagefaultsGoal﹣ 1 kHz update loop (1 ms period)﹣ Less than 3% jitter (30 μs)Code﹣ ros2/demos - pendulum control48

ROS 2 Real-time Benchmarking: MemoryZero runtime allocationsstatic void * testing malloc( size t size, const void * caller) {if (running) {throw std::runtime error("Called malloc from real-time context!" );}// . allocate and return pointer.}Zero major pagefaults during runtime﹣﹣Some minor pagefaults on the first iteration of the loop, none afterConclusion: all required pages allocated before execution starts49

ROS 2 Real-time Benchmarking: ResultsLatency (ns)% of update rateMin16200.16%Max350943.51%Mean45670.46%No stress1,070,650 cycles observedTimeseriesJitter histogram50

ROS 2 Real-time Benchmarking: ResultsLatency (ns)Stress applied:stress --cpu 2 --io 27,345,125 cycles observed3 instances of overrun observed% of update er histogram51

Closing remarks– Systems subject to real-time constraints are veryrelevant in robotics– ROS2 will allow user to implement such systems with a proper RTOS, and carefully written user code– Initial results based on ROS2 alpha are encouraging inverted pendulum demo– Design discussions and development are ongoing! ROS SIG Next-Generation ROS ros2 Github organization52

Selected references–[Biggs, G.] ROS2 design article on node lifecycle (under review)–[Bruyninckx, H.] Real Time and Embedded Guide–[Kay, J.] ROS2 design article on Real-time programming–[National Instruments] What is a Real-Time Operating System (RTOS)?–[OMG] OMG RTC Specification–[ROS Control] ROS Control, an Overview–[RTT] Orocos RTT component builder's manual–[RT PREEMPT] Real-Time Linux Wiki–[Xenomai] Xenomai knowledge base53

Avoid sources of non-determinism in real-time code - Memory allocation and management ( malloc, new ) Pre-allocate resources in the non real-time path Real-time safe O(1) allocators exist - Blocking synchronization primitives (e.g. mutex) Real-time safe alternatives exist (e.g. lock-free) - Printing, logging ( printf, cout )