An engineer's path from "how do processes talk" — through the DDS middleware, QoS as the real-time knob, executors and callback groups, lifecycle and composable nodes — to the discipline of shipping a robot stack. Current on ROS 2 Kilted (2025) with Jazzy (LTS, 2024) as the workhorse.
A modern robot is a distributed system. Its perception runs on a stereo camera and a LiDAR, often on a GPU-loaded box. Its localization runs on a separate CPU to keep the GPU free for vision. Its planner wants the highest-priority thread on yet another core. Its drivers talk to a CAN bus at 1 kHz. None of these processes are on the same machine, none share memory, and none trust each other. And yet the loop must close at 100 Hz.
This is exactly the graph problem: heterogeneous processes, heterogeneous machines, typed contracts, and a hard real-time budget on the critical edge. A chat app can tolerate 200 ms of jitter; a motor servo cannot. A web service can restart when it crashes; a flight controller cannot lose a sample. ROS 2 is the framework that turns this into a solvable problem.
Why is this not just Kafka, or gRPC, or ZeroMQ? Those give you pub/sub or request/reply, sure — but they hand you a single knob: "reliable or not". A robot needs more. It needs to say: "this topic is unreliable on purpose, this one is reliable on purpose; the sensor sample has to appear within 20 ms of publication, or the frame is dropped; the parameter store must outlive the process that publishes it; the whole graph must tolerate one node dying without tearing down the others." Those are the knobs the ROS 2 middleware exposes.
ROS 2 is that middle layer. It is not the wire protocol, and it is not the runtime — it is a thin abstraction over DDS that makes the abstractions above (nodes, topics, QoS, services, actions) feel like a language rather than a configuration. Everything else in this document is what you build on top of that.
Every ROS 2 program is composed of four communication shapes. Knowing which one you are writing against is the first skill. They are not interchangeable — they encode different timing contracts.
Many publishers, many subscribers, decoupled in time. Fire-and-forget unless QoS says otherwise. Continuous stream.
One request, one reply. Blocking from the caller's point of view. Short, transactional.
Goal + feedback stream + terminal result, plus a cancel path. Minutes-long, cancellable.
Typed key/value store owned by the node; settable at runtime. State, not traffic.
Why four and not one? Each one encodes a different answer to the question "when does the caller learn the outcome?". A topic publisher has no answer — the message is fire-and-forget. A service caller blocks until the reply. An action client gets an ongoing stream plus a terminal value. A parameter is not even a message — it is state that outlives the sender.
The mapping is mechanical:
The action primitive deserves a footnote: it is not a service on steroids. It is three topics and one service bundled into a typed contract — goal (submit a request), feedback (progress stream), result (terminal), and a cancel path. Navigation, whole-body control, and any task where the operator must be able to say "stop" are actions, not services.
Here is the fact that most beginner tutorials hide: ROS 2 does not move a byte of data. It is a thin library — the ROS Middleware Interface (RMW) — that translates your create_publisher calls into calls on a Data Distribution Service implementation, e.g. CycloneDDS Fast DDS eProsima. DDS is an OMG standard (DDS-DCPS) that has been shipping in aerospace and defense for two decades. It gives you everything ROS 2 pretends to offer: discovery, reliability, durability, deadlines, security, multicast, cross-language, cross-vendor.
So the real architecture is three layers: your code → rcl/rclcpp/rclpy (a tiny adapter that hides DDS) → RMW (a stable C API) → DDS vendor (Cyclone, Fast DDS, or eProsima). This layering is why switching DDS vendors is a single environment variable (ROS_DISTRO aside, RMW_IMPLEMENTATION) — the ABI you depend on is the RMW, not the DDS implementation.
The DDS discovery mechanism is worth understanding because it is where most "why can't I see the topic?" bugs live. Every DDS domain (a ROS_DOMAIN_ID integer, 0–232) is a logical partition of the network. When a participant starts, it sends an SPDP packet (Standard Participant Discovery Protocol) on UDP multicast to announce: "I am here, on domain 0, with these types, at this address." Peers reply; the two negotiate an RTPS endpoint pairing and begin sending serialized samples directly to each other.
Why this matters: multicast is fire-and-forget from the network's perspective. If your switch drops multicast, or your firewall blocks UDP 7400/7401 (the default SPDP ports), your two machines will not discover each other and you will get a silent failure: every ros2 topic list returns nothing. The fix is usually one of three things — set ROS_DOMAIN_ID to something non-zero, disable multicast and switch to PEERS-based discovery, or open UDP multicast on the switch.
The wire format is CDR (Common Data Representation) — the OMG standard binary serialization. Every message type you define is translated (at build time) into a CDR-serializable struct, and every RMW serializes it into that format before UDP. That is also why cross-language works: your C++ node and your Python node agree on the bytes, not on the language.
If you take one thing from ROS 2, make it this: QoS is the real-time contract. Without QoS you get a pub/sub system that is essentially best-effort UDP. With QoS you get a pub/sub system that can be deliberately made real-time, deliberately made lossy, deliberately made durable, or deliberately made reliable. The knob is per topic, per endpoint, per message.
The six policies that matter in practice are reliability, durability, history, deadline, lifespan, and liveliness. Let's build them up.
Reliability answers "what happens when a sample is dropped on the wire?" Two choices:
reliable — DDS retransmits on nack until the subscriber has it or gives up.best_effort — DDS drops and moves on.For a servo command, you want reliable — losing a command is catastrophic. For a LiDAR scan at 20 Hz, you want best_effort — a dropped frame is tolerable, and retrying would add jitter that would hurt the deadline of every other scan.
Durability answers "what if a subscriber joins late?" Three choices:
volatile — only samples published after subscription are delivered (the default).transient_local — the publisher keeps one (or N) last sample in memory and replays it to new subscribers.transient — as above, but samples are re-delivered even after the publisher dies.The transient_local policy is how a parameter server works: the first /tf frame is published before your listener subscribes, but you still receive it. And it is how a launch file gets its initial parameters into a just-started node.
History answers "how many samples does the middleware remember?"
keep_last(N) — the buffer holds the last N samples; older ones are evicted.keep_all — the buffer grows; the middleware back-pressures the publisher when the subscriber is slow.This is the buffer you are choosing when you set depth=10 on your subscriber. It is also the knob that determines your memory budget:
For a 4 MB stereo depth image at keep_last(5), that is 20 MB per subscriber — and every slow subscriber on a topic multiplies that cost. This is why camera topics ship with small history depths.
Deadline is the only QoS policy with meaningful numbers. It is a bound on the maximum time between two consecutive samples. If the publisher does not publish within the deadline, DDS raises a deadline missed event. That is how you convert "the sensor is stuck" into a signal your supervisor node can act on.
The way to use it: if your perception loop runs at 100 Hz (period $T_p = 10$ ms) and you allow jitter of at most $J_{\max}$, set the deadline to
Not $T_p$ itself — the deadline is a violation signal, not a pacing signal. If you set it equal to the period, you will trigger spurious deadline misses on any thread migration or GC pause, and the noise will drown out real faults.
Lifespan caps how long a sample is valid after publication — a stale sensor sample should not reach the planner. Liveliness asks DDS to watch the publisher: if the process dies or its executor stalls, DDS raises a liveliness-lost event. Together they are how a node says "my samples are worth X seconds, and if I stop being a node, notice."
QoS is checked pairwise between a publisher and a subscriber. Two endpoints match only if their policies are compatible. The compatibility rules are asymmetric:
| Policy | Compatibility rule | Wrong direction → symptom |
|---|---|---|
| Reliability | PUB ≥ SUB (PUB reliable is compatible with SUB best_effort, not the reverse) | PUB best_effort + SUB reliable → no data |
| Durability | PUB ≥ SUB (transient > transient_local > volatile) | Missed initial samples, "cold start" bugs |
| History | Independent (each endpoint picks its own buffer) | Subscriber sees a different history than publisher |
best_effort and the subscriber (via the default rclcpp::QoS) declares reliable. The compatibility check fails, the subscription is not offered, and you get silence. Fix: either raise the publisher to reliable, or drop the subscriber to best_effort.A node is a self-contained program that publishes, subscribes, calls, and serves. But a node is also a set of callbacks: one per subscription, one per service, one per parameter change, one per timer, one per action goal. Something has to decide which callback runs next, and on which thread. That something is the executor.
A single-threaded executor is the default. It spins: check every guard condition, run whichever callback is ready, repeat. Deterministic, easy to reason about, and it runs your 100 Hz timer on the same thread as your 1 kHz timer — so a slow 1 kHz callback will block the 100 Hz one. That is the trap.
A multi-threaded executor gives each callback its own thread (up to N). It fixes the blocking, but it introduces a second trap: callbacks in the same group are still serialized. Callbacks in different groups can run concurrently.
Every callback you register is assigned to a callback group. Two groups exist by default:
The rule is: concurrent execution is granted between groups, not within groups. So to parallelize three independent subscriptions while keeping their internal work serialized, you give each subscription its own Reentrant group:
auto g_sensor = node->create_callback_group(rclcpp::CallbackGroupType::Reentrant); auto g_vision = node->create_callback_group(rclcpp::CallbackGroupType::Reentrant); auto g_control = node->create_callback_group(rclcpp::CallbackGroupType::Reentrant); node->create_subscription<Scan>("/scan", QoS(1), cb_scan, g_sensor); node->create_subscription<Image>("/camera/image", QoS(1), cb_vision, g_vision); node->create_wall_timer(100ms, cb_control, g_control);
Now a stuck perception callback cannot starve the control callback. That is the whole trick.
Every callback has three costs stacked on it:
And if it shares a group with a slow callback, add:
For a servo command at 1 kHz, you are budgeting microseconds. The rule of thumb: never put a control-critical callback in the same group as a callback that can block (vision, logging, file I/O). Give the control path its own group and its own thread — this is the single most valuable piece of advice in the whole document.
ROS 2 messages are declared in a small IDL — not JSON, not protobuf, not Thrift — with its own syntax. It looks like:
# msg/LaserScan.msg — ROS 2 message definition std_msgs/Header header # timestamp + frame_id float32 range_min float32 range_max float32 angle_min float32 angle_max float32 angle_increment float32[] ranges # one range per beam float32[] intensities # optional per-beam intensity
Why its own IDL and not protobuf? Three reasons. First, it has built-in array types (float32[]) and constants — no repeated keyword, no enum ceremony. Second, it is versioned by content: the type description is a hash, so a message definition change is a wire-format change that DDS detects automatically. Third, it is cross-language by construction: rosidl generates C++, Python, and (via plugins) Rust and Go bindings from the same source.
A .msg is a data structure — a topic type. A .srv is a request + response pair — a service type. A .action is three things bundled together:
# action/NavigateTo.action # Goal — what the client asks for geometry_msgs/PoseStamped pose --- # Result — what the server returns bool success string failure_reason --- # Feedback — what the server streams geometry_msgs/Pose current_pose float32 remaining_distance
The --- separator defines the boundary between the three halves. The generated code exposes one class per half, and the client/server APIs are built on top.
Every interface has a type description — a canonical JSON document that includes the IDL hash and every field. When two nodes on different machines try to talk, they exchange type descriptions and compare hashes. If they differ, DDS refuses to connect. That is why editing LaserScan.msg and forgetting to rebuild a dependent node is a loud failure, not a silent one — DDS prints a type-mismatch error.
Two more node shapes exist for when the plain Node is not enough.
A plain node starts and immediately begins publishing. A lifecycle node has a state machine, and transitions are explicit:
Why this matters: on a real robot, you launch 30 nodes in a specific order — driver → filter → perceiver → planner. If the planner starts before the perceiver, it gets garbage. A lifecycle node solves that: the launch file can call transition_config, transition_activate, etc. for each node in order, and each node's on_configure can validate its parameters (or fail cleanly) before it starts publishing. Failure is a state, not a crash.
A plain node is a separate process. A composable node is a plain node compiled into a shared library and loaded by a component container — one process that hosts many nodes. The payoff: intra-process communication is zero-copy (a pointer, not a serialized UDP packet), and one executor schedules all nodes in the process. The cost: any node crash takes down the whole process, so composable nodes are only for nodes that you trust to be well-behaved.
Real-time guidance: put the hot path (control, planning) in a composable node with its own RT-scheduled thread; put the cold path (vision, logging, UI) in separate processes. The composable process owns the CPU core; the rest of the system is best-effort.
ROS 2 ships a small but complete build/run/visualize/replay stack.
Colcon is not CMake; it is an orchestrator around CMake (or setuptools for Python). It walks your src/ tree, finds each package's package.xml, resolves dependencies, and runs each package's build tool in the right order. Your build command is always:
colcon build --symlink-install --packages-select my_package
The --symlink-install flag is important for development: it makes msg, srv, and action files symlinks into build/, so you do not have to rebuild after editing a message definition — the C++/Python bindings are already there, and the new definition is picked up on the next source install/setup.bash.
A launch file is a Python script that says "start these nodes with these parameters, in this order, and wait for them to reach this state before starting the next." It is not a Makefile and not a Docker Compose — it is the state machine of a deployment. A typical launch file:
from launch import LaunchDescription from launch_ros.actions import Node def generate_launch_description(): return LaunchDescription([ Node(package='lidar_driver', executable='scan_pub', parameters=[{'frame_id': 'laser', 'rate_hz': 20}] ), Node(package='nav2_bringup', executable='navigate_to_pose', parameters=['nav2_params.yaml'] ), ])
rviz2 is not a debugging tool; it is a topic consumer that renders sensor_msgs/LaserScan, sensor_msgs/Image, geometry_msgs/Pose, and arbitrary marker arrays as a live 3D scene. It is the fastest way to see whether your data is real. Every ROS 2 developer lives in rviz2 for the first week.
rosbag2 records every topic into a SQLite database with a compressed message format (Cyclone's mcap is the modern default). Replay is just publishing from a bag at a controlled rate. That is how you debug a real-time bug: capture the failing run, replay at 0.5× speed, watch the callbacks, fix, replay again.
The ros2 CLI is the swiss-army knife. Learn six commands and you can debug 90% of issues:
ros2 topic list # what exists? ros2 topic echo /scan --once # what does one sample look like? ros2 topic hz /scan # what is the actual rate? ros2 node list # what nodes are running? ros2 node info /localizer # what topics does it use? ros2 param list /localizer # what parameters are set?
ros2 CLI is your first-line debugger.Out of the box, ROS 2 is wide open: any process on the network can subscribe to any topic. DDS ships an optional DDS-Security plugin (OMG standard) that adds three layers:
Turning it on is one environment variable:
export ROS_SECURITY_ENABLE=true export ROS_SECURITY_KEYSTORE=/etc/ros/security export ROS_AUTH_PUBLIC_KEY=/etc/ros/security/pki/ca/root.crt
Behind that is a directory of per-participant certificates and an ACL YAML file that says, e.g., "the controller participant may write to /cmd_vel but may not read /camera/image_raw." Every DDS packet carries a signature; the peer verifies it against the ACL before processing the sample. Failure is silent at the DDS layer — the topic simply never connects.
Why this matters: on a shared research robot, or a robot in a public building, an attacker with network access can inject commands without security. DDS-Security is the same model TLS uses, applied to the DDS wire. It is not optional for anything leaving the lab.
Let's build the smallest useful ROS 2 program and see every concept come together.
We define one message (a simplified laser reading) and one service (a manual stop):
# msg/Proximity.msg std_msgs/Header header float32 nearest_distance # meters float32 bearing # radians, -pi..pi
# nodes/proximity_pub.py import rclpy from rclpy.node import Node from my_pkg.msg import Proximity class Pub(Node): def __init__(self): super().__init__('proximity_pub') self.pub = self.create_publisher(Proximity, '/proximity', rclpy.qos.qos_profile_sensor_data) # 20 Hz wall-clock timer; best_effort, keep_last(5) self.create_timer(1/20, self.tick) def tick(self): msg = Proximity() msg.header.stamp = self.get_clock().now().to_msg() msg.nearest_distance = 0.42 msg.bearing = 0.0 self.pub.publish(msg) if __name__ == '__main__': rclpy.init(); rclpy.spin(Pub())
# nodes/avoidance_ctrl.py import rclpy from rclpy.node import Node from my_pkg.msg import Proximity from geometry_msgs.msg import Twist class Ctrl(Node): def __init__(self): super().__init__('avoidance_ctrl') self.declare_parameter('brake_distance', 0.5) self.brake = self.get_parameter('brake_distance').value self.create_subscription(Proximity, '/proximity', self.on_prox, qos_profile='rclpy.qos.qos_profile_sensor_data') self.vel_pub = self.create_publisher(Twist, '/cmd_vel', 10) self.last_stamp = None def on_prox(self, msg): twist = Twist() twist.linear.x = 0.0 if msg.nearest_distance < self.brake: twist.linear.x = -0.2 # reverse self.get_logger().warn(f'obstacle at {msg.nearest_distance:.2f} m') self.vel_pub.publish(twist) if __name__ == '__main__': rclpy.init(); rclpy.spin(Ctrl())
# launch/avoidance.launch.py from launch_ros.actions import Node def generate_launch_description(): return LaunchDescription([ Node(package='my_pkg', executable='proximity_pub'), Node(package='my_pkg', executable='avoidance_ctrl', parameters=[{'brake_distance': 0.5}] ), ])
With three files and two commands you have a real-time distributed system:
colcon build --symlink-install — compile.ros2 launch my_pkg avoidance.launch.py — deploy.ros2 topic hz /cmd_vel — verify the 20 Hz rate reaches the control topic.ros2 param set /avoidance_ctrl brake_distance 0.3 — retune without restart.ros2 topic echo /proximity — inspect a sample.ros2 bag record /proximity /cmd_vel -o run1 — capture for replay.best_effort, the subscriber (via the default rclcpp::QoS) says reliable. DDS refuses the connection; nothing is logged at the ROS layer. Diagnosis: ros2 topic info /scan -v prints the publisher's and subscriber's QoS; they must satisfy the compatibility rules.MutuallyExclusive group; a single 200 ms vision callback blocks the 100 Hz control callback for 200 ms. Fix: put the control callback in its own Reentrant group and run a MultiThreadedExecutor. Better: put the control path in a composable node with RT priority.export ROS_DOMAIN_ID=42 (moves you off the default domain 0), (b) export ROS_MULTICAST_ENABLED=false with ROS_DISCOVERY_SERVER=<ip>:1181 (switch to a discovery server), (c) open UDP 7400/7401. Diagnosis: run two ros2 topic list commands on different machines; if one sees nothing, it is discovery, not QoS.spin() loop blocks your main thread.spin() and then do more work in the same thread, your work never runs. Use spin_until_future_complete() for synchronous waits, or spin_once() inside a loop where you also do other work. The rule: spin is the scheduler; it owns the thread.source install/setup.bash.my_pkg.msg types. Every new shell needs the workspace sourced: source /path/to/ws/install/setup.bash. Add it to ~/.bashrc (with the correct path).best_effort as "unreliable" instead of "deliberately lossy."best_effort is correct — you want the latest sample, not the 30 old ones. Switching to reliable would add retransmission jitter to a stream that is already faster than the network. The rule: match the QoS to the data's tolerance for loss, not to its importance.transient_local durability just to catch the late subscriber. A parameter is cheaper: it is a typed value owned by the node, settable at runtime via ros2 param set, and it lives on the node's lifecycle, not on a topic. Use a topic when the value is traffic; use a parameter when it is state.ROS 2 is a thin abstraction over DDS: you describe a graph of nodes and their communication contracts (topics, services, actions, parameters), and the RMW layer translates them into DDS calls on Cyclone, Fast DDS, or eProsima. QoS is the real-time knob — per topic, per endpoint, with reliability (retransmit or not), durability (replay to late joiners or not), history (buffer depth), and deadline (violation signal) as the four knobs that matter; when publisher and subscriber QoS disagree, DDS silently refuses the connection. Underneath each node, an executor schedules callbacks and callback groups decide which callbacks can run in parallel; isolate the control path in its own group and, for the hot path, in a composable node with a real-time-scheduled thread. Interfaces are declared in a small IDL (.msg, .srv, .action), code-generated into every language and hashed at runtime so a schema change is a loud failure. Lifecycle nodes give you a state machine for startup/teardown/error, and DDS-Security gives you TLS-plus-ACL for anything leaving the lab. With colcon, launch, rviz2, rosbag2, and the ros2 CLI, you have a complete build-deploy-visualize-replay-debug stack — and the discipline of the framework is to pick the cheapest primitive that fits the timing, give the control path its own scheduler, and remember that DDS is the real thing and ROS 2 is the shape you put on top of it.
Further reading
rviz2 is the right tool for the developer on the robot. Foxglove Studio is the right tool for everyone else — the stakeholder, the remote engineer, the QA engineer on a different continent, and you three months later trying to remember what went wrong. It is a web-based visualization client that speaks a JSON-over-WebSocket protocol, so it does not need DDS, does not need to be on the same machine, and does not need to know anything about your message types beyond what the bridge tells it.
Foxglove does not talk DDS. Something has to translate your ROS 2 graph into its WebSocket protocol — and the recommended thing is the foxglove_bridge, a C++ node written against the Foxglove SDK. It auto-discovers every topic over DDS, reads each message's type description, and re-publishes them as WebSocket frames on port 8765.
| Mode | Data path | Use case |
|---|---|---|
| Live | ROS 2 → foxglove_bridge (8765) → WebSocket → Studio | Real-time visualization, teleop, debugging a running robot |
| Recorded | ros2 bag record → .mcap → drag-drop into Studio | Replay, analysis, sharing a captured run |
# Install the bridge (binary packages exist for Humble, Jazzy, Kilted) sudo apt install ros-$ROS_DISTRO-foxglove-bridge # Launch — auto-subscribes to every topic ros2 launch foxglove_bridge foxglove_bridge_launch.xml
Add it to your launch file so the bridge comes up with the stack:
Node(package='foxglove_bridge', executable='foxglove_bridge'),
Then in Foxglove Studio: Open connection → Foxglove WebSocket → ws://localhost:8765 (same machine) or ws://<robot-ip>:8765 (same LAN). For remote access without a VPN or port-forward, run the bridge behind Tailscale — the pattern recommended in the official docs.
The single most common Foxglove mistake is adding ten panels and getting a wall of noise. A good layout has a clear visual hierarchy: one hero panel and three supporting panels.
Pick visualization_msgs/Marker, PointCloud2, OccupancyGrid, PoseArray, Path. Add your TF frames and "follow" the robot's base_link. This is the picture the audience sees.
Pick sensor_msgs/Image or CompressedImage. Foxglove decodes compressed images on the client — real bandwidth savings over Wi-Fi.
Pick numeric paths inside topics: /cmd_vel/linear/x, /battery/percentage. One clean line that says "the robot is doing the right thing."
Log panel for rcl_interfaces/Log warnings; Raw Messages for one interesting topic (e.g. /diagnostics) in a JSON tree.
Optional but high-value: URDF Viewer (renders /robot_description in the 3D panel), Teleop (publish geometry_msgs/Twist on /cmd_vel right from the browser — the single most impressive feature in a demo), Diagnostics (stale/OK/error per diagnostic_msgs/DiagnosticArray), Charts (time-series with units and thresholds).
# Record with the modern MCAP format (Jazzy/Kilted default) ros2 bag record -a -o run1 # Older distros: install the MCAP storage plugin first sudo apt install ros-$ROS_DISTRO-rosbag2-storage-mcap ros2 bag record -s mcap --all -o run1
Drag run1.mcap into Foxglove and you get full playback controls (play/pause, seek, speed) — scrub at 0.25× to catch a missed obstacle or a planner stall.
.db3? MCAP captures message definitions inline, so custom types replay correctly on any machine. .db3 does not — you will get "unknown message type" errors unless the machine opening the file has your workspace built. The mcap CLI can convert old files: mcap convert input.db3 output.mcap.Three things separate a "this is just rviz" view from a "this looks like a product demo":
/robot_description and let the URDF viewer render it. A mesh that moves is ten times more convincing than a frame axis.cmd_vel.linear.x going positive then zeroing at the obstacle, or odom.pose.position.x tracking a reference. One clean line, not eight squiggles.Foxglove references