The Modern Robotics Software Stack: From ROS to Foundation Models

Ask a robotics engineer in 2020 what software their humanoid or arm ran, and you'd get a fragmented answer: a custom motion planner, some vendor-specific SDK, a perception pipeline stitched together from OpenCV and a homegrown neural net, and a lot of glue code holding it together. Ask that same question today, and the answer is starting to sound consistent across companies — a stack with recognizable layers, shared components, and a common vocabulary. That convergence is new, and it's worth understanding because it changes who can build robots and how fast.

This piece walks through that stack layer by layer: the middleware that lets robot parts talk to each other, the simulators that train and test behavior before it touches hardware, the foundation models that are replacing hand-coded control logic, and the edge runtimes that get all of it running on a battery-powered machine with a few hundred watts to spare.

What's Inside a Robotics Software Stack

A robotics software stack is the layered set of software systems that take a robot from "collection of motors and sensors" to "machine that does useful work autonomously or semi-autonomously." It's useful to think of it as four broad layers, each solving a different problem:

• Middleware and communication — how sensors, actuators, and compute nodes exchange messages in real time. This is the plumbing.
• Perception and world modeling — turning camera, lidar, and force-torque data into a representation the robot can reason about.
• Planning and control — deciding what to do next and translating that decision into joint torques or wheel velocities.
• Learning and policy — the models (increasingly foundation models) that generate or inform those decisions, trained in simulation and fine-tuned on real-world data.

Historically, each layer was built and maintained separately, often by different teams using incompatible tools. A perception team might publish detections in a custom binary format that the planning team then had to reverse-engineer. What's changed over the past two years is that these layers have started to standardize on shared interfaces — a robot equivalent of the web's HTTP-and-JSON convergence, except still younger and messier.

The Middleware Layer: ROS and ROS 2

The Robot Operating System (ROS) is not an operating system in the Linux-kernel sense — it's a set of libraries and conventions for building robot software as a graph of communicating processes, called nodes. A camera driver node publishes image messages; a perception node subscribes to them and publishes detections; a planner subscribes to detections and publishes velocity commands. This publish-subscribe pattern, combined with a standard set of message types, is what let robotics researchers share code instead of rewriting drivers for every new sensor.

ROS 1, released in the mid-2000s out of Willow Garage, became the de facto standard in robotics research but had real limitations for production use: no real-time guarantees, a single point of failure in its master node, and weak security. ROS 2 was built from the ground up on top of DDS (Data Distribution Service), an industrial pub-sub middleware standard, to address exactly those gaps — real-time performance, no single master, and support for the kind of multi-robot, multi-vendor deployments that industrial and commercial robotics need.

A few things are worth knowing about where ROS sits in the modern stack:

ROS 2 is now the default choice for new robot programs that need to integrate third-party components, and most major simulators and hardware vendors ship ROS 2 drivers or bridges. But it's still fundamentally a middleware layer — it moves messages around and gives you a standard vocabulary for sensors and actuators. It doesn't tell the robot what to do. That's the job of the layers above it, and it's where the most change has happened recently.

Simulation and the Sim-to-Real Pipeline

Training or testing a policy directly on physical hardware is slow, expensive, and occasionally destructive — a robot arm that hasn't learned to avoid singularities can break itself or whatever it's holding. Simulation solves this by letting a robot fail thousands or millions of times in a physics engine before it ever touches real hardware.

Modern robotics simulators do more than render 3D scenes; they model rigid-body dynamics, contact forces, friction, and increasingly sensor noise and lighting variation, so that a policy trained in simulation transfers to the real world without needing to be retrained from scratch. This transfer problem — known as the "sim-to-real gap" — has been one of the central open problems in robotics for over a decade.

The current generation of simulators leans on GPU-accelerated physics to run thousands of parallel environments simultaneously, which is what makes it feasible to train reinforcement-learning policies or generate the synthetic demonstration data that foundation models need. Two techniques dominate how teams close the sim-to-real gap:

1. Domain randomization — deliberately varying textures, lighting, friction coefficients, and sensor noise during training so the policy learns to be robust to the difference between simulation and reality, rather than overfitting to one simulator's exact physics.
2. System identification — measuring a real robot's actual physical parameters (motor torque curves, joint friction, mass distribution) and feeding those back into the simulator so it more closely matches the specific hardware the policy will run on.

Simulation has also become the primary source of training data for imitation-learning and foundation-model approaches, because collecting real-world robot demonstrations — a human physically teleoperating a robot arm through thousands of pick-and-place tasks — is far more expensive per data point than generating synthetic rollouts. That data-generation role is part of why simulation, once a research convenience, is now a first-class layer in the production stack rather than an optional testing step.

Foundation Models: VLAs and Generalist Policies

The most visible shift in robotics software over the last two years is the move from hand-engineered control pipelines to learned, general-purpose policies — specifically vision-language-action models (VLAs). A VLA takes in camera images and a natural-language instruction ("pick up the red mug and place it on the shelf") and outputs low-level robot actions directly, without a human writing separate perception, planning, and grasping modules for each task.

This is a genuine architectural break from the previous generation of robot software, which relied on modular pipelines: a perception module detects objects, a planner computes a trajectory, a controller executes it, and each module is trained or tuned separately. VLAs instead train a single large model — typically built on top of vision-language model backbones similar to those used in multimodal chat assistants — on large datasets of robot demonstrations paired with language instructions, so that generalization comes from the model's scale and data diversity rather than from a human anticipating every edge case in the pipeline logic.

What foundation models add to the stack, concretely:

• Instruction following — the ability to accept a new task in plain language rather than requiring a new hand-coded behavior for each task variant.
• Cross-embodiment transfer — some models are trained on data from multiple robot types (arms, mobile manipulators, humanoids) and can transfer skills across them, reducing the amount of robot-specific data needed.
• Few-shot adaptation — fine-tuning a generalist policy on a small amount of task-specific demonstration data, rather than training a narrow policy from scratch.
• Unified perception-to-action reasoning — closing the gap between "the robot sees a cluttered shelf" and "the robot decides how to reach into it," without hand-off between separately trained modules.

None of this replaces the middleware and simulation layers below it — a VLA still needs ROS 2 (or an equivalent) to get sensor data in and actuator commands out, and it's still trained and validated largely in simulation before deployment. What it replaces is the hand-written planning and control logic that used to sit between perception and actuation. That's a meaningful shift in engineering effort: instead of writing and tuning a grasp planner, teams now curate demonstration data and fine-tune a model.

Edge Runtimes: Getting Intelligence onto the Robot

A foundation model with billions of parameters is not something you casually run on a robot's onboard compute. Robots have hard constraints that cloud inference doesn't: limited power budgets, latency requirements measured in milliseconds for anything involving contact or balance, and no guarantee of network connectivity in a warehouse, field, or home. This is why the edge runtime layer — the software that takes a trained model and makes it run efficiently on embedded compute — has become as strategically important as the model itself.

Edge deployment for robotics typically involves several compounding techniques:

The practical effect is a two-tier inference pattern showing up across the industry: a smaller, fast policy running directly on the robot's onboard GPU or NPU for low-latency reactive control, paired with a larger model — sometimes cloud-hosted, sometimes running asynchronously on the same device — that handles higher-level task reasoning and re-plans less frequently. This mirrors a pattern familiar from other edge-AI domains: keep the tight control loop local and fast, push the expensive reasoning to whatever budget of compute and latency you can afford.

Why This Is Converging Now

Each of these layers existed in some form for years — ROS dates to 2007, physics simulators predate that, and robot learning research has used neural networks since at least the 2010s. What's new is that they've started fitting together as a recognizable, reusable stack rather than one-off research systems, largely because 2025 and 2026 saw simultaneous maturity across all three of the layers above middleware at once.

Simulators became fast and accurate enough — through GPU-parallelized physics — to generate the volume of training data that foundation models require. Vision-language-action models became capable enough, riding the same scaling techniques that improved general-purpose multimodal models, to serve as genuine generalist controllers rather than research demos. And edge runtimes matured enough, through quantization and distillation techniques developed for on-device AI more broadly, to actually run the resulting models within a robot's power and latency budget. Any one of those three developments alone wouldn't have produced a usable stack — a great simulator with no deployable model is a research tool, and a great model with no edge runtime is a cloud demo tethered to a lab bench. It's the alignment of all three around the same period that turned "robotics software" from a pile of bespoke systems into something closer to a standard architecture that new teams can adopt rather than reinvent.

Practical Implications for Builders and Businesses

For companies evaluating robotics investments — whether building robots or deploying them — this convergence changes the calculus in a few concrete ways.

First, the build-versus-buy line has moved. Teams no longer need to write a custom grasp planner or motion-control pipeline from scratch to get a working manipulation system; they can start from an open or licensed foundation model and fine-tune it on task-specific demonstration data. That lowers the barrier to entry for narrow applications but raises the importance of good data collection, since fine-tuning quality now depends heavily on how well a company can capture representative demonstrations of its actual task.

Second, simulation infrastructure has become a genuine competitive asset rather than a nice-to-have. Companies that invest in high-fidelity, GPU-accelerated simulation environments tailored to their specific robots and tasks can generate far more training data, far more cheaply, than companies relying solely on real-world data collection. That's a build decision worth making deliberately rather than treating simulation as an afterthought before a hardware demo.

Third, the middleware choice still matters even though it's less visible than the model layer. Standardizing on ROS 2 (or a comparable modern middleware) makes it dramatically easier to integrate third-party sensors, swap hardware vendors, and hire engineers who already know the ecosystem, compared to maintaining a proprietary communication layer.

Fourth, deployment planning needs to account for the edge runtime layer early, not as an afterthought. A model that performs well in simulation or on a cloud GPU can fail entirely on the actual robot if nobody budgeted for quantization, latency testing, and power-draw validation on the target hardware. Teams that treat "get the model working" and "get the model working on the robot" as the same milestone are usually wrong about their timeline.

A rough way to think about where to focus engineering effort by team stage:

• Early-stage / prototype: Use existing middleware (ROS 2) and off-the-shelf simulators; don't build custom infrastructure yet. Focus effort on defining the task narrowly enough that a foundation model can be fine-tuned on a modest data collection budget.
• Scaling to production: Invest in simulation fidelity specific to your task and hardware; this is where domain randomization and system identification start paying off in reduced sim-to-real failures.
• Deploying at volume: Invest heavily in the edge runtime layer — quantization, the cloud-edge split architecture, and real hardware latency/power testing — since this is where most late-stage surprises occur.

Limitations, Open Questions, and What to Watch Next

The stack described above is real and increasingly standard, but it's far from solved. A few limitations are worth being honest about.

Sim-to-real transfer still fails in ways that are hard to predict. Domain randomization and system identification reduce the gap, but contact-rich tasks — anything involving deformable objects, granular materials, or fine manipulation — remain notoriously difficult to simulate accurately, which means policies trained mostly in simulation can still behave unreliably on exactly those tasks in the real world.

Foundation models for robotics also inherit a version of the reliability problem seen in language models: they can produce confident, plausible-looking actions that are subtly wrong, and unlike a chatbot's wrong answer, a robot's wrong action can damage equipment or injure someone nearby. Safety validation for learned, non-interpretable control policies is an open area, and most deployments today still wrap model outputs in hand-coded safety envelopes — hard limits on force, velocity, and workspace — rather than trusting the model's judgment unconstrained.

Data remains the bottleneck underneath all of it. Cross-embodiment datasets are growing, but high-quality, diverse, real-world robot demonstration data is still scarce relative to what text and image models had available, and it's far more expensive to collect since it requires physical robots and often human teleoperation.

Looking ahead, a few developments are worth tracking:

• Whether cross-embodiment foundation models genuinely reduce the data needed for new robot types, or whether each new hardware platform still effectively requires its own fine-tuning dataset.
• How edge hardware (NPUs, robot-specific inference chips) evolves to close the gap between what foundation models need and what's deployable within realistic power budgets.
• Whether safety and verification tooling for learned policies matures fast enough to keep pace with capability, particularly for deployments around people.
• Whether middleware standardization holds — ROS 2 is dominant today, but as more compute-heavy learned components enter the stack, new communication patterns optimized for model inference rather than classical control loops could emerge.

FAQ

What is the robotics software stack, in simple terms?

It's the layered software that turns a robot's raw sensors and motors into autonomous behavior — typically middleware for communication between components, simulation for training and testing, learned models (increasingly foundation models) for perception and decision-making, and an edge runtime layer that gets those models running efficiently on the robot's onboard compute.

Is ROS 2 required to build a robot today?

No, but it's the closest thing to a default choice for new commercial and research robots because of its ecosystem of drivers, simulators, and hardware support. Some companies still use proprietary middleware, particularly for latency-critical control loops, but most integrate with ROS 2 at some layer for interoperability.

What's the difference between a VLA and a traditional robot control pipeline?

A traditional pipeline chains together separately engineered modules — perception, planning, control — each tuned by hand for specific tasks. A vision-language-action model is a single learned model trained end-to-end on demonstration data that maps camera images and language instructions directly to robot actions, generalizing across tasks through data and scale rather than hand-coded logic for each one.

Why can't robots just run large models directly without an edge runtime layer?

Because robots have hard limits on power, latency, and often connectivity that most large models weren't designed around. Techniques like quantization, distillation, and hybrid cloud-edge inference are what make it possible to run capable models within a robot's actual power budget and the millisecond-scale latency many control loops require.

What is the sim-to-real gap?

It's the performance drop that happens when a policy trained in a physics simulator is deployed on a real robot, caused by differences between simulated and real-world dynamics, sensor noise, and visual appearance. Domain randomization and system identification are the two main techniques used to reduce it.

Do foundation models eliminate the need for simulation in robotics?

No — simulation has become more important, not less, because it's the primary way teams generate the large volumes of training and fine-tuning data that foundation models require, alongside being used to validate policies before real-world deployment.

How is a business supposed to decide where to invest in this stack?

It depends on stage: early-stage teams generally benefit most from using existing middleware and simulators rather than building custom infrastructure, while teams scaling toward production deployment should invest more specifically in simulation fidelity and edge runtime engineering tailored to their actual hardware and tasks.

Teams navigating where to invest across this stack — middleware, simulation, model fine-tuning, or edge deployment — can get hands-on help scoping that from Woyce Technologies.