Ultrasound Is About to Get a Lot Smarter: NVIDIA and Siemens Healthineers Team Up on Raw2Insights

Ultrasound has always been the workhorse of medical imaging—safe, real-time, portable, and cheap. But the way we turn those sound waves into pictures is surprisingly crude. The traditional beamforming pipeline makes a bunch of simplifying assumptions, like a constant speed of sound throughout the body. Anyone who’s had an ultrasound knows the body isn’t that uniform. Tissues, fat, bone—they all bend and slow sound differently.

NVIDIA and researchers from Siemens Healthineers just released something that might finally fix this. It’s called NV-Raw2Insights-US, and the idea is refreshingly direct: instead of working from the finished image, learn directly from the raw sensor data. The raw channel data is the closest thing to how sound actually interacts with the body. Everything else is a reconstruction, and reconstruction means information loss.

The model estimates the speed of sound for each patient in real time. That might not sound flashy, but it’s a big deal. If you know the local sound speed, you can refocus the image adaptively. What used to require complex, iterative computation now happens in a single AI pass. No more one-size-fits-all assumptions. The system generates a personalized map of sound speed for each patient and uses it to correct the image live.

NVIDIA calls this class of models Raw2Insights, and this is the first real application. The vision is end-to-end AI for ultrasound—skip the traditional pipeline entirely, go straight from raw data to clinical insight. I like that they’re actually shipping something concrete instead of just talking about it.

The Hard Part: Getting Raw Data Out

Raw ultrasound channel data isn’t normally accessible on clinical scanners. It’s high-bandwidth, and the hardware wasn’t designed to stream it out. NVIDIA solved this with Holoscan Sensor Bridge (HSB), an open-source FPGA IP that taps into the DisplayPort outputs of an ACUSON Sequoia scanner. The data gets packetized and sent over Ethernet to an NVIDIA IGX system for processing. They call this Data over DisplayPort, and it’s clever—uses existing scanner ports without invasive hardware mods.

Once the data hits GPU memory, inference runs on a Blackwell-class GPU. The sound-speed estimate streams back to the scanner, and the live image improves. All in real time. That’s the kind of tight feedback loop that makes this practical for clinical use.

What This Unlocks

The architecture is modular. Once you have raw channel data in GPU memory, you can plug in new AI models without changing the hardware. Software-defined ultrasound. Continuous improvement through updates, not hardware swaps. That’s the long-term play.

NVIDIA is releasing the model weights, the dataset, and the GitHub repo. So if you’re building on top of this, you can start today. Links are in the original post.

My Take

This is genuinely interesting, but let’s not oversell it. The model estimates sound speed for adaptive focusing—that’s one piece of the puzzle. It doesn’t replace the full imaging pipeline yet. The Raw2Insights vision is ambitious, but we’re at step one. Also, the hardware requirements are non-trivial. Blackwell GPU, IGX system, FPGA bridge—this isn’t something you’d deploy in a rural clinic tomorrow.

Still, the direction is right. Ultrasound has been stuck in the same basic paradigm for decades. Learning directly from raw physics data, rather than from reconstructed images, is the kind of shift that could eventually make ultrasound more diagnostic and less operator-dependent. Siemens Healthineers being involved gives it clinical credibility.

I’ll be watching to see how fast the community builds on this. The modular architecture is smart, but it only matters if people actually use it.