The Transparent Patient

By Roland Piquepaille

Several companies are currently developing 'augmented reality' systems to help surgeons to simultaneously see inside and outside their patients. In this short article titled "And No, It's Not for Seeing Through Clothes," Fast Company describes a solution from Siemens. This system consists of a custom video-see-through head-mounted display (HMD), two color video cameras attached to the HMD that provide ultrasound, magnetic resonance (MR), or computer tomography (CT) pre-recorded images, and a third infrared video camera for tracking what's doing the surgeon. Such systems could become available in three to five years. But they will not be cheap. A complete augmented reality system should cost as much as $400,000.

Here are some short excerpts from the Fast Company article.

While current medical-imaging techniques such as ultrasound, magnetic resonance (MR), and computer tomography (CT) harvest a wealth of data from inside our bodies, the resulting images can be viewed only on light boards and computer screens. To a neurosurgeon plunging a seven-inch needle into a patient's brain, that's clearly an imperfect solution: He has to take his eye off the incision to see where he's headed.

Siemens's response, called "augmented reality," starts with a headset that overlays prerecorded ultrasound, MR, or CT images with real-time video captured by a pair of cameras just above the physician's eyes. A third infrared camera, also mounted on the headset, spatially orients the video in relation to a set of optical tracking markers placed around the patient's body. The resulting picture is projected onto two tiny screens positioned directly in front of the physician's eyes.

Below are two previous examples of Siemens technologies that will make patients appear transparent and may one day allow surgeons to operate through micro robots on a cellular level. (Credit: Siemens)

Using a head mounted display and marker bridge, a researcher checks how closely the image of a tumor corresponds with the position of an actual (mock-up) tumor. What the researcher sees is shown on the monitors with the tumor marked in red.

Virtual (also called "optical") biopsies may provide an inexpensive method of pin-pointing individual cancer cells, thus opening the door to the potential of cellular surgery. The biopsies use a novel fluorescent agent that is activated when in contact with tumor specific enzymes.

Both of these images were extracted from the Fall 2001 issue of Pictures of the Future, a former online magazine from Siemens. Here is a short excerpt of what wrote Arthur F. Pease four years ago.

Probably the most far-reaching of these nascent technologies is "in situ visualization." Also known as augmented reality image guidance, in situ visualization can use a head mounted display (HMD) or semi-transparent plate to superimpose 3D computer images of anatomical structures on the actual environment. The images may originate from just about any digital diagnostic modality, and can be dynamically introduced into the surgeon's field of vision. Otherwise invisible structures such as deep seated tumors appear in their exact sizes, shapes and positions vis-à-vis visible objects such as the surface of a patient's head, anchored in their real-world structures with an accuracy of +1/-1 mm. In short, in situ visualization is a revolutionary step that sets the stage for the symbiosis of all digital imaging technologies and opens the door to the transparent patient.

Now, let's return to 2004 for more technical papers with the abstract of a paper named "An Augmented Reality System for MRI-Guided Needle Biopsies: Initial Results in a Swine Model."

The system for augmented reality (AR) visualization enables the physician to perceive MR images in-situ. It consists of a custom video-see-through head-mounted display (HMD), two color video cameras attached to the HMD that provide a stereo view of the scene, and a third video camera for tracking. Optical marker sets are used for viewpoint adjustment and needle tracking. The system runs on a PC and achieves real-time performance (30 fps) with a latency of 0.1 sec, generating a stable augmentation with no apparent jitter visible in the composite images.

Finally, if you're interested by augmented reality systems developed at Siemens, one of them was awarded the U.S. patent number 6,856,324 on February 15, 2005 under the name "Augmented reality guided instrument positioning with guiding graphics." Here is a direct link to this patent.

Sources: Lucas Conley, Fast Company, Issue 96, July 2005, Page 32; and various web sites

Related stories can be found in the following categories.

• Innovation
  
• Medicine
  
• Virtual Reality
  
• Vision and Visualization Apps