By: Charles Dresser
A New Reality
Augmented reality (AR) is the union of a consistent set of artificial perceptions (often digitally generated) added to the worldly perceptions of a user. AR may be contrasted with virtual reality (VR), where a consistent set of artificial perceptions replace the worldly perceptions of the user.
Modest forms of AR have benefitted the lives of Americans for decades. For instance, 1st & ten—the artificial first down line presented on NFL broadcasts—is an example of AR, first broadcast in 1998¹.
The Potential For AR Technology
More ambitious AR applications involve heads-up displays that display virtual imagery on top of the real-world seen by a user. Optimistic proponents for AR have forecast that AR equipped glasses will replace desktop monitors and even smartphones. A pair of AR glasses may prevent the bowing of one’s head to interface with the virtual world of a smartphone and allow wearers to engage with their surroundings while maintaining digital contact. Notably, Google Glass, Microsoft HoloLens, and Magic Leap One have all offered to the public a means to merge real and virtual imagery. Even Apple has been rumored to have been working on this application for years. With the potential to displace approximately 7-Billion smartphone users, huge sums of capital have been spent trying to develop this technology. So why must we all still looking down at our phones—like a drinking bird toy—every time we check an email?
There are many technical challenges associated with the optical design of an AR headset, including occlusion, convergence-accommodation, eye motion, and the like. However, a single law of physics may reign supreme as the greatest challenge.
The Etendue Problem
Etendue is a French word that translates to “extent” in English. Etendue is a measure of the extent, or spread, of light. Technically etendue is the product of an area of a pupil (i.e., pi r-squared) and the solid angle subtended by the light entering the pupil (i.e., size of the cone of light entering the pupil). The etendue problem is analogous to the law of conservation of energy and dictates how much light can fit in a hole. Large fields of view and large pupils require large etendues.
The Great Extent of the Human Eye
The pupil of the human eye can reach diameters of ~20mm and angular fields of view greater than 100o (up to 120o horizontally). For this reason, the human eye has a huge etendue. For comparison an optical system that has a comparable etendue is a full-frame camera lens².
Sony Full Frame 24x36mm Cine Camera
The Small Extent of a Waveguide
HoloLens and Magic Leap One each employ waveguides for optical propagation of the virtual imagery. The etendue of an optical waveguide is a function of the cross-sectional area and the acceptance cone of the waveguide. Light entering at a too steep of angle will not propagate within the waveguide. The acceptance cone of a waveguide is typically about +/-30o (compared with 100o FOV of the human eye); and the width of a waveguide is going to be no greater than the thickness of the AR glasses (a few millimeters—compared with 20mm for the human eye).
Accordingly, the etendue of a waveguide optical system is far less than that of the human eye. This leaves the optical designer in the unenviable position of having to choose between high resolution and wide field of view. This trade-off is analogous to looking at small image (like a small field of view) or the scaled up pixelated version of the same image (low resolution).
Small image (FOV)
Pixelated image (low resolution)
For this reason, the fields of view of the currently available AR headsets have been too small for wide market adoption. However, a review of the patent filings in the field shows potential solutions may exist.
Patent Trends Related To AR and Head Mounted Displays
Patent applications related to AR and head mounted displays, on the whole, continue to trend upward, albeit at a decreasing rate. As patent applications do not usually publish until 18 months after their filing date only incomplete information is available for those applications filed after 2019. For this reason, applications with filing dates in 2020, 2021, or 2022 have not been included in the below graphs.
Review of these patent applications suggest that potential technological advances may address the problems of a waveguide propagation.
Foveated Imaging and Beam Steering Laser Projection
Most of the images perceived by human sight is very poor resolution. Only the retina, with its 5o field of view, senses at a high resolution; and only the fovea, a fraction of the area of the retina, perceives at a resolution high enough for reading. A dynamic eye-tracking projection system may take advantage of this limitation of human vision to present high resolution only when and where an eye can see it. This is called foveated or foveal-imaging.
An exemplary AR patent application, U.S. Pat. App. No. 16/158,625, “SYSTEM FOR A METHOD OF PROJECTING AUGMENTATION IMAGERY IN A HEAD-MOUNTED DISPLAY” by D. Palacios et al., of Augmenteum Inc., describes a head-mounted AR display that provides foveated imaging through use of a novel wavefront corrector, as well as beam splitting cubes. Palacios teaches the capability of each technology allows projection of imagery with an etendue comparable to that of the human eye. For instance, the beam splitting cubes allow for a wide field of view and a large pupil through which a pico projector may scan an image using low powered lasers. Technologies like that taught by Palacios will likely advance the efforts of those who long for widespread adoption of near-eye AR applications.
1. https://en.wikipedia.org/wiki/1st_%26_Ten_(graphics_system)
2. https://caldwelllaw.com/app/uploads/2022/03/ilm1-2017iwk-141.pdf
3. https://www.fdtimes.com/2017/06/08/sony-full-frame-24x36mm-cine-camera/