In this article, we demonstrate the usage of OpticStudio’s TrueFreeForm surface type in designing an eye-tracking subsystem within a wedge prism for AR/VR applications. In particular, the grid-based sag optimization functionality of TrueFreeForm is demonstrated to achieve a sub-aperture sag optimization, greatly improving imaging quality for the eye-tracking subsystem.

Authored By Zachary Derocher

Introduction

The TrueFreeForm surface is a sequential surface type in OpticStudio which uses a superposed combination of polynomial and grid-based sag characterizations. In addition, the TrueFreeForm surface also supports optimization of the sag values at each data point in the grid sag data file, allowing for a unique, non-parameterized optimization methodology. This can be advantageous in cases where spatially-selective optimization is desired, or a given polynomial function cannot well represent the desired sag structure.

Background

TrueFreeForm allows surface sag to be defined using polynomial functions including biconic toroidal, even asphere, Zernike standard polynomial, and extended polynomial, as well as a grid-based sag definition. There are several advantages to having access to all these sag functions within the same surface. For instance, freeform surfaces optimized using extended polynomial, biconic toroidal, or grid-based definitions can now be externally toleranced for irregularity via Zernike coefficients. Perhaps the most interesting advantage of TrueFreeForm is the ability to optimize the sag profile through setting points in the sag grid as variables.

Grid-based sag optimization has some interesting advantages over traditional parametric (polynomial function-based) optimization. Because the default grid sag interpolation algorithm used is a bicubic spline method, the domain of influence of any individual point in the sag grid extends only two points away. This means that if we optimize the sag of just one point in the data grid, only the local surface structure is affected, and the distant surface structure is perfectly preserved. This is demonstrated in the following figure, where a 1D spline model is shown for simplicity The three curves are defined via cubic spline between 5 points. The leftmost point varies between the three curves, and the effect only extends two points away.

Considering the full 2D grid of sag values which are used to define a surface structure, this means that by optimizing the sag at a single point (shown in red below), we only affect the surface structure within a rectangular region of 5x5 points>

This brings two revelations. Firstly, with grid sag optimization, we can optimize sub-regions of a given surface, whereas with parametric optimization we necessarily alter the entire surface structure with any change to a parameter value. Secondly, we can generate unique surface geometries, which might be very difficult or impossible with a given limited-order polynomial function.

There are two main considerations to be aware of when using the grid-based optimization approach. Because the sag grid utilizes cubic interpolation, this means that the curve defined between a set of points might be limited in the geometric form it can achieve. Additionally, there is a fine balance in the optimal density of sag points to define in the data grid; too many, and the optimization can become slowed down (too many variables), and sampling becomes an issue (see below). Too few data points, and the optimization might not be able to achieve an optimal surface structure. Care and consideration must be used when definition the data grid to be used with this optimization method.

Case study in TrueFreeForm grid optimization

Setup

The basis of this case study was a freeform wedge prism type optical see-through head mounted display. It is based on a patent application submitted by Gao C. and Hua H., which also supports an inlaid near-infrared eye-tracking subsystem utilizing the same three optical surfaces, as well as a wedged singlet to provide a focus. Our initial design form was created in OpticStudio using parametric optimization (extended polynomial) of the three optical surfaces. Because the TrueFreeForm surface supports an extended polynomial function, the TrueFreeForm surface type can be used at outset. The nominal microdisplay visual system was optimized, and the eye-tracking subsystem was subsequently inlaid, initially without any re-correction of the parametric freeform optical surfaces.

Note that in this design, for simplicity, the see-through portion of the system was ignored, and the design was only optimized at a single wavelength.

Because the three optical surfaces were initially optimized only for the visual display system performance, ignoring the eye-tracking system performance, the image quality was reasonable for the F/3 visual system NIR image quality, but quite poor for the NIR eye-tracking system. Here, the MTF at fields sampled across the full FOV of the visual display system is shown, as well as the full field astigmatism for the NIR eye tracking subsystem. The latter had an average astigmatism of roughly 1.4 waves, and a total average RMS wavefront error of 1.42 waves across the full field of view.

One traditional approach to rectify this imbalance in performance is to re-optimize the three freeform surfaces. However, any improvement in the NIR eye tracking system will necessarily come at the expense of performance in the visual display system. Instead, TrueFreeForm enables a more novel approach. A sub-aperture optimization on S3, the top surface of the prism, was used to improve the eye-tracking subsystem performance without compromising the performance of the microdisplay system.

Defining the variable sag grid

Note the strong spatial separation between the surface S3 footprints of the imaging path (green) of the NIR ray bundle and the microdisplay illumination on S3 (blue):

With the extended polynomial terms of TrueFreeform surface S3 already optimized for the microdisplay system, we superposed a grid sag definition (initially just empty zeros) into the surface sag profile; only the subregion of surface S3 which catches the NIR rays as they approach the eye-tracking camera would be re-optimized. Care was used to ensure that such an optimization has no impact on the microdisplay ray bundle footprint on S3. It was noted previously that the dominant aberration in that subsystem was field-independent astigmatism. As such, a relatively low-order surface profile correction was considered sufficient to greatly improve optical performance.

With these considerations in mind, a blank (all-zero) grid sag file was generated. The system has left/right symmetry, which was utilized; we optimized the right side of the grid sag profile, and any changes were picked up on the left side by TrueFreeForm’s symmetry options. As such, the variable region was chosen for points on the right only.

The points in the grid sag definition are demarked above (right) with black “x” marks. Note that the variable region chosen, highlighted in light blue, is 4x4 points. For the bicubic spline interpolation, the region of influence for a given variable is known to extend a further 2 points in all directions (region of influence highlighted orange above). With these selections of grid density and variable region, we could be sure that the surface profile would remain fixed in the area illuminated by the microdisplay ray paths, meaning the nominal microisplay system performance would be preserved during this optimization.

The grid sag variables were selected by highlighting the desired points using the Grid Point Selector tool, as shown in the following image.

Optimizing the sag grid

Once the sag grid was defined and a variable subsection selected, the Merit Function could be built. Localizing the surface optimization to a sub-aperture means that we could remove all variables except for our grid sag points (and the NIR detector position). All of the optimization targets previously used for the initial microdisplay visual system optimization were also removed.

In building the Merit Function, we used a simple geometric spot-size default Merit Function. Note that the Rectangular Array pupil integration method was chosen, because Gaussian Quadrature would leave large areas of the pupil unsampled. That is not acceptable in grid-based sag optimization; the limited domain of influence of each variable sag-point must be adequately sampled to ensure some fields or pupil positions aren’t preferentially optimized. This means that in general, not only do we need relatively high pupil sampling, but we must also use relatively high field sampling (especially in this system, where the surface to optimize is near a focal plane, meaning there is strong field separation of the footprints on this surface).

After setting the fields and pupil sampling, we also added some constraints on the allowed maximum slope and sag departure from the nominal surface profile. This was done with the GOPT operand, which checks the sag and local slope at each point in the grid, and allows constraints on the maximum or minimum values

Optimized results

After the optimization, we saw a result in the sag profile which closely matched expectation; a local sagittal curvature was added in the region of interest, which serves to counteract the dominant field-constant astigmatism, while the non-local surface profile remained unchanged.

We can prove that the performance of the microdisplay visual system is preserved by viewing the before/after MTF performance; the plots are identical, demonstrating that our grid variable selection was adequate to isolate the NIR eye tracking subsystem optimization.

The performance of the eye tracking subsystem to greatly improved through this optimization. The astigmatism was reduced by several orders of magnitude across the field (from 1.38 waves average at 850nm, to 0.0509 waves average at 850nm). The MTF performance after optimization was vastly improved (to a field-average of 0.45 at 40 lp/mm), allowing a well-resolved image of the user’s entire eyebox in the eye tracking system. This should allow us to more accurately pick up the NIR reflections from the user’s pupil in order to help guide software optimizations of the projected image.

Final thoughts

We’ve seen here that grid sag optimization was highly successful in performing a local surface optimization and greatly improving optical performance of a subsystem, while preserving the non-local sag profile. The drawbacks of using this approach were that the balance of sampling (both field and pupil) and the density of the variable sag grid must be carefully considered. In general, this optimization methodology requires higher density ray sampling and larger numbers of variables, resulting in slower optimization runs.

We must also be very aware, as with any freeform optimization, of manufacturability. Here, we used only a few GOPT operands to control sag and slope offset within the variable region. But because OpticStudio has to stitch the edges of the optimized variable subgrid with the rest of the grid sag structure, we can end up with large slope effects just outside the fringes of the variable grid. This effect can be seen in the optimized result here; the variable region is roughly 5mm across in the X direction, but there are slope variations extending well beyond that bound (to be exact, an additional 2 sag points in the grid, in all directions, as this is the domain of influence of the variable region). It would be more robust to check the local slope at every sag point in the domain of influence of the variable sag region, and limit, for instance, the RMS slope across that space. This can be done with the addition of SSAG operands, or slope equivalent calculations, at the grid point coordinates within the domain of influence.

KA-01890