How to debug a DLL or EXE

This article shows two techniques to debug a User Defined DLL or a User Defined EXE.
- The first one is to write "debugged" values in a text file.
- The second one is to attach the Visual Studio debugger to a running process. It is shown here using Microsoft Visual Studio Community 2017.

Authored By Michael Humphreys, Sandrine Auriol and Tim Gustafson


Article Attachments


This article shows how to debug User Defined DLL or EXE created in Visual Studio in C, C++ or C#. The steps to create those User Defined features are not detailed here.

Debug using a text file

One way to debug a User Defined DLL or a User Defined EXE is to write "debugged" values in a text file. Let's add some debugging lines to the user-defined scattering function {Zemax}\DLL\SurfaceScatter\Lambertian.c. The modified file called Lambertian_DebugText.cpp can be found in the attachments.

First, to write in a text file, the following libraries are added at the top of the code:

#include <fstream>
#include <sstream>
#include <iostream>
#include <shlobj.h>
using namespace std;

Then a "DebugBool" parameter is defined:

int __declspec(dllexport) APIENTRY UserParamNames(char *data)
    if (i == 3) strcpy(data, "DebugBool");
    return 0;

Depending on the value of the "DebugBool" parameter, a file is created and values can be written in that file for further analysis:

     bool debug = data[54] != 0;


     CHAR my_documents[MAX_PATH];
     fstream debuglog;
     string filepath;
     if (debug)
         SHGetFolderPath(NULL, CSIDL_PERSONAL, NULL, SHGFP_TYPE_CURRENT, my_documents);
         filepath = string(my_documents) + "\\Zemax\\DLL\\SurfaceScatter\\debug.txt";, ios::out | ios::app); // debuglog.close();


     debuglog << debugged_value << "\t";

As can be seen, writing a text file to debug is an easy method.

Debug using Visual Studio debugger

Compile the DLL or EXE in Debug Mode

Let's see how to compile a DLL or EXE in Debug Mode. For example, let's compile a user defined surface (DLL). For more details, see the article "How to compile a User-Defined DLL".


Debug mode

Once compiled, 2 files need to be copied from Project\x64\Debug to {Zemax}\DLL\Surfaces: the DLL and the PDB file. Normally (when using the DLL in release mode), only the DLL file is copied. But the PDB file (Program database) is needed here because it stores debugging information about programs.

Copy DLL and PDB

Every time the DLL is modified and recompiled, it will have to be manually copied again from Project\x64\Debug to {Zemax}\DLL\Surfaces.

To avoid this, the "Project\x64\Debug" output folder can be changed to {Zemax}\DLL\Surfaces:

In C++: in Visual Studio, go to Solution Explorer. Select the project. Right-click, go to Properties. The project Property pages opens. Under Configuration Properties, select General. Change the Target Name.

In C#: in Visual Studio, go to Project Preferences > Build > Output directory.

Back to OpticStudio

Open OpticStudio and select the User Defined Surface or Operand.

  • User Defined Surface DLL: change Surface 2 to a User Defined Surface. Surface 2 is using the DLL in Debug mode:
    User-defined surface in OpticStudio
  • User Defined Operand UDOC: select the UDOC operand in the Merit Function.
    Make sure to set the Timeout (ms) to a high value so that the UDOC can be debugged in Visual Studio without having OpticStudio throw a timeout error.
    UDOC in OpticStudio

At this point, do not update the system or the Merit Function yet. An update will execute the user defined surface or UDOC. If there are errors in the code, it may cause a crash.

Attach to Process

Back to Visual Studio, everything is ready to start debugging. If the user extension is a DLL, the debugger can be attached to the OpticStudio.exe process. If the user extension is an EXE, the debugger can be attached to our application process.

DLL Method – Attach to OpticStudio

The debugger can be attached to the OpticStudio running process :


A window appears. Click Select.


Select Native Code:


In Available Processes, make sure to select the appropriate OpticStudio process. It may be easier to have only one OpticStudio instance open.  


Then click Attach.

EXE Method – Attach Directly To UDOC

To debug a UDOC, the UDOC application itself is attached to the process rather than OpticStudio. Therefore, a “pause” is set in the UDOC code. It can be a Console.ReadKey() for C# or system(“pause”) for C++ to pause the actual execution. 
In C#, the code might look like below:

Attach to UDOC

If the Merit Function is updated, a Command Line Window will open and display a message :

Update Merit Function

Now, the UDOC can be attached to the debugger. Make sure to have the correct code selected for the UDOC. Below, “Managed code” is selected as the code is in C++.

UDOC debugger

Note that with a UDOC, the process needs to be attached every time the Merit Function is updated.

Set Breakpoints

Then let's set a few breakpoints inside the DLL code to investigate the values being passed.


Run the DLL/EXE

In OpticStudio, clicking on the User Defined Surface or updating the Merit Function with the UDOC operand will run the DLL or EXE and start the debugger. Back to Visual Studio, a yellow arrow is displayed on the breakpoint. It shows the code that will be executed next.

The window DEBUG > WINDOWS > LOCALS displays the values being passed between the DLL and OpticStudio. In that window, two types of data structures are passed: User Data (UD) and Fixed Data (FD).


User Data UD

It contains the rays x, y, z locations on a plane tangent to the surface vertex (the DLL will compute and return them for the real surface), the ray direction cosines l, m, n and the surface normals ln, mn, nn (which are computed  and returned) and other data like index, transmission, path length.


Fixed Data FD

It mainly contains data the DLL cannot change: the surface data in the Lens Data Editor (radius, conic, parameters, etc.), the ray wavelength, polarization state and Type and Numb parameters indicating what data OpticStudio wants from a given call to the DLL.


For example, FD -> n2 shows the index of refraction of N_BK7 being passed from OpticStudio to the DLL.

Stop the debugger

To stop the debugger, click on the red Square in Visual Studio.

Stop debugger

Distributing the DLL/EXE in Release Mode

After debugging the DLL/EXE, the configuration can be changed back from Debug to Release. The libraries for Debug configurations are not redistributable while the libraries for Release configurations are redistributable.



Was this article helpful?
1 out of 2 found this helpful



Article is closed for comments.