Dynamically Loadable Modules

LINKIMAGE can be used to make IDL load your system routines in a simple and efficient manner. However, it quickly becomes inconvenient if you are adding more than a few routines. Furthermore, the limitation that the LINKIMAGE call must happen before any code that calls it is compiled makes it difficult to use and complicates the process of redistributing your routines to others. IDL offers an alternative method of packaging your system routines, called Dynamically Loadable Modules (DLMs), that address these and other problems.

This section covers the following topics:

DLM Concepts

How DLMs Work

The Module Description (.dlm) File

The IDL_Load() function

DLM Example

Packaging and Installing DLMs

DLM Concepts

The IDL_SYSFUN_DEF2 structure, which is described in Registering Routines, contains all the information required by IDL for it to be able to compile calls to a given system routine and call it:

A routine signature (Name, minimum and maximum number of arguments, if the routine accepts keywords).

A pointer to a compiled language function (usually C) that supplies the standard IDL system routine interface (argc, argv, argk) and which implements the desired operation.

IDL does not require the actual code that implements the function until the routine is called: It is able to compile other routines and statements that reference it based only on its signature.

DLMs exploit this fact to load system routines on an "as needed" basis. The routines in a DLM are not loaded by IDL unless the user calls one of them. A DLM consists of two files:

A module description file (human readable text) that IDL reads when it starts running. This file tells IDL the signature for all system routines contained in the loadable module.

A sharable library that implements the actual system routines.This library must be coded to present a specific IDL mandated interface (described below) that allows IDL to automatically load it when necessary without user intervention.

DLMs are a powerful way to extend IDL's built-in system routines. This form of packaging offers many advantages:

Unlike LINKIMAGE, IDL automatically discovers DLMs when it starts up without any user intervention. This makes them easy to install — you simply copy the two files into a directory on your system where IDL will look for them.

DLM routines work exactly like standard built-in routines, and are indistinguishable from them. There is no need for the user to load them (for example, using LINKIMAGE) before compiling code that references them.

As the amount of code added to IDL grows, using sharable libraries in this way prevents name collisions in unrelated compiled code from fooling the linker into linking the wrong code together. DLMs thus act as a firewall between unrelated code. For example, there are instances where unrelated routines both use a common third party library, but they require different versions of this library. A specific example is that the HDF support in IDL requires its own version of the NetCDF library. The NetCDF support uses a different incompatible version of this library with the same names. Use of DLMs allows each module to link with its own private copy of such code.

Since DLMs are separate from the IDL program, they can be built and distributed on their own schedule independent of IDL releases.

System routines packaged as DLMs are effectively indistinguishable from routines built-into IDL by ITT Visual Information Solutions.

Use of sharable libraries in this manner has ample precedent in the computer industry. Most modern operating systems use loadable kernel modules to keep the kernel small while the functionality grows. The same technique is used in user programs in the form of sharable libraries, which allows unrelated programs to share code and memory space (e.g. a single copy of the C runtime library is used by all running programs on a given system).

How DLMs Work

IDL manages DLMs in the following manner:

When IDL starts, it looks in the current working directory for module definition (.dlm) files. It reads any file found and adds the routines and structure definitions thus defined to its internal routine and structure lookup tables as "stubs". In the system routine dispatch table, stubs are entries that inform IDL of the routine's existence, but which lack an actual compiled function to call. They contain sufficient information for IDL to properly compile calls to the routines, but not to actually call them. Similarly, stub entries in the structure definition table allow IDL to know that the DLM supplies the structure definition, but the actual definition is not present.

After it looks in the current working directory, IDL searches !DLM_PATH for .dlm files and adds them to the table in the same manner. The default value of !DLM_PATH is the directory in the IDL distribution where the binary executables are kept (bin/bin.platform), followed by the idlde/plugins directory, both in the IDL installation. This default can be changed by defining the IDL_DLM_PATH preference (this is similar to the way the IDL_PATH preference works with !PATH). This process happens once at startup, and never again. This means that IDL's knowledge of loadable modules is static and unchangeable once the session is underway. This is very different from the way !PATH works, and reflects the static nature of built-in routines. The format of .dlm files is discussed in The Module Description (.dlm) File.

Warning
If you redefine the IDL_DLM_PATH preference, be sure to include the token <IDL_DEFAULT>. IDL will not run correctly if the default DLM directories are not included in !DLM_PATH.

See Packaging and Installing DLMs for additional information about how IDL selects sharable libraries on different platforms.

The IDL session then continues in the usual fashion until a call to a routine from a loadable module occurs. At that time, the IDL interpreter notices the fact that the routine is a stub, and loads the sharable library for the loadable module that supplies the routine. It then looks up and calls a function named IDL_Load(), which is required to exist, from the library. It's job is to replace the stubs from that module with real entries (by using IDL_SysRtnAdd()) and otherwise prepare the module for use.

Once the module is loaded, the interpreter looks up the routine that caused the load one more time. If it is still a stub then the module has failed to load properly and an error is issued. Normally, a full routine entry is found and the interpreter successfully calls the routine.

At this point the module is fully loaded, and cannot be distinguished from a compiled part of IDL. A module is only loaded once, and additional calls to any routine, or access to any structure definition, from the module are made immediately and without requiring any additional loading.

The Module Description (.dlm) File

The module description file is a simple text file that is read by IDL when it starts. Module description files have the file suffix .dlm. The information in the .dlm file tells IDL everything it needs to know about the routines supplied by a loadable module. With this information, IDL can compile calls to these routines and otherwise behave as if it contains the actual routine. The loadable module itself remains unloaded until a call to one of its routines is made, or until the user forces the module to load by calling the IDL DLM_LOAD procedure.

Empty lines are allowed in .dlm files. Comments are indicated using the # character. All text from a # to the end of the line is ignored by IDL and is for the user's benefit only.

All other lines start with a keyword indicating the type of information being conveyed, possibly followed by arguments. The syntax of each line depends on the keyword. Possible lines are:

MODULE Name

Gives the name of the DLM. This should always be the first non-comment line in a .dlm file. There can only be one MODULE line.

MODULE JPEG

DESCRIPTION DescriptiveText

Supplies a short one line description of the purpose of the module. This information is displayed by HELP, /DLM. This line is optional.

DESCRIPTION IDL JPEG support

VERSION VersionString

Supplies a version string that can be used by the IDL user to determine which version of the module will be used. IDL does not interpret this string, it only displays it as part of the HELP, /DLM output. This line is optional.

VERSION 6a

BUILD_DATE DateString

If present, IDL will display this information as part of the output from HELP, /DLM. IDL does not parse this string to determine the date, it is simply for the users benefit. This line is optional.

BUILD_DATE JAN 8 1998

SOURCE SourceString

A short one line description of the person or organization that is supplying the module. This line is optional.

SOURCE ITT Visual Information Solutions

CHECKSUM CheckSumValue

This directive is used by ITT Visual Information Solutions to sign the authenticity of the DLMs supplied with IDL releases. It is not required for user-written DLMs.

STRUCTURE StructureName

There should be one STRUCTURE line in the DLM file for every named structure definition supplied by the loadable module. If you refer to such a structure before the DLM is loaded, IDL uses this information to cause the DLM to load. The IDL_Init() function for the DLM will define the structure.

GLOBAL_SYMBOLS

This line is optional. Including this line in the DLM file will cause the shared library to load all of its symbols (functions or procedures) as globally accessible rather than locally accessible. If a symbol is globally accessible, then libraries that are loaded later will be able to access the symbol. In practice, adding this line to the DLM file will cause IDL to set the RTLD_GLOBAL flag when calling the dlopen() operating system function to load the module.

On Microsoft Windows and Macintosh OS X systems, symbols are automatically loaded as global. A GLOBAL_SYMBOLS line in the DLM file will be quietly ignored.

Use caution when making a DLM's symbols globally accessible. Judicious naming of the DLM's symbol names will help ensure that symbols exported by the DLM will not cause namespace collisions with symbols from other libraries.

FUNCTION RtnName [MinArgs] [MaxArgs] [Options...]

PROCEDURE RtnName [MinArgs] [MaxArgs] [Options...]

There should be one FUNCTION or PROCEDURE line in the DLM file for every IDL routine supplied by the loadable module. These lines give IDL the information it needs to compile calls to these routines before the module is loaded.

RtnName

The IDL user level name for the routine. The routine name can be a simple procedure or function name (e.g. MY_PROCEDURE or MY_FUNCTION), or the name of an object method (e.g. MY_OBJECT::PROCEDURE_METHOD or MY_OBJECT::FUNCTION_METHOD).

MinArgs

The minimum number of arguments accepted by this routine. If not supplied, 0 is assumed.

MaxArgs

The maximum number of arguments accepted by this routine. If not supplied, 0 is assumed.

Options

Zero or more of the following:

OBSOLETE

IDL should issue a warning message if this routine is called and !WARN.OBS_ROUTINE is set.

KEYWORDS

This routine accepts keywords as well as plain arguments.

For example, a procedure named READ_JPEG that accepts a minimum of one argument, a maximum of three arguments, and also accepts keyword arguments would have the following definition in the .dlm file:

PROCEDURE    READ_JPEG   1   3     KEYWORDS

The IDL_Load() function

Every loadable module sharable library must export a single symbol called IDL_Load(). This function is called when IDL loads the module, and is expected to do all the work required to load real definitions for the routines supplied by the function and prepare the module for use. This always requires at least one call to IDL_SysRtnAdd(). It usually also requires a call to IDL_MessageDefineBlock() if the module defines any messages. Any other initialization needed would also go here:

int IDL_Load(void)

This function takes no arguments. It is expected to return True (non-zero) if it was successful, and False (0) if some initialization step failed.

DLM Example

This example creates a loadable module named TESTMODULE.

Note
Code for this example is included in the external/dlm subdirectory of the IDL installation.

TESTMODULE provides 2 routines:

TESTFUN

A function that issues a message indicating that it was called, and then returns the string "TESTFUN" This function accepts between 0 and IDL_MAXPARAMS arguments, but it does not use them for anything.

TESTPRO

A procedure that issues a message indicating that it was called. This procedure accepts between 0 and IDL_MAX_ARRAY_DIM arguments, but it does not use them for anything.

The intent of this example is to show the support code required to write a DLM for a completely trivial application. This framework can be easily adapted to real modules by replacing TESTFUN and TESTPRO with other routines.

The first step is to create the module definition file for TESTMODULE, named testmodule.dlm:

MODULE testmodule 
DESCRIPTION Test code for loadable modules 
VERSION 1.0 
SOURCE ITT Visual Information Solutions 
BUILD_DATE JAN  8 1998 
FUNCTION TESTFUN 0 IDL_MAXPARAMS 
PROCEDURE TESTPRO 0 IDL_MAX_ARRAY_DIM

The next step is to write the code for the sharable library. The contents of testmodule.c are shown in the following figure. Comments in the code explain what each step is doing.

Table 15-7: testmodule.c
C	1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56	#include <stdio.h> #include "idl_export.h" /* Define message codes and their corresponding printf(3) format * strings. Note that message codes start at zero and each one is * one less that the previous one. Codes must be monotonic and * contiguous. / static IDL_MSG_DEF msg_arr[] = { #define M_TM_INPRO 0 { "M_TM_INPRO", "%NThis is from a loadable module procedure." }, #define M_TM_INFUN -1 { "M_TM_INFUN", "%NThis is from a loadable module function." }, }; / The load function fills in this message block handle with the * opaque handle to the message block used for this module. The other * routines can then use it to throw errors from this block. / static IDL_MSG_BLOCK msg_block; / Implementation of the TESTPRO IDL procedure / static void testpro(int argc, IDL_VPTR argv) { IDL_MessageFromBlock(msg_block, M_TM_INPRO, IDL_MSG_RET); } /* Implementation of the TESTFUN IDL function / static IDL_VPTR testfun(int argc, IDL_VPTR argv) { IDL_MessageFromBlock(msg_block, M_TM_INFUN, IDL_MSG_RET); return IDL_StrToSTRING("TESTFUN"); } int IDL_Load(void) { /* These tables contain information on the functions and procedures * that make up the TESTMODULE DLM. The information contained in these * tables must be identical to that contained in testmodule.dlm. / static IDL_SYSFUN_DEF2 function_addr[] = { { testfun, "TESTFUN", 0, IDL_MAXPARAMS, 0, 0}, }; static IDL_SYSFUN_DEF2 procedure_addr[] = { { (IDL_SYSRTN_GENERIC) testpro, "TESTPRO", 0, IDL_MAX_ARRAY_DIM, 0, 0}, }; / Create a message block to hold our messages. Save its handle where * the other routines can access it. / if (!(msg_block = IDL_MessageDefineBlock("Testmodule", IDL_CARRAY_ELTS(msg_arr), msg_arr))) return IDL_FALSE; / Register our routine. The routines must be specified exactly the same * as in testmodule.dlm. */ return IDL_SysRtnAdd(function_addr, TRUE, IDL_CARRAY_ELTS(function_addr)) && IDL_SysRtnAdd(procedure_addr, FALSE, IDL_CARRAY_ELTS(procedure_addr)); }

Table 15-7: testmodule.c

#include <stdio.h> 
#include "idl_export.h" 
  
/* Define message codes and their corresponding printf(3) format 
 * strings. Note that message codes start at zero and each one is 
 * one less that the previous one. Codes must be monotonic and 
 * contiguous. */ 
static IDL_MSG_DEF msg_arr[] = { 
#define M_TM_INPRO                       0 
  {  "M_TM_INPRO",   "%NThis is from a loadable module procedure." }, 
#define M_TM_INFUN                       -1   
  {  "M_TM_INFUN",   "%NThis is from a loadable module function." }, 
}; 
  
/* The load function fills in this message block handle with the 
 * opaque handle to the message block used for this module. The other 
 * routines can then use it to throw errors from this block. */ 
static IDL_MSG_BLOCK msg_block; 
  
/* Implementation of the TESTPRO IDL procedure */ 
static void testpro(int argc, IDL_VPTR *argv) 
{ IDL_MessageFromBlock(msg_block, M_TM_INPRO, IDL_MSG_RET); } 
  
/* Implementation of the TESTFUN IDL function */ 
static IDL_VPTR testfun(int argc, IDL_VPTR *argv) 
{ 
  IDL_MessageFromBlock(msg_block, M_TM_INFUN, IDL_MSG_RET); 
  return IDL_StrToSTRING("TESTFUN"); 
} 
  
int IDL_Load(void) 
{ 
  /* These tables contain information on the functions and procedures 
   * that make up the TESTMODULE DLM. The information contained in these 
   * tables must be identical to that contained in testmodule.dlm. 
   */ 
  static IDL_SYSFUN_DEF2 function_addr[] = { 
    { testfun, "TESTFUN", 0, IDL_MAXPARAMS, 0, 0}, 
  }; 
  static IDL_SYSFUN_DEF2 procedure_addr[] = { 
    { (IDL_SYSRTN_GENERIC) testpro, "TESTPRO", 0, IDL_MAX_ARRAY_DIM, 0, 0}, 
  }; 
  
  /* Create a message block to hold our messages. Save its handle where 
   * the other routines can access it. */ 
  if (!(msg_block = IDL_MessageDefineBlock("Testmodule", 
                                           IDL_CARRAY_ELTS(msg_arr), 
                                           msg_arr))) return IDL_FALSE; 
  
  /* Register our routine. The routines must be specified exactly the same 
   * as in testmodule.dlm. */ 
  return IDL_SysRtnAdd(function_addr, TRUE,  
                       IDL_CARRAY_ELTS(function_addr)) 
    && IDL_SysRtnAdd(procedure_addr, FALSE,  
                     IDL_CARRAY_ELTS(procedure_addr)); 
}

If building a DLM for Microsoft Windows, a linker definition file (testmodule.def) is also needed. All of these files, along with the commands required to build the module can be found in the dlm subdirectory of the external directory of the IDL distribution.

Once the loadable module is built, you can cause IDL to find it by doing one of the following:

Move to the directory containing the .dlm and sharable library for the module.

Define the IDL_DLM_PATH preference to include the directory.

Running IDL to demonstrate the resulting module:

IDL> HELP,/DLM,'testmodule' 
** TESTMODULE - Test code for loadable modules (not loaded) 
Version:1.0,Build Date:JAN 8 1998,Source:ITT Visual Information 
Solutions. 
Path: /home/user/testmodule/external/testmodule.so 
IDL> testpro 
% Loaded DLM: TESTMODULE. 
% TESTPRO: This is from a loadable module procedure. 
IDL> HELP,/DLM,'testmodule' 
** TESTMODULE - Test code for loadable modules (loaded) 
Version:1.0,Build Date:JAN 8 1998,Source:ITT Visual Information 
Soluctions. 
Path: /home/user/testmodule/external/testmodule.so 
IDL> print, testfun() 
% TESTFUN: This is from a loadable module function. 
TESTFUN

The initial HELP output shows that the module starts out unloaded. The call to TESTPRO causes the module to be loaded. As IDL loads the module, it prints an announcement of the fact (similar to the way it announces the .pro files it automatically compiles to satisfy calls to user routines). Once the module is loaded, subsequent calls to HELP show that it is present. Calls to routines from this module do not cause the module to be reloaded (as evidenced by the fact that calling TESTFUN did not cause an announcement message to be issued).

Packaging and Installing DLMs

Once you have created sharable library (.so or .dll) and module description (.dlm) files, you will need to ensure that the files are installed in a location where IDL can find and load the libraries. Your approach may be slightly different depending on whether your dynamically loadable module supports a single platform or multiple platforms.

Single-Platform DLMs

If your module will be installed only on computers of a single architecture (32-bit Windows machines, for example, or 64-bit Linux machines), the process is relatively simple:

Create the sharable library file. The file will have the extension .dll for Microsoft Windows platforms, or .so for UNIX-like platforms (Macintosh, Linux, Solaris).

Create the module description file (.dlm) as described The Module Description (.dlm) File.

Place both the sharable library file and the module description file in a directory included in IDL's IDL_DLM_PATH preference. See Installing DLMs Using the IDL Workbench Update Mechanism for additional notes.

Restart IDL.

Even if your module supports only one platform, consider following the naming rules described in How IDL Selects the Correct Sharable Library File. Using the multi-platform naming rules incurs no performance penalty, and may save effort if you end up supporting other platforms in the future.

Multi-Platform DLMs

If your module will be installed on computers of different architectures, you must create a unique sharable library file for each architecture. To install the DLM on a user's machine, you have the following options:

Create Platform-Specific Installations

If you create a separate installation package for each architecture, creating a multi-platform DLM is essentially just creating a series of Single-Platform DLMs, one for each platform. Use caution with this approach, since you will have to ensure that if your end-user installs more than one platform's version of the DLM, the module description and shared library files for the different platforms are installed in the correct directories.

Create a Multi-Platform Installation

You can create a single installation package that supports multiple architectures if you follow a simple set of naming rules when creating your sharable library files. To create a multi-platform installation package:

Create a sharable library file for each platform, following the naming rules described in How IDL Selects the Correct Sharable Library File.

Create a single module description file (.dlm) as described in The Module Description (.dlm) File.

Place the module description file and all of the sharable libraries in a single directory included in IDL's IDL_DLM_PATH preference. See Installing DLMs Using the IDL Workbench Update Mechanism for additional notes.

Restart IDL.

How IDL Selects the Correct Sharable Library File

When IDL starts, it searches for DLMs in the directories included in IDL's IDL_DLM_PATH preference as described in How DLMs Work. When IDL finds a module description file, it adds the routines and structure definitions defined by the DLM to its internal routine and structure lookup tables.

It is not until later, when a user calls a routine defined by the DLM, that IDL actually loads the sharable library. At this point, IDL searches for a sharable library file built for the current platform.

Note
IDL's ability to search for platform-specific library file names was introduced in IDL 7.1.

IDL uses the following process to search for the sharable library file:

IDL constructs the base name of the library file by removing the .dlm suffix from the module definition file's name.

To the library's base name, IDL appends a platform-specific string. The specific strings are shown in Table 15-8 below. The string is the concatenation of the name of the platform's platform-specific bin subdirectory, along with the suffix .dll on Windows systems or .so on all UNIX-based systems.

For example, if the name of the DLM file is

my_module.dlm

then the platform-specific sharable library file name for a 64-bit Linux platform would be

my_module.linux.x86_64.so

IDL searches the directory that contains the module definition file (.dlm) for a library file with the platform-specific sharable library file name. If it finds a matching file, it loads the library and executes the routine called by the user.

If IDL does not find the platform-specific library file, it searches the directory that contains the module definition file (.dlm) for a library file with the same base name as the module definition file, replacing the .dlm extension with the suffix .dll or .so.

For example, if the name of the DLM file is

my_module.dlm

then the generically-named sharable library file name would be

my_module.dll

on a Windows system, or

my_module.so

on a UNIX system.

If IDL finds a generically-named sharable library file (with either the .dll or the .so extension), it attempts to load the library and execute the routine called by the user. Note that IDL will only be able to successfully load the library if the generically-named library file was built for the current platform.

If IDL fails to find either the platform-specific sharable library file or the generically-named library file, it will issue one of the following error messages:

If a platform-specific sharable library file for a different platform exists in the same directory, the error message is

Dynamically loadable module is unavailable on this platform: 
my_module.

If no platform-specific sharable library files for any platform are present, the error message is

Dynamically loadable module failed to load: my_module.

The first message indicates that the DLM exists but is not supported for the current platform, the second indicates that the DLM does not exist, despite the presence of the .dlm file.

One benefit of this file naming and search procedure is that you can distribute a DLM package that includes library files for several platforms in a single directory. IDL will load the correct shared library for the end-user's platform, or provide a sensible error message if the platform is not supported.

Platform-Specific Sharable Library File Suffixes

The following table lists the platform-specific file suffixes for IDL's supported platforms:

Table 15-8: Sharable Library File Suffixes
Platform	Sharable Library File Suffix
Windows 32-bit	`.x86.dll`
Windows 64-bit	`.x86_64.dll`
Solaris SPARC 32-bit	`.solaris2.sparc.so`
Solaris SPARC 64-bit	`.solaris2.sparc64.so`
Solaris x86 64-bit	`.solaris2.x86_64.so`
Linux 32-bit	`.linux.x86.so`
Linux 64-bit	`.linux.x86_64.so`
Macintosh OS X PPC 32-bit	`.darwin.ppc.so`
Macintosh OS X Intel 32-bit	`.darwin.i386.so`
Macintosh OS X Intel 64-bit	`.darwin.x86_64.so`

Example DLM Distribution

For example, suppose you have created a dynamically loadable module named my_cool_module, and created sharable libraries for Windows (32- and 64-bit) and Linux (32- and 64-bit) but not for Macintosh OS X or Solaris. Your DLM installation directory would contain the following files:

my_cool_module.dlm 
my_cool_module.x86.dll 
my_cool_module.x86_64.dll 
my_cool_module.linux.x86.so 
my_cool_module.linux.x86_64.so

If a user on a Macintosh OS X or Solaris system attempts to call a routine from the my_cool_module DLM, the fact that sharable libraries for other platforms exist informs IDL that the DLM intentionally does not provide support for that platform. If a user on one of these unsupported platforms attempts to use the functionality from the DLM, IDL will issue the message

Dynamically loadable module is unavailable on this platform: 
my_cool_module.

Installing DLMs Using the IDL Workbench Update Mechanism

Beginning in IDL 7.1, the idlde/plugins subdirectory of the IDL installation is automatically added to the list of directories in the IDL_DLM_PATH preference. This, coupled with IDL's ability to select a sharable library file based on a platform-specific file name, allows you to distribute multi-platform DLMs using the IDL Workbench update mechanism. See IDL Workbench Questions and Answers for more on the Workbench update mechanism.

Briefly stated, you can distribute your module using the Workbench update mechanism if you create an Eclipse update site that includes (among other things) a plug-in that contains your module description file and sharable libraries. When an end-user installs the plug-in, the module description and library files are placed in the idlde/plugins subdirectory of the IDL installation automatically, and are thus found by the DLM search mechanism the next time IDL starts. See IDL Plug-in Wizard for information on how to generate a plug-in and update site.