Reading and Writing Very Large Files
IDL on all platforms is able to read and write data from files up to 231-1 bytes in length. On some platforms, it is also able to read and write data from files longer than this limit.
To see if IDL on your platform supports large files, use the following:
If IDL prints the number 64, the platform supports large files. For more information, see "!VERSION" (IDL Reference Guide).
Warning
Macintosh systems that use the UNIX File System (UFS) rather than the default Mac OS Extended Filesystem (HFS+) will not be able to access large files, even though IDL itself will report the ability to do so. This is a limitation of the file system, not of IDL.
When reading and writing to files smaller than this limit, there is no difference in behavior between the platforms that can and those that cannot handle larger files. IDL uses longword integers for file position arguments (e.g. POINT_LUN, FSTAT) and keywords, as before. However, when dealing with files that exceed this limit, IDL uses signed 64-bit integers in order to be able to properly represent the offset. Consider the following example:
;Open the file OPENW, 1, 'test.dat' ;Initial position should be 0. POINT_LUN, -1, POS ;Print the position and its type. HELP, POS ;Move the file pointer past the signed 32-bit boundary. POINT_LUN, 1, '000000ffffffffff'x ;The position is now too large to represent as a longword. POINT_LUN, -1, POS ;Print the position and its type. HELP, POS CLOSE, 1
Executing these statements results in the following output:
Initially, the file position is 0, which fits easily into a 32-bit integer. Once the file position exceeds the range of a signed 32-bit number, IDL automatically shifts to the 64-bit integer type.
Limitations of Large File Support
There are limitations on IDL's support for very large files that must be understood by the IDL programmer:
- On any platform, the amount of data that IDL can transfer in a single operation is limited by the amount of memory it can allocate. On most platforms, IDL is a 32-bit program, and as such, can theoretically address up to 2^31-1 bytes of memory (approximately 2.3GB). On these 32-bit platforms, reading, writing, and processing data larger than this limit must be done in multiple operations. Most systems do not have 2.3 GB of memory available, and other programs running on the system also compete for the same memory, so the actual memory available is likely to be considerably smaller.
- The ability to read or write to very large files is constrained by the ability of the underlying file system to support such files. Many platforms can only support large files on certain file systems. For example, many platforms will be unable to support these operations on NFS mounted file systems because NFS version 3 and later must be in use on both client and server. Some platforms can only support such operations on special large file systems, and only if they are mounted using the appropriate mount options. Consult your system documentation to determine the limitations present on your system and the procedures for supporting very large file.
To see if your platform is 32- or 64-bit, use the following:
PRINT, !VERSION.MEMORY_BITSIF "32" is returned, your platform is 32-bit. If "64" is returned, your platform is 64-bit. For more information, see "!VERSION" (IDL Reference Guide).
