7. Loading `sqw` and `dnd` objects to memory

Majority of Horace algorithms accept the name of file containing binary sqw or dnd object as the source of the data for operations. This is necessary as not every sqw object may fit to memory. When an object can fit memory it is convenient to place it for speed and convenience operating with it. Horace provides two methods of placing sqw objects in memory: read or load operations and sqw/dnd object construction.

load is standard operation, which allow loading sqw/dnd objects or arrays of such objects previously saved from MATLAB to standard MATLAB .mat file using MATLAB save operation. The objects operated this way must fit memory.

read is standard MATLAB command allowing users loading various data in memory. To allow operations with Horace .sqw files without overloading standard read command, Horace introduces read_horace family of commands.

7.1. read_horace

Reads sqw or dnd data from a file. The object type is determined from the contents of the file.

output = read_horace(filename);

The returned variable is an sqw or dnd object.

7.2. read_sqw

Reads sqw data from a file.

output = read_sqw(filename);

The returned variable is an sqw object.

7.3. read_dnd

As read_sqw, but reads dnd data saved to file. If the file contains a full sqw dataset, then only the binned data will be read.

output = read_dnd(filename);

The returned variable is an dnd object.

7.4. Constructing `sqw` object from filename

Calling sqw constructor with the name of binary sqw file is equivalent to invoking read_sqw function.

output = sqw(sqw_filename);

8. Saving sqw objects from memory and creating filebacked objects

8.1. save

There are two ways of saving sqw or dnd objects to files on disk.

First – use MATLAB save command, which would save objects from memory into MATLAB .mat files, and second one – using Horace overload for save command in storing objects in sqw file format. MATLAB command works the same as for any MATLAB variable:

save('filename','variable1_name','variable2_name',...);

The benefit of this way of storing data is the possibility of storing multiple objects in a single .mat file.

Note that the method works for objects in memory so if you use it to save filebacked sqw objects you will probably obtain unexpected results, as main part of filebacked sqw object is not loaded in memory.

Warning

Saving filebacked objects using MATLAB ``save`` command is dangerous!

One can say that filebacked objects can be created in two ways. First way – “primary” filebacked objects build over existing sqw files or saved with permanent file name. (see below). These objects are backed by permanent .sqw files which stay on disk after objects are deleted from MATLAB session. Second way – “secondary” filebacked objects obtained as the result of various operations with filebacked sqw objects, e.g. large cuts which do not fit memory – (see File- and memory-backed cuts for more details about filebacked cuts), the result of unary or binary operations between “primary” filebacked sqw objects, results of filebacked PageOp algorithms, etc.. These objects are backed by temporary files with extension .tmp_XXXXXXX and temporary files get deleted when such objects go out of scope.

MATLAB save command saves part of filebacked object stored in memory. The operation is saving the reference to the file containing the pixels – the name and path to the file the object is backed by. As temporary sqw (.tmp_XXXXXXX) file backing the object gets deleted, .mat file for stored “secondary” type of filebacked object will contain incorrect reference to the missing file. .mat file for the “primary” filebacked object will contain reference to existing .sqw file, so can be restored back until the referred file exists and is not moved or renamed. Obviously, this way of saving filebacked sqw objects is also not very reliable.

The only reliable way of saving filebacked sqw object is to use Horace save command, which stores sqw object in binary Horace .sqw file format. The command for this is:

save(sqw_object, filename);

This method saves single object into Horace binary file with extension .sqw, so if you have filebacked sqw object, the method would correctly write this object. It will be possible to restore the object later by accessing appropriate .sqw file. If your filebacked object is backed by temporary file, the object will not be physically saved (long operation) as the major part of this object is already located in file. The file contents will be synchronized with the data in memory and temporary file will be renamed to the name, you have provided as the second input for the save command.

You, of course, can also use Horace save command to create Horace binary .sqw files from sqw/dnd objects in memory.

See File- and memory-backed cuts to read a bit more about filebacked and memory based cuts and Horace Config for the information on how to set up the size of memory based object.

8.2. Create filebacked objects from data on disk

If your sqw file is big enough (see mem_chunk_size and fb_scale_factor from “hor_config” class for numerical meaning of “big enough”, the command:

fb_obj = sqw('filename');

will create filebacked object fb_obj. You can operate with filebacked object exactly as with memory based object, but many operations which involve operations with pixels will be slower. Alternatively, you may create filebacked object regardless of its size using command:

fb_obj = read_sqw('filename','-filebacked');

Note, that this command invoked without -filebacked is equivalent to sqw('filename') and

mb_obj = read_sqw('filename','-force_pix_location');

will try to load sqw object in memory regardless of its size on disk, so will fail if the object is to big to fit the memory.

The filebacked objects created this way, unlike filebacked objects created as the result of the operations with filebacked objects or large cut operations, are backed by permanent files which would not be deleted if the object in memory gets deleted.

7. Loading sqw and dnd objects to memory