Create Library In Memory With The MEMLIB Option
In many data heavy SAS operations it is crucial to reduce I/O operations. I/O is the most common bottleneck in production ETLs. Therefore, you should think about where your data reside. If you have reasonably sized data sets in a library you access frequently, you should think of letting the library reside in memory. This post shows and example of how to do just that with the MEMLIB option to the LIBNAME Statement.
Regular Disk SAS Library
First, let us take a look at a regular library. We simply define the library RegLib and points to my Documents folder.
libname RegLib 'c:\Users\Peter\Documents\';
Next, let us perform a few common data tasks within the library. First we create a data set with 10mio observations. Next, we create a second data set, which simply reads the observations and writes them to a new data set. These operations are for demonstration and comparing purposes only.
data RegLib.InMemoryData; do x=1 to 10e7; output; end; run; data RegLib.InMemoryData1; set RegLib.InMemoryData; run;
These data steps take about 4 and 8 seconds to run respectively (Real time).
In Memory Library
libname RamLib "c:\Users\Peter\Desktop\" memlib;
Next, we run the exact same code as above, but directs the data sets to the in memory library. Now, if you encounter an error here, it is most likely because you do not have the user right to lock pages in memory in your operating system. Use the Error: Cannot adjust the token privileges with error 1300 guide to resolve the problem.
data RamLib.InMemoryData; do x=1 to 10e7; output; end; run; data RamLib.InMemoryData1; set RamLib.InMemoryData; run;
Now, this code took about 3 and 6 seconds respectively to run. That is a stunning 25% decrease in wall clock time. All due to a simple little option in your libname statement.
The MEMLIB Option is a great way to let your data reside in memory. This post demonstrates how to read an entire library into memory directly in the LIBNAME Statement. This is a very simple introduction to the option. For a more thorough explanation, see the article Why Aren’t You Using MEMLIB? by Erik Dilts. For other performance tips, see the Performance Category at my blog.
If you like the MEMLIB Option and its capabilities, you may also like the closely related MEMCACHE Option.
You can download the entire code from this post here.