towards making CompCache adaptive, CompCache is a compressed in-memory swap device for Linux developed by Nitin Gupta,

Goal : Study different possibilities of making the CompCache adaptive.

In systems that support paging, and under memory pressure, the kernel uses disk swapping to enlarge accessible memory space (swapping). The “Compressed Memory Swap” is a solution that improves the performance of that mechanism.
CompCache is an initiative to introduce the memory compression mechanism to the Linux kernel, (project initiated by Nitin Gupta). In CompCache, when memory is needed and there is no free space, the system chooses a victim page to be replaced, that page is not sent to disk swap but is compressed and kept in memory (sent to a special memory block device called the CompCache), in this way we can free memory (by compressing pages) and at the same time reduce access to the backing store (because pages are compressed and kept in memory). So the main goal of CompCache is to reduce disk accesses (expensive I/O) by using memory data compression (pages).

The Adaptive Compressed Memory Swap (Adaptive CM-swap) is a research branch that aims to make the CompCache adaptive; We are studying different approaches that can be used to enable the compcache to be adaptive; by enabling the CompCache to change it's size at runtime (in the current implementation the size of compcache is Predefined), and to choose the appropriate size, this should be done carefully, if the compcache is small, the system will not fully take advantage of memory compression, whereas if it's big, it'll heart the system performance. our goal is :

• study different heuristics and algorithms that can be used to define the compcache size at runtime;
• how can we determine the best size? (what are the parameters)

“Adaptive, Compressed Memory Swap (CM-Swap) using complementary decision factor” , Embedded Systems Conference - SSNE08 , Algeria - 2008)

Memory Compression is very useful, because:
– It reduces accesses to the backing store (local or distant disks) by storing pages in memory in a compressed format, and so improves the system performance.
– Memory Compression improves performance for programs that have huge memory needs, since these programs need to access large data sets randomly; these large arrays cannot fit into memory, and using the classical swapping scheme leads to very bad performance.
– Memory Compression in Embedded Systems: Memory is a critical resource in many embedded systems. Increasing memory often increases packaging, cooling costs, size, and energy consumption. CM-swap (Memory compression) is used to save memory in systems with low or no disk space (diskless systems); it helps those embedded systems to run more programs without the need of any disk.

This work is not yet intended to be included in the CompCache, this is a research branch; Work on CompCache now focus more on making it small and very light so that it gets included in the Linux kernel mainline, for the next months I'll be working on making the decision factor better and perhaps I'll make some changes on the Rodrigo method and adapt it on the CompCache,

Libjpeg Tests
Image processing Performance Improvements using CompCache
Author : Baghdadi Riyadh,

System resources have been for a long time the most important obstacle for image processing, not only because memories are not large enough to hold huge images, but alos becuase of the gap in speed between memories and CPUs; Image Processing applications are known to be intensive-memory and intensive-CPU consummers, and this lead us to think about reducing the gap between CPU speed and memory speed,
One novel solution is the usage of memory compression, to enlarge memory capacities, and at the same time reduce I/O and thus give the CPU the possibility to achive more in less time; (no need for waiting). In the next test we focus on comparing between two systems, one with CompCache activated and the other with CompCache disactivated; Our goal is to show how much the usage of compcache will nhance image processing performance.

• Hardware:

  For tests we used a Desktop Computer with\\
  CPU     : Intel E2180 2.00GHz CPU, (Core2 Duo)\\
  RAM    : 256MB,\\
  Swap    : 1GB,\\
  Disk      : 250GB SATAII disk (7200rpm);\\

• Software:

        Linux Distro     : Ubuntu Hardy Heron (8.04)\\
        kernel version: 2.6.26-rc8\\
        CompCache patch:

patch_compcache_with_notify_support_2.6.26-rc8;

In this test we use Libjpeg utilities for image processing (we focus on convertion images from JPEG to BMP), to evalute the effect of using compcache on this type of applications;
The Independent JPEG Group's JPEG library (Libjpeg) [1] is a library for handling JPEG files, it contains tools that convert from JPEG format to other formats (cjpeg & djpeg tools);
we use djpeg tool to convert a JPEG 5785 x 8429 DirectClass 8-bit file, of size 42.3974mb, from JPEG format to BMP;
To put more pressur on memoy we launch many instances (many processes) of djpeg at the same time, and this cause an intensive demand for memory, memory gets full quickly and the system start using swap partition (in our case it uses the compcache).

          djpeg  -bmp  -outfile test.out.bmp  test.jpg &

we launch 2 instances of djpeg;

* Results with compcache notactivated :

Instance | Reel time | CPU time in sec | System CPU time | faults

instance1| 21.45 | 6.89 | 0.53 | 1819

Instance2| 21.50 | 7.75 | 0.63 | 2099

Average | 21.47 | 7.32 | 0.58 | 1959

* when compcache was activated wee have these results :

Instance | Reel time | CPU time in sec | System CPU time | faults

instance1| 20.23 | 6.72 | 1.00 | 2426

Instance2| 20.23 | 6.77 | 0.86 | 2461

Average | 20.23 | 6.81 | 1.71 | 3522

this test shows a little improvements when using compcache;

The next test focus on how much compcache can make the system able to run more processes then an other ordinary system; we launch here (exactly as previous tests) many instances of djpeg and see if all processes success to run until they finish their job;

* compcache is not activated, we load 8 instances of djpeg at the same time, to convert the test file from jpeg to bmp;

This test shows that a system with 220MB of RAM fails to run 8 instances of djpeg at the same time, after a short time (28 sec) the memory becomes full and the OOM killer stops the first two instances of djpeg to free some memory,
It is the job of the Linux “oom killer” to sacrifice one or more processes in order to free up memory for the system when all else fails;

* compcache is now activated, we repeat the same test;

By Using compcache the system could run successfully all the 8 processes, and no process were killed; This shows clearly that compcache increased the capacity of the system by 25%, making it able to run up to 8 instances of djpeg instead of 6 instances.

USAGE : decompress_jpeg_file <FILE> <N>

where N is the number of times for decompressing <FILE> to a bmp file, results are thrown to /dev/null, and results of the test are sent to a log

file with the same name with the test file :$FILE.log -results are ready to be plotted;
- use a jpeg image of very big size (i used one with 42mb)

Using CompCache we could speed up JPEG conversion and enhance the system maximum cappacity for running processes by 25% and thus the system could run 25% more processes whereas without compcache the OOM killer was called;

[1]. The Independent JPEG Group's JPEG library for JPEG manipulation
http://www.ijg.org/