CP Proposal - Benchmark + source code

[Thorsten W. Schmidt]

Ok, here the requested Benchmark and the source code to generate it.

Benchmark:
1. Filled about 72.5% (365M) of my memory with data and programs. So that
we have something we can swap out. Although we do not want it to be swaped
out.
2. 'swapoff -a' and then 'swapon -a' to empty the swap.
3. Measure the memory consumption (swap, cache) with top
4. Open a file with 398863775 bytes ten times and with one of the flags
O_DIRECT, O_DSYNC, O_STREAMING, POSIX_FADV_NOREUSE turned on.
5. Measure the memory consumption (swap, cache) with 'top'


Memory and cache before and after the process is given in the following.

-----------------------------------------------------------------------
------ Open file without any special settings. -----
-----------------------------------------------------------------------
Mem:    515636k total,   511892k used,     3744k free,      704k buffers
Swap:  1004020k total,        0k used,  1004020k free,   134452k cached

real    0m13.297s
real    0m11.946s
real    0m11.328s
real    0m11.972s
real    0m12.793s
real    0m11.944s
real    0m10.969s
real    0m12.992s
real    0m12.456s
real    0m14.132s

Mem:    515636k total,   511028k used,     4608k free,     1064k buffers
Swap:  1004020k total,   167940k used,   836080k free,   275160k cached

-----------------------------------------------------------------------
------ Open file with O_DIRECT -----
-----------------------------------------------------------------------
Mem:    515636k total,   512060k used,     3576k free,     2660k buffers
Swap:  1004020k total,        0k used,  1004020k free,   135064k cached

real    0m8.108s
real    0m7.925s
real    0m8.425s
real    0m8.413s
real    0m8.356s
real    0m8.346s
real    0m8.398s
real    0m8.122s
real    0m8.488s
real    0m8.114s

Mem:    515636k total,   511412k used,     4224k free,     2940k buffers
Swap:  1004020k total,      832k used,  1003188k free,   134116k cached

-----------------------------------------------------------------------
------ Open file with O_STREAMING -----
-----------------------------------------------------------------------
Mem:    515636k total,   511248k used,     4388k free,     1472k buffers
Swap:  1004020k total,        0k used,  1004020k free,   135020k cached

real    0m13.229s
real    0m12.343s
real    0m11.680s
real    0m12.705s
real    0m11.032s
real    0m11.673s
real    0m11.515s
real    0m12.172s
real    0m13.220s
real    0m11.796s

Mem:    515636k total,   510688k used,     4948k free,      508k buffers
Swap:  1004020k total,   149800k used,   854220k free,   280876k cached

-----------------------------------------------------------------------
------ Open file with O_DSYNC -----
-----------------------------------------------------------------------
Mem:    515636k total,   510536k used,     5100k free,      912k buffers
Swap:  1004020k total,        0k used,  1004020k free,   112136k cached

real    0m13.252s
real    0m15.784s
real    0m13.447s
real    0m16.383s
real    0m13.745s
real    0m10.723s
real    0m11.523s
real    0m12.655s
real    0m13.037s
real    0m11.516s

Mem:    515636k total,   510600k used,     5036k free,      552k buffers
Swap:  1004020k total,   153716k used,   850304k free,   262120k cached

-----------------------------------------------------------------------
------ Open file with POSIX_FADV_NOREUSE -----
-----------------------------------------------------------------------
Mem:    515636k total,   511508k used,     4128k free,      756k buffers
Swap:  1004020k total,        0k used,  1004020k free,   136316k cached

real    0m15.426s
real    0m15.103s
real    0m12.279s
real    0m10.582s
real    0m10.357s
real    0m17.865s
real    0m12.230s
real    0m12.818s
real    0m12.665s
real    0m13.621s

Mem:    515636k total,   511888k used,     3748k free,      820k buffers
Swap:  1004020k total,   133696k used,   870324k free,   259788k cached

-----------------------------------------------------------------------
------ Summary -----
-----------------------------------------------------------------------

             Standard|O_DIRE|O_STREAM|O_DSYNC|POSIX_FADV_NOREUSE
------------+--------+------+--------+-------+----------
Cache change|+140708k|-948k |+145856k|149985k|+123472k
Swap change |+167940k|+832k |+149800k|153716k|+133696k
Avg. Time   | 13.38s |8.26s | 12.14s | 13.21s| 13.28s

As you can see, opening files with O_DIRECT is much faster as they are not
cached and the kernel is not bothered with swapping.
Turing off the cache is only a good idea, if you are sure you will not
need the files any more or if you copy one or more large files. "Large"
starts for *me* at about 5% of my total physical RAM. Bur for most cases
it is faster to turn on O_DIRECT always.

Greetings
  Thorsten W. Schmidt


Here the code to create the benchmark:

// see
//
// compile with
//   g++ streaming.cpp

// http://kerneltrap.org/node/460
// fd = open(file, ... | O_STREAMING);

#define O_STREAMING 04000000


#include <stdio.h>
#include <fcntl.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <unistd.h>

int main ( int argc, char ** argv )
{
  if ( argc < 2 )
  {
    printf ("Usage:\n %s file [advice]\n\n", argv[0] );
    printf (" file     give a large file to read\n" );
    printf (" advice   0 standard open\n");
    printf ("          1 direct open");
    printf ("          2 stream open\n");
    printf ("          3 posix no reuse od data\n");
    printf ("          4 dsync open\n");
    printf ("\n");
    exit (-1);
  }

  int   fd = -1;
  char* filename = argv[1];
  int   use_streaming = 0;
  long  fileLen = 0;

  // Check commandline parameter
  if ( argc > 2)
  {
    use_streaming = atoi (argv[2]);
  }

  // open the file
  switch (use_streaming)
  {
    case 0:
      if ( ( fd = open ( filename, O_LARGEFILE | O_RDONLY )) == -1 )
      {
        printf ("Could not open file %s\n", filename);
        exit (-1);
      }
      printf ("Open file without any special settings.\n");
      break;
    case 1:
      if ( ( fd = open ( filename, O_DIRECT | O_LARGEFILE | O_RDONLY )) ==
-1 )
      {
        printf ("Could not open file %s\n", filename);
        exit (-1);
      }
      printf ("Open file with O_DIRECT.\n");
      break;
    case 2:
      if ( ( fd = open ( filename, O_STREAMING | O_LARGEFILE | O_RDONLY ))
== -1 )
      {
        printf ("Could not open file %s\n", filename);
        exit (-1);
      }
      printf ("Open file without O_STREAMING.\n");
      break;
    case 3:
      if ( ( fd = open ( filename, O_LARGEFILE | O_RDONLY )) == -1 )
      {
        printf ("Could not open file %s\n", filename);
        exit (-1);
      }
      posix_fadvise ( fd, 0, 0, POSIX_FADV_NOREUSE );
      printf ("Open file with POSIX_FADV_NOREUSE.\n");
      break;
    case 4:
      if ( ( fd = open ( filename, O_DSYNC | O_LARGEFILE | O_RDONLY )) == -1 )
      {
        printf ("Could not open file %s\n", filename);
        exit (-1);
      }
      printf ("Open file with O_DSYNC.\n");
      break;
  }
  fileLen=lseek(fd, 0L, SEEK_END);
  printf ("Filesize: %d\n", fileLen);
  lseek(fd, 0L, SEEK_SET);

  // read the file and do something with the date. Otherwise it could
happen that it is not paged in or ...
  char data[256];
  for ( long i=0; i<fileLen/128; i++)
  {
    read ( fd, data, 128 );
    for (int j=0;j<127;j++) data[j]=0;
  }

  close (fd);
}