Reading data files

[John W. Eaton]

On 11-Nov-1998, Jens Thoms Toerring <address@hidden> wrote:

|    I am using Octave often for simulations of experimental data. Therfore, I
| also have to read in these data. The data files usually start with a header
| with the experimental conditions, commented out with either a '%' or a
| '#'. While this leads to no problem e.g. with Matlab, Octave doesn't seem
| to like these header lines and complains (file name is `a1')
| 
| error: load: a1: inconsistent number of columns near line 2
| error: load: unable to extract matrix size from file `a1'
| error: evaluating index expression near line 1, column 1
| 
| Is there a way to avoid having to edit each data file to remove the headers?
| (Of course, on could easily delete the header with grep or friends, but this
| would mean to have every file twice and, besides, I forget to go through
| this procedure constantly...)

Here is a patch that allows comments in otherwise numbers-only data
files.  With it, I'm able to load a file with the following contents
without error.

  % this is a test
  # this is a test
          # this is a test
          % this is a test
   # this is a test
   % this is a test
  1 2 3# this is a test
  1 2 3 # this is a test
  1 2 3 # this is a test

  1 2 3% this is a test
  1 2 3 % this is a test
  1 2 3 % this is a test
  4 5 6
  % this is a test
  # this is a test
          # this is a test
          % this is a test
   # this is a test
   % this is a test
  7 8 9
   % this is a test

It does not provide the more general capability of passing the data
through an arbitrary filter.  Perhaps that will also be built in to
the load command someday.  Until then, techniques like those used in
the aload function Dirk mentioned should be sufficient.

Thanks,

jwe

Wed Nov 11 14:07:27 1998  John W. Eaton  <address@hidden>

        * load-save.cc (get_mat_data_input_line): New function.
        (get_lines_and_columns): Use it here.
        (read_mat_ascii_data): And here and do our own reading instead of
        using Matrix::operator<<.


*** src/load-save.cc~   Wed Oct 28 20:47:30 1998
--- src/load-save.cc    Wed Nov 11 14:10:25 1998
***************
*** 921,939 ****
    return name;
  }
  
! static void
! get_lines_and_columns (istream& is, const string& filename, int& nr, int& nc)
  {
!   streampos pos = is.tellg ();
  
!   int file_line_number = 0;
  
!   nr = 0;
!   nc = 0;
! 
!   while (is && ! error_state)
      {
!       string buf;
  
        char c;
        while (is.get (c))
--- 921,936 ----
    return name;
  }
  
! static string
! get_mat_data_input_line (istream& is)
  {
!   string retval;
  
!   bool have_data = false;
  
!   do
      {
!       retval = "";
  
        char c;
        while (is.get (c))
***************
*** 941,948 ****
          if (c == '\n')
            break;
  
!         buf += c;
        }
  
        file_line_number++;
  
--- 938,979 ----
          if (c == '\n')
            break;
  
!         if (c == '%' || c == '#')
!           {
!             // skip to end of line
!             while (is.get (c) && c != '\n')
!               ;
! 
!             break;
!           }
! 
!         if (! is.eof ())
!           {
!             if (! have_data && c != ' ' && c != '\t')
!               have_data = true;
! 
!             retval += c;
!           }
        }
+     }
+   while (! (have_data || is.eof ()));
+ 
+   return retval;
+ }
+ 
+ static void
+ get_lines_and_columns (istream& is, const string& filename, int& nr, int& nc)
+ {
+   streampos pos = is.tellg ();
+ 
+   int file_line_number = 0;
+ 
+   nr = 0;
+   nc = 0;
+ 
+   while (is && ! error_state)
+     {
+       string buf = get_mat_data_input_line (is);
  
        file_line_number++;
  
***************
*** 1026,1032 ****
        {
          Matrix tmp (nr, nc);
  
!         is >> tmp;
  
          if (is)
            {
--- 1057,1085 ----
        {
          Matrix tmp (nr, nc);
  
!         if (nr < 1 || nc < 1)
!           is.clear (ios::badbit);
!         else
!           {
!             double d;
!             for (int i = 0; i < nr; i++)
!               {
!                 string buf = get_mat_data_input_line (is);
! 
!                 istrstream tmp_stream (buf.c_str ());
! 
!                 for (int j = 0; j < nc; j++)
!                   {
!                     tmp_stream >> d;
!                     if (is)
!                       tmp.elem (i, j) = d;
!                     else
!                       goto done;
!                   }
!               }
!           }
! 
!       done:
  
          if (is)
            {