Re: [Libcdio-devel] [PATCH] UDF+ISO image extraction sample (and a plan for upcoming changes)

[Rocky Bernstein]

On Sun, Jan 22, 2012 at 7:01 PM, Pete Batard <address@hidden> wrote:

> Alright, now that I have straightened out a working UDF+ISO extraction in
> my application (or at least what looks like one so far), I am in a position
> to try to feed back a few fixes and improvements to libcdio.
>

Great!


>
> Naturally, I don't expect all the modifications I have currently applied
> to the source to work in my app, to be transposable as is, especially as I
> took some liberties here and there. But at the very leats, I should be able
> to provide details on how I addressed the specific issues that I had, and
> we can try to take it from there.
>

If it improves your usage it will probably improve other's usage. From what
I've seen so far from the changes I don't see way things are going to get
worse for people.


>
> Overall, even though I almost gave up on libcdio in despair a few days
> ago, not much seems to be needed to get UDF support going.
>

Good to hear.


>
> For people interested in testing the upcoming changes, the first thing we
> are going to need is a new sample that does image extraction, which is
> provided in the attached patch. It's heavily based on what the existing iso
> and udf samples already do for directory listing and file extraction, the
> main different being that it avoid using ftruncate because ultimately we
> would need an ftruncate64 (we'll be handling some files that are > 4GB in
> size) and this becomes a cross-platform headache, especially if one plans
> to eventually support MSVC compilers.
> The current sample does not do timestamp/premissions preservation and does
> not deal with Unicode (unless the underlying calls support UTF-8, which
> wouldn't be the case on Windows), as this isn't really something we should
> have issues with. One I have sorted these in my app, I'll probably update
> the sample though.
>

Sample programs don't have to be perfect and handle all cases. On the other
hand, if there is something that is going to have to be coded again and
again, then perhaps the code should be put in the library.



>
> Before I go ahead an apply it to mainline, I'd appreciate if you could
> review the sample to confirm that it looks OK. Unless I get a green light,
> I'm not planning to push anything to mainline for another day or two.
>

To make it easier for others to test things out, I've created a pbatard
branch and committed the patch here and the next one to reset
_udf->i_position.

You can commit future changes to the pbatard branch, and then when folks
feel the time is read this can be merged in the master branch.

>
> The second item we will need besides that sample, is an UDF image that can
> exhibit issues that need to be addressed. The one I would encourage
> everyone to use would be the "Windows 8 Developer Preview with developer
> tools English" since:
> 1. It is freely available from Microsoft
> 2. It is more than 4 GB in size (=> will test 32 bit limitations)
> 3. It also contains a file that is more than 4 GB in size (=> more 32 bit
> test, as well as extended attributes use)
> This image can be downloaded at: [1] (direct link [2]).
>

We can't include this in the distribution. In fact anything that is 4GB in
size can't be included in the distribution.

Conceivably one could add some sort of Makefile target to issue a
retrieval. Then there is the thing that is unsettling to me that this link
could be broken at any time.

Independent of that one wants the ability to test code. I don't think it
reasonable to ask people to download 4GB just to test UDF handling,
although it may be fine to suggest that for more complete or realistic
tests.

Specifically, it is not clear that one couldn't come up with a very small
UDF image that exhibits the _udf->i_position problem of your next patch.



> With the sample and the image file above, we should then be able to:
>
> 1. Make UDF extraction work on Linux 32 bit (note that 64 bit Linux may
> not exhibit the 32 bit problems I've seen, so I'd strongly encourage to use
> 32 bit Linux for tests)
>
> 2. Make UDF extraction work on MinGW32 as well as other platforms that
> don't have _FILE_OFFSET_BITS 64 or transparent large file support
>
> 3. Make UDF and ISO9660 image extraction work with MSVC on Windows
>
>
> As you may guess, my objective is to bridge as much as the gap as possible
> between the libcdio source I use in my app (which is compiled with both
> MinGW and MSVC), and official. Eventually, I'd like to be able to use
> mainline files as is, to easily benefit from future libcdio fixes/updates.
>
> Finally, because I am only interested in image extraction at the moment,
> and it should be enough to keep us busy for some time (especially if we add
> MSVC support), I'm going to be very restrictive in the scope of libcdio for
> this whole exercise.


Perfectly reasonable. I and perhaps others will test with the other options.


> In effect, the following options are what I'll use on all platforms (which
> you may want to add to your autogen.sh for testing)
>
> ./configure --enable-maintainer-mode --disable-cddb \
>  --disable-vcd-info --disable-cxx --disable-cpp-progs \
>  --without-cd-drive --without-cd-info --without-cd-paranoia \
>  --without-cdda-player --without-cd-read


> With this sorted out, we are ready to start patching libcdio...
>
> Regards,
>
> /Pete
>
> [1] 
> http://msdn.microsoft.com/en-**us/windows/apps/br229516<http://msdn.microsoft.com/en-us/windows/apps/br229516>
> [2] http://wdp.dlws.microsoft.com/**WDPDL/**9B8DFDFF736C5B1DBF956B89D8A9D4
> **FD925DACD2/**WindowsDeveloperPreview-64bit-**English-Developer.iso<http://wdp.dlws.microsoft.com/WDPDL/9B8DFDFF736C5B1DBF956B89D8A9D4FD925DACD2/WindowsDeveloperPreview-64bit-English-Developer.iso>
>
> From 4616028005dfedc6580baa75d5390a9696921ffc Mon Sep 17 00:00:00 2001
> From: Pete Batard <address@hidden>
> Date: Sun, 22 Jan 2012 02:00:37 +0000
> Subject: [PATCH] Add ISO9660+UDF image extraction example
>
> ---
>  example/Makefile.am |    5 +-
>  example/extract.c   |  270
> +++++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 274 insertions(+), 1 deletions(-)
>  create mode 100644 example/extract.c
>
> diff --git a/example/Makefile.am b/example/Makefile.am
> index de65b5b..995f619 100644
> --- a/example/Makefile.am
> +++ b/example/Makefile.am
> @@ -26,7 +26,7 @@ paranoia_progs = paranoia paranoia2
>  endif
>  if BUILD_EXAMPLES
>  noinst_PROGRAMS = audio cdchange cdtext device discid drives eject \
> -                 isofile isofile2 isofuzzy isolist isolsn \
> +                 extract isofile isofile2 isofuzzy isolist isolsn \
>                  mmc1 mmc2 mmc2a mmc3 $(paranoia_progs) tracks \
>                  sample3 sample4 udf1 udffile cdio-eject
>  endif
> @@ -54,6 +54,9 @@ drives_LDADD          = $(LIBCDIO_LIBS) $(LTLIBICONV)
>  eject_DEPENDENCIES    = $(LIBCDIO_DEPS)
>  eject_LDADD           = $(LIBCDIO_LIBS) $(LTLIBICONV)
>
> +extract_DEPENDENCIES  = $(LIBISO9660_LIBS) $(LIBUDF_LIBS) $(LIBCDIO_DEPS)
> +extract_LDADD         = $(LIBISO9660_LIBS) $(LIBUDF_LIBS) $(LIBCDIO_LIBS)
> $(LTLIBICONV)
> +
>  cdio_eject_DEPENDENCIES = $(LIBCDIO_DEPS)
>  cdio_eject_LDADD        = $(LIBCDIO_LIBS) $(LTLIBICONV)
>
> diff --git a/example/extract.c b/example/extract.c
> new file mode 100644
> index 0000000..30644db
> --- /dev/null
> +++ b/example/extract.c
> @@ -0,0 +1,270 @@
> +/*
> +  Copyright (C) 2012 Pete Batard <address@hidden>
> +  Based on samples copyright (c) 2003-2011 Rocky Bernstein <address@hidden
> >
> +
> +  This program is free software: you can redistribute it and/or modify
> +  it under the terms of the GNU General Public License as published by
> +  the Free Software Foundation, either version 3 of the License, or
> +  (at your option) any later version.
> +
> +  This program is distributed in the hope that it will be useful,
> +  but WITHOUT ANY WARRANTY; without even the implied warranty of
> +  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +  GNU General Public License for more details.
> +
> +  You should have received a copy of the GNU General Public License
> +  along with this program.  If not, see <http://www.gnu.org/licenses/>.
> +*/
> +
> +/* Extract the full content of either an UDF or ISO9660
> +   TODO: timestamp preservation, file permissions, Unicode
> + */
> +
> +#include <stdio.h>
> +#include <string.h>
> +#include <malloc.h>
> +#include <errno.h>
> +
> +#include <cdio/cdio.h>
> +#include <cdio/logging.h>
> +#include <cdio/iso9660.h>
> +#include <cdio/udf.h>
> +
> +#if defined(_WIN32)
> +#include <direct.h>
> +#else
> +#include <sys/stat.h>
> +#include <sys/types.h>
> +#define _mkdir(a) mkdir(a, S_IRWXU)
> +#endif
> +
> +#ifndef MIN
> +#define MIN(a,b) (((a) < (b)) ? (a) : (b))
> +#endif
> +
> +#define print_vd_info(title, fn)     \
> +  if (fn(p_iso, &psz_str)) {         \
> +    printf(title ": %s\n", psz_str); \
> +  }                                  \
> +  free(psz_str);                     \
> +  psz_str = NULL;
> +
> +const char *psz_extract_dir;
> +
> +static int udf_extract_files(udf_t *p_udf, udf_dirent_t *p_udf_dirent,
> const char *psz_path)
> +{
> +  FILE *fd = NULL;
> +  int i_length;
> +  char* psz_fullpath;
> +  const char* psz_basename;
> +  udf_dirent_t *p_udf_dirent2;
> +  uint8_t buf[UDF_BLOCKSIZE];
> +  int64_t i_read, i_file_length;
> +
> +  if ((p_udf_dirent == NULL) || (psz_path == NULL))
> +    return 1;
> +
> +  while (udf_readdir(p_udf_dirent)) {
> +    psz_basename = udf_get_filename(p_udf_dirent);
> +    i_length = 3 + strlen(psz_path) + strlen(psz_basename) +
> strlen(psz_extract_dir);
> +    psz_fullpath = (char*)calloc(sizeof(char), i_length);
> +    if (psz_fullpath == NULL) {
> +      fprintf(stderr, "Error allocating file name\n");
> +      goto out;
> +    }
> +    i_length = snprintf(psz_fullpath, i_length, "%s%s/%s",
> psz_extract_dir, psz_path, psz_basename);
> +    if (i_length < 0) {
> +      goto out;
> +    }
> +    printf("Extracting: %s\n", psz_fullpath);
> +    if (udf_is_dir(p_udf_dirent)) {
> +      _mkdir(psz_fullpath);
> +      p_udf_dirent2 = udf_opendir(p_udf_dirent);
> +      if (p_udf_dirent2 != NULL) {
> +        if (udf_extract_files(p_udf, p_udf_dirent2,
> &psz_fullpath[strlen(psz_extract_dir)]))
> +          goto out;
> +      }
> +    } else {
> +      fd = fopen(psz_fullpath, "wb");
> +      if (fd == NULL) {
> +        fprintf(stderr, "  Unable to create file\n");
> +        goto out;
> +      }
> +      i_file_length = udf_get_file_length(p_udf_dirent);
> +      while (i_file_length > 0) {
> +        memset(buf, 0, UDF_BLOCKSIZE);
> +        i_read = udf_read_block(p_udf_dirent, buf, 1);
> +        if (i_read < 0) {
> +          fprintf(stderr, "  Error reading UDF file %s\n",
> &psz_fullpath[strlen(psz_extract_dir)]);
> +          goto out;
> +        }
> +        fwrite(buf, (size_t)MIN(i_file_length, i_read), 1, fd);
> +        if (ferror(fd)) {
> +          fprintf(stderr, "  Error writing file\n");
> +          goto out;
> +        }
> +        i_file_length -= i_read;
> +      }
> +      fclose(fd);
> +      fd = NULL;
> +    }
> +    free(psz_fullpath);
> +  }
> +  return 0;
> +
> +out:
> +  if (fd != NULL)
> +    fclose(fd);
> +  free(psz_fullpath);
> +  return 1;
> +}
> +
> +static int iso_extract_files(iso9660_t* p_iso, const char *psz_path)
> +{
> +  FILE *fd = NULL;
> +  int i_length, r = 1;
> +  char psz_fullpath[4096], *psz_basename;
> +  const char *psz_iso_name = &psz_fullpath[strlen(psz_extract_dir)];
> +  unsigned char buf[ISO_BLOCKSIZE];
> +  CdioListNode_t* p_entnode;
> +  iso9660_stat_t *p_statbuf;
> +  CdioList_t* p_entlist;
> +  size_t i;
> +  lsn_t lsn;
> +  int64_t i_file_length;
> +
> +  if ((p_iso == NULL) || (psz_path == NULL))
> +    return 1;
> +
> +  i_length = snprintf(psz_fullpath, sizeof(psz_fullpath), "%s%s/",
> psz_extract_dir, psz_path);
> +  if (i_length < 0)
> +    return 1;
> +  psz_basename = &psz_fullpath[i_length];
> +
> +  p_entlist = iso9660_ifs_readdir(p_iso, psz_path);
> +  if (!p_entlist)
> +    return 1;
> +
> +  _CDIO_LIST_FOREACH (p_entnode, p_entlist) {
> +    p_statbuf = (iso9660_stat_t*) _cdio_list_node_data(p_entnode);
> +    /* Eliminate . and .. entries */
> +    if ( (strcmp(p_statbuf->filename, ".") == 0)
> +      || (strcmp(p_statbuf->filename, "..") == 0) )
> +      continue;
> +    iso9660_name_translate(p_statbuf->filename, psz_basename);
> +    if (p_statbuf->type == _STAT_DIR) {
> +      _mkdir(psz_fullpath);
> +      if (iso_extract_files(p_iso, psz_iso_name))
> +        goto out;
> +    } else {
> +      printf("Extracting: %s\n", psz_fullpath);
> +      fd = fopen(psz_fullpath, "wb");
> +      if (fd == NULL) {
> +        fprintf(stderr, "  Unable to create file\n");
> +        goto out;
> +      }
> +      i_file_length = p_statbuf->size;
> +      for (i = 0; i_file_length > 0; i++) {
> +        memset(buf, 0, ISO_BLOCKSIZE);
> +        lsn = p_statbuf->lsn + i;
> +        if (iso9660_iso_seek_read(p_iso, buf, lsn, 1) != ISO_BLOCKSIZE) {
> +          fprintf(stderr, "  Error reading ISO9660 file %s at LSN %lu\n",
> +            psz_iso_name, (long unsigned int)lsn);
> +          goto out;
> +        }
> +        fwrite(buf, (size_t)MIN(i_file_length, ISO_BLOCKSIZE), 1, fd);
> +        if (ferror(fd)) {
> +          fprintf(stderr, "  Error writing file\n");
> +          goto out;
> +        }
> +        i_file_length -= ISO_BLOCKSIZE;
> +      }
> +      fclose(fd);
> +      fd = NULL;
> +    }
> +  }
> +  r = 0;
> +
> +out:
> +  if (fd != NULL)
> +    fclose(fd);
> +  _cdio_list_free(p_entlist, true);
> +  return r;
> +}
> +
> +int main(int argc, char** argv)
> +{
> +  iso9660_t* p_iso = NULL;
> +  udf_t* p_udf = NULL;
> +  udf_dirent_t* p_udf_root;
> +  char *psz_str = NULL;
> +  char vol_id[UDF_VOLID_SIZE] = "";
> +  char volset_id[UDF_VOLSET_ID_SIZE+1] = "";
> +  int r = 0;
> +
> +  cdio_loglevel_default = CDIO_LOG_DEBUG;
> +
> +  if (argc < 3) {
> +    fprintf(stderr, "Usage: extract <iso_image> <extraction_dir>\n");
> +    return 1;
> +  }
> +
> +  psz_extract_dir = argv[2];
> +  if (_mkdir(psz_extract_dir) == 0) {
> +    printf("Creating directory: %s\n", psz_extract_dir);
> +  } else if (errno != EEXIST) {
> +    fprintf(stderr, "Unable to create extraction directory %s\n",
> psz_extract_dir);
> +    return 1;
> +  }
> +
> +  /* First try to open as UDF - fallback to ISO if it failed */
> +  p_udf = udf_open(argv[1]);
> +  if (p_udf == NULL)
> +    goto try_iso;
> +
> +  p_udf_root = udf_get_root(p_udf, true, 0);
> +  if (p_udf_root == NULL) {
> +    fprintf(stderr, "Couldn't locate UDF root directory\n");
> +    goto out;
> +  }
> +  vol_id[0] = 0; volset_id[0] = 0;
> +
> +  /* Show basic UDF Volume info */
> +  if (udf_get_volume_id(p_udf, vol_id, sizeof(vol_id)) > 0)
> +    fprintf(stderr, "Volume id: %s\n", vol_id);
> +  if (udf_get_volume_id(p_udf, volset_id, sizeof(volset_id)) >0 ) {
> +    volset_id[UDF_VOLSET_ID_SIZE]='\0';
> +    fprintf(stderr, "Volume set id: %s\n", volset_id);
> +  }
> +  fprintf(stderr, "Partition number: %d\n", udf_get_part_number(p_udf));
> +
> +  /* Recursively extract files */
> +  r = udf_extract_files(p_udf, p_udf_root, "");
> +
> +  goto out;
> +
> +try_iso:
> +  p_iso = iso9660_open(argv[1]);
> +  if (p_iso == NULL) {
> +    fprintf(stderr, "Unable to open image '%s'.\n", argv[1]);
> +    goto out;
> +  }
> +
> +  /* Show basic ISO9660 info from the Primary Volume Descriptor. */
> +  print_vd_info("Application", iso9660_ifs_get_application_id);
> +  print_vd_info("Preparer   ", iso9660_ifs_get_preparer_id);
> +  print_vd_info("Publisher  ", iso9660_ifs_get_publisher_id);
> +  print_vd_info("System     ", iso9660_ifs_get_system_id);
> +  print_vd_info("Volume     ", iso9660_ifs_get_volume_id);
> +  print_vd_info("Volume Set ", iso9660_ifs_get_volumeset_id);
> +
> +  r = iso_extract_files(p_iso, "");
> +
> +out:
> +  if (p_iso != NULL)
> +    iso9660_close(p_iso);
> +  if (p_udf != NULL)
> +    udf_close(p_udf);
> +
> +  return r;
> +}
> --
> 1.7.8.msysgit.0
>
>
>