[Bug-wget] Segfault with WARC + CDX

[Gijs van Tulder]

Hi,

There's a bug in the warc_find_duplicate_cdx_record function. If you 
provide a file with CDX records, Wget can segfault if a record is not 
found in the CDX file. In fact, the deduplication now only works if 
*every* new record can be found in the CDX index.

The segmentation fault is generated on these lines in src/warc.c:

  hash_table_get_pair (warc_cdx_dedup_table, sha1_digest_payload, &key,
                       &rec_existing);
  if (rec_existing != NULL && strcmp (rec_existing->url, url) == 0)

Other than the code expects hash_table_get_pair does not set 
rec_existing to NULL if no record is found. So instead of checking for 
NULL, the function should check if the return value of 
hash_table_get_pair is non-zero:

  int found = hash_table_get_pair (warc_cdx_dedup_table, 
sha1_digest_payload,
                                   &key, &rec_existing);
  if (found && strcmp (rec_existing->url, url) == 0)

The attached patch makes this change. The deduplication works better.

Regards,

Gijs

0001-warc-Fix-segfault-if-CDX-record-is-not-found.patch
Description: Text Data