Re: Malformed sav file - need help for investigation

[Ben Pfaff]

I have now made this change in PSPP, so that it should successfully read
these files, issuing only a warning. That means that you can now use
"pspp-convert" to fix them, although it will lose the weighting variable
(if that's the problem) or the value label (if that's the problem). So, it
would still be better to fix the library that produces the incorrect files.

On Wed, Jun 10, 2020 at 11:27 AM Ben Pfaff <blp@cs.stanford.edu> wrote:

> Ah. I can easily explain what the problem is.
>
> SPSS data files have a variable record for each variable. For string
> variables with a width greater than 8, they actually have a variable record
> for every 8 columns of width (rounding up). Lots of places in data files
> refer to variables by the index that includes such extra records. So, if
> you have a string variable with width 24, and it's the third variable in
> the file following two numeric variables, then it will have indexes 2, 3,
> and 4. When such a variable is referenced by index, only index 2 is
> supposed to be used. PSPP complains if index 3 or 4 is used instead, and
> SPSS (apparently) crashes instead.
>
> Maybe I should make PSPP more tolerant, just giving a warning instead of
> an error. If so, then it would be easy to fix up a file with this error by
> using pspp-convert.
>
> On Wed, Jun 10, 2020 at 10:24 AM Ignácz, István <istvan.ignacz@neticle.com>
> wrote:
>
>> Thanks, Ben for the quick response!
>>
>> Yes, the ultimate goal is to add a patch to the package which fixes the
>> problem or throws an error if the user of the package (in this case me)
>> provides invalid input. I think I'm facing the later.
>>
>> One significant difference between the test file and my case is that in my
>> case no variables and no cases are recognized so the reading process
>> basically crashes.
>> I think I've found the part where the PSPP ends up in a 'goto error'
>> (sys-file-reader.813) but I lack the context of what's going on here. I
>> guess the value labels could be misconfigured? Maybe there's a value in a
>> record that doesn't have a label in the variable settings?
>>
>> If I would know a little bit more about what happens in that part and why
>> that causes an error, maybe I could force my (and tiamo's) code to
>> reliably
>> produce the error.
>>
>> Thanks,
>> István
>>
>> On Wed, 10 Jun 2020 at 18:51, Ben Pfaff <blp@cs.stanford.edu> wrote:
>>
>> > I'm not sure exactly what you're looking for, but PSPP has a test case
>> > that intentionally provokes this error message. I'm attaching the file
>> it
>> > produces.
>> >
>> > I guess the ultimate goal here should be to fix tiamo's package?
>> >
>> > On Wed, Jun 10, 2020 at 9:42 AM Ignácz, István <
>> istvan.ignacz@neticle.com>
>> > wrote:
>> >
>> >> Greetings,
>> >>
>> >> I think this is not the usual topic on this list, excuse me if you
>> find it
>> >> off-topic.
>> >>
>> >> I'm facing a tough bug in my software and I'm looking for some help to
>> be
>> >> able to take the next steps.
>> >>
>> >> Our software generates sav files via tiamo's package:
>> >> https://packagist.org/packages/tiamo/spss
>> >> Usually, everything works fine, but in some cases, we generate
>> malformed
>> >> files. In this case, SPSS crashes without any notice, however, PSPP at
>> >> least gives a useful error message:
>> >> Variable index n refers to long string continuation.
>> >>
>> >> I cannot reproduce this issue on my side, so I tried to reverse
>> engineer
>> >> the PSPP codebase to find any cases when it could occur. So far not
>> with
>> >> much luck.
>> >>
>> >> Could you help me with some example cases?
>> >>
>> >> Thanks for your help,
>> >> István
>> >>
>> >
>>
>> --
>>
>> *István Ignácz*
>> Software Engineer @ Neticle
>>
>> +36 30 460 5356
>> istvan.ignacz@neticle.com
>> neticle.com
>>
>> <https://neticle.com/mediaintelligence/en/>  <https://zurvey.io/>
>> <https://api.neticle.com/>  <https://data.neticle.com/documentation>
>>
>> Neticle Labs Ltd.
>> Address: H-1082 Budapest, Leonardo da Vinci  41. (2nd floor/12)
>> Billing address: H-1213 Budapest, Veto street 10.
>>
>