Re: [lmi] rate_table_tool: merge a whole directory

[Greg Chicares]

On 2016-12-01 02:03, Greg Chicares wrote:
[...]
> New hypothesis: The original binary tables may be slightly inaccurate.
> Certainly some were added by means of the old gcc-compiled program
> that I built from hacked SOA sources. Very likely others were added
> by means of a 1990s binary distributed by the SOA, which would at best
> have been compiled with some old version of "turbo C++" (which supplied
> its own C RTL), or at worst with "turbo pascal".

Proven.

Here's a narrative explanation of the session pasted below (redacted:
proprietary table name replaced with "PROPRIETARY", and some data
with "REDACTED"). Using the binary "built from hacked SOA sources"
(above), in an msw-xp VM, I extracted every table from the latest
proprietary database. Then I merged them all into a new rate database
(changing a rejected comment in one table). Finally, I copied that
new database to GNU/Linux, verified its md5sums, and used it to rerun
the system test. Results of that system test matched one I had run
yesterday using a database regenerated using Vadim's tool instead of
the SOA-based one: every single file was identical. This proves that
all anomalies observed are explained by the hypothesis above, and
strongly confirms that Vadim's new rate-table tool behaves the same
way as the ancient one. Honi soit qui mal y pense--I shouldn't have
assumed that that tool was the cause of the regressions.

(BTW, I obtained permission from the author to use the code that
SOA had distributed; the reason why I never published it in our
nongnu.org repository is that it copied some CRC32 code whose
provenance was non-free.)

/cygdrive/c/soaqx[0]$ls -l PROPRIETARY.*
-rwxr-xr-x 1 earl None 6151032 Jul 22  2015 PROPRIETARY.dat
-rwxr-xr-x 1 earl None   34452 Jul 22  2015 PROPRIETARY.ndx
/cygdrive/c/soaqx[0]$md5sum PROPRIETARY.*
74c18f30345311052ce23c6202dd26f2 *PROPRIETARY.dat
041a54dd70bf5406396079c04ff5adb4 *PROPRIETARY.ndx
/cygdrive/c/soaqx[0]$mkdir rate_tables
/cygdrive/c/soaqx[0]$cd rate_tables
# Highest table number is 999.
/cygdrive/c/soaqx/rate_tables[0]$for z in `seq 1 999`; do ../table_utilities -a 
-f ../PROPRIETARY -e $z; done
# That creates an empty file for each nonexistent table; delete them:
/cygdrive/c/soaqx/rate_tables[0]$for z in *.txt; do [ -s $z ] || rm $z; done
/cygdrive/c/soaqx/rate_tables[0]$for z in *.txt; do ../table_utilities -a -f 
regenerated -m $z; done

27.txt
Error translating ASCII file; misplaced colon before
 Order is as in REDACTED
[file utils.cpp, line 41]

Error: Line of text ignored in ASCII file:
sorted alphabetically.
[file utils.cpp, line 41]

# Fix that table by combining all "Comments" in a single line.
# The table is irrelevant anyway--it's just ancient premtax rates.
# Now regenerate again:

/cygdrive/c/soaqx/rate_tables[0]$rm regenerated.dat regenerated.ndx
/cygdrive/c/soaqx/rate_tables[0]$for z in *.txt; do ../table_utilities -a -f 
regenerated -m $z; done
/cygdrive/c/soaqx/rate_tables[0]$ls -l regenerated.*
-rwxr-xr-x 1 earl None 6149018 Dec  1 03:02 regenerated.dat
-rwxr-xr-x 1 earl None   34452 Dec  1 03:02 regenerated.ndx
/cygdrive/c/soaqx/rate_tables[0]$md5sum regenerated.*
93fb4b97c8be47abfe560b3609529072 *regenerated.dat
07a335b0e5d15ab79f2b736d2cfa50ac *regenerated.ndx