status of utf-8.el, etc [Re: Several serious problems]

[Kenichi Handa]

In article <address@hidden>, Dave Love <address@hidden> writes:
> Richard Stallman <address@hidden> writes:
>>      For instance, the RC version of mule-utf-8 doesn't translate
>>      cyrillic-iso8859-5, and the Cyrillic coding systems don't translate
>>      mule-unicode-0100-24ff.
>>  
>>  We could consider adding that support in RC.  Is it a safe change?

> It won't break anything if done correctly, but I don't remember how
> much of a change it is relative to the 21.2 code and I don't know who
> might have been testing it, if anyone.

I noticed some combinations of unify-8859-on-encoding-mode,
utf-8-fragment-on-decoding, and utf-8-translate-cjk doesn't
work in HEAD.  So, I made a fairly comprehensive testsuite
for testing them (attached at the tail).

As the testsuite revealed several bugs, before working on
RC, I decided to fix them in HEAD at first.  I've finished
these:

(1) Fixing the following bugs.

(1-1) unify-8859-on-encoding-mode can't be turned off
safely.  For instance, then, iso-latin-1 can't encode
Latin-1 chars.

(1-2) utf-8-translate-cjk can never be turned off once
turned on.

(1-3) When utf-8-fragment-on-decoding is non-nil, utf-16-*
doesn't encode CJK chars correctly even if
utf-8-translate-cjk is non-nil.

(1-4) encode-char/decode-char don't reflect utf-8-translate-cjk.


(2) Renaming tables/variables.   We should have cleaner
    names before people starting to use it.

(2-1) As utf-8-fragment-on-decoding and utf-8-translate-cjk are
also applicable to utf-16, I cut off "-8" from them.

(2-2) Make translation table names and their body
char-tables different to avoid confusion.

The result is as follows:

(2-2-1) Translation-tables and translation-hash-tables (not variable)

old                                     new
---                                     ---
ucs-mule-to-mule-unicode                utf-translation-table-for-encode
  (mule-utf-8/16 use it for encoding)

utf-translation-table-for-decode        utf-translation-table-for-decode
  (mule-utf-8/16 use it for decoding)

utf-8-subst-rev-table                   utf-subst-table-for-encode
  (mule-utf-8/16 use it for encoding)

utf-8-subst-table                       utf-subst-table-for-decode
  (mule-utf-8/16 use it for decoding)

(2-2-2) Mapping tables (variables) populating above.

old                                     new
---                                     ---
ucs-mule-to-mule-unicode                ucs-mule-to-mule-unicode
  (this populates utf-translation-table-for-encode
   when unify-8859-on-encoding-mode is non-nil)

utf-8-subst-table                       ucs-unicode-to-mule-cjk
  (this populates utf-subst-table-for-decode
   when utf-translate-cjk is non-nil)

utf-8-subst-rev-table                   ucs-mule-cjk-to-unicode
  (this populates utf-subst-table-for-encode
   when utf-translate-cjk is non-nil)

utf-8-fragmentation-table               utf-fragmentation-table
  (this populates utf-translation-table-for-decode
   when utf-fragment-on-decoding is non-nil)

--not_exist--                           utf-defragmentation-table
  (this populates utf-translation-table-for-encode
   when unify-8859-on-encoding-mode is nil
   and utf-fragment-on-decoding is non-nil)

utf-8-translation-table-for-decode      --deleted--


Don't you have better ideas for these names?   If not, I'll
install the changes soon.

---
Ken'ichi HANDA
address@hidden

Two files: utf-test.el and result.txt.

result.txt is the result of loading utf-text.el, running M-x
utf-testsuite RET, and viewing the variable
utf-testsuite-result in the current Emacs.  After my
modification, all elements are `t'.

begin 664 temp.tar.gz
M'XL(`'9=D3T``^U:6W/;-A9V9O=%>address@hidden>LIU-3K<DEJ:OMZ>ZJJ:?U)K%G
M7'?[X&0F%`7)M"G2(address@hidden>Q(LOL;M*!\<C&I=SP\&'`Q!2PA8ZHS$S
MJ+_U7&1:ICD:#;9,TS:MTG\DVQH/H0R/@3W$#N"W+7.P1<QG\ZA`2<R<B)"M
M"R>8.RU\=_5_I71P0/3O=.*&<R]8[A,O#G7;M&U]//,8[^IVM3E=W$",?(=Y
address@hidden@54M)5S145Q1W2`Z&0RW-,M\J]W?=/L]=8$[8<++B-*KZIBHCD5&U?$
M+KWX,T!L4I6\=*Z=@,943UG(8$+Z?9#EPFX8Q(PD<DG$B<=`D$=&CV]C1E=Q
MEY!SC)&,!51)address@hidden>?[GEMJ9!>.IS,O'MDF5C,_BSRK
MQ*<Z.C<IU:R1/J/address@hidden&`0]<%945.-KGVN&HAQ/)P@"address@hidden,!`S^
M,T88U!G4&=09*S';'<,address@hidden(9,1JN&3,T9P.&,DU<_?):E--!RB(?H:C^
address@hidden&C_YC3K`P,address@hidden/G!J6`M"K0/K\R>[)S&^&SW>DPG(>F
M3\H<>'ZG`P_2\EE7O,ZR_7)W]N^7VH_;[:7'F-P$7AQV_YV-0[]72(%_/(+/
M>$O$ZUY"UIY]UXS!DBA,OJV?ZHZ;L'I(V&5(/&:6GQP^7QLO!F+XSK8'VS6(
M_V;7(DSB^\YR!1DOSIY,address@hidden(A^E:A];T3FA9Q2A9]4EA.DT30'-I7<U]
M5UB3S5FFO[FT22![3"*RZL#ST&CG*>K+U5E[_<<#H(R]/?TPB<+Z;6Z(R)L.
MGC_,]F#P9TV`,/,#&address@hidden@B3RIVDE_-&A(;@]0E@),'.2F
M_O6%(address@hidden,+.27UDT_)MQ_SZ9'E^K/37[Y`R9Y5
M2$?WDY2B`BC9Z?[H1&HLGO<Y5D0JVBBL/&R:[,'_'5<PU[99WLY*DZ7]/9\L
M6:['RE]?/%*#9=K#!J0TBLIL<O!R]PR;";ZAAH$#[Y3;;P[/address@hidden/R>L3\GIZ
M)address@hidden&&QF;3L0BF[T<GG\KV<@59?WOQ9#K[
address@hidden)T+\NNA%[]Y[4^E2::;H,V"`B;P`M@')D"D/=]VQ@,`&-P9.K+
MLPWVU_ZO(.CX`4(`85N>=YIY>^GE'MX,EB_&(AHG/D,address@hidden"FEJ
MM7*">5G"address@hidden(:Z+(S(.7GUR_14-_73PU]_>W-&#,,0+6_2EO=&M]S@
MQ5Q]E!L#Y5ZP)`O0)UAKK)S\='3\<]%.VO2V:&BMJ=&4,,,MBJ'B#2<1-YR9
M%J*]91>$^G1%`R[]H>U>=*=7]?KMR4^')9]YP[C@,6\X?B)_O8#WKL(Y)<<&
M(<>HT#2,L=$]PW;J!"@&>GF/$Y,9]<-/^S#W1PMX=PJ7.-_'address@hidden";S%
MK<Z/+V&@\UM''#$W`!P`&[!R`C:C3UY,=SDH/8Q!`.S8NZ[9SC1#)!>1L\30
MHFY^BXG./4JM553+(B>(X3T0)NCRZI[ZUL)address@hidden).#;EVXIC.29RX
M+HWC1>+[M^B<N0]3%$4``_^6S"F#`G"!$AY"address@hidden/.)"=RY6]2QAY%,4
M!DLNS#O(S'&O4+-UM^:BN-2+DC#.?4#(7=*`L$1.-=8!TPL2ARLJL\**!S1D
MA.U""'G,$+,.P\Z(<OC/DF4*13<!:P'K'JX<-S:V,0^!`:VT<O1KLI/=<8O$
MID7T8^)%-.U(address@hidden,1T9!?\P)&(address@hidden;/P,)RL(157/T^`'1V
ML-=NX"FO[IX\[VLQ91_KDZCXED6N?6X<address@hidden(Q^,J[9/Y.706ALKAF
M?8]\L%RX#G35`<OUACU,.,B2&;G,%4`[F.'":.6R;&<B]6>GA$P%7^.3S$\/
MEAM"5IN%23"7,S:I7<+<address@hidden:3`^4.VFKEAR$"F</W@@/P>1E?$B=`P
M!Q7B"]"8N'%N'_7#(O\=EG$"ZY(E42"address@hidden(I[T;XAHN%P>/3
MENBL7A8%%W2'*QV"J<->ZN$"B<F'-NENIR/'N,MC^,^:<<(6C3NZY/R@"<YB
MH!O"+"0:.01#^TRU>F??S[M2PBV[5>HJ^5,5:G'$RAW)YB)#*JX'address@hidden
MKVKV':PJBHLR@(6=<address@hidden:BX^4($%IP`PG5:*;7Q0O,$NM1<5QRSB
M#]Q=4'/):*S['B!6<R*Z$%]/EK,CXVE9B(G%7,XEME26C5JXH7T/>0$F,'52
MV))FVC(=\3+^?%<0*B&G\&5+:)D%]A`^:address@hidden>E7C%P#(++\ZB03'<V
ME,RTI%]FEB33WJ)TFG-38?0R!Y%P9T57'T&(R!!GR.2=L,$X?N:R-)&QD#*;
M])7W8C_O$MY=Y_9[63]2R2W9"B(2,address@hidden)ZH.Y4]Q+.?R\..^OG!1PE\^[
MZT\+HMLL#`HM68V64I;F8\4:(Z;WBF=&SF))T_Q-*FULF1I\#<GC7UI$8@<3
MZ*W#L,1#NM+A/"B1(A*(Z$X74&6E\&-`<;'<"?T<^`)!HK$5E0)/^3?W(O8[
ML(W)X7.&'YIX:C`J$9HOFY2YS%M@;L:L"&H^;+,H2[*TQ9M8<9GD^+(*:;PX
MYW<,/!?(:H6I?_#$RY3869]YKK+N]S^"PV"?V?/]QJC]]U_F<#@:I[__&@T'
MIOC]EZE^__5'4!V>X#WFQO$3_I8;7X2?`O'Z9W2[ZHY$W9&H.Q)U1_)<=R3=
M[G\Q\>QWS\_/UV^DF?A[#YZ>,address@hidden:#S?&&>PK2VXC^E3`U0#?*8!?DF[
M&M0?.:@_+12Q)!PL/&5/I1V?K7TPOMI7,46*%"E2I$B1(D6*%"E2I$B1(D6*
7%"E2I$B1(D6*%"GZ(OH?U(],address@hidden
`
end