Re: extended format is wrong?

[Anmol Sethi]

I found https://github.com/mauke/unibilium

It supports extended capabilities but its code is much easier for me to 
understand. I’m gonna try and implement it again tomorrow.
> On Apr 26, 2016, at 10:03 PM, Anmol Sethi <address@hidden> wrote:
> 
> I’d test to see how ncurses handles it, and fix my problems from there, but I 
> cannot enable tracing for some reason. I have 0 experience with c, sorry! I 
> set the environment variable, I see the tracing set, but it outputs nothing. 
> And I do have the right flags set.
> 
>> On Apr 26, 2016, at 10:02 PM, Anmol Sethi <address@hidden> wrote:
>> 
>> Hello!
>> 
>> I’m writing a terminfo library for go. Someone here suggested I add support 
>> for the extended storage format. I’ve tried and I’m running into quite a few 
>> problems. Here is an example xterm terminfo file from my system in hex. I’ve 
>> only included the extended format divided up.
>> 
>> 
>> EXTENDED:                             
>> Header:
>> 01 00 00 00 39 00 73 00 a2 02 
>> Values are 1, 0, 57, 115, and 674 in decimal.
>> 
>> According to term(5), that means 1 bool cap, 0 numeric caps, 57 string caps, 
>> 115 bytes for the extended string table and 674 bytes for the last offset of 
>> the extended string table. Ok so I attempted at writing a reader with this 
>> information and turns out those values are wrong or I am not understanding 
>> any of this correctly. Before I begin, what exactly is the difference 
>> between the 4th and 5th short integers of the header? One is the byte length 
>> of the string table and the other is the last offset of the string table. 
>> I’m not 100% what exactly that means. I’m gonna assume that the first one is 
>> the size in which the string cap values are contained and the second is the 
>> entire size of the table. The ncurses source defines the 4th short as 
>> ext_str_size and the 5th as ext_str_limit. Funny thing is, the source 
>> doesn’t even use the 4th short.
>> 
>> Bool:
>> 01 00 
>> 
>> Anyways, after the header, there is the boolean section. Since there is only 
>> one boolean value, I read the byte and then I noticed the old quirk, I need 
>> to skip the extra null byte inserted to keep everything on word boundaries. 
>> Ok so done and done.
>> 
>> String Section:
>> ff ff 00 00 07 00 0e 00 15 00 1c 00 23 00 2a 00 31 00 38 00 3f 00 46 00 4d 
>> 00 54 00 5b 00 62 00 69 00
>> 70 00 77 00 7e 00 85 00 8c 00 93 00 9a 00 a1 00 a8 00 af 00 b6 00 bd 00 c4 
>> 00 cb 00 d2 00 d9 00 e0 00
>> e7 00 ee 00 f5 00 fc 00 03 01 0a 01 11 01 18 01 1f 01 26 01 2d 01 34 01 3b 
>> 01 42 01 49 01 50 01 57 01
>> 5e 01 65 01 ff ff ff ff ff ff ff ff 
>> 
>> Next the numeric capabilities were 0 so I went straight to the string caps. 
>> Now here I read in these 114 bytes, or 57 shorts.
>> 
>> Rest:
>> 00 00 03 00 06 00 0b 00 10 00
>> 15 00 1a 00 1f 00 23 00 28 00 2d 00 32 00 37 00 3c 00 42 00 48 00 4e 00 54 
>> 00 5a 00 60 00 66 00 6c 00
>> 72 00 78 00 7d 00 82 00 87 00 8c 00 91 00 97 00 9d 00 a3 00 a9 00 af 00 b5 
>> 00 bb 00 c1 00 c7 00 cd 00
>> d3 00 d9 00 df 00 e5 00 eb 00 f1 00 f7 00 fd 00 03 01 09 01 0d 01 12 01 17 
>> 01 1c 01 21 01 26 01 2a 01
>> 2e 01 32 01 1b 5b 33 3b 33 7e 00 1b 5b 33 3b 34 7e 00 1b 5b 33 3b 35 7e 00 
>> 1b 5b 33 3b 36 7e 00 1b 5b
>> 33 3b 37 7e 00 1b 5b 31 3b 32 42 00 1b 5b 31 3b 33 42 00 1b 5b 31 3b 34 42 
>> 00 1b 5b 31 3b 35 42 00 1b
>> 5b 31 3b 36 42 00 1b 5b 31 3b 37 42 00 1b 5b 31 3b 33 46 00 1b 5b 31 3b 34 
>> 46 00 1b 5b 31 3b 35 46 00
>> 1b 5b 31 3b 36 46 00 1b 5b 31 3b 37 46 00 1b 5b 31 3b 33 48 00 1b 5b 31 3b 
>> 34 48 00 1b 5b 31 3b 35 48
>> 00 1b 5b 31 3b 36 48 00 1b 5b 31 3b 37 48 00 1b 5b 32 3b 33 7e 00 1b 5b 32 
>> 3b 34 7e 00 1b 5b 32 3b 35
>> 7e 00 1b 5b 32 3b 36 7e 00 1b 5b 32 3b 37 7e 00 1b 5b 31 3b 33 44 00 1b 5b 
>> 31 3b 34 44 00 1b 5b 31 3b
>> 35 44 00 1b 5b 31 3b 36 44 00 1b 5b 31 3b 37 44 00 1b 5b 36 3b 33 7e 00 1b 
>> 5b 36 3b 34 7e 00 1b 5b 36
>> 3b 35 7e 00 1b 5b 36 3b 36 7e 00 1b 5b 36 3b 37 7e 00 1b 5b 35 3b 33 7e 00 
>> 1b 5b 35 3b 34 7e 00 1b 5b
>> 35 3b 35 7e 00 1b 5b 35 3b 36 7e 00 1b 5b 35 3b 37 7e 00 1b 5b 31 3b 33 43 
>> 00 1b 5b 31 3b 34 43 00 1b
>> 5b 31 3b 35 43 00 1b 5b 31 3b 36 43 00 1b 5b 31 3b 37 43 00 1b 5b 31 3b 32 
>> 41 00 1b 5b 31 3b 33 41 00
>> 1b 5b 31 3b 34 41 00 1b 5b 31 3b 35 41 00 1b 5b 31 3b 36 41 00 1b 5b 31 3b 
>> 37 41 00 41 58 00 58 4d 00
>> 6b 44 43 33 00 6b 44 43 34 00 6b 44 43 35 00 6b 44 43 36 00 6b 44 43 37 00 
>> 6b 44 4e 00 6b 44 4e 33 00
>> 6b 44 4e 34 00 6b 44 4e 35 00 6b 44 4e 36 00 6b 44 4e 37 00 6b 45 4e 44 33 
>> 00 6b 45 4e 44 34 00 6b 45
>> 4e 44 35 00 6b 45 4e 44 36 00 6b 45 4e 44 37 00 6b 48 4f 4d 33 00 6b 48 4f 
>> 4d 34 00 6b 48 4f 4d 35 00
>> 6b 48 4f 4d 36 00 6b 48 4f 4d 37 00 6b 49 43 33 00 6b 49 43 34 00 6b 49 43 
>> 35 00 6b 49 43 36 00 6b 49
>> 43 37 00 6b 4c 46 54 33 00 6b 4c 46 54 34 00 6b 4c 46 54 35 00 6b 4c 46 54 
>> 36 00 6b 4c 46 54 37 00 6b
>> 4e 58 54 33 00 6b 4e 58 54 34 00 6b 4e 58 54 35 00 6b 4e 58 54 36 00 6b 4e 
>> 58 54 37 00 6b 50 52 56 33
>> 00 6b 50 52 56 34 00 6b 50 52 56 35 00 6b 50 52 56 36 00 6b 50 52 56 37 00 
>> 6b 52 49 54 33 00 6b 52 49
>> 54 34 00 6b 52 49 54 35 00 6b 52 49 54 36 00 6b 52 49 54 37 00 6b 55 50 00 
>> 6b 55 50 33 00 6b 55 50 34
>> 00 6b 55 50 35 00 6b 55 50 36 00 6b 55 50 37 00 6b 61 32 00 6b 62 31 00 6b 
>> 62 33 00 6b 63 32 00
>> 
>> Alright, now the rest of it should be the string table right? Well it turns 
>> out that it isn’t. Here look, this is the rest of the extended format (it is 
>> in a different format but it should be clear whats going on):
>> 
>> First 116 bytes:
>> "\x00\x00\x03\x00\x06\x00\v\x00\x10\x00\x15\x00\x1a\x00\x1f\x00#\x00(\x00-\x002\x007\x00<\x00B\x00H\x0
>> 0N\x00T\x00Z\x00`\x00f\x00l\x00r\x00x\x00}\x00\x82\x00\x87\x00\x8c\x00\x91\x00\x97\x00\x9d\x00\xa3\x00
>> \xa9\x00\xaf\x00\xb5\x00\xbb\x00\xc1\x00\xc7\x00\xcd\x00\xd3\x00\xd9\x00\xdf\x00\xe5\x00\xeb\x00\xf1\x
>> 00\xf7\x00\xfd\x00\x03\x01\t\x01\r\x01\x12\x01\x17\x01\x1c\x01!\x01&\x01*\x01.\x012\x01"
>> 
>> String Table:
>> "\x1b[3;3~\x00\x
>> 1b[3;4~\x00\x1b[3;5~\x00\x1b[3;6~\x00\x1b[3;7~\x00\x1b[1;2B\x00\x1b[1;3B\x00\x1b[1;4B\x00\x1b[1;5B\x00
>> \x1b[1;6B\x00\x1b[1;7B\x00\x1b[1;3F\x00\x1b[1;4F\x00\x1b[1;5F\x00\x1b[1;6F\x00\x1b[1;7F\x00\x1b[1;3H\x
>> 00\x1b[1;4H\x00\x1b[1;5H\x00\x1b[1;6H\x00\x1b[1;7H\x00\x1b[2;3~\x00\x1b[2;4~\x00\x1b[2;5~\x00\x1b[2;6~
>> \x00\x1b[2;7~\x00\x1b[1;3D\x00\x1b[1;4D\x00\x1b[1;5D\x00\x1b[1;6D\x00\x1b[1;7D\x00\x1b[6;3~\x00\x1b[6;
>> 4~\x00\x1b[6;5~\x00\x1b[6;6~\x00\x1b[6;7~\x00\x1b[5;3~\x00\x1b[5;4~\x00\x1b[5;5~\x00\x1b[5;6~\x00\x1b[
>> 5;7~\x00\x1b[1;3C\x00\x1b[1;4C\x00\x1b[1;5C\x00\x1b[1;6C\x00\x1b[1;7C\x00\x1b[1;2A\x00\x1b[1;3A\x00\x1
>> b[1;4A\x00\x1b[1;5A\x00\x1b[1;6A\x00\x1b[1;7A\x00AX\x00XM\x00kDC3\x00kDC4\x00kDC5\x00kDC6\x00kDC7\x00k
>> DN\x00kDN3\x00kDN4\x00kDN5\x00kDN6\x00kDN7\x00kEND3\x00kEND4\x00kEND5\x00kEND6\x00kEND7\x00kHOM3\x00kH
>> OM4\x00kHOM5\x00kHOM6\x00kHOM7\x00kIC3\x00kIC4\x00kIC5\x00kIC6\x00kIC7\x00kLFT3\x00kLFT4\x00kLFT5\x00k
>> LFT6\x00kLFT7\x00kNXT3\x00kNXT4\x00kNXT5\x00kNXT6\x00kNXT7\x00kPRV3\x00kPRV4\x00kPRV5\x00kPRV6\x00kPRV
>> 7\x00kRIT3\x00kRIT4\x00kRIT5\x00kRIT6\x00kRIT7\x00kUP\x00kUP3\x00kUP4\x00kUP5\x00kUP6\x00kUP7\x00ka2\x
>> 00kb1\x00kb3\x00kc2\x00”
>> 
>> The first 116 bytes are actually not part of the string table. I have no 
>> idea what purpose they serve. If I skip these 116 bytes, I get to the actual 
>> string table and the offsets from before work perfectly fine. I noticed that 
>> 114 + 116 is 230. So 230 bytes need to be read to get to the string table. I 
>> noticed that the 4th short is 115, and 115 * 2 = 230. So essentially I’m 
>> using the 4th short multiplied by 2 as the length of the offsets. But I take 
>> the 3rd short of the header, aka the number of string caps and use only that 
>> number of shorts from the buffer. As in I read in those extra 116 bytes but 
>> I never use them. This seems to work fine across different files but I don’t 
>> think its correct. I think I’m misunderstanding something.
>> 
>> Could someone please help me understand the extended format and what I am 
>> doing wrong?
>> 
>> If you need any further clarification on my problem, please do not hesitate 
>> to ask.
>> 
>> Here is the current extended reader code. 
>> https://github.com/nhooyr/terminfo/blob/master/read.go#L126
>> 
>> First I make sure we’re on a word boundary. Then I read in the 10 bytes for 
>> the reader. Next I print it out. Then I read in the bool section. Then I 
>> read in the numeric section (its 0 in this case). Next I read the string 
>> offset section but for the size, I use r.h[lenExtTable] * 2 which means, 
>> take the 4th short (the size of the string table section apparently..) and 
>> multiple it by 2, and use that. Then I use the last offset as the size of 
>> the actual string table and print out all of the extended values. You can 
>> easily run this on your own computer by setting the TERM to xterm, running 
>> “go get github.com/nhooyr/terminfo” and then run "go test 
>> github.com/nhooyr/terminfo -v”. You should see all of the string caps being 
>> printed out.
>> 
>