RE: ms-terminal in terminfo.src

[Mike Griese]

Hey Thomas, Jurgen,

First off, let me first say THANK YOU so much for all the information you've 
compiled on invisible-island.net. I cannot tell you how many times I've been to 
that site over the last four years working on the VT emulation of the Windows 
Console. It's been an invaluable resource.

I don't think I have any issues with the actual values in the entry. That 
largely seems accurate for what we currently have support for. I think what I'm 
more worried about is this part:

# The task manager shows this as "OpenConsole.exe", which differs
# from the "Windows Command Processor" used for the command-prompt.
#
# The settings dialog does not work (unless the end user expects to open
# profiles.json in Visual Studio).  There is no documentation, of course.
#
# Testing via an ssh connection, using openssh:
# - the program sets TERM to cygwin if the tab is set to PowerShell,
#   and to xterm-256color if "Legacy".  However, in the latter, more tests
#   fail in vttest, which does not pay attention to TERM.

I'm going to try and explain a bunch of related things here so bear with me.

First off, before the Windows Terminal, there was only ever one console program 
on Windows, and that was conhost.exe. Conhost.exe is the program that's 
responsible for being the terminal for both cmd.exe ("Command Prompt") and 
powershell.exe. 

When we released the Windows Terminal as an open-source project this last 
April, we also released the source for conhost.exe with it. However, when you 
build conhost.exe externally, it builds as "OpenConsole.exe". The reason for 
this is unimportant, this is mostly just an artifact of an iterative design. 
OpenConsole.exe is the _exact same thing_ as conhost.exe.

When you launch a console (commandline) application on Windows, the OS will 
automatically create a conhost.exe for that application, and list the shell as 
the parent application of the conhost. However, when you launch OpenConsole.exe 
directly, we have some code that'll automatically launch cmd.exe on your 
behalf. In that scenario, OpenConsole.exe ends up being the parent of cmd.exe, 
which is why OpenConsole shows up differently from conhost.exe in the task 
manager.

I'd also be wary of using the openssh ssh.exe that ships inbox with Windows to 
run VT tests. In my experience their implementation has been less than perfect 
for whatever reason. We've gotten our fair share of bugs filed on us that are 
actually bugs in their emulation layer. Running WSL directly within conhost 
works _great_ as a test bed for running vttest et. al, and I've found the ssh 
that's provided with WSL distros to be more reliable. 

So far, I've only really been talking about conhost.exe, the Console. The 
Windows Terminal is actually an even more complicated story.

Conhost.exe is not _only_ responsible for being the terminal window, but it's 
also responsible for being the Console API server. The Console API on Windows 
is composed of some 70+ APIs, some of which make sense as VT sequences and 
others that are totally insane. The Console API also includes a lot of features 
that would probably make more sense as a `readline` like library, rather than 
an integral part of the console. 

Rather than force terminal authors to deal with the quirks of the Console API, 
we instead created Conpty, which acts as a translation layer from the Console 
API to VT sequences that would be understood by any existing terminal. At the 
core on conpty is conhost.exe, still acting as the Console API server, but now 
it's _rendering_ its state to a stream of characters and VT sequences. These 
characters are emitted to another pipe which a terminal program can read to 
recreate the state of the console buffer. 

The Windows Terminal actually then uses the conpty API and the stream of chars 
conpty generates, not the output of the application directly. The Windows 
Terminal is actually a pretty weak terminal emulator all by itself, because 
conhost.exe is doing most of the heavy lifting, and reducing the number of 
sequences the Terminal needs to know about to like, 10 total. All of this _is_ 
supposed to be transparent to the end user, but many of the bugs in the Windows 
Terminal's "terminal emulation" are actually due to bugs in the conpty layer, 
not necessarily conhost's handling of those sequences. For example, mouse input 
works pretty well in conhost.exe, but conpty isn't quite yet equipped to handle 
it. 

Neither conhost nor Windows Terminal are setting TERM, so I'd presume that the 
TERM value mentioned is being set by openssh. I know that WSL will set TERM to 
xterm-256color. If we wanted to be _technically_ correct, then we'd have 
separate entries for both conhost and the Windows Terminal. Though, due to the 
aforementioned translation being done by conpty, this could become confusing 
very quickly. If we leave anything in for the Windows Terminal, I'd want to 
make sure it was definitely marked **Experimental** or **Preview** or something 
to that effect.

The settings UI is not done yet - that's correct. When a user tries to open the 
settings, we'll try to open the user's configured .json file editor. Visual 
Studio does override this on install, but the user is free to use whatever 
editor they like. As of v0.3, we'll open notepad if they don't have a .json 
editor installed. We're still adding more documentation, but you can find the 
start of the documentation [here]( 
https://github.com/microsoft/terminal/blob/master/doc/user-docs/index.md).

It was also mentioned that changing the color palette is tricky - That should 
be a lot better nowadays actually. We released the [ColorTool]( 
https://github.com/Microsoft/Terminal/tree/master/src/tools/ColorTool) a few 
years ago, which should make changing the colors much easier. Additionally, we 
support OSC 4 (and similar sequences) for changing the palette at runtime.

Do you actually get bug reports for ncurses for users using conhost.exe and/or 
the Windows Terminal? I'd certainly be curious to see those - our VT emulation 
is getting better every release, but it still has a long way to go. Getting 
more bug reports would help us be able to track the important things we need to 
work on next. The list of diffs you've provided between `ms-terminal` and 
`xterm-256color` are a super helpful place to start next 😊


Let us know if there's anything we can do to help here!

Mike Griese



-----Original Message-----
From: Thomas Dickey <address@hidden> 
Sent: Friday, August 2, 2019 3:52 PM
To: Mike Griese <address@hidden>
Cc: address@hidden; Windows Command Line Team <address@hidden>
Subject: Re: ms-terminal in terminfo.src

On Fri, Aug 02, 2019 at 12:59:42PM +0000, Mike Griese via Bug-ncurses wrote:
> To Whom It May Concern,
> 
> My name is Mike Griese, I work on the Windows Terminal Team.  It's 
> come to our attention that there's a new 
> ms-terminal<https://invisible-island.net/ncurses/terminfo.src.html#tic
> -ms-terminal> in terminfo.src, though I'm fairly certain no one in our 
> organization submitted that entry.  It looks to me like the entry was 
> submitted in response to our announcement of the Windows Terminal.  
> However there looks to be some problems with the listing, particularly 
> in the comment with it.  Is there any way we could at very least 
> update the comment to be more accurate?

Actually the comment's accurate.  It says what version was examined,

        # Windows 10 1903
        # Version 0.2.1715.0

and lists several issues which led to omitting various features of the 
"xterm-256color" terminal description to give a terminal description which 
would work properly with the Windows Terminal.

If someone uses that version of Windows Terminal with TERM=xterm-256color, I'll 
be seeing bug reports and negative comments about ncurses.

(If there's a newer released version of Windows Terminal, I can update).

You might find these helpful:

https://invisible-island.net/ncurses/tack/
https://invisible-island.net/vttest/

--
Thomas E. Dickey <address@hidden> https://invisible-island.net 
ftp://ftp.invisible-island.net