BUG #16108: Colorization to the output of command-line has unproperly behaviors at Windows platform

The following bug has been logged on the website:

Bug reference: 16108
Logged by: Haiying Tang
Email address: tanghy(dot)fnst(at)cn(dot)fujitsu(dot)com
PostgreSQL version: 12.0
Operating system: Windows
Description:

Hello

I found the following release notes in PG12 is not working properly at
Windows.
> •Add colorization to the output of command-line utilities

Following the release note, I've set the the environment variable PG_COLOR
to auto, then I run pg_dump command with an incorrect passwd.
However, the command-line output is not colorized as the release notes
said.

Before PG_COLOR=auto is set: pg_dump: error: connection to database
"tanghy.fnst" failed: FATAL:
After PG_COLOR=auto is set: pg_dump: error: connection
to database "tanghy.fnst" failed: FATAL

I think the colorization to the output of command-line is not supported at
Windows.
Maybe function "pg_logging_init" at source "src\common\logging.c" should add
a platform check.
Besides, the related release note of PG12 should add some description about
it.

Best Regards,
Tang

On Tue, Nov 12, 2019 at 9:30 PM PG Bug reporting form
<noreply(at)postgresql(dot)org> wrote:
> The following bug has been logged on the website:
>
> Bug reference: 16108
> Logged by: Haiying Tang
> Email address: tanghy(dot)fnst(at)cn(dot)fujitsu(dot)com
> PostgreSQL version: 12.0
> Operating system: Windows
> Description:
>
> Hello
>
> I found the following release notes in PG12 is not working properly at
> Windows.
> > •Add colorization to the output of command-line utilities
>
> Following the release note, I've set the the environment variable PG_COLOR
> to auto, then I run pg_dump command with an incorrect passwd.
> However, the command-line output is not colorized as the release notes
> said.
>
> Before PG_COLOR=auto is set: pg_dump: error: connection to database
> "tanghy.fnst" failed: FATAL:
> After PG_COLOR=auto is set: [01mpg_dump: [0m [01;31merror: [0mconnection
> to database "tanghy.fnst" failed: FATAL
>
> I think the colorization to the output of command-line is not supported at
> Windows.
> Maybe function "pg_logging_init" at source "src\common\logging.c" should add
> a platform check.
> Besides, the related release note of PG12 should add some description about
> it.

Based on this:

https://en.wikipedia.org/wiki/ANSI_escape_code#DOS_and_Windows

... I wonder if it works if you use the new Windows Terminal, and I
wonder if it would work on the older thing if we used the
SetConsoleMode() flag it mentions.

On Tue, Nov 12, 2019 at 9:39 AM Thomas Munro <thomas(dot)munro(at)gmail(dot)com> wrote:

>
> ... I wonder if it works if you use the new Windows Terminal, and I
> wonder if it would work on the older thing if we used the
> SetConsoleMode() flag it mentions.
>
>
In order to make it work both things are needed, setting the console mode
and a terminal that supports it. Please find attached a patch for so.

Regards,

Juan José Santamaría Flecha

>In order to make it work both things are needed, setting the console mode and a terminal that supports it.

Your patch worked fine on windows which supports VT100. But the bug still happened when set PG_COLOR="always" at Windows Terminal that not support VT100. Please see the attached file “Test_result.png” for the NG result. (I used win7 for this test)

To fix the above bug, I made some change to your patch. The new one works fine on my win7(VT100 not support) and win10(VT100 support).

Also, in this new patch(v1), I added some doc change for Windows not support Colorization. Please find the attached patch for so.

Regards,
Tang

From: Juan José Santamaría Flecha <juanjo(dot)santamaria(at)gmail(dot)com>
Sent: Wednesday, November 13, 2019 4:00 AM
To: Thomas Munro <thomas(dot)munro(at)gmail(dot)com>
Cc: PG Bug reporting form <noreply(at)postgresql(dot)org>; PostgreSQL mailing lists <pgsql-bugs(at)lists(dot)postgresql(dot)org>; Tang, Haiying/唐 海英 <tanghy(dot)fnst(at)cn(dot)fujitsu(dot)com>
Subject: Re: BUG #16108: Colorization to the output of command-line has unproperly behaviors at Windows platform

On Tue, Nov 12, 2019 at 9:39 AM Thomas Munro <thomas(dot)munro(at)gmail(dot)com<mailto:thomas(dot)munro(at)gmail(dot)com>> wrote:

... I wonder if it works if you use the new Windows Terminal, and I
wonder if it would work on the older thing if we used the
SetConsoleMode() flag it mentions.

In order to make it work both things are needed, setting the console mode and a terminal that supports it. Please find attached a patch for so.

Regards,

Juan José Santamaría Flecha

Thanks for testing. I am opening a new item in the next commitfest for this
topic.

On Fri, Nov 15, 2019 at 5:23 AM Tang, Haiying <tanghy(dot)fnst(at)cn(dot)fujitsu(dot)com>
wrote:

> >In order to make it work both things are needed, setting the console mode
> and a terminal that supports it.
>
>
>
> Your patch worked fine on windows which supports VT100. But the bug still
> happened when set PG_COLOR="always" at Windows Terminal that not support
> VT100. Please see the attached file “Test_result.png” for the NG result. (I
> used win7 for this test)
>
>
> To fix the above bug, I made some change to your patch. The new one works
> fine on my win7(VT100 not support) and win10(VT100 support).
>

My understanding of the "always" logic is that it has to be enabled no
matter what, even if not supported in current output.

Also, in this new patch(v1), I added some doc change for Windows not
> support Colorization. Please find the attached patch for so.
>
>
>
You cannot change the release notes, if anything it will be added to 12.2
patch notes. It should be added to the 21 (!) utilities that specify the
PG_COLOR usage, but I am not so sure that adding a note stating this
feature requires Windows 10 >= 1511 update is really a Postgres business.

Please find attached a version that supports older Mingw versions and SDKs.

Regards,

Juan José Santamaría Flecha

Hello everyone.

> Please find attached a version that supports older Mingw versions and SDKs.

I have checked the patch source code and it seems to be working. But a
few moments I want to mention:

I think it is not good idea to mix the logic of detecting the fact of
TTY with enabling of the VT100 mode. Yeah, it seems to be correct for
current case but a little confusing.
Maybe is it better to detect terminal using *isatty* and later call
*enable_vt_mode*?

Also, it seems like if GetConsoleMode returns
ENABLE_VIRTUAL_TERMINAL_PROCESSING flag already set - we could skip
SetConsoleMode call (not a big deal of course).

Thanks,
Michail.

P.S.

Also, should we enable vt100 mode in case of PG_COLOR=always? I think yes.

On Tue, Feb 18, 2020 at 11:39 PM Michail Nikolaev <
michail(dot)nikolaev(at)gmail(dot)com> wrote:

>
> I have checked the patch source code and it seems to be working. But a
> few moments I want to mention:
>

Thanks for looking into this.

> I think it is not good idea to mix the logic of detecting the fact of
> TTY with enabling of the VT100 mode. Yeah, it seems to be correct for
> current case but a little confusing.
> Maybe is it better to detect terminal using *isatty* and later call
> *enable_vt_mode*?
>

Most of what enable_vt_mode() does is actually detecting the terminal, but
I can see why that is confusing without better comments.

> Also, it seems like if GetConsoleMode returns
> ENABLE_VIRTUAL_TERMINAL_PROCESSING flag already set - we could skip
> SetConsoleMode call (not a big deal of course).
>

Agreed.

The patch about making color by default [1] introduces the
function terminal_supports_color(), that I think is relevant for this
issue. Please find attached a new version based on that idea.

Also, adding Peter to weight on this approach.

[1] https://commitfest.postgresql.org/27/2406/

Regards,

Juan José Santamaría Flecha

Hello.

> The patch about making color by default [1] introduces the function terminal_supports_color(), that I think is relevant for this issue. Please find attached a new version based on that idea.

I am not sure it is good idea to mix both patches because it adds some
confusion and makes it harder to merge each.
Maybe is it better to update current patch the way to reuse some
function later in [1]?

Also, regarding comment
> It is disabled by default, so it must be enabled to use color outpout.

It is not true for new terminal, for example. Maybe it is better to
rephrase it to something like: "Check if TV100 support if enabled and
attempt to enable if not".

[1] https://www.postgresql.org/message-id/flat/bbdcce43-bd2e-5599-641b-9b44b9e0add4(at)2ndquadrant(dot)com

On Sat, Feb 22, 2020 at 9:09 PM Michail Nikolaev <michail(dot)nikolaev(at)gmail(dot)com>
wrote:

>
> I am not sure it is good idea to mix both patches because it adds some
> confusion and makes it harder to merge each.
> Maybe is it better to update current patch the way to reuse some
> function later in [1]?
>

The patch was originaly reported for Windows, but looking into Peter's
patch, I think this issue affects other systems unless we use stricter
logic to detect a colorable terminal when using the "auto" option.
Probably, the way to go is leaving this patch as WIN32 only and thinking
about a future patch.

> Also, regarding comment
> > It is disabled by default, so it must be enabled to use color outpout.
>
> It is not true for new terminal, for example. Maybe it is better to
> rephrase it to something like: "Check if TV100 support if enabled and
> attempt to enable if not".
>

The logic I have seen on new terminals is that VT100 is supported but
disabled. Would you find clearer? "Attempt to enable VT100 sequence
processing. If it is not possible consider it as unsupported."

Please find attached a patch addressing these comments.

Regards,

Juan José Santamaría Flecha

Hello.

Looks totally fine to me now.

So, I need to mark it as "ready to commiter", right?

On Wed, Feb 26, 2020 at 11:48 AM Michail Nikolaev <
michail(dot)nikolaev(at)gmail(dot)com> wrote:

>
> Looks totally fine to me now.
>
> So, I need to mark it as "ready to commiter", right?
>

Yes, that's right. Thanks for reviewing it.

Regards

On Mon, Feb 24, 2020 at 06:56:05PM +0100, Juan José Santamaría Flecha wrote:
> The patch was originaly reported for Windows, but looking into Peter's
> patch, I think this issue affects other systems unless we use stricter
> logic to detect a colorable terminal when using the "auto" option.
> Probably, the way to go is leaving this patch as WIN32 only and thinking
> about a future patch.

It is better to not mix issues. You can actually bump on similar
coloring issues depending on your configuration, with OSX or even
Linux.

> The logic I have seen on new terminals is that VT100 is supported but
> disabled. Would you find clearer? "Attempt to enable VT100 sequence
> processing. If it is not possible consider it as unsupported."
>
> Please find attached a patch addressing these comments.

I was reading the thread for the first time, and got surprised first
with the argument about "always" which gives the possibility to print
incorrect characters even if the environment does not allow coloring.
However, after looking at logging.c, the answer is pretty clear what
always is about as it enforces colorization, so this patch looks
correct to me.

On top of that, and that's a separate issue, I have noticed that we
have exactly zero documentation about PG_COLORS (the plural flavor,
not the singular), but we have code for it in common/logging.c..

Anyway, committed down to 12, after tweaking a few things.
--
Michael

On Mon, Mar 2, 2020 at 7:48 AM Michael Paquier <michael(at)paquier(dot)xyz> wrote:

>
> On top of that, and that's a separate issue, I have noticed that we
> have exactly zero documentation about PG_COLORS (the plural flavor,
> not the singular), but we have code for it in common/logging.c..
>

Yeah, there is nothing about it prior to [1]. So, this conversation will
have to be carried over there.

> Anyway, committed down to 12, after tweaking a few things.
>

Thank you.

[1]
https://www.postgresql.org/message-id/bbdcce43-bd2e-5599-641b-9b44b9e0add4@2ndquadrant.com

Regards,

Juan José Santamaría Flecha