bug#53941: 27.2; socks + tor dont work with https

[J.P.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Kangas <stefankangas@gmail.com>
>> Date: Wed, 6 Sep 2023 15:25:19 -0700
>> Cc: "J.P." <jp@neverwas.me>, larsi@gnus.org, gnuhacker@member.fsf.org, 
>>      53941@debbugs.gnu.org
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> > I'm really uncomfortable with installing these changes before the 
>> >> > release
>> >> > branch is cut.  The changes are hardly trivial, some controversial even 
>> >> > to
>> >> > my eyes, even though I'm no expert on network connections.
>> >>
>> >> Well, I myself am just about the furthest thing from (an expert), which
>> >> certainly doesn't comport well with dropping rash changes at the
>> >> eleventh hour. (That was rather disrespectful on my part, so shame on
>> >> me.) As such, if it's easier to revisit this once things settle down,
>> >> just ignore this email and I'll re-ping you sometime down the road.
>> >
>> > Yes, please do.  The branch is cut now, so if we agree on installing such
>> > changes on master, it's now up to you when to post another version of these
>> > changes with the requisite fixes.
>> 
>> Could we revisit this now?  It sounds like something we'd want to fix.
>
> No objections from me, but I don't think we saw "another version"
> posted.  J.P.?

Hi, unfortunately, there is no other version of any real note, at least
not yet. The attached version is slightly updated but lacks substantial
progress in the areas that matter most. As of now, the only patches I'd
be comfortable offering would be the first two, which aren't even
directly related to this bug.

In addition to providing some cosmetic refactoring [1], the third patch
is mainly just a working placeholder for the socks.el side of a
hypothetical integration with the URL framework (currently imagined as
an overhauled version of `socks-open-connection' made newly compatible
with the `url-proxy' and `open-network-stream' interfaces). A sketch of
the URL side of such an arrangement can be found in the fourth patch,
but it isn't well thought out (or thoroughly researched).

AFAIR, the OP's main complaint concerns the lack of a transparent and
easily configurable experience for proxying built-in Emacs applications
over SOCKS, perhaps compared to what you'd get with a typical web
browser. I think we can probably agree that users shouldn't have to
customize both `socks'- and `url'-owned options or write extra code to
achieve a working setup, which is how things are currently (AFAICT).

If we're to solve this using the `url' library's existing public API, it
may need to be more clearly defined in a few areas. For example, at
least one function (`url-http-find-free-connection') uses the `host' and
`service' parameters of `url-open-stream' for dialing HTTP proxies. But
if other protocols are meant to do the same, they'll need to somehow
encode both proxy- and logical-endpoint addresses into those two params
or find some other means of conveying the same info. The fourth patch
currently uses `url-using-proxy' as a dynamic variable for this purpose,
but perhaps that's unwise. It's quite possible I'm breaking something.

There's also the issue of DNS lookups for verifying domain certs. Last I
looked, we can't run `nsm' checks without involving the system resolver,
which may be a deal breaker for the more privacy minded. If that's true,
we may want to find an acceptable way of cluing folks in to the
situation.

As thing stand, I haven't really invested enough in understanding how
`url' works to take the lead in planning a comprehensive integration
strategy (sorry). But, perhaps others with open bugs in the same area
[2] might be interested in lending some expertise or insight. Thanks.


[1] The third patch also attempts to untangle some of the hairiness
    brought about by the `socks-override-functions' flag. The original
    picture looks like:

    s-o-f:    `socks-override-functions', flag var
    o-n-s:    `open-network-stream', standard (non-SOCKS) opener
    o-n-s*:   `open-network-stream' with s--o-n-s as :around advice
    s-o-c:    `socks-open-connection', proxy (SOCKS) opener
    s-o-n-s:  `socks-open-network-stream', semi o-n-s compatible thin
               wrapper around s--o-n-s
    s--o-n-s: `socks--open-network-stream': workhorse with SOCKS/non
               code paths

    | s-o-f | entry point         | no-route | route          |
    |-------+---------------------+----------+----------------|
    | t     | o-n-s* -> s--o-n-s  | o-n-s    | s-o-c -> o-n-s |
    | t/nil | s-o-n-s -> s--o-n-s | o-n-s    | s-o-c -> o-n-s |

    I've changed the above to:

    s--o-n-s: thin wrapper around `s-o-n-s'
    s-o-n-s:  o-n-s compatible workhorse with SOCKS/non code paths

    | s-o-f | entry point                   | no-route | route          |
    |-------+-------------------------------+----------+----------------|
    | t     | o-n-s* -> s--o-n-s -> s-o-n-s | o-n-s    | s-o-c -> o-n-s |
    | t/nil | s-o-n-s                       | o-n-s    | s-o-c -> o-n-s |


[2]  Bug#62598: "29.0.60; url-https-proxy-connect doesn't support
     multi-stage auth to proxies"

     https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62598

0000-v2-v3.diff
Description: Text Data

0001-Don-t-hard-code-server-ports-in-SOCKS-tests.patch
Description: Text Data

0002-Improve-SOCKS-error-handling-and-add-support-for-4a.patch
Description: Text Data

0003-POC-Simplify-network-stream-openers-in-socks.el.patch
Description: Text Data

0004-POC-Integrate-the-socks-and-url-libraries.patch
Description: Text Data