bug#35969: 26.2, Excorporate

[Thomas Fitzsimmons]

I think there should be no need to remove test-home, so you won't have
to re-run the M-x package-install steps.

Thomas

Thomas Fitzsimmons <fitzsim@fitzsim.org> writes:

> OK, can you apply my patch, then run "make" again and confirm it works?
>
> Thanks,
> Thomas
>
> Collin Day <dcday137@gmail.com> writes:
>
>> Did a git pull and followed instructions above.
>>
>> There are some issues.  First, I need to run M-x package-install <RET>
>> excorporate <RET>  four times because I see (each line after each
>> invocation)
>>
>> package--with-response-buffer-1:
>> https://elpa.gnu.org/packages/url-http-ntlm-2.0.4.el: Method not
>> allowed
>> package--with-response-buffer-1:
>> https://elpa.gnu.org/packages/fsm-0.2.1.el: Method not allowed
>> package--with-response-buffer-1:
>> https://elpa.gnu.org/packages/excorporate-0.8.3.tar: Method not
>> allowed
>>
>> After the 4th call it compiles and becomes available.
>>
>> M-x customize-group <RET> excorporate, set up for no autoconfig right
>> now, hit apply for current sessions, save not availiable....
>>
>> M-x excorporate <RET>
>>
>> enter uname and password
>>
>> Contacting host: outlook.office365.com:443
>> error in process filter: exco--parse-xml-in-current-buffer: Server
>> response is not an XML document
>> error in process filter: Server response is not an XML documen
>>
>> *http outlook.office.365.com:443*
>
> [...]
>
>> So it appears not to work, at least on the current head as of today....
>>
>> Thanks!
>>
>> -C
>>
>> On Tue, Jul 9, 2019 at 4:09 PM Collin Day <dcday137@gmail.com> wrote:
>>>
>>> Sorry, I have not had the chance to.  A lot has been going on at my
>>> place of employment.  It has crossed my mind, and as soon as I have
>>> a chance, I will try what you suggested above.  Thanks, sorry for
>>> the inconvenience.
>>>
>>> On Tue, Jul 9, 2019, 3:53 PM Thomas Fitzsimmons <fitzsim@fitzsim.org> wrote:
>>>>
>>>> Hi,
>>>>
>>>> Were you able to complete building Emacs?  I'd like to know if my patch
>>>> is needed on top of Andreas's to make Excorporate work through your
>>>> proxy.
>>>>
>>>> I'm not sure why the built version would interfere with the current
>>>> installation.  If you were doing "make install" to a common prefix, that
>>>> might explain it.  Instead you can try something like this:
>>>>
>>>> cd emacs-master [your github.com checkout]
>>>> make
>>>> mkdir test-home
>>>> HOME=`pwd`/test-home ./src/emacs -Q
>>>>
>>>> That will ensure you're only running the built Emacs, and completely
>>>> ignoring the packages installed in your home directory.
>>>>
>>>> Thomas
>>>>
>>>> "tenspd137 ." <dcday137@gmail.com> writes:
>>>>
>>>> > I haven't tried the first item on the list yet - ie reloading Andreas'
>>>> > function.  i have tried to build emacs from
>>>> > https://github.com/emacs-mirror/emacs.git because I can't reach the
>>>> > official repo from my firewall for whatever reason.  I am sure it is
>>>> > something here.  Anyway, I was having problems because my built
>>>> > version (27.0.x) was getting seemingly mixed up with my current
>>>> > installation.  I can try again later, but right now things are pretty
>>>> > busy at work.
>>>> >
>>>> > Thanks.
>>>> >
>>>> > On Tue, Jun 18, 2019 at 10:26 PM Thomas Fitzsimmons 
>>>> > <fitzsim@fitzsim.org> wrote:
>>>> >>
>>>> >> Hi,
>>>> >>
>>>> >> "tenspd137 ." <dcday137@gmail.com> writes:
>>>> >>
>>>> >> > So - I am not sure if I did it correctly, but I copied this function
>>>> >> > with Andreas' changes into a file:
>>>> >> >
>>>> >> > (defun url-https-proxy-after-change-function (_st _nd _length)
>>>> >> >   (let* ((process-buffer (current-buffer))
>>>> >> >          (proc (get-buffer-process process-buffer)))
>>>> >> >     (goto-char (point-min))
>>>> >> >     (when (re-search-forward "^\r?\n" nil t)
>>>> >> >       (backward-char 1)
>>>> >> >       ;; Saw the end of the headers
>>>> >> >       (setq url-http-end-of-headers (set-marker (make-marker) 
>>>> >> > (point)))
>>>> >> >       (url-http-parse-response)
>>>> >> >       (cond
>>>> >> >        ((null url-http-response-status)
>>>> >> >         ;; We got back a headerless malformed response from the
>>>> >> >         ;; server.
>>>> >> >         (url-http-activate-callback)
>>>> >> >         (error "Malformed response from proxy, fail!"))
>>>> >> >        ((= url-http-response-status 200)
>>>> >> >         (if (gnutls-available-p)
>>>> >> >             (condition-case e
>>>> >> >                 (let ((tls-connection (gnutls-negotiate
>>>> >> >                                        :process proc
>>>> >> >                                        :hostname (url-host 
>>>> >> > url-current-object)
>>>> >> >                                        :verify-error nil)))
>>>> >> >                   ;; check certificate validity
>>>> >> >                   (setq tls-connection
>>>> >> >                         (nsm-verify-connection tls-connection
>>>> >> >                                                (url-host 
>>>> >> > url-current-object)
>>>> >> >                                                (url-port 
>>>> >> > url-current-object)))
>>>> >> >                   (with-current-buffer process-buffer (erase-buffer))
>>>> >> >                   (set-process-buffer tls-connection process-buffer)
>>>> >> >                   (setq url-http-after-change-function
>>>> >> >                         'url-http-wait-for-headers-change-function)
>>>> >> >                   (set-process-filter tls-connection 
>>>> >> > 'url-http-generic-filter)
>>>> >> >                   (process-send-string tls-connection
>>>> >> >                                        ;; Use the non-proxy form of 
>>>> >> > the request
>>>> >> >                                        (let (url-http-proxy)
>>>> >> >                                          (url-http-create-request))))
>>>> >> >               (gnutls-error
>>>> >> >                (url-http-activate-callback)
>>>> >> >                (error "gnutls-error: %s" e))
>>>> >> >               (error
>>>> >> >                (url-http-activate-callback)
>>>> >> >                (error "error: %s" e)))
>>>> >> >           (error "error: gnutls support needed!")))
>>>> >> >        (t
>>>> >> >         (url-http-debug "error response: %d" url-http-response-status)
>>>> >> >         (url-http-activate-callback))))))
>>>> >> >
>>>> >> > and then loaded it before running excorporate.  After that, I did M-x
>>>> >> > excorporate, and the minibuffer returns:  error in process filter:
>>>> >> > Server response is not an XML document
>>>> >>
>>>> >> In this scenario, if you immediately (without restarting Emacs/reloading
>>>> >> anything) re-run M-x excorporate does it still fail?  I just want to
>>>> >> make sure that's not a transient failure.  If it does fail the second
>>>> >> time, can you post the HTTP response from the server?
>>>> >>
>>>> >> > When I do the similar test by loading the url-http-create-request with
>>>> >> > Thomas's changes, I can get a connection and grab my schedule
>>>> >> > through the proxy.
>>>> >>
>>>> >> OK.
>>>> >>
>>>> >> > Let me know if I need to try something different.
>>>> >>
>>>> >> Are you in a position to build Emacs master tip and retry the experiment
>>>> >> without patching anything?
>>>> >>
>>>> >> Thanks,
>>>> >> Thomas
>>>> >>
>>>> >> > On Mon, Jun 17, 2019 at 4:08 PM Thomas Fitzsimmons 
>>>> >> > <fitzsim@fitzsim.org> wrote:
>>>> >> >>
>>>> >> >> Hi,
>>>> >> >>
>>>> >> >> Good to hear that the patch I posted worked!
>>>> >> >>
>>>> >> >> Yes, that's the patch that Andreas's commit
>>>> >> >> 84613dae5c34ea742dd9a3e56f5acb55f604b483 applied.  From what I can 
>>>> >> >> tell,
>>>> >> >> you will not have that in Emacs 26.2.
>>>> >> >>
>>>> >> >> Can you try reverting my patch and applying Andreas's patch, and see 
>>>> >> >> if
>>>> >> >> Excorporate still works through the proxy?
>>>> >> >>
>>>> >> >> Thanks,
>>>> >> >> Thomas
>>>> >> >>
>>>> >> >> "tenspd137 ." <dcday137@gmail.com> writes:
>>>> >> >>
>>>> >> >> > The patch Thomas seems to work from behind the proxy.  My current
>>>> >> >> > emacs version is 26.2, so I would think it would include the commit
>>>> >> >> > Andreas is talking about....  I went and looked it up - is this the
>>>> >> >> > correct commit?
>>>> >> >> >
>>>> >> >> > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
>>>> >> >> > index 53798f7..817c5ce 100644
>>>> >> >> > --- a/lisp/url/url-http.el
>>>> >> >> > +++ b/lisp/url/url-http.el
>>>> >> >> > @@ -1412,7 +1412,9 @@ The return value of this function is the
>>>> >> >> > retrieval buffer."
>>>> >> >> > 'url-http-wait-for-headers-change-function)
>>>> >> >> > (set-process-filter tls-connection 'url-http-generic-filter)
>>>> >> >> > (process-send-string tls-connection
>>>> >> >> > - (url-http-create-request)))
>>>> >> >> > + ;; Use the non-proxy form of the request
>>>> >> >> > + (let (url-http-proxy)
>>>> >> >> > + (url-http-create-request))))
>>>> >> >> > (gnutls-error
>>>> >> >> > (url-http-activate-callback)
>>>> >> >> > (error "gnutls-error: %s" e))
>>>> >> >> >
>>>> >> >> > Thanks!
>>>> >> >> >
>>>> >> >> > -C
>>>> >> >> >
>>>> >> >> >
>>>> >> >> > On Sat, Jun 15, 2019 at 1:41 AM Andreas Schwab 
>>>> >> >> > <schwab@linux-m68k.org> wrote:
>>>> >> >> >>
>>>> >> >> >> On Jun 14 2019, Thomas Fitzsimmons <fitzsim@fitzsim.org> wrote:
>>>> >> >> >>
>>>> >> >> >> > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
>>>> >> >> >> > index 00803a103a..723d111d58 100644
>>>> >> >> >> > --- a/lisp/url/url-http.el
>>>> >> >> >> > +++ b/lisp/url/url-http.el
>>>> >> >> >> > @@ -329,7 +329,10 @@ url-http-create-request
>>>> >> >> >> >               ;; The request
>>>> >> >> >> >               (or url-http-method "GET") " "
>>>> >> >> >> >               (url-http--encode-string
>>>> >> >> >> > -              (if using-proxy (url-recreate-url 
>>>> >> >> >> > url-http-target-url) real-fname))
>>>> >> >> >> > +              (if (and using-proxy
>>>> >> >> >> > +                       (not (equal "https" (url-type 
>>>> >> >> >> > url-http-target-url))))
>>>> >> >> >> > +                  (url-recreate-url url-http-target-url)
>>>> >> >> >> > +                real-fname))
>>>> >> >> >>
>>>> >> >> >> That should already be handled by commit 84613dae5c.
>>>> >> >> >>
>>>> >> >> >> Andreas.
>>>> >> >> >>
>>>> >> >> >> --
>>>> >> >> >> Andreas Schwab, schwab@linux-m68k.org
>>>> >> >> >> GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 
>>>> >> >> >> A9DA AEC1
>>>> >> >> >> "And now for something completely different."