www-authenticate

[mentaal]

Hi there, I tried using your library and I am also trying to use your library within a different github project https://github.com/Crypt0s/python-webdav
I get the following error:
File "<corporate_path_here>/python-webdav/requests-ntlm/requests_ntlm/requests_ntlm.py", line 57, in retry_using_http_NTLM_auth
auth_header_value = response2.headers[auth_header_field]
File "<corporate_path_here>/python-webdav/requests/requests/structures.py", line 77, in getitem
return self._store[key.lower()][1]
KeyError: 'www-authenticate'

Any ideas what this error means?

[Lukasa]

It means no Auth challenge header was received. =)

[mentaal]

Hi Lukasa. Thanks for the very prompt response. Ok so you may have to dumb this down a little for me. Basically I am trying to download a file over webdav from a sharepoint site and I think that the problem is that the site is using ntlm authentication. So if I am not getting an Auth challenge header does that mean that the site is not using NTLM?

[Lukasa]

I suspect so. I'm not sure what your code looks like, but if you can get hold of the Response object that requests produces you could print its headers attribute and see what headers came back. Otherwise, if you're familiar with Wireshark or tcpdump you could analyse the network traffic to see the headers.

[sigmavirus24]

Looking at the project you linked @mentaal it seems like it was last updated in December but was still depending on requests 0.11.1. This library is maintained for requests >= 2.2.0 and is likely highly incompatible with the version of requests that library is using.

[mentaal]

Ok guys thanks for the feedback here. Lukasa, I'll try what you suggested.
Sigmavirus24, that's a good point alright. I saw that as well but I was just hack and slashing and thought I would try it anyway!

Cheers

[TheTechromancer]

I'm running into the same error with the following response, where the www-authenticate header is capitalized.

HTTP/2 401 Unauthorized
Server: Microsoft-IIS/10.0
Request-Id: deadbeef-140d-4546-9cc8-45da370c730a
X-Soap-Enabled: True
X-Wssecurity-Enabled: True
X-Wssecurity-For: None
X-Oauth-Enabled: True
X-Owa-Version: 15.2.986.15
Www-Authenticate: Basic realm="evilcorp.com"
Www-Authenticate: Negotiate
Www-Authenticate: NTLM
X-Powered-By: ASP.NET
X-Feserver: EXCHANGE
Date: Sat, 05 Feb 2022 01:24:14 GMT
Content-Length: 0

[ERRR] Traceback (most recent call last):
  File "/home/bls/Downloads/code/trevorspray/trevorspray/lib/proxy.py", line 130, in run
    valid, exists, locked, msg = self.check_cred(user, password, enumerate_users)
  File "/home/bls/Downloads/code/trevorspray/trevorspray/lib/proxy.py", line 275, in check_cred
    response = request(
  File "/home/bls/Downloads/code/trevorspray/trevorspray/lib/util/misc.py", line 190, in request
    response = session.send(*args, **kwargs)
  File "/home/bls/.cache/pypoetry/virtualenvs/trevorspray-cQTot07L-py3.10/lib/python3.10/site-packages/requests/sessions.py", line 652, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/home/bls/.cache/pypoetry/virtualenvs/trevorspray-cQTot07L-py3.10/lib/python3.10/site-packages/requests/hooks.py", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/home/bls/.cache/pypoetry/virtualenvs/trevorspray-cQTot07L-py3.10/lib/python3.10/site-packages/requests_ntlm/requests_ntlm.py", line 146, in response_hook
    return self.retry_using_http_NTLM_auth(
  File "/home/bls/.cache/pypoetry/virtualenvs/trevorspray-cQTot07L-py3.10/lib/python3.10/site-packages/requests_ntlm/requests_ntlm.py", line 103, in retry_using_http_NTLM_auth
    auth_header_value = response2.headers[auth_header_field]
  File "/home/bls/.cache/pypoetry/virtualenvs/trevorspray-cQTot07L-py3.10/lib/python3.10/site-packages/requests/structures.py", line 54, in __getitem__
    return self._store[key.lower()][1]
KeyError: 'www-authenticate'

[jvspl]

I'm running into the same issue. It seems the web server sometimes responds with 'www-authenticate' and other times with 'WWW-Authenticate'.

Function response_hook uses a safe get on the headers dictionary to extract www-authenticate. If found, it calls retry_using_http_NTLM_auth with auth_header_field="www-authenticate". In retry_using_http_NTLM_auth, a new request is made and on the response (response2), there is an unsafe call on the headers dictionary: auth_header_value = response2.headers[auth_header_field].

We cannot assume that because response (first response) had a header "www-authenticate", that response2 will also have a header "www-authenticate".

It seems a solution should be in the line of:

1. use response2.headers.get(auth_header_field)
2. Also get the headers values for upper/lower case variations of auth_header_field.
3. response_hook should probably also be more lenient towards upper/lower case variants of www-authenticate and proxy-authenticate.

[jvspl]

It seems my web server issue is NOT due to case sensitivity. The requests package uses a case insensitive dictionary, which handles this just fine.

However, there is an assumption in retry_using_http_NTLM_auth. We cannot assume that because response (first response) had a header "www-authenticate", that response2 will also have a header "www-authenticate"!

In my case, it seems the server sometimes (randomly) responds with an http 400. This triggers an exception in requests_ntlm because the header does not have "www-authenticate".

Instead of quitting gracefully and informing that the web server did not like the request, it raises a KeyError.

HTTP for response2

< GET /something/attributes HTTP/1.1
< Host: 10.0.x.xxx
< User-Agent: python-requests/2.26.0
< Accept-Encoding: gzip, deflate
< Accept: */*
< Connection: Keep-Alive
< Authorization: Negotiate TlRMTVNTUAABAAAAN4II4AAAAAAgAAAAAAAAACAAAAA=
< 

> HTTP/1.1 400 Bad Request
> Content-Length: 0
> Server: Microsoft-HTTPAPI/2.0
> Date: Fri, 26 May 2023 09:18:10 GMT
> 

Error message

<command-890195501745907> in api_call_nolog(url, params)
     15   for attempt in range(retries):
     16       # logger.debug(f"Performing REST API call: {url} with params {params}")
---> 17       response = session.request(
     18           method=method,
     19           url=url,

/databricks/python/lib/python3.9/site-packages/requests/sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    540         }
    541         send_kwargs.update(settings)
--> 542         resp = self.send(prep, **send_kwargs)
    543 
    544         return resp

/databricks/python/lib/python3.9/site-packages/requests/sessions.py in send(self, request, **kwargs)
    660 
    661         # Response manipulation hooks
--> 662         r = dispatch_hook('response', hooks, r, **kwargs)
    663 
    664         # Persist cookies

/databricks/python/lib/python3.9/site-packages/requests/hooks.py in dispatch_hook(key, hooks, hook_data, **kwargs)
     29             hooks = [hooks]
     30         for hook in hooks:
---> 31             _hook_data = hook(hook_data, **kwargs)
     32             if _hook_data is not None:
     33                 hook_data = _hook_data

/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/requests_ntlm/requests_ntlm.py in response_hook(self, r, **kwargs)
    166 
    167             if auth_type is not None:
--> 168                 return self.retry_using_http_NTLM_auth(
    169                     "www-authenticate",
    170                     "Authorization",

<command-890195501745883> in retry_using_http_NTLM_auth_patched(self, auth_header_field, auth_header, response, auth_type, args)
    101     #logger.debug(f"This is where it sometimes breaks.")
    102     try:
--> 103       auth_header_value = response2.headers[auth_header_field]
    104     except:
    105       logger.critical(f"It failed! Header type: {type(response2.headers)}, specific header: '{response2.headers.get(auth_header_field)}', full headers: {response2.headers}. Full response content: {response2.content}")

/databricks/python/lib/python3.9/site-packages/requests/structures.py in __getitem__(self, key)
     52 
     53     def __getitem__(self, key):
---> 54         return self._store[key.lower()][1]
     55 
     56     def __delitem__(self, key):

KeyError: 'www-authenticate'

[jvspl]

A workaround I've found, which might or not conform to what you expect or what you might want - is to return response, with the 401 unauthorised, when response2 does not have the www-authenticate header.

In function retry_using_http_NTLM_auth, instead of (line 132-133)

# get the challenge
auth_header_value = response2.headers[auth_header_field]

We do:

# get the challenge
try:
    auth_header_value = response2.headers[auth_header_field]
except:
    # Failed to get authentication header after first step of NTLM
    # Return original response and quit
    return response

Another, perhaps better way, to handle this could be to raise an Exception stating something like failed to negotiate NTLM with the server and then the HTTP code of the server's response and/or indicating that the www-authenticate header was empty.

Perhaps something like (untested code):

# confirm response2 is valid
response2.raise_for_status()
# get the challenge or raise an error
try:
    auth_header_value = response2.headers[auth_header_field]
except keyError:
    raise ValueError(f"Failed to negotiate NTLM authentication with server. Server response headers do not contain '{auth_header_field}': {response2.headers}")

[sn0wer]

As for me, it was that the server responded with a 400 (Invalid request) :

Using requests debugger (from requests documentation).

I got those logs :

header: Date: Mon, 12 Feb 2024 14:14:12 GMT
header: X-Frame-Options: SAMEORIGIN
header: X-XSS-Protection: 1; mode=block
header: X-Content-Type-Options: nosniff
header: Connection: close
header: Content-Length: 87
header: Content-Type: text/html
DEBUG:urllib3.connectionpool:https://myhost:443 "POST /ews/exchange2.asmx HTTP/1.1" 400 87
Traceback (most recent call last):
  File "C:\xampp\htdocs\ntlm.py", line 37, in <module>
    r = requests.post("https://myhost/ews/exchange2.asmx",auth=HttpNtlmAuth('myuser','mypassword'), data=content, headers=headers)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "Programs\Python\Python312\Lib\site-packages\requests\api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "Programs\Python\Python312\Lib\site-packages\requests\api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "Programs\Python\Python312\Lib\site-packages\requests\sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "Programs\Python\Python312\Lib\site-packages\requests\sessions.py", line 710, in send
    r = dispatch_hook("response", hooks, r, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "Programs\Python\Python312\Lib\site-packages\requests\hooks.py", line 30, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "Programs\Python\Python312\Lib\site-packages\requests_ntlm\requests_ntlm.py", line 168, in response_hook
    return self.retry_using_http_NTLM_auth(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "Programs\Python\Python312\Lib\site-packages\requests_ntlm\requests_ntlm.py", line 133, in retry_using_http_NTLM_auth
    auth_header_value = response2.headers[auth_header_field]
                        ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "Programs\Python\Python312\Lib\site-packages\requests\structures.py", line 52, in __getitem__
    return self._store[key.lower()][1]
           ~~~~~~~~~~~^^^^^^^^^^^^^
KeyError: 'www-authenticate'

To me, if the server responds with Invalid request, we should not expect www-authenticate headers (maybe I'm incorrect, I did not read the NTLM specifications)