Calling self.cont() in RESPMOD handler causes Broken Piper error

[ichramm]

I am testing the icap server using the example respmod_copy.py and Squid 3.5.25.

The only change I made was to put a call to self.cont() before copying the response body into the temporary file: like this:

    def example_RESPMOD(self):
        self.set_icap_response(200)

        self.set_enc_status(b' '.join(self.enc_res_status))
        for h in self.enc_res_headers:
            for v in self.enc_res_headers[h]:
                self.set_enc_header(h, v)

        if not self.has_body:
            self.send_headers(False)
            return

        self.cont() ### <-- Send 100 Continue ###

        # Read everything from the response to a temporary file
        # This file can be placed onto a tmpfs filesystem for more performance
        with tempfile.NamedTemporaryFile(prefix='pyicap.', suffix='.tmp') as upstream:
            self.read_into(upstream)
            if self.preview and not self.ieof:
                self.cont()
                self.read_into(upstream)
            upstream.seek(0)

            # And write it to downstream
            content = upstream.read()
            self.write_chunk(content)

From the OPTIONS message I understand Squid should send only the response headers to the ICAP Server:

self.set_icap_header(b'Preview', b'0')
self.set_icap_header(b'Transfer-Preview', b'*')
self.set_icap_header(b'Transfer-Ignore', b'jpg,jpeg,gif,png,swf,flv')
self.set_icap_header(b'Transfer-Complete', b'')

So, the behavior I expect is the following:

• Squid send the response headers (Threasfer-Preview=* and Preview=0)
• The ICAP Server send the 100 Continue
• Squid send the response body
• The ICAP Server writes to the temporary file and everything does as planned

Nevertheless, the ICAP Server doesn't perform es expected and it fails writing the response:

$ python examples/respmod_copy.py 
127.0.0.1 - - [08/b'May'/2017 16:15:24] "b'OPTIONS icap://127.0.0.1:1344/resp ICAP/1.0'" b'200' b'-'
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 36906)
Traceback (most recent call last):
  File "/usr/lib/python3.6/socketserver.py", line 639, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python3.6/socketserver.py", line 361, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python3.6/socketserver.py", line 696, in __init__
    self.handle()
  File "/usr/lib/python3.6/site-packages/pyicap.py", line 443, in handle
    self.handle_one_request()
  File "/usr/lib/python3.6/site-packages/pyicap.py", line 493, in handle_one_request
    method()
  File "examples/respmod_copy.py", line 67, in resp_RESPMOD
    self.write_chunk(content)
  File "/usr/lib/python3.6/site-packages/pyicap.py", line 213, in write_chunk
    self.wfile.write(l + b'\r\n' + data + b'\r\n')
  File "/usr/lib/python3.6/socketserver.py", line 775, in write
    self._sock.sendall(b)
ConnectionResetError: [Errno 104] Connection reset by peer
----------------------------------------

The error messages indicates that, for some reason, Squid closed the connection.

The only message I get from Squid is this one:

1494271928.277      0 ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff ICAP_OPT/200 258 OPTIONS icap://127.0.0.1:1344/resp - -/127.0.0.1 -
1494271928.278      1 wilderkrieger.localdomain ICAP_ERR_OTHER/100 50 RESPMOD icap://127.0.0.1:1344/resp - -/127.0.0.1 

And there is where I get lost, Squid is configured with ICAP Preview and Persistent Connections enabled. Perhaps Squid is failed due to the 100 continue?

Any help would be greatly appreciated.

Thanks!

Juan

[ichramm]

I found that self.cont() should not be called before reading because the only way to check for the ieof tag is by reading the request body.

The problem arises when the response is larger than the size of the Preview. In that case, the following code is executed:

if self.preview and not self.ieof:
    self.cont()
    self.read_into(upstream)

And then the icap server failes as explained above. It fails because Squid already sent the whole body but the ieof tag was not present. By reading the code of the function readchunk in pyicap.py I noticed that it also sets the variable self.bob.

It is safe to check also for that variable before calling self.cont()?

[uovobw]

@ichramm i am trying to get this to work too, using the test server in examples/respmod_copy.py. did you make any progress?

[aes3219563]

See no_adaptation_required() in pyicap.py. Under the case that icap client does not allow 204, it copies everything to the client. This part works.