Page 2 - i28 proposed replacement text

Henrik Nordstrom wrote:
¶r, 2008-05-24 at 09:52 +0200, Julian Reschke wrote:

>With chunked encoding in use, how does the sender of a message indicates
>an error? (such as when serving a file from the file system, and getting
>an I error)

By closing the connection in the middle of the response without sending
the last-chunk.

Yes, if course (I shouldn't have phrased that as a question).

What I was trying to say is that, when streaming out a response using
chunked encoding, this is the *only* way to signal an error, and

1) should make sure that httpbis still allows it, and

2) potentially point out that this is the only way to do it.

BR, Julian

Henrik Nordstrom wrote:

If no T-E is used and no Content-Length then it's tricker as the receier
then has to fully parse the response and from that judge if it's
complete or not, and then guess if this is due to a server failure or if
the object at the server really is corrupted/truncated


Well, in the generic case, the recipient hasn't got sufficient
information for parsing (think browser doing file download).

BR, Julian

¶r, 2008-05-24 at 09:52 +0200, Julian Reschke wrote:

With chunked encoding in use, how does the sender of a message indicates
an error? (such as when serving a file from the file system, and getting
an I error)

By closing the connection in the middle of the response without sending
the last-chunk.

Regards
Henrik

PGP SIGNATURE
Version: GnuPG v1.4.7 (GNU/Linux)




bMwSYAEmlj8=
=Kupr
PGP SIGNATURE

¶r, 2008-05-24 at 11:51 +0200, Henrik Nordstrom wrote:
¶r, 2008-05-24 at 09:52 +0200, Julian Reschke wrote:

With chunked encoding in use, how does the sender of a message indicates
an error? (such as when serving a file from the file system, and getting
an I error)

By closing the connection in the middle of the response without sending
the last-chunk.

if you meant "Witch chunked encoding NT in use" then it's still the
same, by closing the connection in the middle of the response. How easy
the error is to detect for the recipient then depends on the encoding or
format of the response.

T-E: gzip/deflate both have stream structure and clearly indicates
premature closure (except for the hypothetical case of a sender sending
multiple members in a gzip stream and crashing just between two
members).

If no T-E is used and Content-Length is used then it's trivially
detected by the response beeing too short.

If no T-E is used and no Content-Length then it's tricker as the receier
then has to fully parse the response and from that judge if it's
complete or not, and then guess if this is due to a server failure or if
the object at the server really is corrupted/truncated

Regards
Henrik

PGP SIGNATURE
Version: GnuPG v1.4.7 (GNU/Linux)




gYqCwI/zM6M=
=Gtc2
PGP SIGNATURE

Sat, 24 May 2008, Henrik Nordstrom wrote:

fre, 2008-05-23 at 23:17 +0100, Joe wrote:
>
>What delimits the end of the message in the gzip transfer-coding?
>
Chunked encoding, or if chunked is not used by closing the connection.

Then we must at least say that the encoding used (not chunked) must
give the same characteristics as chunked wrt detection of the end of the
message.

Is this particular case really used in deployed client/servers, and done
the right way ?

--
Baroula que barouleras, au toujou t'entourneras.

~~Yves

ons, 2008-05-28 at 10:41 -0400, Yves Lafon wrote:

Then we must at least say that the encoding used (not chunked) must
give the same characteristics as chunked wrt detection of the end of the
message.

Why? The protocol will not fall down if a message is unexpectedly cut
short.

A note mentioning that this may downgrade the message integrity to the
level of a Connection: close without Content-Length message may be
acceptable, but not a must. In practice most file formats and encodings
easily detect truncation.

Is this particular case really used in deployed client/servers, and done
the right way ?

As already mentioned both gzip and deflate has this property to a
sufficient level.

Regards
Henrik

Wed, 28 May 2008, Henrik Nordstrom wrote:

ons, 2008-05-28 at 10:41 -0400, Yves Lafon wrote:
>
>Then we must at least say that the encoding used (not chunked) must
>give the same characteristics as chunked wrt detection of the end of the
>message.
>
Why? The protocol will not fall down if a message is unexpectedly cut
short.

Essentially for what it below, the need to verify the integrity of the
message. It is needed when you don't cut the connection, but it is a
desirable property even if the connection is cut, to detect an error case
(like cutting the connection in the middle of a chunked stream) from a
normal response.

A note mentioning that this may downgrade the message integrity to the
level of a Connection: close without Content-Length message may be
acceptable, but not a must. In practice most file formats and encodings
easily detect truncation.

partial or absolute detection ? If the format allows multiple concatenated
entries, and integrity check is true if you reach exactly the end of one
entry, you have a highly probable integrity check but not an absolute one.

In any case, a note explaining that it might be difficult to figure out
the integrity of the message in this case would be sufficient.

>Is this particular case really used in deployed client/servers, and done
>the right way ?
>
As already mentioned both gzip and deflate has this property to a
sufficient level.

But it is used that way in deployed servers/clients ? is chunked
encoding always the norm.

--
Baroula que barouleras, au toujou t'entourneras.

~~Yves

¥n 2008-06-02 klockan 03:10 -0400 skrev Yves Lafon:

Essentially for what it below, the need to verify the integrity of the
message.

Which in HTTP is provided by Content-MD5 (even if partly misunderstood
on 206 responses), or by Digest qop=auth-int (if implemented by
anyone)

It is needed when you don't cut the connection, but it is a
desirable property even if the connection is cut, to detect an error case
(like cutting the connection in the middle of a chunked stream) from a
normal response.

Sure, there is many things which is desireable and nice, but it doesn't
make them MUSTs

partial or absolute detection ? If the format allows multiple concatenated
entries, and integrity check is true if you reach exactly the end of one
entry, you have a highly probable integrity check but not an absolute one.

You don't hav an absolute one even if Content-Length matches or chunked
encoding is terminated proper. It all depends on where the failure was
and how communication was to that In any situation where there may be
some form or proxy inbetween (including servers running scripts) both
Content-Length and chunking is synthetic and hop-by-hop. As recipient
you can only be absolutely sure that if there is a mismatch then
something is wrong

As sender you can be reasonably sure that if you explicitly indicate the
length by Content-Length then the recipient will get that, but chunking
does not provide this level of integrity as it's a hop-by-hop feature
which may be added/removed any number of times along the response path.

But it is used that way in deployed servers/clients ? is chunked
encoding always the norm.

Not sure one yet can speak of a norm when talking about
transfer-encoding gzip/chunked

Regards
Henrik

Henrik Nordstrom wrote:
partial or absolute detection ? If the format allows multiple concatenated
entries, and integrity check is true if you reach exactly the end of one
entry, you have a highly probable integrity check but not an absolute one.

You don't hav an absolute one even if Content-Length matches or chunked
encoding is terminated proper. It all depends on where the failure was
and how communication was to that

Agree, except we can at least identify when it's not the protocol
itself introducing undetected failure.

In any situation where there may be
some form or proxy inbetween (including servers running scripts) both
Content-Length and chunking is synthetic and hop-by-hop. As recipient
you can only be absolutely sure that if there is a mismatch then
something is wrong

According to RFC2616, Content-Length is an entity header and an
end-to-end header, therefore should not be synthesised hop-by-hop.
Yeah, I know :-)

-- Jamie

Yves Lafon wrote:
>As already mentioned both gzip and deflate has this property to a
>sufficient level.

But it is used that way in deployed servers/clients ? is chunked
encoding always the norm.

Chunked encoding clearly is not the norm when sending compressed
dynamic content to a HTTP/1.0 client.

-- Jamie

Henrik Nordstrom wrote:
, 2008-06-02 at 11:31 +0100, Jamie Lokier wrote:
Yves Lafon wrote:
>As already mentioned both gzip and deflate has this property to a
>sufficient level.

But it is used that way in deployed servers/clients ? is chunked
encoding always the norm.

Chunked encoding clearly is not the norm when sending compressed
dynamic content to a HTTP/1.0 client.

We are talking about transfer-encoding and message delimiting, not
content-encoding.

True, but the same message integrity issues w.r.t connection aborts
apply to Content-Encoding used for compressed dynamic content. Since
that is how it's done most often right now, it seems relevant.

-- Jamie



Regards
Henrik

¥n, 2008-06-02 at 11:31 +0100, Jamie Lokier wrote:
Yves Lafon wrote:
>As already mentioned both gzip and deflate has this property to a
>sufficient level.

But it is used that way in deployed servers/clients ? is chunked
encoding always the norm.

Chunked encoding clearly is not the norm when sending compressed
dynamic content to a HTTP/1.0 client.

We are talking about transfer-encoding and message delimiting, not
content-encoding.

Regards
Henrik

PGP SIGNATURE
Version: GnuPG v1.4.7 (GNU/Linux)




aW/ymTt8g18=
=BJao
PGP SIGNATURE

¥n, 2008-06-02 at 11:35 +0100, Jamie Lokier wrote:

According to RFC2616, Content-Length is an entity header and an
end-to-end header, therefore should not be synthesised hop-by-hop.
Yeah, I know :-)

Content-Length is that odd entity header which is also a hop-by-hop
header.

See "Non-modifiable Headers" which among other things states

The Content-Length field of a request or response is added or deleted
according to the rules in section 4.4. A transparent proxy MUST
preserve the entity-length (section 7.2.2) of the entity-body,
although it MAY change the transfer-length (section 4.4).

Regards
Henrik

PGP SIGNATURE
Version: GnuPG v1.4.7 (GNU/Linux)




gb6/gaVGA5Y=
=/Zp1
PGP SIGNATURE

Yves Lafon wrote:
My issue is not about interaction between 1.0 and 1.1, but more about
integrity/error detection in 1.1.

In p1 3.4:
<<
Whenever a transfer-coding is applied to a message-body, the set of
transfer-codings MUST include "chunked", unless the message is
terminated by closing the connection. When the "chunked" transfer-
coding is used, it MUST be the last transfer-coding applied to the
message-body. The "chunked" transfer-coding MUST NT be applied more
than once to a message-body. These rules allow the recipient to
determine the transfer-length of the message (Section 4.4).
>>

Should we add there or in Section 4.4 something like:
<<
If the transfer-length is defined by closing the connection, the
integrity of the message is not guaranteed, unless it is a
characteristic of the transfer-coding used. 'Identity' does not have
this characteristic.
>>
?

<<
Because closing is used to abort in some error scenarios, if the
transfer-length is defined by closing the connection, the recipient
cannot be sure the whole message is received, unless the
transfer-coding unambiguously detects truncation. The
transfer-codings 'identity', 'deflate' and 'gzip' do not have this
characteristic, although in most cases 'deflate' and 'gzip' will
detect truncation. Content-encoding and type-specific syntax of
the received entity may also detect truncation, but truncation of
entity syntax does not always indicate a transport error.
>>

-- Jamie

Mon, 2 Jun 2008, Jamie Lokier wrote:

Chunked encoding clearly is not the norm when sending compressed
dynamic content to a HTTP/1.0 client.
>>
>We are talking about transfer-encoding and message delimiting, not
>content-encoding.
>
True, but the same message integrity issues w.r.t connection aborts
apply to Content-Encoding used for compressed dynamic content. Since
that is how it's done most often right now, it seems relevant.

My issue is not about interaction between 1.0 and 1.1, but more about
integrity/error detection in 1.1.

In p1 3.4:
<<
Whenever a transfer-coding is applied to a message-body, the set of
transfer-codings MUST include "chunked", unless the message is
terminated by closing the connection. When the "chunked" transfer-
coding is used, it MUST be the last transfer-coding applied to the
message-body. The "chunked" transfer-coding MUST NT be applied more
than once to a message-body. These rules allow the recipient to
determine the transfer-length of the message (Section 4.4).
>>

Should we add there or in Section 4.4 something like:
<<
If the transfer-length is defined by closing the connection, the
integrity of the message is not guaranteed, unless it is a
characteristic of the transfer-coding used. 'Identity' does not have
this characteristic.
>>
?

--
Baroula que barouleras, au toujou t'entourneras.

~~Yves

tis 2008-06-03 klockan 04:41 -0400 skrev Yves Lafon:

Should we add there or in Section 4.4 something like:
<<
If the transfer-length is defined by closing the connection, the
integrity of the message is not guaranteed, unless it is a
characteristic of the transfer-coding used. 'Identity' does not have
this characteristic.
>>
?

I don't see why as this isn't a guaranteed even if chunking is used in
the hop to the receiver.

Chain may be

server [Connection: close, no TE] -proxy [Transfer-Encoding: chunked]
-recipient

In which case receiving a properly terminated chunked encoding says
nothing about if the message was truncated in transit or not.

Denying upgrading from Connection: close to Transfer-Encoding: chunked
would work around this, but at a significant performance penalty and
also uncertain as such upgrading is actively done by deployed RFC2616
compliant implementations.

Integrity is not a property of the transport. The transport is
hop-by-hop and must only guarantee internal consistency, allowing
unambigious parsing of the data stream. Integrity checks should be done
at the message level.

But I am in favor for stating that if the transfer encoding could not be
properly decoded (decompression error in gzip/deflate, missing end chunk
in chunked etc) or if network errors is detected while receiving the
message (i.e. read error or similar) the message SHULD be considered
invalid and the connection closed. Implementations MUST NT try to
continue using a connection after a transport error including
transfer-encoding related issues.

Regards
Henrik

Henrik Nordstrom wrote:
Integrity is not a property of the transport. The transport is
hop-by-hop and must only guarantee internal consistency, allowing
unambigious parsing of the data stream. Integrity checks should be done
at the message level.

True, but there is no practical mechanism in HTTP at the moment to
assure even the most basic message integrity: that you received all of it.

Content-MD5 is not useful for dynamically produced entities.
As a trailer it might be possible, but how compatible is that?

Content-Length, even, cannot be used.

Not even for static content. When the server does provide
Content-Length, in your own example the proxy->client hop removes
Content-Length.

I'm thinking that the solution to these is allowing Content-Length in
a chunked trailer, and Content-MD5 too.

-- Jamie

Jamie Lokier wrote:
Henrik Nordstrom wrote:
>Integrity is not a property of the transport. The transport is
>hop-by-hop and must only guarantee internal consistency, allowing
>unambigious parsing of the data stream. Integrity checks should be done
>at the message level.

True, but there is no practical mechanism in HTTP at the moment to
assure even the most basic message integrity: that you received all of it.

Content-MD5 is not useful for dynamically produced entities.
As a trailer it might be possible, but how compatible is that?

Well, unless I'm missing something, it will be hard to send from a
servlet (hey, Servlet EG, are you listening?).

Content-Length, even, cannot be used.

Not even for static content. When the server does provide
Content-Length, in your own example the proxy->client hop removes
Content-Length.

I'm thinking that the solution to these is allowing Content-Length in
a chunked trailer, and Content-MD5 too.

Well, what would they contain in case of a truncated response? Surely
not the length/digest of the actual response, because that wouldn't help
the client finding out about the truncation

Maybe something like "final-status" as a new response header would make
sense. That way, a server could send an initial 2xx, start sending
content, and in case of internal errors could at least signal that
something went fatally wrong

BR, Julian

Tue, 3 Jun 2008, Henrik Nordstrom wrote:

I don't see why as this isn't a guaranteed even if chunking is used in
the hop to the receiver.
>
Chain may be
>
server [Connection: close, no TE] -proxy [Transfer-Encoding: chunked]
-recipient
>
In which case receiving a pro