This was horrible to narrow down, but:

1. I'm doing a POST to a HTTPS url
2. Some of my headers containg iso-8859-1 data
3. The body is sent with transfer-encoding: chunked
4. the "is_utf8" bit was set on the data (although it happens to be
   all in code points < 256).

(changing *any* of these conditions makes the bug go away).

The request headers get corrupted, sent in utf-8 instead of iso-8859-1

some of the data doesn't get sent, messing up the chunked counts, or
even trashing the request headers.

The number of missing bytes seems related to the difference in length
between the iso-8859-1 headers and the incorrect utf-8 versions.

For example my request should look like:
1
®
0
----

But it is sent as:

----

Here's my test program:

----
#! /usr/bin/perl

use strict;
use LWP::UserAgent;

my $agent = LWP::UserAgent->new (agent => 'LWP UTF8 BUG');

# Bug only happens if https
my $req = HTTP::Request->new (POST => 'https://localhost:4433');

# Bug only happens if utf8 bit is set on data to be written
my $body = substr ("\x{f00f}\xae", 1, 1);

print "utf8 bit set\n" if utf8::is_utf8($body);

# Bug only happens with chunked content
my $read_body = sub {
	my $buf = $body;
	$body = "";
	$buf
};

$req->content ($read_body);

# Bug only happens if header with iso-8859-1 data
$req->header (Subject => "\xae" x 12);

my $ret = $agent->request ($req);

# Request sent is malformed - iso-8859-1 data sent as utf-8 and
# bytes missing from output (number of bytes missing equal to
# difference in length between iso-8859-1 and utf-8 representations.
---

found 745823 6.06-1
thanks

Interesting. I can reproduce this on (mostly current) sid with
libwww-perl 6.06-1.

[...]

Quoting HTTP::Request documentation:

     $r->content( $bytes )
           This is used to get/set the content and it is inherited from
           the "HTTP::Message" base class.  See HTTP::Message for details
           and other methods that can be used to access the content.

           Note that the content should be a string of bytes.  Strings in
           perl can contain characters outside the range of a byte.
           The "Encode" module can be used to turn such strings into a
           string of bytes.

So this is not totally unexpected, but the particular failure mode you've
run into is certainly rather horrible.

Possibly the content() method should croak when the UTF8 bit is set?
(I suppose it can't encode the string automatically as it doesn't know
which encoding should be used.)

My pleasure.

Ah, I was going to test that Monday :-)
Interestingly in the my case, although the UTF8 bit is set, the data is
all code points below 256.  In fact the first time i ran into the bug
the data was "XXX".  (Read from a file with "binmode :utf8" on).

Maybe something like

     if (utf8::is_utf8($data)) {
         eval {
             utf8::downgrade ($data);
         };
         croak "content not bytes" if $@;
     }

That's ridiculously over the top.

We could just unconditionaly call "utf8::downgrade ($data);"

Interestingly there is code in HTTP::Message that does that, but we're
going from LWP::Protocol::http::request to
LWP::Protocol::https::Socket->syswrite.

This fixes it for me.

--- /usr/share/perl5/LWP/Protocol/http.pm	2010-01-22 22:44:52.000000000 +0100
+++ /usr/local/share/perl/5.10.1/LWP/Protocol/http.pm	2014-04-26 18:45:56.000000000 +0200
@@ -240,6 +240,7 @@
  	if (ref($content_ref) eq 'CODE') {
  	    my $buf = &$content_ref();
  	    $buf = "" unless defined($buf);
+	    utf8::downgrade ($buf);
  	    $buf = sprintf "%x%s%s%s", length($buf), $CRLF, $buf, $CRLF
  		if $chunked;
  	    substr($buf, 0, 0) = $req_buf if $req_buf

But that's the wrong place to fix it.  The bug is realy in
$socket->syswrite, aka Crypt::SSLeay::Conn::write.

That's where the bug should be fixed.

This patch fixes it for me.