RSS feed having encoding problems with invalid tokens

sneakyjim

Hi,

I was wondering if someeone could help me with a problem i've been stuck on for ages and haven't been able to find the answer to.

I'm trying to syndicate my search engine results with RSS and I have it all setup producing the rss file but when i try to validate the feed using syndic8 or any of the other online validators I get the following errors (I think they're both related).

Error: description contains bad characters.
On Line: Corporate Clients \x95 To provide ongoing administrati....

AND,

Error: not well-formed (invalid token):
On Line: possible start date negotiable

the \x95 and the are giving me the trouble. How can I strip these characters out of the string variables in the PHP code? I've tried str_replace but then I just get the next one along, \x96 giving me trouble, and i don't want to have a replace statement for each code as I don't know where it'll end and what other problem characters I haven't discovered yet.

I think it's to do with encoding but i don't know what.

I've tried utf_encode, utf_decode, htmlentities, htmlspecialchars and i've tried putting the data inside a CDATA block, none of these work. And i've tried changing the <?xml version="1.0" encoding="iso-8859-1" ?> to all the different encoding types.

I would be very grateful for any help.

Thanks

Kerry

sneakyjim

I've now solved the \x95 problem.

The data in my SE comes from a visual basic app. first so I added this function to the app. http://www.devx.com/vb2themax/Tip/19010 and that seems to strip out the problems, unfortunately it also strips out some useful characters like pound signs but it works.

I still have the problem with the little squares, I thought these would have been removed for being extended ASCII. But they're not, and this doesn't work either.

$description = str_replace("", " ", $description);

Does anyone know how to remove these little squares?

Thanks, Kerry

sneakyjim

Well I think at this stage i'm talking to myself but here's my final solution for anyone who's interested or happens to be stumbling across this in google a few years down the line. It's a VBScript function which i've modified (original code linked above). But i think it should be fairly easy to convert to PHP.

My only worry is i've replaced all ascii characters below 32 with a space, these all seem to be carriage returns, device controls. tab, backspace, bell etc.

Maybe a niave question but I don't need these do I? If my data is stored in a field in MySQL and I just want it to be one long continous paragraph of text.

Anyway, here's the code.

Function StripExtendedASCII(source As String) As String
    Dim Index As Long
    Dim bytes() As Byte

' the fastest way to process this string
' is copy it into an array of Bytes
bytes() = source
For Index = 0 To UBound(bytes) Step 2
    ' if this is an extended ASCII character, although 19 is the square so remove that and anything in ascii under 32
    'is tabs etc.
    If bytes(Index) > 127 Or bytes(Index + 1) <> 0 _
        Or bytes(Index) < 32 Then
        'we want to keep the £ sign
        If bytes(Index) <> 163 Then
            ' convert this char to Chr$(255)
            bytes(Index) = 255
            bytes(Index + 1) = 0
        End If
    End If
Next

' return this string, after filtering out all Chr$(255) chars
StripExtendedASCII = Replace(bytes(), Chr$(255), " ")

End Function