The main hassle is that if you want to match a literal backslash you end up needing to escape it for both PHP and PCRE - especially if it's followed by another PCRE-significant character. So [/b] becomes \ so that PCRE recognises it as a literal backslash instead of an escape; but PHP's string parsing will see that as an escaped single backslash and only pass one of them on to PCRE - unless you escape them. Upshot is that you end up with \\.
Some of your parentheses (the outer pairs in both expressions) are redundant; the inner pairs are probably unnecessary also.
Ergo,
$matches = array('/\{\\\\(.+?)\}/','/\\\\(.+?)\b/');
Another ways around it would be to use chr(92) to sneak the backslash past PHP; using \x5c instead; or using \Q..\E to get PCRE to treat stuff between them literally (no metacharacter behaviour).
The other thing that's likely to be causing you grief (at least in what is posted) is that $rtf is double-quoted - so PHP will be parsing all those \r and \t sequences before the string even gets into the variable.
Maybe a more format-aware parser might be simpler to use (e.g., that {\fonttbl{...}{...}} can go, I'm guessing); I'm too rusty with rtf, but it occurs to me that using preg_replace_callback to match a more straightforward pattern that gets refined further in the callback function might be something worth tackling.