$str= preg_replace ('/[start](.*)[end]/i', '<htmltag>\1</htmltag>', $str);
What your regex is saying is this:
Find [ followed by s, t, a, r, t, ] followled by any character (.) of 0 to infi. Followed by [ followed by e, n, d, ].
I think what is happening is that its working correctly! 😉
Try using the ^ and $ before and after [start] and [end] to signify that each of these items are on their own line all alone. Also, maybe you need to tell your regex that (.*) is really on a new line since right now its looking for [start] sdjfhsdkfjhsfj [end].
What you have written for a regex is working exactly how it should be. The issue probably is with (.*) as well since it is saying "hey match one or more to infinity characters between [start] and [end]. Which allows [start][start][end][end] to be valid here.... maybe you need to negate [[start]] and [[end]] so that your final code would be something like:
$str= preg_replace ('/\[start\](.*)[^\[start\]] [^\[end]\]\[end\]/i', '<htmltag>\1</htmltag>', $str);
But I am no regex expert... for real! Plus this is not tested code and I am no pro with PHP regex stuff.