I am not sure specifically what the differences are. However, I can say that I think it is a bad idea to change technologies without know exactly why you are changing. For instance, if there is something specific you are trying to accomplish, PHP/MySQL can't pull it off, and JSP can, then that's a good reason to change.
If you have "heard good things" about JSP and are changing because you are trying to stay on top of current trends... DON'T DO IT! You are just asking for trouble.
Bottom line: if PHP/MySQL is doing everything you need it to, and the system is working fine, leave it alone. You mother was right -- "if you pick at it, you'll just make it worse!"