Since you're dividing by seconds in a year, I assume you must have put in todays date x number of years ago. If that's the case, the only thing I can think of is specifiying the timezone:
date_default_timezone_set('Europe/Dublin');
If that doesn't work (not always 365 days as you've pointed out) you could just take the difference and get the how many years ahead of 1970 and then subtract 1970 (there has to be a more elegant way!)
$age_years = date("Y", $diff ) - 1970;