I've never used fancybox so I don't know how it works. But if you use something like Firefox's "Firebug" plugin or Safari's built in equivalent (IE has one, but it sucked in IE8 and I havn't used 9), you can rclick an element and choose "inspect" or something like that, and it will show you a tree structure of the underlying html code, including modifications made to it by javascript. Come to think of it, there might even be documentation to read to find out these things.
Either way once you know the structure, you can choose an appropriate way of targeting the inner fancybox. For example, if fancyboxes set their class attribute to "fancybox", you could
$('.fancybox .fancybox').close();
assuming close works on a set of matched elements (in this case the set only contains one element, but it may or may not work), else
$('.fancybox .fancybox').first().close();