good question..
first imagine four circles at each corner of the picture, for example the upper left circle's top border is at Y=0 (seen from the top) and its left border is touching X=0, and the rest of the circles positioned symmetrically,
you could use an algorithm for checking if current pixel is in the part between the circles and corresponding corner (blue in attached picture) but the pictures wouldnt be antialiased this way easily..
a perhaps more elegant way would be to have a black and white alpha channel (it has to be the same size of the image, so this is cool if you have a standard size on your images)
and then by comparing pict1[x][y] with alpha[x][y] and seeing if it's black or white or perhaps inbetween you can filter out parts you dont want, and even make that channel in an easy program like photoshop so the shapes can be as irregular as you want..
try hard enough and you you'll make this antialiased !...
oz
edit: better pic