It may be that you have X authentication enabled.
Assuming you're on a secure network and/or have a firewall blocking connections from unauthorised places, try putting
xhost +
In some script which gets run after X server startup (e.g. .Xsession)
This will disable X authentication, allowing a connection from anyone, anywhere (provided they can get past the firewall).
If it's using a Unix socket, you might also want to look at its permissions (although they should general be everyone-everything). The socket is probably in /tmp/.X11-unix/
Mark