Hi
I was talking about this with a colleague yesterday and we've managed to get it working (thanks Paul).
The SSO over Azure will work fine using either the msappproxy.net or a custom domain, provided that the "Translate Urls in headers" app proxy option (under the Application proxy / Advanced settings) is unchecked.
Having looked at the info for this option, (it's a little wooly IMHO) when it's enabled it will try to translate the AAP URL into the internal URL once the authentication is completed, hence the issue I was seeing.
I still don't understand why the tests were working from the "Test this application" option of the SSO config, but that's an issue for another day.