It seems the problem was the User Agent being set in the request to hidden link.
I have fixed it with this filter:
// The default User Agent is 'WordPress/6.3.1; $SITE_URL'
// This fails if the SITE_URL is hidden link
add_filter('http_headers_useragent', function ($user_agent) {
return 'WordPress/6.3.1';
});