This was an issue with a feature "Enable Element Adoption" that is used to adopt wireless Unifi devices. Disabling this feature in the Advanced Wi-FI Settings in Unifi reprovisions the Access Points, and they remain connected after that. There's a bugfix for this in firmware 1.7.2 for the UDM Pro that would allow us to re-enable it.