Handle pod specialization success/failure better. Make sure we only retry on connection failure, otherwise fail immediately.