Restarting the agents will not cause a problem. You will see the hosts drops on vcenter then reconnect once the vpxa re-establishes connectivity.
I would look to find out how Veaam was originally setup, Was it setup on host one or through vcenter.
Are there enough licenses for Veaam, I believe it is licensed by cpu, are those cpu's on host 1 only.
Start by moving the jobs to vcenter if they are running against host 1.