Hi,
The official documentation proposes the following procedure:
To revert to a previous release, uninstall the current SRM server release from the protected and recovery sites,
uninstall the SRM plug-in, and restore the SRM database from the backup you made before you updated the
SRM server. You can then install the previous release and use the restored database.
You can find more info on page 40 in the admin guide.
IMO your plan should work. But you probably will have to run SRM installer in "Modify" mode (same page 40 in the admin guide) to repair connection to vCenter.
Michael.