Recently I did work on a project to upgrade vRealize Automation from 7.0.1 to 7.3.
I would like to share my experience on how the upgrade took place and various issues encountered while implementing it
We had to execute upgrade in two phases
Upgrade from 7.0.1 to 7.2
Once 7.2 validation is complete upgrade to 7.3
Pre-Requisites
Ensure Databases have been backed up. Both IaaS ( Contact your DBA ) and vRA Postgres database.
Take snapshots of all virtual machines ( vRA appliances and IaaS machines ).
Ensure vRA components are in working and stable condition and correct all issues before starting upgrade
Check for any space issues on root partition of appliances
Check the /storage/log subfolder and remove any old archived ZIP files to cleanup space
Ensure you have access to database , if you have restricted access then ensure DBA is around to help you during situation where you need to revert during failure
System must be unavailable for users to submit new request and any applications that query vRealize Automation
Upgrade from 7.0.1 to 7.2
Appliance Upgrade
Create a file called disable-iaas-upgrade under /tmp on all vRA appliances as we do not want to perform an automated IaaS upgrade.
/tmp/disable-iaas-upgrade
Mount 7.2 vRA iso images to appliances (VMware-vR-Appliance-7.2.0.381-4660246-updaterepo.iso)
Ensure under VAMI portal , Update Appliance settings are pointing towards CDRom updates
Click on "Check Updates" , then you would get information on existing version and the version your upgrading to , accept and proceed for upgrade
You may use following logs to see how upgrade is progressing
/opt/vmware/var/log/vami/vami.log
/opt/vmware/var/log/vami/updatecli.log
/var/log/messages
Once upgrade is complete, VAMI asks you to reboot appliances once done vRA appliances are upgraded to 7.2
IaaS Upgrade
Ensure Java version is Java 8 build 91 or above
Download IaaS installer from vRA VAMI ( https://vrapphostname:5480/installer)
Execute IaaS install on all the IaaS nodes starting with
IaaS Websites starting with the one where Model Manager Data is installed and then going ahead with other nodes
Manager Services , ensure active one is upgraded before the passive node
DEM Orchestrator and workers. Ensure you perform one box after another
Agents , one node after another
Management Agents on all nodes ( automatically upgraded ) , no manual intervention needed
Check cluster status
rabbitmqctl cluster_status
Ensure all services are in "REGISTERED" status under VAMI portal of vRA nodes
Upgrade vRO appliances if it's external
Issues encountered during upgrade to 7.2
While starting vRA upgrade by mounting iso image during post upgrade phase of postgresql on appliances we did encounter failure
Exception
+ echo 'Script /etc/bootstrap/postupdate.d/09-90-prepare-psql failed, error status 1'
+ exit 1
+ exit 1
+ trapfunc
+ excode=1
+ test 1 -gt 0
+ vami_update_msg set post-install 'Post-install: failed'
+ test -x /usr/sbin/vami-update-msg
+ /usr/sbin/vami-update-msg set post-install 'Post-install: failed'
+ sleep 1
+ test 1 -gt 0 -o 0 -gt 0
+ vami_update_msg set update-status 'Update failed (code 0-1). Check logs in /opt/vmware/var/log/vami or retry update later.'
+ test -x /usr/sbin/vami-update-msg
+ /usr/sbin/vami-update-msg set update-status 'Update failed (code 0-1). Check logs in /opt/vmware/var/log/vami or retry update later.'
+ exit
21/10/2017 01:12:41 [ERROR] Failed with exit code 256
21/10/2017 01:12:41 [INFO] Update status: Running VMware tools reconfiguration
21/10/2017 01:12:41 [INFO] Running /opt/vmware/share/vami/vami_reconfigure_tools
vmware-toolbox-cmd is /usr/bin/vmware-toolbox-cmd
Configuring VAMI VMware tools service wrapper.
21/10/2017 01:12:41 [INFO] Update status: Done VMware tools reconfiguration
21/10/2017 01:12:41 [INFO] Update status: Error while running post-install scripts
21/10/2017 01:12:41 [ERROR] Failure: updatecli exiting abnormally
21/10/2017 01:12:41 [INFO] Install Finished
Resolution
Ensure Clustering is working as expected between Primary and Secondary vRA nodes
Exception
Model Manager Data upgrade fails with an error stating
[21/10/2017 8:09:09 PM] "E:\Program Files (x86)\VMware\vCAC\Server\Model Manager Data\RepoUtil.exe" Model-Uninstall -f DynamicOps.ManagementModel.dll -v
[21/10/2017 8:09:11 PM] System.InvalidOperationException: File DynamicOps.ManagementModel.dll not found.
Upgrade from 7.2 to 7.3
Resolution
When above failure occurred , we had to revert IaaS node where Model Manager Data was and then the SQL server which was hosting the database
Once done , we had to uninstall and reinstall MSDTC
Post these steps , restarted upgrade which helped in fixing upgrade failure on IaaS
Upgrade from 7.2 to 7.3
Appliance Upgrade
Same steps as the previous upgrade to 7.2. Only difference is that when you start upgrading primary vRA node it automatically upgrades the second one as well.
IaaS Upgrade
Same steps as previous upgrade to 7.2
Issues encountered during upgrade to 7.3
Exception
Though vRA upgrade was successful post reboot of the nodes or appliances apart from component registry , none of the services were showing up as registered
Browsing through logs we did find out that horizon.log under vRA appliance ( Primary ) was full of postgres errors
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [spring/datastore-wireup.xml]: Invocation of init method failed; nested exception is liquibase.exception.MigrationFailedException: Migration failed for change set changelog-0008-2015-H2.xml::1::HHW-58146:
Reason: liquibase.exception.DatabaseException: Error executing SQL ALTER TABLE saas."EncryptionKeys" DROP CONSTRAINT encryptionkeys_pkey: ERROR: constraint "encryptionkeys_pkey" of relation "EncryptionKeys" does not exist:
Caused By: Error executing SQL ALTER TABLE saas."EncryptionKeys" DROP CONSTRAINT encryptionkeys_pkey: ERROR: constraint "encryptionkeys_pkey" of relation "EncryptionKeys" does not exist:
Caused By: ERROR: constraint "encryptionkeys_pkey" of relation "EncryptionKeys" does not exist
2017-10-22 01:41:43,410 ERROR (localhost-startStop-1) [;;;] org.springframework.web.context.ContextLoader - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [spring/datastore-wireup.xml]: Invocation of init method failed; nested exception is liquibase.exception.MigrationFailedException: Migration failed for change set changelog-0008-2015-H2.xml::1::HHW-58146:
Reason: liquibase.exception.DatabaseException: Error executing SQL ALTER TABLE saas."EncryptionKeys" DROP CONSTRAINT encryptionkeys_pkey: ERROR: constraint "encryptionkeys_pkey" of relation "EncryptionKeys" does not exist:
Caused By: Error executing SQL ALTER TABLE saas."EncryptionKeys" DROP CONSTRAINT encryptionkeys_pkey: ERROR: constraint "encryptionkeys_pkey" of relation "EncryptionKeys" does not exist:
Caused By: ERROR: constraint "encryptionkeys_pkey" of relation "EncryptionKeys" does not exist
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1628)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
Resolution
If you encounter this issue it does not mean that your upgrade failed. After the upgrade there are new liquibase files which are downloaded who performs DB changes if any between versions. The reason of failure is because CONSTRAINT was not present
Check details below
Login into vRA database
set schema 'saas'
\x to set expanded display ON
\d+ "EncrytionKeys"
Output of above commands would be
Before you create a constraint you need to verify if there are duplicate key id's. If yes , then your alter command will fail with following error
ALTER TABLE "EncryptionKeys" ADD CONSTRAINT "encryptionkeys_pkey" PRIMARY KEY(id);
ERROR: could not create unique index "encryptionkeys_pkey"
DETAIL: Key (id)=(1) is duplicated.
So now identify number of id's present in this table. In my case there were 8 where first two had same key ID as described below
id | 1
uuid | 6e4c6867-377e-4ce8-9f76-102da90951a7
keyContainer | __GLOBAL_REST_KEY_CONTAINER__
algorithmId | bbdec4a9-dbaa-49bd-95ed-11fd6cff26f3
keySize | 1024
id | 1
uuid | 6d1d29ae-4bb6-487e-a902-2ddb798f9c3c
keyContainer | VSPHERE.LOCAL
algorithmId | 1f932d40-0204-11e2-a21f-0800200c9a66
keySize | 256
Observe carefully id number is same. Select a number for ID which is not present in the output from previous command and then change the value of key id
Select UUID to change the Key
select id,uuid from "EncryptionKeys" where uuid = '6e4c6867-377e-4ce8-9f76-102da90951a7';
Remember the value of uuid will vary from environment to environment
Update UUID with the new key
update saas."EncryptionKeys" set id=8 where uuid='6e4c6867-377e-4ce8-9f76-102da90951a7';
Verify if the new ID is updated
select id,uuid from "EncryptionKeys" where uuid = '6e4c6867-377e-4ce8-9f76-102da90951a7';
Once done now run the Alter table command to create constraint for the table
ALTER TABLE "EncryptionKeys" ADD CONSTRAINT "encryptionkeys_pkey" PRIMARY KEY(id);
Once done verify this using \d+ "EncryptionKeys" command , you would see now the Primary Key is present.
Restart vRA services so that all services will now be shown as "REGISTERED" successfully on both the nodes if distributed and on single node if simple installation
This issue is pre-dominantly seen on version upgrades from 7.0 and 7.0.1 to 7.3
Happy Upgrades