There was a requirement for us to work on implementing HF8 on a environment which was running on HF3. I created a run-book to explain the procedure. Thought of sharing it with everyone as it would be useful.
Pre-Requisites
For successful patch deployments, perform these prerequisite steps on the target vRealize Automation cluster:
Ensure the Service Account running the 'VMware vCloud Automation Center Management Agent' has the following requirements:
Account must be part of the Local Administrator group.
Account must have 'Log on as a service' enabled in the Local Security Policies.
Account must be formatted as domain\username.
Remove old / obsolete nodes from the Distributed Deployment Information Table. For detailed steps, see Remove a Node from the Distributed Deployment Information Table section of vRealize Automation documentation.
Ensure that Management Agent in all IaaS machines is latest (7.4) version.
In vRA Virtual Appliance nodes, open the /etc/hosts file and locate the entry IPv4 loopback IP Address (127.0.0.1). Ensure that the Fully Qualified Domain Name of the node is added immediately after 127.0.0.1 and before localhost. For example, 127.0.0.1 FQDN_HOSTNAME_OF_NODE localhost
Take snapshots/ backups of all nodes in your vRealize Automation installation.
If your environment uses load balancers for HA, disable traffic to secondary nodes and disable service monitoring until after installing or removing patches and all services are showing REGISTERED.
Obtain the files from below and copy it to the file system available to the browser you use for the vRealize Automation appliance management interface.
Files needed to install HF8
Following files are needed to install HF8 on a vRealize Automation 7.4 environment
vRA-7.4-HF8-patch
self-patch.zip
patchscript.sh
All three files are available under KB: 56618
Implementing vRA 7.4 HF8
As a first step , ensure all the pre-requisites are met. These are mandatory and cannot be skipped.
Now , copy self-patch.zip and patchscript.sh under /tmp of the Master or Primary vRealize Automation appliance
Once they are copied , give required permissions to the new file: chmod +x patchscript.sh
Then run the patchscript.sh
After script execution completes , it would throw a message stating Self-Patch successfully applied
Note : **Ensure the prerequisite script has run prior to running the below procedure to implement the actual patch!**
Log in to the vRealize Automation appliance management interface (https://vrealize-automation-appliance-FQDN:5480) as root. This has to be your Primary or Master node if it's a distributed vRA instance
Click vRA Settings > Patches
Under Patch Management, click the option that you need, and follow the prompts to install a new patch
Once you click on INSTALL , it would start implementing the patch
First creates a local patch repo
2019-01-18T05:08:04.157975+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.commands.cluster.patch.util.PatchUtil.getAllEligiblePatchesAndCreatePatchRepo:48 - Creating local patch repo
Identifies it has to install HF8
2019-01-18T05:11:31.584608+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.commands.cluster.patch.PatchDeployCommand.installPatch:129 - Installing the patch 7342927e-7099-4d8a-bc6b-8ca77c5a876b
It starts applying HF8 after extraction. Also identifies that it has a previous patch installed , which we all know it's HF3 with patch ID : 58ec2da5-823b-440e-b918-fbdf6ff7166f
2019-01-18T05:11:33.705636+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.commands.cluster.patch.util.PatchUtil.getPatchLocation:112 - Patch location: /usr/lib/vcac/patches/repo/contents/vRA-patch/45994cb81454cba76ebe347e9e149e3a2253d74f889b5b667d117e438cbac4/patch-vRA-7.4.10980652.10980652-HF8 2019-01-18T05:11:33.884824+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.services.cluster.patch.PatchHistoryRepository.getLastAppliedPatch:249 - The last applied patch 58ec2da5-823b-440e-b918-fbdf6ff7166f 2019-01-18T05:11:33.885757+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.commands.cluster.patch.PatchDeployCommand.applyPatch:258 - Applying the patch 58ec2da5-823b-440e-b918-fbdf6ff7166f-Reverse
This is when you would see that HF3 is being uninstalled
Initiates HF3 Reverse patching
2019-01-18T05:15:05.134068+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.commands.cluster.patch.util.PatchUtil.publishInstallBundlesForDownloading:190 - Created cafe.patch in: 2019-01-18T05:15:05.633471+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.commands.cluster.patch.util.PatchUtil.publishInstallBundlesForDownloading:195 - Created iaas.patch in: 2019-01-18T05:15:05.634676+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.commands.cluster.patch.PatchExecutor.run:147 - 1. Initiate patching... 2019-01-18T05:15:05.634676+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.commands.cluster.patch.PatchExecutor.initiatePatching:226 - Starting :: initiate patching 2019-01-18T05:15:05.738027+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.services.cluster.patch.PatchHistoryRepository.addPatch:75 - Adding patch 58ec2da5-823b-440e-b918-fbdf6ff7166f-Reverse to history::
Identifies nodes where HF3 reverse has to be applied
2019-01-18T05:15:05.975140+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.commands.cluster.patch.util.PatchUtil.getPatchLocation:112 - Patch location: /usr/lib/vcac/patches/repo/contents/vRA-patch/9a909a94eb9cb15199c686e2e29d8fc83ea5fe3460426e340476544b211dc/patch-vRA-7.4.8182598.8182598-HF3-Reverse 2019-01-18T05:15:06.133201+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.services.cluster.patch.PatchHistoryRepository.syncPatchHistory:287 - Queueing command update-patch-history 2019-01-18T05:15:06.190063+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.services.cluster.impl.ClusterNodeServiceImpl.run:332 - Notifying node with hostname [nukesvra01.nukescloud.com] for command process-cmd... 2019-01-18T05:15:06.190344+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.services.cluster.impl.ClusterNodeServiceImpl.run:332 - Notifying node with hostname [nukesvra02.nukescloud.com] for command process-cmd... 2019-01-18T05:15:06.192237+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.services.cluster.impl.ClusterNodeServiceImpl.run:332 - Notifying node with hostname [nukesvra03.nukescloud.com] for command process-cmd...
Starts a thread to stop services on all of the nodes
2019-01-18T05:15:06.314525+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.platform.rest.client.impl.HttpClientFactory$IdleConnectionEvictor.start:370 - Starting thread Thread[Connection evictor-57136495-4bad-4414-a279-000ae3c34a54,5,main] 2019-01-18T05:15:06.637921+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.services.cluster.impl.ClusterNodeCommunicatorImpl.notifyNode:121 - Notifying an existing cluster node with url: [https://nukesvra01.nukescloud.com:5480/config/process-cmd] for configuration changes. 2019-01-18T05:15:06.646167+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.services.cluster.impl.ClusterNodeCommunicatorImpl.notifyNode:121 - Notifying an existing cluster node with url: [https://nukesvra03.nukescloud.com:5480/config/process-cmd] for configuration changes. 2019-01-18T05:15:06.649133+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.services.cluster.impl.ClusterNodeCommunicatorImpl.notifyNode:121 - Notifying an existing cluster node with url: [https://nukesvra02.nukescloud.com:5480/config/process-cmd] for configuration changes.
Finishes patch iniitiation
2019-01-18T05:15:23.913716+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.services.cluster.patch.PatchHistoryRepository.currentPatch:115 - The current patch is 58ec2da5-823b-440e-b918-fbdf6ff7166f-Reverse 2019-01-18T05:15:23.913716+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.commands.cluster.patch.PatchExecutor.initiatePatching:242 - Finished :: Initiate patching
Starts patch discovery
2019-01-18T05:15:23.913738+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.commands.cluster.patch.PatchExecutor.run:153 - 2. Patch discovery... 2019-01-18T05:15:23.913738+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.commands.cluster.patch.PatchExecutor.discovery:247 - Starting :: component discovery 2019-01-18T05:15:23.924128+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.commands.cluster.patch.PatchExecutor.isAllCommandExecuted:775 - Checking if all commands are executed 2019-01-18T05:15:38.925206+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.commands.cluster.patch.PatchExecutor.validateCommandStatusForFinishLine:788 - Starting:: Command validation 2019-01-18T05:15:38.935453+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.commands.cluster.patch.PatchExecutor.validateCommandStatusForFinishLine:793 - Command status for update-patch-history: COMPLETED
Finishes HF3 reverse patch installation
2019-01-18T05:58:04.331648+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.commands.cluster.patch.PatchExecutor.finalizePatch:739 - Starting :: Finlaize patch 2019-01-18T05:58:04.331758+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.services.cluster.patch.PatchHistoryRepository.currentPatch:115 - The current patch is 58ec2da5-823b-440e-b918-fbdf6ff7166f-Reverse 2019-01-18T05:58:04.336476+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.services.cluster.patch.PatchHistoryRepository.finishPatch:122 - Finishing patch :: 58ec2da5-823b-440e-b918-fbdf6ff7166f-Reverse
As it finished uninstalling HF3 , now it starts installing HF8
2019-01-18T05:58:34.296019+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.commands.cluster.patch.PatchDeployCommand.applyPatch:258 - Applying the patch 7342927e-7099-4d8a-bc6b-8ca77c5a876b 2019-01-18T06:01:49.098522+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.commands.cluster.patch.PatchExecutor.run:147 - 1. Initiate patching... 2019-01-18T06:01:49.098920+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.commands.cluster.patch.PatchExecutor.initiatePatching:226 - Starting :: initiate patching 2019-01-18T06:01:49.175530+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.services.cluster.patch.PatchHistoryRepository.addPatch:75 - Adding patch 7342927e-7099-4d8a-bc6b-8ca77c5a876b to history:: 2019-01-18T06:01:49.829562+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.commands.cluster.patch.util.PatchUtil.getPatchLocation:112 - Patch location: /usr/lib/vcac/patches/repo/contents/vRA-patch/45994cb81454cba76ebe347e9e149e3a2253d74f889b5b667d117e438cbac4/patch-vRA-7.4.10980652.10980652-HF8 2019-01-18T06:02:07.223205+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.services.cluster.patch.PatchHistoryRepository.currentPatch:115 - The current patch is 7342927e-7099-4d8a-bc6b-8ca77c5a876b
Finalizes HF8 as it's installation is complete
2019-01-18T07:19:58.675075+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.commands.cluster.patch.PatchExecutor.finalizePatch:739 - Starting :: Finlaize patch 2019-01-18T07:19:58.675240+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.services.cluster.patch.PatchHistoryRepository.currentPatch:115 - The current patch is 7342927e-7099-4d8a-bc6b-8ca77c5a876b 2019-01-18T07:19:58.689236+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.services.cluster.patch.PatchHistoryRepository.finishPatch:122 - Finishing patch :: 7342927e-7099-4d8a-bc6b-8ca77c5a876b 2019-01-18T07:19:58.689439+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.cli.configurator.services.cluster.patch.PatchHistoryRepository.finishPatch:173 - Set last applied patch to 7342927e-7099-4d8a-bc6b-8ca77c5a876b
Since application of patch is now finished , it starts the services on all of the nodes
2019-01-18T07:20:05.468675+00:00 nukesvra01.nukescloud.com vcac-config: INFO com.vmware.vcac.platform.rest.client.impl.HttpClientFactory$IdleConnectionEvictor.start:370 - Starting thread Thread[Connection evictor-8a21cda8-1edd-4b4e-85d5-82587ad602f6,5,main]
As a final step , enable secondary nodes on the load balancer and ensure all health-checks on the environment are passed.
