Pre-Requisite
Ensure vRSLCM 8.10 PSPACK 1 is implemented
Once done , download / upload the product binary which will enable you to upgrade vRLI to 8.10
vRSLCM 8.10 PSPACK 1 also allows you to upgrade vRNI to 6.8 , we will discuss that in the next blog
Click here for vRSLCM 8.10 PSPACK 1 Release Notes
Environment consists of vRLI 8.8.2
Upgrading this version to 8.10
Trigger Inventory Sync
Check the binary or the pak file
Take Snapshot
Pre-checks would run in the background
Pre-checks complete
TLS Check is performed
SSH Check is performed
Valid IP/FQDN check is performed
Cluster setup check for vRLI
Disk space check on the root filesystemSubmit the request
It's a single node vRealize LogInsight environment , hence there are 8 stages. Let's discuss each one of them in detail
Stage:1 Shutting Down Guest OS
Stage:2 Creating Node Snapshot
Stage:3 Power On VM
Stage:4 vrlihealthcheck
Stage:5 Create snapshot inventory
Stage:6 Start vRealize LogInsight Generic Task
Stage:7 deleteNodeSnapshot
Stage:8 productupgradeinventoryupdateLooking at Stage 6 in detail and what really happens in the background
Reference: vmware_vrlcm.log
### Genetic Task is Initiated ###
2022-10-28 13:28:37.695 INFO [pool-3-thread-42] c.v.v.l.p.v.StartGenericVRLIInstallTask - -- Starting :: Start vRLI Generic Task
2022-10-28 13:28:37.695 INFO [pool-3-thread-42] c.v.v.l.p.a.s.Task - -- Injecting Edge :: OnGenericVrliTaskInitialized
### Upgrade Task is triggered , pak file is identified ###
2022-10-28 13:28:38.265 INFO [pool-3-thread-13] c.v.v.l.p.v.VrliUpgradeTask - -- Starting :: vRLI upgrade task
2022-10-28 13:28:38.266 INFO [pool-3-thread-13] c.v.v.l.d.v.InstallConfigureVRLI - -- Checking if vRLI instance is running
2022-10-28 13:28:38.349 INFO [pool-3-thread-13] c.v.v.l.d.v.InstallConfigureVRLI - -- The vRLI instance https://10.109.44.140 service is running
2022-10-28 13:28:39.234 INFO [pool-3-thread-13] c.v.v.l.d.v.InstallConfigureVRLI - -- Return message for vrli: {"releaseName":"GA","version":"8.8.2-20056468"}
2022-10-28 13:28:39.238 INFO [pool-3-thread-13] c.v.v.l.d.v.InstallConfigureVRLI - -- Return status code for vrli: 200
2022-10-28 13:28:39.240 INFO [pool-3-thread-13] c.v.v.l.p.v.VrliUpgradeTask - -- Current vRLI version is: 8.8.2-20056468
2022-10-28 13:28:39.241 INFO [pool-3-thread-13] c.v.v.l.p.v.VrliUpgradeTask - -- Current vRLI version without build is: 8.8.2
2022-10-28 13:28:39.241 INFO [pool-3-thread-13] c.v.v.l.p.v.VrliUpgradeTask - -- vRLI upgrade spec properties {product=vrli, productId=vrli, masterNodeIP=10.109.44.140, takeSnapshot=true, snapshotWithMemory=false, vCenterHost=vc.cap.org, snapshotNamePrefix=VRLCM_AUTOGENERATED_2735afc8-a188-4779-a58f-50cd3997aa00, version=8.10.0, vrliAdminPassword=KXKXKXKX, isRetainSnapshot=false, snapshotPrefix=VRLCM_AUTOGENERATED_2735afc8-a188-4779-a58f-50cd3997aa00, environmentId=cf8ac4ce-a7a7-4958-8401-50efdf4f1489, environmentName=Production, vcUsername=vrsvc@cap.org, snapshotWithShutdown=true, tenantId=, vrliHostName=li.cap.org, repositoryType=lcmrepository, vrliUpgradePakUrl=http://lcm.cap.org/repo/productBinariesRepo/vrli/8.10.0/upgrade/VMware-vRealize-Log-Insight-8.10.0-20623770.pak, quiesceSnapshot=false, isVcfUser=false, isVcfEnabledEnv=false, vcPassword=KXKXKXKX, rootPassword=KXKXKXKX
2022-10-28 13:28:39.243 INFO [pool-3-thread-13] c.v.v.l.p.v.VrliUpgradeTask - -- vRLI upgrade version from user input 8.10.0
2022-10-28 13:28:39.245 INFO [pool-3-thread-13] c.v.v.l.c.s.ContentLeaseServiceImpl - -- Inside create content lease.
2022-10-28 13:28:39.246 INFO [pool-3-thread-13] c.v.v.l.c.s.ContentLeaseServiceImpl - -- Created lease for the folder with id :: 564ae968-e032-4c87-89a6-2c1525f11ebd.
2022-10-28 13:28:39.261 INFO [pool-3-thread-13] c.v.v.l.p.v.VrliUpgradeTask - -- /data/vm-config/symlinkdir/564ae968-e032-4c87-89a6-2c1525f11ebd
2022-10-28 13:28:39.262 INFO [pool-3-thread-13] c.v.v.l.p.v.VrliUpgradeTask - -- Started Downloading Content Repo
2022-10-28 13:28:39.263 INFO [pool-3-thread-13] c.v.v.l.c.c.ContentDownloadController - -- INSIDE ContentDownloadControllerImpl
2022-10-28 13:28:39.263 INFO [pool-3-thread-13] c.v.v.l.c.c.ContentDownloadController - -- REPO_NAME :: /productBinariesRepo
2022-10-28 13:28:39.264 INFO [pool-3-thread-13] c.v.v.l.c.c.ContentDownloadController - -- CONTENT_PATH :: /vrli/8.10.0/upgrade/VMware-vRealize-Log-Insight-8.10.0-20623770.pak
2022-10-28 13:28:39.264 INFO [pool-3-thread-13] c.v.v.l.c.c.ContentDownloadController - -- URL :: /productBinariesRepo/vrli/8.10.0/upgrade/VMware-vRealize-Log-Insight-8.10.0-20623770.pak
2022-10-28 13:28:39.264 INFO [pool-3-thread-13] c.v.v.l.c.c.ContentDownloadController - -- Decoded URL :: /productBinariesRepo/vrli/8.10.0/upgrade/VMware-vRealize-Log-Insight-8.10.0-20623770.pak
2022-10-28 13:28:39.272 INFO [pool-3-thread-13] c.v.v.l.c.c.ContentDownloadController - -- ContentDTO{BaseDTO{vmid='86af9699-d28e-42b5-ac2b-ff2c32f5fab8', version=8.1.0.0} -> repoName='productBinariesRepo', contentState='PUBLISHED', url='/productBinariesRepo/vrli/8.10.0/upgrade/VMware-vRealize-Log-Insight-8.10.0-20623770.pak'}
2022-10-28 13:28:39.350 INFO [pool-3-thread-13] c.v.v.l.p.v.VrliUpgradeTask - -- Completed Downloading Content Repo and starting InputStream
2022-10-28 13:28:39.351 INFO [pool-3-thread-13] c.v.v.l.p.v.VrliUpgradeTask - -- /data/vm-config/symlinkdir/564ae968-e032-4c87-89a6-2c1525f11ebd/VMware-vRealize-Log-Insight-8.10.0-20623770.pak
### pak file is uploaded to vRLI ###
2022-10-28 13:29:32.013 INFO [pool-3-thread-13] c.v.v.l.d.v.InstallConfigureVRLI - -- Proceeding with manual copy of upgrade pak using ssh.
2022-10-28 13:29:32.141 INFO [pool-3-thread-13] c.v.v.l.u.SshUtils - -- Uploading file --> ssh://root@10.109.44.140/tmp
2022-10-28 13:29:50.961 INFO [pool-3-thread-13] c.v.v.l.u.SshUtils - -- Uploaded file sucessfully
2022-10-28 13:29:52.652 INFO [pool-3-thread-13] c.v.v.l.d.v.InstallConfigureVRLI - -- Return status code for vrli: 200
2022-10-28 13:29:52.654 INFO [pool-3-thread-13] c.v.v.l.d.v.InstallConfigureVRLI - -- pak file upgrade version: 8.10.0-20623770
2022-10-28 13:29:52.655 INFO [pool-3-thread-13] c.v.v.l.p.v.VrliUpgradeTask - -- vRLI upgrade version from pak file: 8.10.0-20623770
### Upgrade is now triggered ###
2022-10-28 13:29:53.108 INFO [pool-3-thread-13] c.v.v.l.p.v.VrliUpgradeTask - -- Pak was loaded successfully. Eula pending. Triggerring upgrade
2022-10-28 13:29:53.109 INFO [pool-3-thread-13] c.v.v.l.d.v.InstallConfigureVRLI - -- pak file successfully uploaded for version: 8.10.0-20623770. Triggerring the upgrade now
2022-10-28 13:29:53.421 INFO [pool-3-thread-13] c.v.v.l.d.v.InstallConfigureVRLI - -- Upgrade triggered. Status is: Upgrading
2022-10-28 13:29:53.421 INFO [pool-3-thread-13] c.v.v.l.d.v.InstallConfigureVRLI - -- Upgrade status: Upgrading. Waiting for upgrade to finish
2022-10-28 13:29:53.421 INFO [pool-3-thread-13] c.v.v.l.d.v.InstallConfigureVRLI - -- vRLI is not up. sleeping for 3 min
Reference: /storage/var/loginsight/upgrade.log
( At this point in time switch over to vRLI logs )
### checks are performed on disk space and certificates ###
2022-10-28 13:29:51,927 loginsight-upgrade INFO Certificate verified: VMware-vRealize-Log-Insight.cert: C = US, ST = California, L = Palo Alto, O = "VMware, Inc."
error 18 at 0 depth lookup:self signed certificate
OK
2022-10-28 13:29:51,951 loginsight-upgrade INFO Signature of the manifest validated: Verified OK
2022-10-28 13:29:52,559 loginsight-upgrade INFO Current version is 8.8.2-20056468 and upgrade version is 8.10.0-20623770. Version Check successful!
2022-10-28 13:29:52,560 loginsight-upgrade INFO Available Disk Space at /tmp: 3440889856
2022-10-28 13:29:52,560 loginsight-upgrade INFO Disk Space Check successful!
2022-10-28 13:29:52,560 loginsight-upgrade INFO Available Disk Space at /storage/core: 16030388224
2022-10-28 13:29:52,560 loginsight-upgrade INFO Disk Space Check successful!
2022-10-28 13:29:52,560 loginsight-upgrade INFO Available Disk Space at /storage/var: 11989639168
2022-10-28 13:29:52,560 loginsight-upgrade INFO Disk Space Check successful!
2022-10-28 13:29:52,561 loginsight-upgrade INFO Loading eula license successful!
2022-10-28 13:29:52,561 loginsight-upgrade INFO Done!
2022-10-28 13:29:53,564 loginsight-upgrade INFO Certificate verified: VMware-vRealize-Log-Insight.cert: C = US, ST = California, L = Palo Alto, O = "VMware, Inc."
error 18 at 0 depth lookup:self signed certificate
OK
### Version Check is done ###
2022-10-28 13:29:53,579 loginsight-upgrade INFO Signature of the manifest validated: Verified OK
2022-10-28 13:29:53,805 loginsight-upgrade INFO Current version is 8.8.2-20056468 and upgrade version is 8.10.0-20623770. Version Check successful!
2022-10-28 13:29:53,806 loginsight-upgrade INFO Available Disk Space at /tmp: 3440914432
2022-10-28 13:29:53,806 loginsight-upgrade INFO Disk Space Check successful!
2022-10-28 13:29:53,806 loginsight-upgrade INFO Available Disk Space at /storage/core: 16030560256
2022-10-28 13:29:53,806 loginsight-upgrade INFO Disk Space Check successful!
2022-10-28 13:29:53,807 loginsight-upgrade INFO Available Disk Space at /storage/var: 11989626880
2022-10-28 13:29:53,808 loginsight-upgrade INFO Disk Space Check successful!
2022-10-28 13:30:53,506 loginsight-upgrade INFO Checksum validation successful!
2022-10-28 13:30:53,512 loginsight-upgrade INFO Attempting to upgrade to version 8.10.0-20623770
### upgrade-driver script is triggered ###
2022-10-28 13:30:53,925 upgrade-driver INFO Starting 'upgrade-driver' script ...
2022-10-28 13:30:53,927 upgrade-driver INFO Start processing the manifest file ...
2022-10-28 13:30:53,927 upgrade-driver INFO Log Insight TO_VERSION is manifest file is 8.10.0-20623770
2022-10-28 13:30:53,927 upgrade-driver INFO Parsed version is 8.10.0-20623770
2022-10-28 13:30:53,927 upgrade-driver INFO Creating file /storage/core/upgrade-version to store upgrade version.
2022-10-28 13:30:53,943 upgrade-driver INFO The file /storage/core/upgrade-version is created successfully.
2022-10-28 13:31:07,318 upgrade-driver INFO Cassandra snapshot run time: 0:00:13.369487
2022-10-28 13:31:07,714 upgrade-driver INFO Start processing key list ...
2022-10-28 13:31:07,714 upgrade-driver INFO Start processing rpm list ...
2022-10-28 13:31:07,714 upgrade-driver INFO Rpm by name upgrade-image-8.10.0-20623770.rpm
2022-10-28 13:32:34,318 upgrade-driver INFO INFO: Running /storage/core/upgrade/kexec-li - Resize|Partition|Boot
... Starting to run kexec-li script ...
Reading and saving /etc/ssh/sshd_config
Reading and saving old ssh keys if key based Authentication is enabled
cp: cannot stat '/root/.ssh//id_rsa': No such file or directory
cp: cannot stat '/root/.ssh//id_rsa.pub': No such file or directory
cp: cannot stat '/root/.ssh//known_hosts': No such file or directory
Reading and saving /etc/hosts
Reading and saving ssh host keys
Reading and saving /var/lib/loginsight-agent/liagent.ini
Reading and saving hostname
Reading and saving old cassandra keystore
Failed copying /usr/lib/loginsight/application/lib/apache-cassandra-*/conf/keystore*
Reading and saving old default keystore
Reading and saving old default truststore
Reading and saving old tomcat configs
chmod ing /storage/core/upgrade/vmdk-extracted-root/usr/lib/loginsight/application/etc/3rd_config/keystore*
chmod ing /storage/core/upgrade/vmdk-extracted-root/usr/lib/loginsight/application/etc/truststore*
Reading and saving old loginsight.conf
Reading and saving old password in /etc/shadow
Root password info root P 07/29/2022 0 365 7 -1
Root password change date is 07/29/2022
Root password is set. Password reset will not be required on first login.
Reading and saving /etc/fstab
Reading and saving cacerts
Copying java.security to java.security.old
Reading and saving network configs
Reading and saving resolv.conf
Lazy partition is sda5
sda partition count is 5
/storage/core/upgrade/kexec-li script run took 74 seconds
Partition sda5 , which is lazy partition, will be formatted and will become root partition
Photon to Photon upgrade flow will be called, where base OS was Photon
... Starting to run photon2photon script ...
Root partition copy took 84 seconds
clean up upgrade-image.rpm
Removing lock file
/storage/core/upgrade/photon2photon-base-photon.sh script run took 87 seconds
### reboots appliance ###
Rebooting...
### After reboot , from journalcrl logs , we can see following statement stating that the upgrade was successful ###
Oct 28 13:36:17 li.cap.org systemd[1]: Started Mark VMware vRealize Log Insight upgrade successful.
-- Subject: Unit loginsight_mark_upgrade_successful.service has finished start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit loginsight_mark_upgrade_successful.service has finished starting up.
--
-- The start-up result is RESULT.
Oct 28 13:36:17 li.cap.org systemd[1]: Started Cleanup after successful upgrade of VMware vRealize Log Insight.
-- Subject: Unit loginsight_cleanup_after_upgrade.service has finished start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit loginsight_cleanup_after_upgrade.service has finished starting up.
--
-- The start-up result is RESULT.
### cleanup done ###
Oct 28 13:36:27 li.cap.org cleanup_after_upgrade.sh[2043]: Removed /etc/systemd/system/graphical.target.wants/loginsight_cleanup_after_upgrade.service.
Oct 28 13:36:27 li.cap.org cleanup_after_upgrade.sh[2043]: Removed /etc/systemd/system/multi-user.target.wants/loginsight_cleanup_after_upgrade.service.
Upgrade is now complete
Important Logs to Refer
vRSLCM Appliance: /var/log/vrlcm/vmware_vrlcm.log
vRLI Applinace: /storage/var/loginsight/upgrade.log
Comments