top of page

Experienced Technology Product Manager adept at steering success throughout the entire product lifecycle, from conceptualization to market delivery. Proficient in market analysis, strategic planning, and effective team leadership, utilizing data-driven approaches for ongoing enhancements.

  • Twitter
  • LinkedIn
White Background

VMware Identity Manager Upgrade from 3.3.6 to 3.3.7 | Deepdive |


 

Goal


Upgrade globalenvironment or VMware Identity Manager from version 3.3.6 to 3.3.7


For this demo, we are using vRSLCM 8.10 PSPACK 15
 

Environment

Nodes

Hostname

Load balancer fqdn

vidmlb.cap.org

first node

vidmone.cap.org

second node

vidmtwo.cap.org

third node

vidmthree.cap.org


 

Pre-requisites

  • Take snapshot of VMware Identity Manager 


globalenvironment product properties


take snapshot using "create snapshot" day-2 action of vRSLCM 



takes around 58 minutes to complete the snapshot task


  • Perform Inventory Sync




 

Upgrade


before request submission phase

  • In order to perform an upgrade, we need the upgraderepo downloaded



  • Once downloaded, we will go ahead and trigger upgrade of vIDM from lifecycle operations pane of Suite Lifecycle


  • An inventory sync must be performed here as well in order to proceed



  • Once Inventory Sync is complete we can click on "proceed" to start the process

  • Remember we already have the upgrade repo downloaded so let's point it to the same link

  • You may opt for product snpashot and also to retain the snapshot after an upgrade

  • Clicking on next will ask you to run a precheck 

SSH connectivity
Disk space on / 
Node health
Opened ports 
Root Password expiration
Version check
Sizing check 

are the ones which are verified during precheck


  • Prechecks are successful as shown above


upgrade triggered phase


  • Review the summary before submitting the request and then click on submit to initiate uprgade


  • Upgrade is now triggered


  • Multiple stages during upgrade are shown as below. Remember the timings mentioned below are from my lab, these might vary in your environment.

Stage

Description

Time

1

Graceful Shutdown of VMware Identity Manager

16m 40s

2

Create Node Snapshot

16s

3

Power On VMware Identity Manager Nodes

6m 17s

4

Remediate VMware Identity Manager Nodes

27m 53s

5

Enable VMware Identity Manager Health Notification

3m 46s

6

Create Snapshot Inventory

575ms

7

Disable VMware Identity Manager Health Notification

2s

8

Stop VMware Identity Manager Services before starting an upgrade

2m 19s

9

Stop VMware Identity Manager Services before starting an upgrade

2m 14s

10

Stop VMware Identity Manager Services before starting an upgrade

2m 14s

11

Upgrade VMware Identity Manager Offline

30m 34s

12

Assign delegate IP to Postgres primary node of VMware Identity Manager

2m 11s

13

Upgrade VMware Identity Manager Offline

30m 8s

14

Upgrade VMware Identity Manager Offline

28m 43s

15

Configure pgpool on VMware Identity Manager Secondary Nodes

3m 35s

16

Verify VMware Identity Manager Postgres Clustering

32s

17

Configure VMware Identity Manager pgpool post upgrade

5m 58s

18

Stop Postgres on VMware Identity Manager

4s

19

Remediate VMware Identity Manager Nodes

20m 28s

20

Product Upgrade Inventory Update

582ms

21

Enable Log Rotation of pgpool services

3s

22

Snapshot Inventory Update

530ms

23

Snapshot Inventory Compute

206ms

25

Snapshot Inventory Update

530ms


Total Time Taken for Upgrade

3 hours 8 mins


  • You might see some steps repeating as it has to perform upgrade on 3 VMware Identity Manager nodes one after another

 

vRSLCM logs deep-dive w.r.t vIDM Upgrade


Reference: /var/log/vrlcm/vmware_vrlcm.log

We will specifically look into Stages 13 to 21 where the upgrade and post upgrade steps occur 



//**** Upgrade vIDM Offline state machine ( stage 11 ) **** //

To begin with old updateoffline.hzn is deleted and new one is copied and permissions are changed 


2023-07-22 12:05:26.565 INFO  [pool-3-thread-23] c.v.v.l.v.c.t.VidmUploadUpgradeScriptTask -  -- Starting :: vIDM upload upgrade script Task
2023-07-22 12:05:26.567 INFO  [pool-3-thread-23] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil -  -- vIDM Configuration property vidmUpgradeBasePath value is obtained from Config Service : /usr/local/horizon/update/
2023-07-22 12:05:26.567 INFO  [pool-3-thread-23] c.v.v.l.v.c.t.VidmUploadUpgradeScriptTask -  -- Target version is 3.3.3 and above, loading new offline upgrade script.
2023-07-22 12:05:26.568 INFO  [pool-3-thread-23] c.v.v.l.v.c.t.VidmUploadUpgradeScriptTask -  -- Upgrade script source folder path :: newupgradescript/
2023-07-22 12:05:26.667 INFO  [pool-3-thread-23] c.v.v.l.u.SshUtils -  -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX
2023-07-22 12:05:26.668 INFO  [pool-3-thread-23] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:05:26.668 INFO  [pool-3-thread-23] c.v.v.l.u.SshUtils -  -- Command: ls /usr/local/horizon/update/updateoffline.hzn
2023-07-22 12:05:26.668 INFO  [pool-3-thread-23] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:05:27.260 INFO  [pool-3-thread-23] c.v.v.l.u.SshUtils -  -- exit-status: 0
2023-07-22 12:05:27.260 INFO  [pool-3-thread-23] c.v.v.l.u.SshUtils -  -- Command executed sucessfully
2023-07-22 12:05:27.261 INFO  [pool-3-thread-23] c.v.v.l.u.SshUtils -  -- Command execution response: {
  "exitStatus" : 0,
  "outputData" : "/usr/local/horizon/update/updateoffline.hzn\n",
  "errorData" : null,
  "commandTimedOut" : false
}
2023-07-22 12:05:27.268 INFO  [pool-3-thread-23] c.v.v.l.u.SshUtils -  -- Found file for upload: /tmp/updateoffline.hzn
2023-07-22 12:05:27.393 INFO  [pool-3-thread-23] c.v.v.l.u.SshUtils -  -- Uploading file --> ssh://root@vidmone.cap.org/usr/local/horizon/update/
2023-07-22 12:05:27.740 INFO  [pool-3-thread-23] c.v.v.l.u.SshUtils -  -- Uploaded file sucessfully
2023-07-22 12:05:27.742 INFO  [pool-3-thread-23] c.v.v.l.v.c.t.VidmUploadUpgradeScriptTask -  -- Uploaded file to the location : /tmp/updateoffline.hzn
2023-07-22 12:05:27.935 INFO  [pool-3-thread-23] c.v.v.l.u.SshUtils -  -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX
2023-07-22 12:05:27.937 INFO  [pool-3-thread-23] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:05:27.938 INFO  [pool-3-thread-23] c.v.v.l.u.SshUtils -  -- Command: chmod 500 /usr/local/horizon/update/updateoffline.hzn
2023-07-22 12:05:27.939 INFO  [pool-3-thread-23] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:05:28.520 INFO  [pool-3-thread-23] c.v.v.l.u.SshUtils -  -- exit-status: 0
2023-07-22 12:05:28.520 INFO  [pool-3-thread-23] c.v.v.l.u.SshUtils -  -- Command executed sucessfully
2023-07-22 12:05:28.521 INFO  [pool-3-thread-23] c.v.v.l.u.SshUtils -  -- Command execution response: {
  "exitStatus" : 0,
  "outputData" : "",
  "errorData" : null,
  "commandTimedOut" : false
}
2023-07-22 12:05:28.522 INFO  [pool-3-thread-23] c.v.v.l.v.c.t.VidmUploadUpgradeScriptTask -  -- File: /tmp/updateoffline.hzn deleted successfully.
2023-07-22 12:05:28.612 INFO  [pool-3-thread-23] c.v.v.l.u.SshUtils -  -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX
2023-07-22 12:05:28.613 INFO  [pool-3-thread-23] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:05:28.613 INFO  [pool-3-thread-23] c.v.v.l.u.SshUtils -  -- Command: sed -i '/^ *read$/d' /usr/local/horizon/update/updateoffline.hzn
2023-07-22 12:05:28.613 INFO  [pool-3-thread-23] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:05:29.119 INFO  [http-nio-8080-exec-2] c.v.v.l.s.n.s.NotificationServiceImpl -  -- Authentication object is not null org.springframework.security.authentication.UsernamePasswordAuthenticationToken@fe908ae6: YXYXYXYX org.springframework.security.core.userdetails.User@c220133a: Username: admin@local; Password: YXYXYXYX Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: LCM_ADMIN; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@957e: RemoteIpAddress: 127.0.0.1; SessionId: null; Granted Authorities: LCM_ADMIN
2023-07-22 12:05:29.212 INFO  [pool-3-thread-23] c.v.v.l.u.SshUtils -  -- exit-status: 0
2023-07-22 12:05:29.212 INFO  [pool-3-thread-23] c.v.v.l.u.SshUtils -  -- Command executed sucessfully
2023-07-22 12:05:29.213 INFO  [pool-3-thread-23] c.v.v.l.u.SshUtils -  -- Command execution response: {
  "exitStatus" : 0,
  "outputData" : "",
  "errorData" : null,
  "commandTimedOut" : false
}
2023-07-22 12:05:29.213 INFO  [pool-3-thread-23] c.v.v.l.p.a.s.Task -  -- Injecting Edge :: OnVidmUploadUpgradeScriptCompletion


vIDM bundle upload task is initiated and completed 


2023-07-22 12:05:29.563 INFO  [pool-3-thread-26] c.v.v.l.v.c.t.VidmUploadUpgradeBundleTask -  -- Starting :: vIDM upgrade bundle upload Task
2023-07-22 12:05:29.563 INFO  [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmInstallTaskUtil -  -- VidmInstallTaskUtil.getLeasedLocalFileLocation :: Start
2023-07-22 12:05:29.565 INFO  [pool-3-thread-26] c.v.v.l.c.s.ContentLeaseServiceImpl -  -- Inside create content lease.
2023-07-22 12:05:29.566 INFO  [pool-3-thread-26] c.v.v.l.c.s.ContentLeaseServiceImpl -  -- Created lease for the folder with id :: 8a233ccd-ccf9-4ddb-adff-358c20c20fe1.
2023-07-22 12:05:29.577 INFO  [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmInstallTaskUtil -  -- /data/vm-config/symlinkdir/8a233ccd-ccf9-4ddb-adff-358c20c20fe1
2023-07-22 12:05:29.577 INFO  [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmInstallTaskUtil -  -- Started Downloading Content Repo
2023-07-22 12:05:29.577 INFO  [pool-3-thread-26] c.v.v.l.c.c.ContentDownloadController -  -- INSIDE ContentDownloadControllerImpl
2023-07-22 12:05:29.578 INFO  [pool-3-thread-26] c.v.v.l.c.c.ContentDownloadController -  -- REPO_NAME :: /productBinariesRepo
2023-07-22 12:05:29.578 INFO  [pool-3-thread-26] c.v.v.l.c.c.ContentDownloadController -  -- CONTENT_PATH :: /vidm/3.3.7/upgrade/identity-manager-3.3.7.0-21173100-updaterepo-lcm.tar.gz
2023-07-22 12:05:29.578 INFO  [pool-3-thread-26] c.v.v.l.c.c.ContentDownloadController -  -- URL :: /productBinariesRepo/vidm/3.3.7/upgrade/identity-manager-3.3.7.0-21173100-updaterepo-lcm.tar.gz
2023-07-22 12:05:29.578 INFO  [pool-3-thread-26] c.v.v.l.c.c.ContentDownloadController -  -- Decoded URL :: /productBinariesRepo/vidm/3.3.7/upgrade/identity-manager-3.3.7.0-21173100-updaterepo-lcm.tar.gz
2023-07-22 12:05:29.581 INFO  [pool-3-thread-26] c.v.v.l.c.c.ContentDownloadController -  -- ContentDTO{BaseDTO{vmid='da06426f-5f18-4985-b640-38d711fd597f', version=8.1.0.0} -> repoName='productBinariesRepo', contentState='PUBLISHED', url='/productBinariesRepo/vidm/3.3.7/upgrade/identity-manager-3.3.7.0-21173100-updaterepo-lcm.tar.gz'}
2023-07-22 12:05:29.582 INFO  [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmInstallTaskUtil -  -- Completed Downloading from Content Repo and starting InputStream
2023-07-22 12:05:29.582 INFO  [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmInstallTaskUtil -  -- /data/vm-config/symlinkdir/8a233ccd-ccf9-4ddb-adff-358c20c20fe1/identity-manager-3.3.7.0-21173100-updaterepo-lcm.tar.gz
2023-07-22 12:09:56.628 INFO  [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmInstallTaskUtil -  -- Output stream completed
*
*
*
*
2023-07-22 12:09:56.637 INFO  [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmInstallTaskUtil -  -- VidmInstallTaskUtil.getLeasedLocalFileLocation upgradeFileLocation : /data/vm-config/symlinkdir/8a233ccd-ccf9-4ddb-adff-358c20c20fe1/identity-manager-3.3.7.0-21173100-updaterepo-lcm.tar.gz
2023-07-22 12:09:56.649 INFO  [pool-3-thread-26] c.v.v.l.v.c.t.VidmUploadUpgradeBundleTask -  -- Started :: Copying upgrade bundle to vIDM with hostname: vidmone.cap.org
2023-07-22 12:09:56.652 INFO  [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil -  -- vIDM Configuration property vidmMinDiskspaceFor333UpgradeInKb value is obtained from Config Service : 7340032
2023-07-22 12:09:56.653 INFO  [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil -  -- vIDM Configuration property vidmOptMinDiskspaceFor333UpgradeInKb value is obtained from Config Service : 7
2023-07-22 12:09:56.654 INFO  [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil -  -- vIDM Configuration property vidmExpandDiskspaceFor333UpgradeKb value is obtained from Config Service : https://kb.vmware.com/s/article/81220
2023-07-22 12:09:56.655 INFO  [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil -  -- Checking if required free space of 7340032 GB is available in /var or / partiton in vIDM host vidmone.cap.org
2023-07-22 12:09:56.750 INFO  [pool-3-thread-26] c.v.v.l.u.SshUtils -  -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX
2023-07-22 12:09:56.751 INFO  [pool-3-thread-26] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:09:56.751 INFO  [pool-3-thread-26] c.v.v.l.u.SshUtils -  -- Command: df | grep -w "/" | awk '{print $4}'
2023-07-22 12:09:56.751 INFO  [pool-3-thread-26] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:09:57.340 INFO  [pool-3-thread-26] c.v.v.l.u.SshUtils -  -- exit-status: 0
2023-07-22 12:09:57.340 INFO  [pool-3-thread-26] c.v.v.l.u.SshUtils -  -- Command executed sucessfully
2023-07-22 12:09:57.343 INFO  [pool-3-thread-26] c.v.v.l.u.SshUtils -  -- Command execution response: {
  "exitStatus" : 0,
  "outputData" : "8693676\n",
  "errorData" : null,
  "commandTimedOut" : false
}
2023-07-22 12:09:57.343 INFO  [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil -  -- Partition / has disk space of 8 GB
2023-07-22 12:09:57.343 INFO  [pool-3-thread-26] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil -  -- Partition / has required disk space of 7340032 GB to store vIDM upgrade bundle
2023-07-22 12:09:57.345 INFO  [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil -  -- vIDM ENDPOINT HOST :: vidmone.cap.org
2023-07-22 12:09:57.345 INFO  [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil -  -- COMMAND :: mkdir -p /var/tmp/2cd58d3a-c41b-4109-8a00-dec658ce9a6b/
2023-07-22 12:09:57.444 INFO  [pool-3-thread-26] c.v.v.l.u.SshUtils -  -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX
2023-07-22 12:09:57.445 INFO  [pool-3-thread-26] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:09:57.445 INFO  [pool-3-thread-26] c.v.v.l.u.SshUtils -  -- Command: mkdir -p /var/tmp/2cd58d3a-c41b-4109-8a00-dec658ce9a6b/
2023-07-22 12:09:57.446 INFO  [pool-3-thread-26] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:09:58.528 INFO  [pool-3-thread-26] c.v.v.l.u.SshUtils -  -- exit-status: 0
2023-07-22 12:09:58.528 INFO  [pool-3-thread-26] c.v.v.l.u.SshUtils -  -- Command executed sucessfully
2023-07-22 12:09:58.529 INFO  [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil -  -- Command Status code :: 0
2023-07-22 12:09:58.529 INFO  [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil -  -- ====================================================
2023-07-22 12:09:58.530 INFO  [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil -  -- Output Stream ::
2023-07-22 12:09:58.530 INFO  [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil -  -- ====================================================
2023-07-22 12:09:58.530 INFO  [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil -  -- 
2023-07-22 12:09:58.530 INFO  [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil -  -- ====================================================
2023-07-22 12:09:58.530 INFO  [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil -  -- Error Stream ::
2023-07-22 12:09:58.531 INFO  [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil -  -- ====================================================
2023-07-22 12:09:58.532 INFO  [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil -  -- null
2023-07-22 12:09:58.532 INFO  [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil -  -- ====================================================
2023-07-22 12:09:58.534 INFO  [pool-3-thread-26] c.v.v.l.u.SshUtils -  -- Found file for upload: /data/vm-config/symlinkdir/8a233ccd-ccf9-4ddb-adff-358c20c20fe1/identity-manager-3.3.7.0-21173100-updaterepo-lcm.tar.gz
2023-07-22 12:09:58.735 INFO  [pool-3-thread-26] c.v.v.l.u.SshUtils -  -- Uploading file --> ssh://root@vidmone.cap.org/var/tmp/2cd58d3a-c41b-4109-8a00-dec658ce9a6b/
*
*
*
2023-07-22 12:15:05.647 INFO  [pool-3-thread-26] c.v.v.l.v.c.t.VidmUploadUpgradeBundleTask -  -- Completed :: Copying upgrade bundle to vIDM host: vidmone.cap.org
2023-07-22 12:15:05.647 INFO  [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil -  -- vIDM ENDPOINT HOST :: vidmone.cap.org
2023-07-22 12:15:05.647 INFO  [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil -  -- COMMAND :: chmod 644 /var/tmp/2cd58d3a-c41b-4109-8a00-dec658ce9a6b/identity-manager-3.3.7.0-21173100-updaterepo-lcm.tar.gz && chown root:KXKXKXKX /var/tmp/2cd58d3a-c41b-4109-8a00-dec658ce9a6b/identity-manager-3.3.7.0-21173100-updaterepo-lcm.tar.gz
2023-07-22 12:15:06.264 INFO  [pool-3-thread-26] c.v.v.l.u.SshUtils -  -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX
2023-07-22 12:15:06.266 INFO  [pool-3-thread-26] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:15:06.266 INFO  [pool-3-thread-26] c.v.v.l.u.SshUtils -  -- Command: chmod 644 /var/tmp/2cd58d3a-c41b-4109-8a00-dec658ce9a6b/identity-manager-3.3.7.0-21173100-updaterepo-lcm.tar.gz && chown root:KXKXKXKX /var/tmp/2cd58d3a-c41b-4109-8a00-dec658ce9a6b/identity-manager-3.3.7.0-21173100-updaterepo-lcm.tar.gz
2023-07-22 12:15:06.266 INFO  [pool-3-thread-26] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:15:07.356 INFO  [pool-3-thread-26] c.v.v.l.u.SshUtils -  -- exit-status: 0
2023-07-22 12:15:07.356 INFO  [pool-3-thread-26] c.v.v.l.u.SshUtils -  -- Command executed sucessfully
2023-07-22 12:15:07.356 INFO  [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil -  -- Command Status code :: 0
2023-07-22 12:15:07.356 INFO  [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil -  -- ====================================================
2023-07-22 12:15:07.356 INFO  [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil -  -- Output Stream ::
2023-07-22 12:15:07.356 INFO  [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil -  -- ====================================================
2023-07-22 12:15:07.356 INFO  [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil -  -- 
2023-07-22 12:15:07.356 INFO  [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil -  -- ====================================================
2023-07-22 12:15:07.357 INFO  [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil -  -- Error Stream ::
2023-07-22 12:15:07.357 INFO  [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil -  -- ====================================================
2023-07-22 12:15:07.357 INFO  [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil -  -- null
2023-07-22 12:15:07.357 INFO  [pool-3-thread-26] c.v.v.l.v.d.h.VidmUtil -  -- ====================================================
2023-07-22 12:15:07.357 INFO  [pool-3-thread-26] c.v.v.l.p.a.s.Task -  -- Injecting Edge :: OnVidmUpgradeBundleUploadCompletion


Starts vIDM offline upgrade task 

2023-07-22 12:15:07.961 INFO  [pool-3-thread-27] c.v.v.l.v.c.t.VidmOfflineUpgradeTask -  -- Starting :: vIDM offline upgrade Task
2023-07-22 12:15:07.962 INFO  [pool-3-thread-27] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil -  -- vIDM Configuration property vidmUpgradeBasePath value is obtained from Config Service : /usr/local/horizon/update/
2023-07-22 12:15:07.963 INFO  [pool-3-thread-27] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil -  -- vIDM Configuration property timeoutCount value is obtained from Config Service : 45
2023-07-22 12:15:07.963 INFO  [pool-3-thread-27] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil -  -- vIDM Configuration property serverAliveCountMax value is obtained from Config Service : 3
2023-07-22 12:15:07.964 INFO  [pool-3-thread-27] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil -  -- vIDM Configuration property serverAliveInterval value is obtained from Config Service : 60000
2023-07-22 12:15:07.964 INFO  [pool-3-thread-27] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil -  -- vIDM Configuration property timeoutSleep value is obtained from Config Service : 60000
2023-07-22 12:15:07.965 INFO  [pool-3-thread-27] c.v.v.l.v.c.t.VidmOfflineUpgradeTask -  -- Version to upgrade: 3.3.7 source version :: 3.3.6.0
2023-07-22 12:15:07.965 INFO  [pool-3-thread-27] c.v.v.l.v.c.t.VidmOfflineUpgradeTask -  -- Offline Upgrade to version 3.3.3 and above.
2023-07-22 12:15:08.090 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX
2023-07-22 12:15:08.097 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:15:08.098 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- Command: touch /usr/local/horizon/conf/flags/elasticsearchReindexed.flag
2023-07-22 12:15:08.099 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:15:39.128 INFO  [http-nio-8080-exec-5] c.v.v.l.s.n.s.NotificationServiceImpl -  -- Authentication object is not null org.springframework.security.authentication.UsernamePasswordAuthenticationToken@fe908ae6: YXYXYXYX org.springframework.security.core.userdetails.User@c220133a: Username: admin@local; Password: YXYXYXYX Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: LCM_ADMIN; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@957e: RemoteIpAddress: 127.0.0.1; SessionId: null; Granted Authorities: LCM_ADMIN
2023-07-22 12:16:08.184 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- exit-status: 0
2023-07-22 12:16:08.191 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- Command executed sucessfully
2023-07-22 12:16:08.193 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- output content :
2023-07-22 12:16:08.321 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX
2023-07-22 12:16:08.324 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:16:08.324 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- Command: rm -rf /db/elasticsearch/horizon
2023-07-22 12:16:08.325 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:16:40.145 INFO  [http-nio-8080-exec-3] c.v.v.l.s.n.s.NotificationServiceImpl -  -- Authentication object is not null org.springframework.security.authentication.UsernamePasswordAuthenticationToken@fe908ae6: YXYXYXYX org.springframework.security.core.userdetails.User@c220133a: Username: admin@local; Password: YXYXYXYX Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: LCM_ADMIN; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@957e: RemoteIpAddress: 127.0.0.1; SessionId: null; Granted Authorities: LCM_ADMIN
2023-07-22 12:17:08.420 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- exit-status: 0
2023-07-22 12:17:08.427 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- Command executed sucessfully
2023-07-22 12:17:08.428 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- output content :
2023-07-22 12:17:08.526 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX
2023-07-22 12:17:08.527 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:17:08.527 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- Command: echo "vidmone.cap.org" > /usr/local/horizon/conf/flags/opensearch.masternode
2023-07-22 12:17:08.528 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:17:41.119 INFO  [http-nio-8080-exec-9] c.v.v.l.s.n.s.NotificationServiceImpl -  -- Authentication object is not null org.springframework.security.authentication.UsernamePasswordAuthenticationToken@fe908ae6: YXYXYXYX org.springframework.security.core.userdetails.User@c220133a: Username: admin@local; Password: YXYXYXYX Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: LCM_ADMIN; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@957e: RemoteIpAddress: 127.0.0.1; SessionId: null; Granted Authorities: LCM_ADMIN
2023-07-22 12:18:08.612 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- exit-status: 0
2023-07-22 12:18:08.619 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- Command executed sucessfully
2023-07-22 12:18:08.619 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- output content :
2023-07-22 12:18:08.620 INFO  [pool-3-thread-27] c.v.v.l.v.c.t.VidmOfflineUpgradeTask -  -- Manifest cmd formed :: /usr/local/horizon/update/configureupdate.hzn manifest --set-version 3.3.6.0
2023-07-22 12:18:08.824 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX
2023-07-22 12:18:08.826 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:18:08.827 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- Command: /usr/local/horizon/update/configureupdate.hzn manifest --set-version 3.3.6.0
2023-07-22 12:18:08.827 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:18:42.131 INFO  [http-nio-8080-exec-3] c.v.v.l.s.n.s.NotificationServiceImpl -  -- Authentication object is not null org.springframework.security.authentication.UsernamePasswordAuthenticationToken@fe908ae6: YXYXYXYX org.springframework.security.core.userdetails.User@c220133a: Username: admin@local; Password: YXYXYXYX Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: LCM_ADMIN; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@957e: RemoteIpAddress: 127.0.0.1; SessionId: null; Granted Authorities: LCM_ADMIN
2023-07-22 12:18:46.551 INFO  [http-nio-8080-exec-9] c.v.v.l.a.c.AuthznCustomObjectMapper -  -- ConfigParamsDTO : ConfigParamsDTO [configname=lcm.vidm.registered, configvalue=true]
2023-07-22 12:18:46.553 INFO  [http-nio-8080-exec-9] c.v.v.l.a.c.AuthznCustomObjectMapper -  -- ConfigParamsDTO : ConfigParamsDTO [configname=lcm.firstboot.admin.password.changed, configvalue=true]
2023-07-22 12:19:08.920 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  --     Setting backrev version in manifest-installed.xml to: 3.3.6.0

2023-07-22 12:19:08.928 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- exit-status: 0
2023-07-22 12:19:08.928 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- Command executed sucessfully
2023-07-22 12:19:08.929 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- output content :    Setting backrev version in manifest-installed.xml to: 3.3.6.0

2023-07-22 12:19:08.929 INFO  [pool-3-thread-27] c.v.v.l.v.c.t.VidmOfflineUpgradeTask -  -- Fetching upgrade binaries from /var/tmp/2cd58d3a-c41b-4109-8a00-dec658ce9a6b/
2023-07-22 12:19:08.930 INFO  [pool-3-thread-27] c.v.v.l.v.c.u.VidmFileChangeUtil -  -- Creating upgrade script - /opt/vmware/horizon/vrslcm_vidm_upgrade.sh on vIDM host - vidmone.cap.org
2023-07-22 12:19:09.022 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX
2023-07-22 12:19:09.023 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:19:09.024 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- Command: echo "/usr/local/horizon/update/updateoffline.hzn -f /var/tmp/2cd58d3a-c41b-4109-8a00-dec658ce9a6b/identity-manager-3.3.7.0-21173100-updaterepo-lcm.tar.gz
echo \$? > /opt/vmware/horizon/vidmUpgradeCommandExitCode.log" > /opt/vmware/horizon/vrslcm_vidm_upgrade.sh && chmod 777 /opt/vmware/horizon/vrslcm_vidm_upgrade.sh
2023-07-22 12:19:09.024 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:19:09.611 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- exit-status: 0
2023-07-22 12:19:09.612 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- Command executed sucessfully
2023-07-22 12:19:09.612 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- Command execution response: {
  "exitStatus" : 0,
  "outputData" : "",
  "errorData" : null,
  "commandTimedOut" : false
}
2023-07-22 12:19:09.613 INFO  [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil -  -- Command exit code :: 0
2023-07-22 12:19:09.613 INFO  [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil -  -- ==============================================
2023-07-22 12:19:09.613 INFO  [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil -  -- Command output stream ::
2023-07-22 12:19:09.613 INFO  [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil -  -- ==============================================
2023-07-22 12:19:09.613 INFO  [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil -  -- Command error stream :: null
2023-07-22 12:19:09.613 INFO  [pool-3-thread-27] c.v.v.l.v.c.t.VidmOfflineUpgradeTask -  -- Invoking upgrade script using the command :: nohup /opt/vmware/horizon/vrslcm_vidm_upgrade.sh > /opt/vmware/horizon/vrslcm_vidm_upgrade.log 2>&1 &
2023-07-22 12:19:09.614 INFO  [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil -  -- vIDM ENDPOINT HOST :: vidmone.cap.org
2023-07-22 12:19:09.614 INFO  [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil -  -- COMMAND :: nohup /opt/vmware/horizon/vrslcm_vidm_upgrade.sh > /opt/vmware/horizon/vrslcm_vidm_upgrade.log 2>&1 &
2023-07-22 12:19:09.697 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX
2023-07-22 12:19:09.698 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:19:09.698 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- Command: nohup /opt/vmware/horizon/vrslcm_vidm_upgrade.sh > /opt/vmware/horizon/vrslcm_vidm_upgrade.log 2>&1 &
2023-07-22 12:19:09.698 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:20:09.815 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- exit-status: 0
2023-07-22 12:20:09.822 INFO  [pool-3-thread-27] c.v.v.l.u.SshUtils -  -- Command executed sucessfully
2023-07-22 12:20:09.823 INFO  [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil -  -- Command Status code :: 0
2023-07-22 12:20:09.823 INFO  [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil -  -- ====================================================
2023-07-22 12:20:09.823 INFO  [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil -  -- Output Stream ::
2023-07-22 12:20:09.824 INFO  [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil -  -- ====================================================
2023-07-22 12:20:09.824 INFO  [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil -  -- 
2023-07-22 12:20:09.824 INFO  [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil -  -- ====================================================
2023-07-22 12:20:09.824 INFO  [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil -  -- Error Stream ::
2023-07-22 12:20:09.824 INFO  [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil -  -- ====================================================
2023-07-22 12:20:09.824 INFO  [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil -  -- null
2023-07-22 12:20:09.825 INFO  [pool-3-thread-27] c.v.v.l.v.d.h.VidmUtil -  -- ====================================================
2023-07-22 12:20:09.825 INFO  [pool-3-thread-27] c.v.v.l.v.c.t.VidmOfflineUpgradeTask -  -- Upgrade initiated successfully.
2023-07-22 12:20:09.825 INFO  [pool-3-thread-27] c.v.v.l.p.a.s.Task -  -- Injecting Edge :: OnVidmOfflineUpgradeCompletion


vRSLCM waits for uprgade to complete. There are 45 retries. So we expect upgrade to be completed


2023-07-22 12:20:10.366 INFO  [pool-3-thread-25] c.v.v.l.v.c.t.VidmRebootTask -  -- Starting :: vIDM reboot Task
2023-07-22 12:20:10.367 INFO  [pool-3-thread-25] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil -  -- vIDM Configuration property timeoutSleep value is obtained from Config Service : 60000
2023-07-22 12:20:10.368 INFO  [pool-3-thread-25] c.v.v.l.v.c.t.u.VidmUpgradeTaskUtil -  -- vIDM Configuration property timeoutCount value is obtained from Config Service : 45
2023-07-22 12:20:10.368 INFO  [pool-3-thread-25] c.v.v.l.v.c.u.VidmFileChangeUtil -  -- Checking vIDM upgrade status on vIDM host - vidmone.cap.org
2023-07-22 12:20:10.456 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX
2023-07-22 12:20:10.460 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:20:10.461 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- Command: ls /opt/vmware/horizon/vidmUpgradeCommandExitCode.log
2023-07-22 12:20:10.462 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:20:11.004 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- exit-status: 2
2023-07-22 12:20:11.004 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- Command executed sucessfully
2023-07-22 12:20:11.005 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- Command execution response: {
  "exitStatus" : 2,
  "outputData" : "",
  "errorData" : null,
  "commandTimedOut" : false
}
2023-07-22 12:20:11.005 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- Command exit code :: 2
2023-07-22 12:20:11.005 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- ==============================================
2023-07-22 12:20:11.006 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- Command output stream ::
2023-07-22 12:20:11.006 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- ==============================================
2023-07-22 12:20:11.006 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- Command error stream :: null
2023-07-22 12:20:11.006 INFO  [pool-3-thread-25] c.v.v.l.v.c.u.VidmFileChangeUtil -  -- Waiting for upgrade to complete, sleeping for 60000 milliseconds
2023-07-22 12:21:11.006 INFO  [pool-3-thread-25] c.v.v.l.v.c.u.VidmFileChangeUtil -  -- Number of retries left to check vIDM upgrade status on the node - vidmone.cap.org : 44
2023-07-22 12:21:11.164 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX
2023-07-22 12:21:11.170 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:21:11.171 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- Command: ls /opt/vmware/horizon/vidmUpgradeCommandExitCode.log
2023-07-22 12:21:11.172 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:21:11.715 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- exit-status: 2
2023-07-22 12:21:11.716 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- Command executed sucessfully
2023-07-22 12:21:11.716 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- Command execution response: {
  "exitStatus" : 2,
  "outputData" : "",
  "errorData" : null,
  "commandTimedOut" : false
}
2023-07-22 12:21:11.717 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- Command exit code :: 2
2023-07-22 12:21:11.718 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- ==============================================
2023-07-22 12:21:11.718 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- Command output stream ::
2023-07-22 12:21:11.718 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- ==============================================
2023-07-22 12:21:11.718 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- Command error stream :: null
2023-07-22 12:21:11.718 INFO  [pool-3-thread-25] c.v.v.l.v.c.u.VidmFileChangeUtil -  -- Waiting for upgrade to complete, sleeping for 60000 milliseconds
2023-07-22 12:22:11.718 INFO  [pool-3-thread-25] c.v.v.l.v.c.u.VidmFileChangeUtil -  -- Number of retries left to check vIDM upgrade status on the node - vidmone.cap.org : 43


vIDM upgrade completes finally. 



2023-07-22 12:25:13.798 INFO  [pool-3-thread-25] c.v.v.l.v.c.u.VidmFileChangeUtil -  -- Number of retries left to check vIDM upgrade status on the node - vidmone.cap.org : 40
2023-07-22 12:25:13.889 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX
2023-07-22 12:25:13.889 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:25:13.889 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- Command: ls /opt/vmware/horizon/vidmUpgradeCommandExitCode.log
2023-07-22 12:25:13.890 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:25:14.439 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- exit-status: 2
2023-07-22 12:25:14.440 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- Command executed sucessfully
2023-07-22 12:25:14.440 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- Command execution response: {
  "exitStatus" : 2,
  "outputData" : "",
  "errorData" : null,
  "commandTimedOut" : false
}
2023-07-22 12:25:14.440 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- Command exit code :: 2
2023-07-22 12:25:14.441 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- ==============================================
2023-07-22 12:25:14.441 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- Command output stream ::
2023-07-22 12:25:14.441 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- ==============================================
2023-07-22 12:25:14.441 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- Command error stream :: null
2023-07-22 12:25:14.441 INFO  [pool-3-thread-25] c.v.v.l.v.c.u.VidmFileChangeUtil -  -- Waiting for upgrade to complete, sleeping for 60000 milliseconds
2023-07-22 12:26:14.441 INFO  [pool-3-thread-25] c.v.v.l.v.c.u.VidmFileChangeUtil -  -- Number of retries left to check vIDM upgrade status on the node - vidmone.cap.org : 39
2023-07-22 12:26:14.526 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX
2023-07-22 12:26:14.527 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:26:14.527 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- Command: ls /opt/vmware/horizon/vidmUpgradeCommandExitCode.log
2023-07-22 12:26:14.527 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:26:15.075 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- exit-status: 0
2023-07-22 12:26:15.076 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- Command executed sucessfully
2023-07-22 12:26:15.076 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- Command execution response: {
  "exitStatus" : 0,
  "outputData" : "/opt/vmware/horizon/vidmUpgradeCommandExitCode.log\n",
  "errorData" : null,
  "commandTimedOut" : false
}
2023-07-22 12:26:15.077 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- Command exit code :: 0
2023-07-22 12:26:15.077 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- ==============================================
2023-07-22 12:26:15.077 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- Command output stream ::/opt/vmware/horizon/vidmUpgradeCommandExitCode.log

2023-07-22 12:26:15.077 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- ==============================================
2023-07-22 12:26:15.077 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- Command error stream :: null
2023-07-22 12:26:15.077 INFO  [pool-3-thread-25] c.v.v.l.v.c.u.VidmFileChangeUtil -  -- File: /opt/vmware/horizon/vidmUpgradeCommandExitCode.log found
2023-07-22 12:26:15.294 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX
2023-07-22 12:26:15.301 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:26:15.302 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- Command: cat /opt/vmware/horizon/vidmUpgradeCommandExitCode.log
2023-07-22 12:26:15.302 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:26:15.848 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- exit-status: 0
2023-07-22 12:26:15.848 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- Command executed sucessfully
2023-07-22 12:26:15.849 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- Command execution response: {
  "exitStatus" : 0,
  "outputData" : "0\n",
  "errorData" : null,
  "commandTimedOut" : false
}
2023-07-22 12:26:15.849 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- Command exit code :: 0
2023-07-22 12:26:15.850 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- ==============================================
2023-07-22 12:26:15.850 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- Command output stream ::0

2023-07-22 12:26:15.850 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- ==============================================
2023-07-22 12:26:15.850 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- Command error stream :: null
2023-07-22 12:26:15.850 INFO  [pool-3-thread-25] c.v.v.l.v.c.u.VidmFileChangeUtil -  -- Upgrade command exit status obtained from /opt/vmware/horizon/vidmUpgradeCommandExitCode.log is 0. vIDM upgrade completed successfully
2023-07-22 12:26:15.851 INFO  [pool-3-thread-25] c.v.v.l.v.c.u.VidmFileChangeUtil -  -- Number of retries left to check vIDM upgrade status on the node - vidmone.cap.org : 38
2023-07-22 12:26:15.930 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- Executing command on the host: vidmone.cap.org , as user: root:KXKXKXKX
2023-07-22 12:26:15.930 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:26:15.930 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- Command: find /usr/local/horizon/conf -iname update.success
2023-07-22 12:26:15.930 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- ------------------------------------------------------
2023-07-22 12:26:16.479 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- exit-status: 0
2023-07-22 12:26:16.480 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- Command executed sucessfully
2023-07-22 12:26:16.480 INFO  [pool-3-thread-25] c.v.v.l.u.SshUtils -  -- Command execution response: {
  "exitStatus" : 0,
  "outputData" : "/usr/local/horizon/conf/update.success\n",
  "errorData" : null,
  "commandTimedOut" : false
}
2023-07-22 12:26:16.481 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- Command exit code :: 0
2023-07-22 12:26:16.481 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- ==============================================
2023-07-22 12:26:16.481 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- Command output stream ::/usr/local/horizon/conf/update.success

2023-07-22 12:26:16.481 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- ==============================================
2023-07-22 12:26:16.481 INFO  [pool-3-thread-25] c.v.v.l.v.d.h.VidmUtil -  -- Command error stream :: null
2023-07-22 12:26:16.481 INFO  [pool-3-thread-25] c.v.v.l.v.c.t.VidmRebootTask -  -- File: /usr/local/horizon/conf/update.success found. Upgrade completed successfully!
2023-07-22 12:26:16.481 INFO  [pool-3-thread-25] c.v.v.l.v.c.t.VidmRebootTask -  -- vIDM upgrade command exit status: 0. vIDM upgrade completed successfully


Takes around 11 minutes on the primary node 

There are some post uprgade actions performed and then it goes ahead with further steps

 

VMware Identity Manager Upgrade logs


Reference: /opt/vmware/horizon/vrslcm_vidm_upgrade.log

/***** Identifies that the upgrade. isthrough vRSLCM *****/

Upgrade via LCM
dualbootupdate-3.3.7.0-21173100.tar.gz
identity-manager-3.3.7.0-21173100-updaterepo.zip
Archive:  /var/tmp/2cd58d3a-c41b-4109-8a00-dec658ce9a6b/identity-manager-3.3.7.0-21173100-updaterepo.zip
   creating: manifest/
  inflating: manifest/manifest-latest.xml  
  inflating: manifest/manifest-repo.xml  
  inflating: manifest/manifest-latest.xml.sha256  
 extracting: manifest/manifest-latest.xml.sig  
   creating: package-pool/
  inflating: package-pool/tar-1.30-5.ph3.x86_64.rpm  
  inflating: package-pool/procps-ng-3.3.15-3.ph3.x86_64.rpm  
  inflating: package-pool/irqbalance-1.4.0-5.ph3.x86_64.rpm  
  inflating: package-pool/iptables-1.8.3-6.ph3.x86_64.rpm  
  inflating: package-pool/device-mapper-event-2.02.187-2.ph3.x86_64.rpm  
  inflating: package-pool/python3-prettytable-0.7.2-6.ph3.noarch.rpm  
  inflating: package-pool/lsscsi-0.30-1.ph3.x86_64.rpm  
  inflating: package-pool/python2-libs-2.7.17-7.ph3.x86_64.rpm  
  inflating: package-pool/python3-xml-3.7.5-23.ph3.x86_64.rpm  
  inflating: package-pool/python3-six-1.12.0-1.ph3.noarch.rpm  
  inflating: package-pool/svadmin-rpm-3.3.7.0-21173100.noarch.rpm  
  inflating: package-pool/python3-requests-2.24.0-1.ph3.noarch.rpm  
  inflating: package-pool/libmnl-1.0.4-4.ph3.x86_64.rpm  
  inflating: package-pool/cracklib-2.9.7-2.ph3.x86_64.rpm  
  inflating: package-pool/sudo-1.9.5-5.ph3.x86_64.rpm  
  inflating: package-pool/bzip2-1.0.8-2.ph3.x86_64.rpm  
  inflating: package-pool/bc-1.07.1-3.ph3.x86_64.rpm  
  inflating: package-pool/filesystem-1.1-4.ph3.x86_64.rpm  
  inflating: package-pool/sed-4.5-2.ph3.x86_64.rpm  
*
*
*
*
  inflating: package-pool/horizon-database-rpm-3.3.7.0-21173100.noarch.rpm  
  inflating: package-pool/libcap-2.25-8.ph3.x86_64.rpm  
  inflating: package-pool/python3-lxml-4.9.1-1.ph3.x86_64.rpm  
  inflating: package-pool/python2-2.7.17-7.ph3.x86_64.rpm  
  inflating: package-pool/horizon-preconfig-standard-3.3.7.0-21173100.noarch.rpm  
  inflating: package-pool/libuv-1.34.2-2.ph3.x86_64.rpm  
The product RID matches so continue
Continuing...
Server is up, running the update
root     18068  8.0  0.0  19112 14548 ?        S    12:19   0:00 python -m SimpleHTTPServer 8008
root     18072  0.0  0.0   5484  1820 ?        S    12:19   0:00 grep SimpleHTTPServer
Checking website
Website found, validating RID
Website validated
Installing custom provider-runtime
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
Updating update rpm to 3.3.7.0
3.3.7.0
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
Getting the Current version...
Checking Space
FREESPACE 4715420 KB

Checking for updates...

/dualbootupdate.tar.gz
Current version: 3.3.6.0
Updating all VMs to version: 3.3.7.0

Update started, current version : 3.3.6.0 Time : Sat Jul 22 12:20:01 UTC 2023
%_pkgverify_level digest
%_pkgverify_level none
Running preupdate
Saving manifest.xml
This is an LCM deployment. Assuming that the /usr/local/horizon/update/reindexingIndices.hzn script is already run and proceeding...
Skipping managenodes.inc in case of Upgrade via LCM
LCM added master hostname before upgrade : vidmone.cap.org
You are required to change your password immediately (password expired)
su: Authentication token is no longer valid; new one required
(Ignored)
ERROR: KDC configuration not yet initialized. Use the init subcommand.
You are required to change your password immediately (password expired)
su: Authentication token is no longer valid; new one required
(Ignored)
horizon-elasticsearch : normal uninstall
horizon-elasticsearch : normal uninstall
iNode usage on /var at %
runSqlVerification 
12:20:07.608 [main] INFO com.vmware.horizon.utils.KeystoreUtilities - Failed to load non-fips bc provider, keystore conversions will not be available.
12:20:08.283 [main] DEBUG com.vmware.horizon.utils.KeystoreUtilities - Loading BCFKS stream from BCFIPS
SUCCESS: Connection verified for driver version: 42.2.12
Updating the vm.
Saving manifest.xml
Installing version - 
3.3.7.0 Build 21173100
..................................................Connection to sfcbd lost
Attempting to reconnect: 1
.....................................................Waiting on upgrade process to complete: 

Vami upgrade process completed.
Restoring default provider-runtime

%_pkgverify_level digest
%_pkgverify_level digest
Running postupdate 0 false
Updating IDM CA keystore with JRE certificates
Import command completed:  151 entries successfully imported, 0 entries failed or cancelled
Configuring opensearch.yml file : vidmone.cap.org

Resetting RabbitMQ
Upgrade provisioning adapters
%d [%thread] %-5level %logger - %msg%n%d [%thread] %-5level %logger - %msg%nError : Error creating bean with name 'scopedTarget.cacheService' defined in com.vmware.horizon.cache.config.CacheServiceConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.horizon.cache.CacheService]: Factory method 'getCacheService' threw exception; nested exception is org.hibernate.exception.GenericJDBCException: could not prepare statement

renamed '/usr/local/horizon/conf/iptables.cfg.rpmsave' -> '/usr/local/horizon/conf/iptables.cfg'
<79>Jul 22 12:25:30 root: updateiptables.hzn Discovering IP Address
Unable to retrieve IP list, using current ip list
No current ip list defined, using own ip
<79>Jul 22 12:25:30 root: updateiptables.hzn Processing port roles
<79>Jul 22 12:25:31 root: updateiptables.hzn Setup iptables locally
<79>Jul 22 12:25:31 root: setupIPTables.hzn begin iptable setup
<79>Jul 22 12:25:31 root: setupIPTables.hzn allow any ip to access tcp ports: 22 443 5432 6443 7443 80 8080 8443 9000 9300 9694 9898 9999
<79>Jul 22 12:25:32 root: setupIPTables.hzn vApp access only to tcp ports: 40002 40003 5701 9300
<79>Jul 22 12:25:32 root: setupIPTables.hzn finish with static rules
<79>Jul 22 12:25:33 root: setupIPTables.hzn save firewall rules
<79>Jul 22 12:25:33 root: setupIPTables.hzn end iptable setup
<79>Jul 22 12:25:33 root: updateiptables.hzn finish processing iptables
Rebuilding manifest file
Update completed, current version : 3.3.7.0 Time : Sat Jul 22 12:25:38 UTC 2023
Update complete, please reboot the VM.


 

Important Logs during Upgrade



vRealize Suite Lifecycle

/var/log/vrlcm/vmware_vrlcm.log

VMware Identity Manager

/opt/vmware/var/log/vami/updatecli.log

 

/opt/vmware/var/log/vami/vami.log

 

/opt/vmware/horizon/vrslcm_vidm_upgrade.log



 




737 views0 comments
bottom of page