Detailed Analysis available here :https://tinyurl.com/vmvtrn66
There are 2 Types of enumerations which occur
Full Enumeration
Incremental Enumeration
After a Cloud Account is added , a full enumeration is performed. Post successful initial enumeration , every other enumeration which is performed after an account is added is an incremental one.
Full enumeration occurs every 24 hours and Incremental occurs every 10 minutes
Activities performed during Full Enumeration and Incremental enumeration is pretty much the same. Just that we collect only changes when we perform the latter action.
If we do math....
143 Attempts of Full Enumeration in 24 hours vSphere Enumeration Deep-Dive ( First one after creating Cloud Account , A Full Enumeration )
Prepares for Enumeration
Starts direct Resource Enumeration by sending request to endpoint adapter
Starts collecting tags from vSphere endpoint
After tags collection , it starts collecting supervisor cluster information
Then comes Datacenter enumeration
Fetches and creates folder objects
Fetches and creates Storage Pod objects
Fetches and creates Datastore objects
Fetches and creates Network Objects
Opaque Network objects are created
Fetches Cluster information and creates it
Hosts information is collected and created
Resource Pool objects are created
Collects the VM's information
Creates VM Objects
Storage Policy information would be collected
Enumeration is marked as finished
Incremental Enumeration
Tasks performed during incremental enumeration is pretty much the same. But only the changes are collected from vCenter
Few example statements during incremental enumeration are below
021-08-02T14:12:53.727Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='scheduling-1' user='' org='' trace='cf6b1ebb-6b1c-4f98-a0c8-a494e27c24e0' parent='' span='575e1683-be35-48e7-bb05 -84f6e7887d95'] c.v.a.s.v.VSphereIncrementalEnumerationController.doIncrementalEnumeration:325 - Starting incremental enumeration of /provisioning/vsphere/incremental-enumeration-adapter/1663dbad-82c5-481f-b624-0cb 80d725f3e
2021-08-02T14:12:53.745Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-30' user='' org='' trace='cf6b1ebb-6b1c-4f98-a0c8-a494e27c24e0' parent='' span='575e1683-be35-48 e7-bb05-84f6e7887d95'] c.v.a.s.v.VSphereIncrementalEnumerationController.lambda$doIncrementalEnumeration$8:337 - /resources/endpoints/1663dbad-82c5-481f-b624-0cb80d725f3e: The following regions will be collected: [ ["Datacenter:datacenter-3"]]
2021-08-02T14:12:53.745Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-30' user='' org='' trace='cf6b1ebb-6b1c-4f98-a0c8-a494e27c24e0' parent='' span='575e1683-be35-48 e7-bb05-84f6e7887d95'] c.v.a.s.v.VSphereIncrementalEnumerationController.lambda$doIncrementalEnumeration$6:353 - Successfully started incremental enumeration of /provisioning/vsphere/incremental-enumeration-adapter/1663dbad-82c5-481f-b624-0cb80d725f3e
*
*
2021-08-02T14:12:54.049Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-2' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.TagEnumerationService.collectIdentifiers:113 - [vc.nukescloud.com]: Begin collecting tags [com.vmware.photon.controller.model.adapters.vsphere.enumeration.VsphereEnumerationAdapterService$EnumerationContext@49071a12]: Start collecting supervisor resources.
*
*
*
2021-08-02T14:12:54.506Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-5' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.VsphereEnumerationAdapterService.lambda$enumerateDatacenter$25:717 - [vc.nukescloud.com][nukescloud] Starting datacenter enumeration.
2021-08-02T14:12:54.506Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-5' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.DatacenterEnumerationService.enumerate:287 - [[vc.nukescloud.com]:[nukescloud]] Starting datacenter enumeration.
*
*
*
2021-08-02T14:12:54.710Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-3' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.processors.Processor.logUpdateSuccess:17
9 - [vc.nukescloud.com]:[nukescloud]:[is-tse-d12]: Successfully updated resource [Name: [is-tse-d12], mo-ref: [datastore-1014], capacity: [461977419776] bytes, free capacity: [460461178880] bytes]
2021-08-02T14:12:54.715Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-1' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.processors.Processor.logUpdateSuccess:17
9 - [vc.nukescloud.com]:[nukescloud]:[is-tse-d14]: Successfully updated resource [Name: [is-tse-d14], mo-ref: [datastore-1024], capacity: [461977419776] bytes, free capacity: [460461178880] bytes]
2021-08-02T14:12:54.715Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-6' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.processors.Processor.logUpdateSuccess:17
9 - [vc.nukescloud.com]:[nukescloud]:[is-tse-d11]: Successfully updated resource [Name: [is-tse-d11], mo-ref: [datastore-1005], capacity: [461977419776] bytes, free capacity: [460461178880] bytes]
2021-08-02T14:12:54.720Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-4' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.processors.Processor.logUpdateSuccess:179 - [vc.nukescloud.com]:[nukescloud]:[is-tse-d13]: Successfully updated resource [Name: [is-tse-d13], mo-ref: [datastore-1015], capacity: [461977419776] bytes, free capacity: [460461178880] bytes]
2021-08-02T14:12:54.720Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-7' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.processors.Processor.logUpdateSuccess:179 - [vc.nukescloud.com]:[nukescloud]:[shareddsone]: Successfully updated resource [Name: [shareddsone], mo-ref: [datastore-46], capacity: [6596801331200] bytes, free capacity: [2023744339968] bytes]
*
*
*
2021-08-02T14:12:54.825Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-6' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.DatacenterEnumerationService.lambda$processPortgroups$15:591 - [vc.nukescloud.com]:[nukescloud]: Total portgroups: [2]
2021-08-02T14:12:54.846Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-4' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.p.ProcessorDelegate.lambda$getExistingResources$15:354 - [[vc.nukescloud.com]:[nukescloud]]: Fetched 2 existing resources of DistributedVirtualPortgroup...
2021-08-02T14:12:54.847Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-5' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.p.SubnetProcessor.processUpdate:62 - [vc.nukescloud.com]:[nukescloud]: No change detected in subnet [DSwitch-DVUplinks-1218]. Skipping update.
2021-08-02T14:12:54.847Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-4' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.p.SubnetProcessor.processUpdate:62 - [vc.nukescloud.com]:[nukescloud]: No change detected in subnet [DPortGroup]. Skipping update.
2021-08-02T14:12:54.847Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-4' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.p.ResourceBatchProcessor.fetchToBeDeletedResources:193 - [[vc.nukescloud.com]:[nukescloud]] Will delete all resources of type [DistributedVirtualPortgroup] that do not have self links in the list [["/resources/sub-networks/beef0b4a-140e-48ff-937b-2b96d1914974","/resources/sub-networks/1e031928-4404-4eb0-a9dd-25241b7b9975"]].
2021-08-02T14:12:54.849Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-2' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.p.ResourceBatchProcessor.lambda$deleteResources$2:124 - [vc.nukescloud.com:nukescloud Deleting 0 resources of type DistributedVirtualPortgroup.]
2021-08-02T14:12:54.985Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-2' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.DatacenterEnumerationService.lambda$processNetworks$19:633 - [vc.nukescloud.com]:[nukescloud]: Total networks: [2]
2021-08-02T14:12:55.005Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-5' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.p.ProcessorDelegate.lambda$getExistingResources$15:354 - [[vc.nukescloud.com]:[nukescloud]]: Fetched 2 existing resources of Network...
2021-08-02T14:12:55.005Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-3' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.p.NetworkProcessor.processUpdate:93 - [vc.nukescloud.com]:[nukescloud]: No change detected in network [VM Network]. Skipping update.
2021-08-02T14:12:55.005Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-5' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.p.NetworkProcessor.processUpdate:93 - [vc.nukescloud.com]:[nukescloud]: No change detected in network [VM Network 2]. Skipping update.
*
*
*
2021-08-02T14:12:55.369Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-6' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.processors.Processor.logUpdateSuccess:179 - [vc.nukescloud.com]:[nukescloud]:[myworld]: Successfully updated resource [Name: [myworld], mo-ref: [domain-c8], isDRSEnabled: [true], max vcpu count: [48], numCpuCores: [96], total cpuMhz: [210624] totalMemory: [1097922330624] bytes, effectiveMemory: [1050290421760] bytes]
*
*
*
2021-08-02T14:12:55.604Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-8' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.processors.Processor.logUpdateSuccess:179 - [vc.nukescloud.com]:[nukescloud]:[hostfour.nukescloud.com]: Successfully updated resource [Name: [hostfour.nukescloud.com], mo-ref: [host-1023], parent: [ClusterComputeResource:domain-c8], numCpuThreads: [48], cpuMhz: [2194], totalMemory: [274480496640] bytes, availableMemory: [124851847168] bytes]
2021-08-02T14:12:55.609Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-5' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.processors.Processor.logUpdateSuccess:179 - [vc.nukescloud.com]:[nukescloud]:[hosttwo.nukescloud.com]: Successfully updated resource [Name: [hosttwo.nukescloud.com], mo-ref: [host-1010], parent: [ClusterComputeResource:domain-c8], numCpuThreads: [48], cpuMhz: [2194], totalMemory: [274480447488] bytes, availableMemory: [127551881216] bytes]
2021-08-02T14:12:55.609Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-6' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.processors.Processor.logUpdateSuccess:179 - [vc.nukescloud.com]:[nukescloud]:[hostthree.nukescloud.com]: Successfully updated resource [Name: [hostthree.nukescloud.com], mo-ref: [host-1013], parent: [ClusterComputeResource:domain-c8], numCpuThreads: [48], cpuMhz: [2194], totalMemory: [274480447488] bytes, availableMemory: [134771326976] bytes]
2021-08-02T14:12:55.609Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-1' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.processors.Processor.logUpdateSuccess:179 - [vc.nukescloud.com]:[nukescloud]:[hostone.nukescloud.com]: Successfully updated resource [Name: [hostone.nukescloud.com], mo-ref: [host-1004], parent: [ClusterComputeResource:domain-c8], numCpuThreads: [48], cpuMhz: [2194], totalMemory: [274480939008] bytes, availableMemory: [126811029504] bytes]
*
*
*
c.v.p.c.m.a.v.e.DatacenterEnumerationService.lambda$processMachines$46:1015 - [[vc.nukescloud.com]:[nukescloud]] Successfully processed VMs via incremental enumeration, next update version [nothing-changed], incremental enumeration iterations performed [30]
2021-08-02T14:12:56.807Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-7' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.DatacenterEnumerationService.processFirstClassDisks:1200 - [[vc.nukescloud.com]:[nukescloud]] Starting FCD enumeration.
2021-08-02T14:12:56.818Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-23' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.DatacenterEnumerationService.lambda$getExistingDisks$73:1363 - fcd-enum for endpoint [/resources/endpoints/1663dbad-82c5-481f-b624-0cb80d725f3e]: Time Taken to query existing disks, in millis is 6
2021-08-02T14:12:56.841Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-8' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.DatacenterEnumerationService.lambda$fetchDatastores$75:1398 - fcd-enum: Time to fetch datastores in millis in 23
2021-08-02T14:12:56.841Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-8' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.DatacenterEnumerationService.lambda$fetchDatastores$75:1404 - [vc.nukescloud.com] Successfully fetched [5] datastores for [5] datastore IDs...
2021-08-02T14:12:56.947Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-8' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.DatacenterEnumerationService.lambda$enumerateFirstClassDisks$66:1271 - [[vc.nukescloud.com]:[nukescloud]] Successfully created FCDs via enumeration
2021-08-02T14:12:56.947Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-8' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.p.FirstClassDiskProcessorUtils.deleteDisks:304 - /resources/endpoints/1663dbad-82c5-481f-b624-0cb80d725f3e No FCDs to be deleted
2021-08-02T14:12:56.947Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-8' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.DatacenterEnumerationService.lambda$enumerateFirstClassDisks$68:1285 - fcd-enum: Time taken to complete delete of FCDs in millis is 0
2021-08-02T14:12:56.947Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-8' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.DatacenterEnumerationService.lambda$enumerateFirstClassDisks$68:1294 - [[vc.nukescloud.com]:[nukescloud]] Successfully deleted FCDs via enumeration
2021-08-02T14:12:56.986Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-25' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.ClusterReconciliationService.lambda$getUpdatedClusterState$6:138 - [vc.nukescloud.com]: Successfully updated cluster [myworld] in database with selflink [/resources/compute/0da008cc-1069-4f68-a039-1ce17bef9626]
2021-08-02T14:12:56.987Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='xn-index-queries-25' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.ClusterReconciliationService.lambda$reconcileClusterCPUs$4:91 - [vc.nukescloud.com]: Finished updating 1 clusters in database.
2021-08-02T14:12:57.002Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-2' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.DatacenterEnumerationService.lambda$enumerate$0:308 - [vc.nukescloud.com]:[nukescloud]: Finish enumerating datacenter.
2021-08-02T14:12:57.002Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-2' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.DatacenterEnumerationService.lambda$enumerate$0:313 - [vc.nukescloud.com]:[nukescloud]: Total elapsed time metrics: {
"vc.nukescloud.com.nukescloud.Dvs.processing.total.elapsedTime.millis": "158",
"vc.nukescloud.com.nukescloud.machines.processing.total.elapsedTime.millis": "1004",
"vc.nukescloud.com.nukescloud.enumerate-fcd.processing.total.elapsedTime.millis": "135",
"vc.nukescloud.com.nukescloud.storagepods.processing.total.elapsedTime.millis": "5",
"vc.nukescloud.com.nukescloud.resourcepools.processing.total.elapsedTime.millis": "189",
"vc.nukescloud.com.nukescloud.hosts.processing.total.elapsedTime.millis": "238",
"vc.nukescloud.com.nukescloud.opaque-networks.processing.total.elapsedTime.millis": "3",
"vc.nukescloud.com.nukescloud.clusters.processing.total.elapsedTime.millis": "202",
"vc.nukescloud.com.nukescloud.folders.processing.total.elapsedTime.millis": "39",
"vc.nukescloud.com.nukescloud.portgroups.processing.total.elapsedTime.millis": "125",
"vc.nukescloud.com.nukescloud.datastores.processing.total.elapsedTime.millis": "170",
"vc.nukescloud.com.nukescloud.networks.processing.total.elapsedTime.millis": "159"
}
2021-08-02T14:12:57.003Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-3' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.VsphereEnumerationAdapterService.handleUnmanagedDatacenters:753 - [vc.nukescloud.com]: No unmanaged datacenter found this enumeration.
2021-08-02T14:12:57.004Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-3' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.StoragePolicyEnumerationService.enumerate:130 - [vc.nukescloud.com]: Begin storage policy enumeration.
*
*
*
2021-08-02T14:12:57.634Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-7' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.StoragePolicyEnumerationService.lambda$enumerate$0:142 - [vc.nukescloud.com]: Finish storage policy enumeration.
2021-08-02T14:12:57.641Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-5' user='' org='' trace='' parent='' span=''] c.v.p.c.m.a.v.e.VsphereEnumerationAdapterService.lambda$handlePatch$6:319 - vc.nukescloud.com: Total elapsed time metrics: {
"vc.nukescloud.com.tags.processing.total.elapsedTime.millis": "244",
"vc.nukescloud.com.storage-policies.processing.total.elapsedTime.millis": "630",
"vc.nukescloud.com.supervisor-resources.processing.total.elapsedTime.millis": "140",
"vc.nukescloud.com.enumerate-datacenters.processing.total.elapsedTime.millis": "2496"
}
2021-08-02T14:12:57.641Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-5' user='' org='' trace='' parent='' span=''] c.v.p.c.model.adapters.util.TaskManager.log:453 - Patching task http://10.244.0.42:8282/provisioning/vsphere/incremental-enumeration-adapter/callback/1663dbad-82c5-481f-b624-0cb80d725f3e to stage: FINISHED
2021-08-02T14:12:57.641Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-5' user='' org='' trace='' parent='' span=''] c.v.p.c.model.adapters.util.TaskManager.log:453 - Sending with Retry to http://10.244.0.42:8282/provisioning/vsphere/incremental-enumeration-adapter/callback/1663dbad-82c5-481f-b624-0cb80d725f3e, authContext.isSystemUser=true, expiration: 31556889864403199
2021-08-02T14:12:57.641Z INFO provisioning [host='provisioning-service-app-58dd6cc799-785h2' thread='vsphere-enum-5' user='' org='' trace='' parent='' span=''] c.v.a.s.v.VSphereIncrementalEnumerationController.handlePatch:234 - Callback received on /provisioning/vsphere/incremental-enumeration-adapter/callback/1663dbad-82c5-481f-b624-0cb80d725f3e: FINISHED
If you read the above snippets
Only changes are collected during incremental enumeration
We do mention time taken for each enumeration , this can be used for any performance troubleshooting if needed
Comments